Feature/cpu optimizations - #562
Conversation
|
1535a03 to
567e660
Compare
1afbe1f to
8b7a7e2
Compare
There was a problem hiding this comment.
я не могу одобрить мерж, пока исходный код tabby с изменениями не будет запушен и одобрен Андреем Михайловым
There was a problem hiding this comment.
Ждём одобрения патча для исходников tabby. Там изменения оформлены отдельными коммитами sunveil/ispras_tbl_extr#1
- параллельный поиск линеек таблиц,
- чтение пикселя без аллокации int[] на каждый пиксель и
- опциональный флаг -c для компактного data.json.
Первые две правки не меняют вывод движка: data.json побайтово идентичен сборке из main на всех 20 PDF из tests/data/pdf_with_text_layer, включая путь -sp/-ep; третья по умолчанию выключена, dedoc включает её флагом. На 145-страничном документе движок ускоряется с 9.9 с до 3.5 с (поиск линеек 7.19 → 1.04 с), data.json становится меньше на 44%.
Jar в этой ветке пересобран строго из той ветки, что в PR — апстримовый main плюс эти три коммита, больше ничего. Побочный эффект: вместе с ними приезжают изменения апстрима, сделанные после сборки нынешнего jar
Плюс помимо оптимизаций возможно добавим одно изменение, там вполне возможно баг с координатами картинок есть
There was a problem hiding this comment.
Патч добавили, но как я выяснил по тестам, нужен еще один, по сути раскомметировать старую механику, которую закомментили между тем как написаны тесты и тем что я добавил в текущий апстрим патчи, и этой механики не хватает для соответствия тестам. По сути сломал тесты не мой патч, а обновления апстрима, которые мы не вливали до этого к себе. Можно в целом собрать jar с текущим апстримом и раскомменченным кусочком кода и не делать PR еще один, тут я лучше решения твоего спрошу
There was a problem hiding this comment.
думаю, что лучше сделать еще один ПР в апстрим с исправлением
There was a problem hiding this comment.
Сделал, подождем когда примут
|
|
||
| # p = fraction of columns with an ink transition, s = ink density. __get_rid_spaces used to sit between | ||
| # base_line_image and s, but its `len(not_space) > 3` guard is the column count (always > 3 for a correct | ||
| # bbox), so it never stripped anything and only wasted a mean(0) -- dropping it is a no-op. (p_img > 0).mean() |
There was a problem hiding this comment.
возможно, в функции __get_rid_spaces был баг, и нужно было проверять not_space.sum()
прежде чем менять код, лучше потестировать старое поведение с исправлением потенциального бага, потому что текущая реализация определения жирности шрифта далеко не всегда работала хорошо
There was a problem hiding this comment.
Если пробелы действительно начнут убираться, надо проверять и исправлять поведение классификатора, т.к. он затюнен под другую среднюю плотность пикселей, относящихся к буквам. Так что я бы пока так оставил, и решал бы эту проблему отдельной задачкой
There was a problem hiding this comment.
Тогда предлагаю не удалять эту функцию, а просто закомментировать. И создать задачу в джире со ссылкой на этот комментарий
There was a problem hiding this comment.
Сделал, функцию вернул и закомментил
The bold classifier's ValleyEmphasisBinarizer.__get_threshold was ~37 ms/page, dominated by np.histogram (slow on uint8) plus a 254-iteration Python loop with a per-step np.sum. Replace with cv2.calcHist (per-value, SIMD) rebinned to the same 255-bin edges, and compute the cumulative omega/mu via cumsum and the neighbour-window sum via a cumsum difference. ~47x faster on the histogram (36.5 -> 0.77 ms) and the threshold is bit-identical (verified: same counts, zero threshold diff over 20 pages), so the bold output is unchanged. 297-page DAE report: byte-identical structure (nodes 3657, same text_sha), wall ~56.5 -> ~53.6 s. Now that the CPU pool is the wall bottleneck (per the per-process profile), this CPU cut registers on the wall.
…rizer After vectorizing __get_threshold, the two full-page boolean masks+assigns (image[<=t]=0; image[>t]=1, ~4 passes, 6.9 ms/page) became the dominant cost of ValleyEmphasisBinarizer.binarize. Replace with a single cv2.threshold (THRESH_BINARY, SIMD) -> ~0.5 ms. Byte-identical output (297-page DAE report unchanged: nodes 3657, same text_sha). The binarizer is now threshold (cv2.calcHist + cumsum) + binarize (cv2.threshold), with only the necessary BGR->GRAY cvtColor (0.38 ms) left.
__get_color_annotation built the non-white mask with 5 full-slice numpy passes (3 comparisons + 2 ANDs) and took the per-channel mean with 3 separate boolean-index gathers (image_slice[mask, i].mean() for i in range(3)) -- the gathers dominated at ~20 ms/page. Replace with SIMD cv2: cv2.inRange for the mask, then cv2.sumElems(bitwise_and)/countNonZero for the means. 24.6 -> 5.3 ms/page over 67 lines/page. Bit-identical: the uint8 sum is exact in float64 regardless of order, so channel_sum/count matches the old .mean() exactly (verified max |diff| = 0 over 1334 real line-bboxes; 297-page DAE report unchanged: nodes 3657, same text_sha).
…eval
__evaluation_one_bbox_image called __get_rid_spaces on the baseline band before
computing ink density, but its `len(not_space) > 3` guard tests the column count
(= width, always > 3 for a correct bbox), so it never stripped a column and only
burned a wasted mean(0) (~2.75 ms/page). Drop it and use base_line_image
directly. Also fold the transition count `p_img[abs>0]=1; p_img[<0]=0; mean()`
into a single (p_img > 0).mean() -- algebraically identical on the uint8 {0,1}
baseline. 11.6 -> 7.3 ms/page (-37%). Bit-identical: verified max |eval diff| = 0
over 1373 real word-slices; 297-page DAE report unchanged (nodes 3657, same
text_sha). Baseline-band top-2 loop left as-is (vectorizing it saved only
0.15 ms/page and risked tie-break divergence).
HoughLinesP was ~79% of table detection cost (~540 ms/page). It only needs the line angle (scale-invariant) and to draw gap-filling lines, so run it on a downscaled copy (length/gap params scaled) and upscale the line mask back to full resolution; cell OCR and contours stay full-res. New config key table_hough_scale (default 0.5) halves the table stage (534 -> 210 s over the 297-page doc) with tables and text preserved.
recognize_tables_from_image had no early-exit: it ran full contour/Hough detection on every page. Add a ~7 ms/page line-crossing signal that reproduces the detector's OWN line-detection parameters (fixed-225 threshold, short h/v morphology kernels floored at TableTree.min_w/h_cell) and count grid crossings; skip the detector when a page has fewer than table_line_gate_min_cross (default 2) crossings. Recall preserved -- gen_tables cell_recall/tables-per-doc identical to baseline, DAE tables=6 preserved, and gate ON vs OFF is byte- identical over 60 pages (0/69827 annotation diff: on a tableless page the gate returns the original image, which equals the detector's np.copy). Set table_line_gate_min_cross / DEDOC_TABLE_MIN_CROSS to 0 to disable.
…wall dedocutils SkewCorrector rotates the full page once per candidate angle over arange(-45, 46, 1) = 91 rotations/page -- the dominant cost of the scanned-image pipeline. FastSkewCorrector (a drop-in subclass, wired into PdfImageReader) brackets the peak with a coarse 3-degree sweep on a <=512 px thumbnail (rotation cost scales with area) then refines +-4 degrees at 1-degree step on a 1000 px image, and skips the final rotation when best_angle == 0 (rotate_image(x, 0) is an identity warp, so bit-identical there). Same projection-profile scoring, so the returned angle matches the 91-step whenever the grid brackets the same peak: 29/30 exact on the pdf_profiling/skew set (the lone divergence is a >28-degree page where the peak is inherently ambiguous), exact on realistic skew <=12 deg. DAE 297 pages: wall 1339 -> 1093 s (-18%), body word-bag F1 vs the 91-step output 0.99993, table cells identical, tables=6. DEDOC-style config-free.
In-process PDFium rendering is ~7% faster end-to-end than pdf2image/pdftoppm (no poppler subprocess, no per-batch PDF re-parse; document cached in memory, 2x2 erode to thicken glyphs). Added as _split_pdfium with a dispatcher and a pdf2image fallback. Kept OPT-IN (default DEDOC_RENDER=pdftoppm) because on master's Tesseract path it is NOT quality-neutral: PDFium's thinner glyph anti-aliasing shifts Tesseract's output even with the erode (which was tuned for the hybrid recognizer), measured -0.38% gen_texts word-bag F1 and 0.988 body word-bag F1 vs poppler on the 297-page DAE report. The default path (_split_pdftoppm) is the original code, byte-identical to before.
Report for feature/cpu-optimizations: 4 default fixes (metadata, HoughLinesP, table-gate, deskew) = -28.0% on the 297-page DAE report, all bit-identical or quality-validated; opt-in pdfium render (-7.5% more but degrades Tesseract OCR).
auto_tabby ran the tabby Java subprocess twice per document: once in TxtLayerDetector.__classify_all_pages (first 8 pages, no tables) to feed the text-layer classifier, then again in PdfAutoReader.__parse_document for the real extraction. __parse_document already reuses a detection-provided document (TxtLayerResult.document), but __classify_all_pages never populated it. When the whole document fits inside the 8-page detection window, extract it once with the full parameters (all pages + tables, exactly __parse_document's own read) and pass it through for reuse -- eliminating the second JVM startup + re-extraction. Bit-identical output (verified: identical node/table counts and text SHA on prospectus/VVP_6_tables/short_lines/example), ~2x faster on small text-layer documents (5.93->3.36 s, 2.77->1.36 s, ...). Documents larger than the detection window keep the cheap 8-page detection and are unchanged.
The tabby JAR is invoked as a single java subprocess that uses only ~2.25 of 16 cores (measured: 26 s CPU over 11.6 s wall on the 145-page riscv spec), so a large document leaves the machine mostly idle. Split the page range into contiguous chunks run as concurrent tabby subprocesses (the jar already accepts -sp/-ep) and merge. This is bit-identical: tabby numbers pages absolutely and its per-page output is page-local (verified 145/145 pages byte-identical to the single call), so merging is just concatenating the per-chunk `pages` lists; multi-page tables spanning a chunk boundary are still assembled downstream by convert_to_multipages_tables from the page fragments. riscv 145 pages: full parse 14.15 -> 11.30 s (-20%), identical reader output (node/table counts + text/cell SHA); the tabby extraction step alone goes ~11.6 -> ~6.4 s. Each chunk JVM runs -XX:+UseSerialGC -Xmx1024m to keep N concurrent processes from saturating memory bandwidth with parallel GC threads. Default: documents >= 40 pages split into 2-4 chunks (min 20 pages/chunk, cap 4); smaller docs, the GOST-frame path and open-ended ranges keep the single call. Tunables: tabby_parallel_chunks / DEDOC_TABBY_CHUNKS (cap, 1 disables), tabby_parallel_min_pages_per_chunk, DEDOC_TABBY_JVM_ARGS. Also fixes the data.json read to specify encoding=utf-8 (RU-locale Windows cp1251 bug).
BBoxAnnotation.__init__ called json.dumps(bbox.to_relative_dict(...)) on every construction -- once per line bounding box, tens of thousands of times per document (46919 on the 145-page riscv spec, where the json encoder was the single largest post-processing cost, ~0.57 s). Build the fixed 6-key JSON string directly with an f-string instead: Python's str(float) equals repr and json's float encoding, and int str equals json's, so the output is byte-identical to json.dumps (verified over 122820 bbox values). Micro-benchmark: the whole new constructor (3.02 us) is faster than the old json.dumps call alone (4.78 us).
…an + compact JSON Rebuild the tabby table-extraction engine (ispras_tbl_extr.jar) with CPU optimizations. Engine time on a 145-page document drops ~11.8s -> ~3.7s; end-to-end auto_tabby through dedoc ~13.9s -> ~7.4s (~1.9x). - ruling extraction: the per-page ruling detection (a pixel scan of the rendered page) now runs across all available cores; the page render itself stays serialized, so the result does not depend on scheduling order - ruling scan: read the pixel band value directly instead of allocating an int[] for every pixel -- billions of short-lived allocations on a large document - text extraction: drop a per-character list that was built for every glyph and never read by anything - JSON output: build and serialize each page in parallel and emit compact JSON instead of a single pretty-printed pass; data.json shrinks ~45%, which also speeds up the json.load on the Python side The first three keep data.json byte-identical to the previous jar. Compact JSON changes the bytes but not the parsed content (data.json is consumed with json.load, which is whitespace-insensitive): verified parse-identical across all of tests/data/pdf_with_text_layer, and dedoc's final output (structure, tables, annotations) verified identical end-to-end.
…pying Cell splitting (CellSplitter._merge_close_borders / __split_one_cell) copies a cell per cell and per split sub-cell only to rewrite its geometry and flags -- it never touches the line contents. The lines (text + annotations) are by far the heaviest part of a cell, so deep-copying them was pure overhead. Share the line objects (each copy still gets its own list, so list-level edits stay independent) and deep-copy only the small geometry (bbox, contour_coord), which the splitter and Cell.shift do mutate. -0.21s on a 145-page document (auto_tabby, warm). Verified: dedoc's final output (structure + tables + annotations) is identical on the tabby path (riscv-spec-v2.2, VVP_6_tables, VVP_global_table, big_table_with_merged_cells, Document635) and on the image-reader path (three scanned table images compared against the previous deepcopy behaviour).
…remaining pages The textual layer detection already runs a complete tabby extraction of the first 8 pages -- the tabby reader ignores need_pdf_table_analysis, so tables are extracted too -- and then threw that extraction away, leaving the subsequent full read to extract those pages a second time. Hand tabby's raw per-page output over (TxtLayerResult.detected_pages) so PdfTabbyReader extracts only the pages the detection did not cover. The pages are merged at the raw ``pages`` level, *before* the per-page processing, which is what keeps cross-page merging intact: this is the same invariant __process_pdf_parallel already relies on (absolute page numbers, contiguous disjoint ranges). Splitting the work into separate reads per page range instead tears paragraphs that span the boundary, so it is not done that way. Restricted to auto_tabby (under "auto" the remaining pages are read by pdf_txtlayer_reader, which cannot consume tabby's pages), to reads starting at page 1, and to runs without attachments (extracted image files live in the detection read's temporary directory, which is gone by then). Measured (auto_tabby, warm): article (25p) 7.78s -> 6.03s (-23%), riscv-spec-v2.2 (145p) ~-0.2s; the gain scales with the content density of the first 8 pages. Verified identical dedoc output (structure + tables + annotations) on multipage (the 8/9 boundary case), with_changed_header_footer, mongolo, article, riscv-spec-v2.2, and on example.pdf as an unaffected control.
…ction The chunking existed to work around the old jar using only ~2 of the machine's cores: splitting a large document into contiguous page ranges and running several tabby JVMs concurrently was a net win back then. The rebuilt jar parallelizes internally across all cores, so those extra JVMs now only contend with it -- measured on a 145-page document, auto_tabby warm: 8.25s with the default 4 chunks vs 7.36s with chunking off. Reverts the machinery added in 9be9f63 (__parallel_chunk_count, __process_pdf_parallel, __run's jvm_args) together with its tunables (tabby_parallel_chunks, tabby_parallel_min_pages_per_chunk, DEDOC_TABBY_CHUNKS, DEDOC_TABBY_MIN_PAGES_PER_CHUNK, DEDOC_TABBY_JVM_ARGS), restoring the single-call extraction. The raw-pages reuse from the textual layer detection is unaffected: it merges the ``pages`` lists on the same invariant the chunking relied on (page-local output, absolute page numbers). Output verified identical (dedoc's final structure, tables and annotations) on the 145-page document.
Internal working report, not needed in the pull request.
8b7a7e2 to
b0d845d
Compare
This comment was marked as outdated.
This comment was marked as outdated.
The jar in this branch was built from a reconstructed source tree. The engine sources turned out to be public (github.com/sunveil/ispras_tbl_extr), so rebuild it from there instead: upstream main plus the three commits sent as sunveil/ispras_tbl_extr#1, and nothing else. - the per-page ruling detection runs on all cores, the page render stays serialized on the shared PDDocument: ruling extraction 7.19s -> 1.92s on a 145-page document - the ruling scans read the pixel band value directly instead of allocating an int[] per pixel: 1.92s -> 1.04s - new opt-in "-c" flag writes data.json without pretty printing, which this reader now passes: writing data.json 236ms -> 167ms, the file -44% Those three keep the engine output identical (byte-identical without -c, parse-identical with it, over the 20 PDFs of tests/data/pdf_with_text_layer). Rebuilding from the public sources does bring the upstream changes made since the jar dedoc ships was built: the style of a word is taken from its central glyph (fixes bold/font runs attributed to the wrong font), rotated text is no longer dropped but comes out as separate fragments, and the y_top_left of an extracted image is now the raw PDF y instead of a page coordinate.
отправил PR в dedoc-utils, там проблема не только в проходе по уменьшенной картинке, если взять полное разрешение, то на моих данных совпадения полного всё равно не будет. Восстановил одну эвристику хорошую, которая гласит что если в двух лучших точках результат почти одинаковый, то лучше взять среднюю между ними, а не лучшую из двух. Вроде с остальными правками результаты совпадают на моих картинках со старыми версиями |
Reusing the detection extraction for the remaining pages passed tabby's raw per-page output through private "__tabby_raw_pages_in/out" keys in the parameters dict, which made the data flow between the reader and the textual layer detection implicit, and taught the tabby reader about a detail of the detection it should not know. It also had to switch itself off when attachments were requested, because the extracted image files live in the temporary directory of the first read. Skipping the second read for documents that fit in the detection window stays - it needs no channel between the two.
…d test it
The alternative rasterizer was switched on by the DEDOC_RENDER environment
variable, which is not how the rest of the reader is configured. It is
config["pdf_renderer"] now ("pdftoppm" by default, "pdfium" for pypdfium2), and
pypdfium2 is declared in requirements.txt - it was imported without being a
dependency.
tests/unit_tests/test_module_pdf_renderer.py covers what the switch has to
guarantee: both renderers return the same pages at the same resolution for the
scanned test documents, poppler stays the default, and an unknown value falls
back to poppler byte for byte.
The line-crossing gate converted the page to grayscale and so did __rec_tables_from_img right after it, on every page that passed the gate. recognize_tables_from_image does it once now and hands the result to both. The gate threshold moves from the DEDOC_TABLE_MIN_CROSS environment variable to config["table_line_gate_min_cross"], the way the renderer switch does - the config is where the reader is configured.
NastyBoget
left a comment
There was a problem hiding this comment.
Нужно исправить упавшие тесты
The function never stripped anything - its guard checks len(not_space), the number of columns, which is always > 3 for a correct bbox - so removing the call did not change the boldness evaluation. Keep the body next to the code that would use it, with a note on what fixing the guard does, so the open question about it is not lost.
sunveil/ispras_tbl_extr#1 (the performance work this branch carried as a local patch) and #2 (skip text rotated inside an upright page) are both merged upstream, so the jar is now a plain build of upstream main (c203af4) with nothing applied on top of it. Against the jar this branch shipped, exactly one class changes, PDContentExtractor: #2 puts the getDir() != 0 filter back, so text rotated inside an upright page is skipped again instead of coming out as separate fragments. That fixes the one test the pipeline was failing, test_pdf_tabby_reader (test_module_gost_frame_recognizer), where the rotated stamp text of the GOST frame became the first line of the document ('В' instead of '1. Sample text 1').
Upstream 90fd411 ("Fix PDF images coordinates") made ImageExtractor report the
raw PDF y of an extracted image - measured bottom-up from the page corner -
while every other coordinate in data.json is top-down. Checked against
pdfminer.six over the corpus, 2 of 45 image boxes come out right that way
against 34 of 45 before it.
dedoc links an attachment to the paragraph it sits next to by that box, so the
upside-down y silently unlinks images: test_pdf_tabby_images_refs asserts an
"attachment" annotation on each of the three paragraphs of
with_attachments_1.docx.pdf and gets 'style' for the first two.
The regression came in with the jar rebuild in df94059 and has been in this
branch since; the api tests only reach it now that the unit stage passes.
Reproduced outside the api harness against four jars - it fails both with the
jar df94059 shipped and with plain upstream main, and passes with this one -
and the tabby unit modules are unchanged either way.
So the jar is upstream main (c203af4) plus this one commit, which is not sent
upstream yet. It touches ImageExtractor and JaksonWriter; every other class is
upstream's.
… are read from apply_houph_line returned a copy of the page with the gap-filling lines drawn on it, and with the downscale that copy was the HALF-SIZE page pushed back up through INTER_CUBIC. get_contours_cells finds the cell contours on exactly that image, so the whole page went through a lossy 0.5x round trip and every cell boundary moved a little - which moves the crop each cell is OCRed from, and so changes what Tesseract reads out of it. On tests/data/tables (33 documents, first 3 pages, image path), against the full-resolution behaviour: before, scale 0.5 17 / 33 documents changed after, scale 0.5 6 / 33 scale 1.0 0 / 33 (same code path as develop) That is what test_multipage_gost_table_image was failing on: the branch read the last cell of the GOST table differently from develop. On the full document the table now comes out exactly as develop has it, 40 rows, cell for cell. Detection still runs on the downscaled copy - only the endpoints come back to full resolution before they are drawn, and minLineLength/maxLineGap are scaled with the image (they were not, so a half-size page silently demanded lines twice as long). apply_houph_line 113.6 -> 32.1 ms/page. Table stage, min of 3, ms/page (develop -> this): page with a table 311.0 -> 227.5 lines but no table 185.5 -> 103.1 no lines at all 56.2 -> 6.1 (the line-crossing gate, unchanged) Verified against a local dedoc API: tests/api_tests/test_api_module_table_recognizer and test_api_misc_multipage_table, 27 tests, behave identically at 0.5 and 1.0 - same 3 failures either way, all local (CRLF, a Cyrillic/Latin 'a' from a newer Tesseract, a Windows path separator in an upload name).
558da69 to
1bd06f6
Compare
|
Поправил тесты, главная проблема была в том что у меня не совсем тот же набор библиотек на винде, что были на CI, и этого хватало чтобы у меня работало с минимальным количеством фиксов, а для подтверждения в CI нужен был еще 1 фикс в jar, PR я создал, ждем чтобы добавили всё |
I reworked tabby pipeline to be more optimal for CPU inference.
I decompiled jar and reworked it with some optimizations + some parts of java code became parallel. Can provide decompiled code, didn't push it here because there is no source code for previous version.
Also added some CPU-oriented optimizations, like vectorization for bold classifier, two-staged skew correction and tested pdftoppm vs pdfium(so currently we have choice between faster version without perfect match and older one which is definitely compatible with everything)
Please check what fixes are relevant for merging to master, on my system it made more than x2 speed improvement for documents with correct text layer