Skip to content

Feature/cpu optimizations - #562

Open
OligerMan wants to merge 29 commits into
developfrom
feature/cpu-optimizations
Open

Feature/cpu optimizations#562
OligerMan wants to merge 29 commits into
developfrom
feature/cpu-optimizations

Conversation

@OligerMan

Copy link
Copy Markdown
Collaborator

I reworked tabby pipeline to be more optimal for CPU inference.

I decompiled jar and reworked it with some optimizations + some parts of java code became parallel. Can provide decompiled code, didn't push it here because there is no source code for previous version.
Also added some CPU-oriented optimizations, like vectorization for bold classifier, two-staged skew correction and tested pdftoppm vs pdfium(so currently we have choice between faster version without perfect match and older one which is definitely compatible with everything)

Please check what fixes are relevant for merging to master, on my system it made more than x2 speed improvement for documents with correct text layer

@OligerMan
OligerMan requested review from NastyBoget and Travvy88 July 16, 2026 15:12
@NastyBoget NastyBoget added the enhancement New feature or request label Jul 27, 2026
@NastyBoget
NastyBoget changed the base branch from master to develop July 27, 2026 08:00
@NastyBoget

Copy link
Copy Markdown
Collaborator
  1. На будущее: pr делаем в develop (возможно потом пересмотрим)
  2. Пайплайны должны пройти
  3. Инструкция по разработке https://dedoc.readthedocs.io/en/latest/contributing/contributing.html

@NastyBoget
NastyBoget force-pushed the feature/cpu-optimizations branch from 1535a03 to 567e660 Compare July 31, 2026 08:57
Comment thread dedoc/readers/pdf_reader/pdf_auto_reader/txtlayer_detector.py Outdated
@NastyBoget
NastyBoget force-pushed the feature/cpu-optimizations branch 5 times, most recently from 1afbe1f to 8b7a7e2 Compare July 31, 2026 10:03

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

я не могу одобрить мерж, пока исходный код tabby с изменениями не будет запушен и одобрен Андреем Михайловым

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ждём одобрения патча для исходников tabby. Там изменения оформлены отдельными коммитами sunveil/ispras_tbl_extr#1

  1. параллельный поиск линеек таблиц,
  2. чтение пикселя без аллокации int[] на каждый пиксель и
  3. опциональный флаг -c для компактного data.json.

Первые две правки не меняют вывод движка: data.json побайтово идентичен сборке из main на всех 20 PDF из tests/data/pdf_with_text_layer, включая путь -sp/-ep; третья по умолчанию выключена, dedoc включает её флагом. На 145-страничном документе движок ускоряется с 9.9 с до 3.5 с (поиск линеек 7.19 → 1.04 с), data.json становится меньше на 44%.

Jar в этой ветке пересобран строго из той ветки, что в PR — апстримовый main плюс эти три коммита, больше ничего. Побочный эффект: вместе с ними приезжают изменения апстрима, сделанные после сборки нынешнего jar

Плюс помимо оптимизаций возможно добавим одно изменение, там вполне возможно баг с координатами картинок есть

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Патч добавили, но как я выяснил по тестам, нужен еще один, по сути раскомметировать старую механику, которую закомментили между тем как написаны тесты и тем что я добавил в текущий апстрим патчи, и этой механики не хватает для соответствия тестам. По сути сломал тесты не мой патч, а обновления апстрима, которые мы не вливали до этого к себе. Можно в целом собрать jar с текущим апстримом и раскомменченным кусочком кода и не делать PR еще один, тут я лучше решения твоего спрошу

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

думаю, что лучше сделать еще один ПР в апстрим с исправлением

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Сделал, подождем когда примут

Comment thread dedoc/readers/pdf_reader/pdf_base_reader.py
Comment thread dedoc/readers/pdf_reader/pdf_auto_reader/txtlayer_detector.py Outdated

# p = fraction of columns with an ink transition, s = ink density. __get_rid_spaces used to sit between
# base_line_image and s, but its `len(not_space) > 3` guard is the column count (always > 3 for a correct
# bbox), so it never stripped anything and only wasted a mean(0) -- dropping it is a no-op. (p_img > 0).mean()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

возможно, в функции __get_rid_spaces был баг, и нужно было проверять not_space.sum()

прежде чем менять код, лучше потестировать старое поведение с исправлением потенциального бага, потому что текущая реализация определения жирности шрифта далеко не всегда работала хорошо

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Если пробелы действительно начнут убираться, надо проверять и исправлять поведение классификатора, т.к. он затюнен под другую среднюю плотность пикселей, относящихся к буквам. Так что я бы пока так оставил, и решал бы эту проблему отдельной задачкой

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Тогда предлагаю не удалять эту функцию, а просто закомментировать. И создать задачу в джире со ссылкой на этот комментарий

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Сделал, функцию вернул и закомментил

Comment thread dedoc/readers/pdf_reader/pdf_image_reader/table_recognizer/table_recognizer.py Outdated
OligerMan and others added 12 commits August 3, 2026 16:21
The bold classifier's ValleyEmphasisBinarizer.__get_threshold was ~37 ms/page,
dominated by np.histogram (slow on uint8) plus a 254-iteration Python loop with
a per-step np.sum. Replace with cv2.calcHist (per-value, SIMD) rebinned to the
same 255-bin edges, and compute the cumulative omega/mu via cumsum and the
neighbour-window sum via a cumsum difference. ~47x faster on the histogram
(36.5 -> 0.77 ms) and the threshold is bit-identical (verified: same counts,
zero threshold diff over 20 pages), so the bold output is unchanged.

297-page DAE report: byte-identical structure (nodes 3657, same text_sha),
wall ~56.5 -> ~53.6 s. Now that the CPU pool is the wall bottleneck (per the
per-process profile), this CPU cut registers on the wall.
…rizer

After vectorizing __get_threshold, the two full-page boolean masks+assigns
(image[<=t]=0; image[>t]=1, ~4 passes, 6.9 ms/page) became the dominant cost
of ValleyEmphasisBinarizer.binarize. Replace with a single cv2.threshold
(THRESH_BINARY, SIMD) -> ~0.5 ms. Byte-identical output (297-page DAE report
unchanged: nodes 3657, same text_sha). The binarizer is now threshold
(cv2.calcHist + cumsum) + binarize (cv2.threshold), with only the necessary
BGR->GRAY cvtColor (0.38 ms) left.
__get_color_annotation built the non-white mask with 5 full-slice numpy passes
(3 comparisons + 2 ANDs) and took the per-channel mean with 3 separate
boolean-index gathers (image_slice[mask, i].mean() for i in range(3)) -- the
gathers dominated at ~20 ms/page. Replace with SIMD cv2: cv2.inRange for the
mask, then cv2.sumElems(bitwise_and)/countNonZero for the means. 24.6 -> 5.3
ms/page over 67 lines/page. Bit-identical: the uint8 sum is exact in float64
regardless of order, so channel_sum/count matches the old .mean() exactly
(verified max |diff| = 0 over 1334 real line-bboxes; 297-page DAE report
unchanged: nodes 3657, same text_sha).
…eval

__evaluation_one_bbox_image called __get_rid_spaces on the baseline band before
computing ink density, but its `len(not_space) > 3` guard tests the column count
(= width, always > 3 for a correct bbox), so it never stripped a column and only
burned a wasted mean(0) (~2.75 ms/page). Drop it and use base_line_image
directly. Also fold the transition count `p_img[abs>0]=1; p_img[<0]=0; mean()`
into a single (p_img > 0).mean() -- algebraically identical on the uint8 {0,1}
baseline. 11.6 -> 7.3 ms/page (-37%). Bit-identical: verified max |eval diff| = 0
over 1373 real word-slices; 297-page DAE report unchanged (nodes 3657, same
text_sha). Baseline-band top-2 loop left as-is (vectorizing it saved only
0.15 ms/page and risked tie-break divergence).
HoughLinesP was ~79% of table detection cost (~540 ms/page). It only
needs the line angle (scale-invariant) and to draw gap-filling lines, so
run it on a downscaled copy (length/gap params scaled) and upscale the
line mask back to full resolution; cell OCR and contours stay full-res.
New config key table_hough_scale (default 0.5) halves the table stage
(534 -> 210 s over the 297-page doc) with tables and text preserved.
recognize_tables_from_image had no early-exit: it ran full contour/Hough
detection on every page. Add a ~7 ms/page line-crossing signal that reproduces
the detector's OWN line-detection parameters (fixed-225 threshold, short h/v
morphology kernels floored at TableTree.min_w/h_cell) and count grid crossings;
skip the detector when a page has fewer than table_line_gate_min_cross (default
2) crossings. Recall preserved -- gen_tables cell_recall/tables-per-doc
identical to baseline, DAE tables=6 preserved, and gate ON vs OFF is byte-
identical over 60 pages (0/69827 annotation diff: on a tableless page the gate
returns the original image, which equals the detector's np.copy). Set
table_line_gate_min_cross / DEDOC_TABLE_MIN_CROSS to 0 to disable.
…wall

dedocutils SkewCorrector rotates the full page once per candidate angle over
arange(-45, 46, 1) = 91 rotations/page -- the dominant cost of the scanned-image
pipeline. FastSkewCorrector (a drop-in subclass, wired into PdfImageReader)
brackets the peak with a coarse 3-degree sweep on a <=512 px thumbnail (rotation
cost scales with area) then refines +-4 degrees at 1-degree step on a 1000 px
image, and skips the final rotation when best_angle == 0 (rotate_image(x, 0) is
an identity warp, so bit-identical there). Same projection-profile scoring, so
the returned angle matches the 91-step whenever the grid brackets the same peak:
29/30 exact on the pdf_profiling/skew set (the lone divergence is a >28-degree
page where the peak is inherently ambiguous), exact on realistic skew <=12 deg.
DAE 297 pages: wall 1339 -> 1093 s (-18%), body word-bag F1 vs the 91-step
output 0.99993, table cells identical, tables=6. DEDOC-style config-free.
In-process PDFium rendering is ~7% faster end-to-end than pdf2image/pdftoppm
(no poppler subprocess, no per-batch PDF re-parse; document cached in memory,
2x2 erode to thicken glyphs). Added as _split_pdfium with a dispatcher and a
pdf2image fallback. Kept OPT-IN (default DEDOC_RENDER=pdftoppm) because on
master's Tesseract path it is NOT quality-neutral: PDFium's thinner glyph
anti-aliasing shifts Tesseract's output even with the erode (which was tuned for
the hybrid recognizer), measured -0.38% gen_texts word-bag F1 and 0.988 body
word-bag F1 vs poppler on the 297-page DAE report. The default path
(_split_pdftoppm) is the original code, byte-identical to before.
Report for feature/cpu-optimizations: 4 default fixes (metadata, HoughLinesP,
table-gate, deskew) = -28.0% on the 297-page DAE report, all bit-identical or
quality-validated; opt-in pdfium render (-7.5% more but degrades Tesseract OCR).
auto_tabby ran the tabby Java subprocess twice per document: once in
TxtLayerDetector.__classify_all_pages (first 8 pages, no tables) to feed the
text-layer classifier, then again in PdfAutoReader.__parse_document for the real
extraction. __parse_document already reuses a detection-provided document
(TxtLayerResult.document), but __classify_all_pages never populated it.

When the whole document fits inside the 8-page detection window, extract it once
with the full parameters (all pages + tables, exactly __parse_document's own
read) and pass it through for reuse -- eliminating the second JVM startup +
re-extraction. Bit-identical output (verified: identical node/table counts and
text SHA on prospectus/VVP_6_tables/short_lines/example), ~2x faster on small
text-layer documents (5.93->3.36 s, 2.77->1.36 s, ...). Documents larger than
the detection window keep the cheap 8-page detection and are unchanged.
The tabby JAR is invoked as a single java subprocess that uses only ~2.25 of 16
cores (measured: 26 s CPU over 11.6 s wall on the 145-page riscv spec), so a
large document leaves the machine mostly idle. Split the page range into
contiguous chunks run as concurrent tabby subprocesses (the jar already accepts
-sp/-ep) and merge. This is bit-identical: tabby numbers pages absolutely and
its per-page output is page-local (verified 145/145 pages byte-identical to the
single call), so merging is just concatenating the per-chunk `pages` lists;
multi-page tables spanning a chunk boundary are still assembled downstream by
convert_to_multipages_tables from the page fragments.

riscv 145 pages: full parse 14.15 -> 11.30 s (-20%), identical reader output
(node/table counts + text/cell SHA); the tabby extraction step alone goes
~11.6 -> ~6.4 s. Each chunk JVM runs -XX:+UseSerialGC -Xmx1024m to keep N
concurrent processes from saturating memory bandwidth with parallel GC threads.

Default: documents >= 40 pages split into 2-4 chunks (min 20 pages/chunk, cap
4); smaller docs, the GOST-frame path and open-ended ranges keep the single
call. Tunables: tabby_parallel_chunks / DEDOC_TABBY_CHUNKS (cap, 1 disables),
tabby_parallel_min_pages_per_chunk, DEDOC_TABBY_JVM_ARGS. Also fixes the
data.json read to specify encoding=utf-8 (RU-locale Windows cp1251 bug).
BBoxAnnotation.__init__ called json.dumps(bbox.to_relative_dict(...)) on every
construction -- once per line bounding box, tens of thousands of times per
document (46919 on the 145-page riscv spec, where the json encoder was the
single largest post-processing cost, ~0.57 s). Build the fixed 6-key JSON string
directly with an f-string instead: Python's str(float) equals repr and json's
float encoding, and int str equals json's, so the output is byte-identical to
json.dumps (verified over 122820 bbox values). Micro-benchmark: the whole new
constructor (3.02 us) is faster than the old json.dumps call alone (4.78 us).
OligerMan and others added 7 commits August 3, 2026 16:21
…an + compact JSON

Rebuild the tabby table-extraction engine (ispras_tbl_extr.jar) with CPU
optimizations. Engine time on a 145-page document drops ~11.8s -> ~3.7s;
end-to-end auto_tabby through dedoc ~13.9s -> ~7.4s (~1.9x).

- ruling extraction: the per-page ruling detection (a pixel scan of the rendered
  page) now runs across all available cores; the page render itself stays
  serialized, so the result does not depend on scheduling order
- ruling scan: read the pixel band value directly instead of allocating an int[]
  for every pixel -- billions of short-lived allocations on a large document
- text extraction: drop a per-character list that was built for every glyph and
  never read by anything
- JSON output: build and serialize each page in parallel and emit compact JSON
  instead of a single pretty-printed pass; data.json shrinks ~45%, which also
  speeds up the json.load on the Python side

The first three keep data.json byte-identical to the previous jar. Compact JSON
changes the bytes but not the parsed content (data.json is consumed with
json.load, which is whitespace-insensitive): verified parse-identical across all
of tests/data/pdf_with_text_layer, and dedoc's final output (structure, tables,
annotations) verified identical end-to-end.
…pying

Cell splitting (CellSplitter._merge_close_borders / __split_one_cell) copies a
cell per cell and per split sub-cell only to rewrite its geometry and flags --
it never touches the line contents. The lines (text + annotations) are by far
the heaviest part of a cell, so deep-copying them was pure overhead.

Share the line objects (each copy still gets its own list, so list-level edits
stay independent) and deep-copy only the small geometry (bbox, contour_coord),
which the splitter and Cell.shift do mutate.

-0.21s on a 145-page document (auto_tabby, warm). Verified: dedoc's final output
(structure + tables + annotations) is identical on the tabby path
(riscv-spec-v2.2, VVP_6_tables, VVP_global_table, big_table_with_merged_cells,
Document635) and on the image-reader path (three scanned table images compared
against the previous deepcopy behaviour).
…remaining pages

The textual layer detection already runs a complete tabby extraction of the
first 8 pages -- the tabby reader ignores need_pdf_table_analysis, so tables are
extracted too -- and then threw that extraction away, leaving the subsequent
full read to extract those pages a second time.

Hand tabby's raw per-page output over (TxtLayerResult.detected_pages) so
PdfTabbyReader extracts only the pages the detection did not cover. The pages
are merged at the raw ``pages`` level, *before* the per-page processing, which
is what keeps cross-page merging intact: this is the same invariant
__process_pdf_parallel already relies on (absolute page numbers, contiguous
disjoint ranges). Splitting the work into separate reads per page range instead
tears paragraphs that span the boundary, so it is not done that way.

Restricted to auto_tabby (under "auto" the remaining pages are read by
pdf_txtlayer_reader, which cannot consume tabby's pages), to reads starting at
page 1, and to runs without attachments (extracted image files live in the
detection read's temporary directory, which is gone by then).

Measured (auto_tabby, warm): article (25p) 7.78s -> 6.03s (-23%),
riscv-spec-v2.2 (145p) ~-0.2s; the gain scales with the content density of the
first 8 pages. Verified identical dedoc output (structure + tables +
annotations) on multipage (the 8/9 boundary case), with_changed_header_footer,
mongolo, article, riscv-spec-v2.2, and on example.pdf as an unaffected control.
…ction

The chunking existed to work around the old jar using only ~2 of the machine's
cores: splitting a large document into contiguous page ranges and running
several tabby JVMs concurrently was a net win back then. The rebuilt jar
parallelizes internally across all cores, so those extra JVMs now only contend
with it -- measured on a 145-page document, auto_tabby warm: 8.25s with the
default 4 chunks vs 7.36s with chunking off.

Reverts the machinery added in 9be9f63 (__parallel_chunk_count,
__process_pdf_parallel, __run's jvm_args) together with its tunables
(tabby_parallel_chunks, tabby_parallel_min_pages_per_chunk, DEDOC_TABBY_CHUNKS,
DEDOC_TABBY_MIN_PAGES_PER_CHUNK, DEDOC_TABBY_JVM_ARGS), restoring the
single-call extraction. The raw-pages reuse from the textual layer detection is
unaffected: it merges the ``pages`` lists on the same invariant the chunking
relied on (page-local output, absolute page numbers).

Output verified identical (dedoc's final structure, tables and annotations) on
the 145-page document.
Internal working report, not needed in the pull request.
@NastyBoget
NastyBoget force-pushed the feature/cpu-optimizations branch from 8b7a7e2 to b0d845d Compare August 3, 2026 13:21
Comment thread dedoc/readers/pdf_reader/pdf_image_reader/fast_skew_corrector.py Outdated
@NastyBoget

This comment was marked as outdated.

The jar in this branch was built from a reconstructed source tree. The engine
sources turned out to be public (github.com/sunveil/ispras_tbl_extr), so rebuild
it from there instead: upstream main plus the three commits sent as
sunveil/ispras_tbl_extr#1, and nothing else.

- the per-page ruling detection runs on all cores, the page render stays
  serialized on the shared PDDocument: ruling extraction 7.19s -> 1.92s on a
  145-page document
- the ruling scans read the pixel band value directly instead of allocating an
  int[] per pixel: 1.92s -> 1.04s
- new opt-in "-c" flag writes data.json without pretty printing, which this
  reader now passes: writing data.json 236ms -> 167ms, the file -44%

Those three keep the engine output identical (byte-identical without -c,
parse-identical with it, over the 20 PDFs of tests/data/pdf_with_text_layer).

Rebuilding from the public sources does bring the upstream changes made since
the jar dedoc ships was built: the style of a word is taken from its central
glyph (fixes bold/font runs attributed to the wrong font), rotated text is no
longer dropped but comes out as separate fragments, and the y_top_left of an
extracted image is now the raw PDF y instead of a page coordinate.
@OligerMan

Copy link
Copy Markdown
Collaborator Author

skew corrector нужно будет поправить - я бы предложила искать итоговый точный угол на исходной картинке (без ее уменьшения)

отправил PR в dedoc-utils, там проблема не только в проходе по уменьшенной картинке, если взять полное разрешение, то на моих данных совпадения полного всё равно не будет. Восстановил одну эвристику хорошую, которая гласит что если в двух лучших точках результат почти одинаковый, то лучше взять среднюю между ними, а не лучшую из двух. Вроде с остальными правками результаты совпадают на моих картинках со старыми версиями

OligerMan and others added 4 commits September 1, 2026 19:31
Reusing the detection extraction for the remaining pages passed tabby's raw
per-page output through private "__tabby_raw_pages_in/out" keys in the
parameters dict, which made the data flow between the reader and the textual
layer detection implicit, and taught the tabby reader about a detail of the
detection it should not know. It also had to switch itself off when attachments
were requested, because the extracted image files live in the temporary
directory of the first read.

Skipping the second read for documents that fit in the detection window stays -
it needs no channel between the two.
…d test it

The alternative rasterizer was switched on by the DEDOC_RENDER environment
variable, which is not how the rest of the reader is configured. It is
config["pdf_renderer"] now ("pdftoppm" by default, "pdfium" for pypdfium2), and
pypdfium2 is declared in requirements.txt - it was imported without being a
dependency.

tests/unit_tests/test_module_pdf_renderer.py covers what the switch has to
guarantee: both renderers return the same pages at the same resolution for the
scanned test documents, poppler stays the default, and an unknown value falls
back to poppler byte for byte.
The line-crossing gate converted the page to grayscale and so did
__rec_tables_from_img right after it, on every page that passed the gate.
recognize_tables_from_image does it once now and hands the result to both.

The gate threshold moves from the DEDOC_TABLE_MIN_CROSS environment variable to
config["table_line_gate_min_cross"], the way the renderer switch does - the
config is where the reader is configured.
@NastyBoget
NastyBoget self-requested a review September 8, 2026 13:22

@NastyBoget NastyBoget left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Нужно исправить упавшие тесты

The function never stripped anything - its guard checks len(not_space), the
number of columns, which is always > 3 for a correct bbox - so removing the call
did not change the boldness evaluation. Keep the body next to the code that
would use it, with a note on what fixing the guard does, so the open question
about it is not lost.
sunveil/ispras_tbl_extr#1 (the performance work this branch carried as a local
patch) and #2 (skip text rotated inside an upright page) are both merged
upstream, so the jar is now a plain build of upstream main (c203af4) with
nothing applied on top of it.

Against the jar this branch shipped, exactly one class changes,
PDContentExtractor: #2 puts the getDir() != 0 filter back, so text rotated
inside an upright page is skipped again instead of coming out as separate
fragments. That fixes the one test the pipeline was failing,
test_pdf_tabby_reader (test_module_gost_frame_recognizer), where the rotated
stamp text of the GOST frame became the first line of the document
('В' instead of '1. Sample text 1').
Upstream 90fd411 ("Fix PDF images coordinates") made ImageExtractor report the
raw PDF y of an extracted image - measured bottom-up from the page corner -
while every other coordinate in data.json is top-down. Checked against
pdfminer.six over the corpus, 2 of 45 image boxes come out right that way
against 34 of 45 before it.

dedoc links an attachment to the paragraph it sits next to by that box, so the
upside-down y silently unlinks images: test_pdf_tabby_images_refs asserts an
"attachment" annotation on each of the three paragraphs of
with_attachments_1.docx.pdf and gets 'style' for the first two.

The regression came in with the jar rebuild in df94059 and has been in this
branch since; the api tests only reach it now that the unit stage passes.
Reproduced outside the api harness against four jars - it fails both with the
jar df94059 shipped and with plain upstream main, and passes with this one -
and the tabby unit modules are unchanged either way.

So the jar is upstream main (c203af4) plus this one commit, which is not sent
upstream yet. It touches ImageExtractor and JaksonWriter; every other class is
upstream's.
… are read from

apply_houph_line returned a copy of the page with the gap-filling lines drawn on
it, and with the downscale that copy was the HALF-SIZE page pushed back up
through INTER_CUBIC. get_contours_cells finds the cell contours on exactly that
image, so the whole page went through a lossy 0.5x round trip and every cell
boundary moved a little - which moves the crop each cell is OCRed from, and so
changes what Tesseract reads out of it.

On tests/data/tables (33 documents, first 3 pages, image path), against the
full-resolution behaviour:

  before, scale 0.5   17 / 33 documents changed
  after,  scale 0.5    6 / 33
  scale 1.0            0 / 33 (same code path as develop)

That is what test_multipage_gost_table_image was failing on: the branch read the
last cell of the GOST table differently from develop. On the full document the
table now comes out exactly as develop has it, 40 rows, cell for cell.

Detection still runs on the downscaled copy - only the endpoints come back to
full resolution before they are drawn, and minLineLength/maxLineGap are scaled
with the image (they were not, so a half-size page silently demanded lines twice
as long). apply_houph_line 113.6 -> 32.1 ms/page.

Table stage, min of 3, ms/page (develop -> this):
  page with a table      311.0 -> 227.5
  lines but no table     185.5 -> 103.1
  no lines at all         56.2 ->   6.1   (the line-crossing gate, unchanged)

Verified against a local dedoc API: tests/api_tests/test_api_module_table_recognizer
and test_api_misc_multipage_table, 27 tests, behave identically at 0.5 and 1.0 -
same 3 failures either way, all local (CRLF, a Cyrillic/Latin 'a' from a newer
Tesseract, a Windows path separator in an upload name).
@OligerMan
OligerMan force-pushed the feature/cpu-optimizations branch from 558da69 to 1bd06f6 Compare September 10, 2026 22:03
@OligerMan

Copy link
Copy Markdown
Collaborator Author

Поправил тесты, главная проблема была в том что у меня не совсем тот же набор библиотек на винде, что были на CI, и этого хватало чтобы у меня работало с минимальным количеством фиксов, а для подтверждения в CI нужен был еще 1 фикс в jar, PR я создал, ждем чтобы добавили всё

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants