Commit f8ecd1a
committed
perf(dtypes): screen unsafe exponents with one blob scan per column
The pandas-segfault guard added on Jul 14 (and its Jul 15 prefilter rework)
still ran a per-cell Python predicate over most text cells: the
case-insensitive 'e' prefilter lowercases every cell and matches ordinary
words, so columns like status or email routed almost entirely through
Series.map. On CI runners this roughly doubled fix_dtypes time and tripped
the CleanBench T5 runtime gate (nightly issue #147), which then blocked the
v1.2.0 release workflow.
Screen each column as a single joined blob first: one C-level join plus one
regex scan for an exponent marker followed by >=3 digits (309 is the
smallest unsafe magnitude). Only columns that screen positive - or object
columns that cannot be joined because they hold non-string values - fall
through to the exact per-cell predicate, which is unchanged. False positives
only cost the precise re-check; false negatives are impossible because any
parse-risky token must contain that signature.
Interleaved local T5 benchmark (median of 4 rounds of 3):
pre-guard green (9c6c4aa): 0.4170 s
main (497c2f6): 0.4284 s
this fix: 0.4159 s1 parent 497c2f6 commit f8ecd1a
2 files changed
Lines changed: 27 additions & 15 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
96 | 96 | | |
97 | 97 | | |
98 | 98 | | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
99 | 108 | | |
100 | 109 | | |
101 | 110 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
45 | 51 | | |
46 | 52 | | |
47 | 53 | | |
| |||
153 | 159 | | |
154 | 160 | | |
155 | 161 | | |
156 | | - | |
157 | | - | |
158 | | - | |
159 | | - | |
160 | | - | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
161 | 167 | | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | | - | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
172 | 175 | | |
173 | 176 | | |
174 | 177 | | |
| |||
0 commit comments