IA-5486 bulk upload import cache

[mestachs]

What problem is this PR solving?

Bulk zip upload: cache instances, org units, entities per batch, bulk insert audit rows

Related JIRA tickets

IA-5486

Changes

Fewer SQL queries per submission in process_mobile_bulk_upload / import_data. No change in behaviour: the tests added in IA-5485 pass unchanged, only their query counts drop.

Cached / prefetched per batch (InstanceImportCache, new)

• Instances: the batch's existing instances loaded in 1 query by uuid (with their entity), plus the ones created during the import. Replaces 1 lookup per instance in import_data, and for an existing instance 2 more lookups, the task's .get() and the load of its entity.
• Org units sent by uuid: looked up once per org unit instead of once per instance.
• Entities: the batch's entities loaded in 1 query by (uuid, entity type), plus the ones created. find_entity() takes the cached list (existing_entities=); its rules (create, duplicates, soft-deleted) are unchanged.

Lists are kept per uuid: older data can have duplicate uuids, which are handled as before (.get() still fails the import on duplicate instances).

Other

• Audit rows (Modification): built with log_modification(..., save=False), then bulk-inserted in chunks of 500 instead of 1 INSERT per instance.
• Validation workflow failing to start: import_data reloads the instance's status, since the in-memory instance is now reused and would otherwise save the rolled-back PENDING.

Result (local, SQL queries / time)

Submissions	All new	Campaign (registrations re-sent + follow-ups)
50	643 → 446, 0.51 → 0.27s	612 → 365, 0.48 → 0.28s
2,000	24,043 → 16,049, 17.2 → 11.1s	23,037 → 13,043, 23.5 → 11.4s

4 fewer queries per new submission, about 5 per re-sent one. S3 calls unchanged. On staging, where S3 dominates, expect roughly 5–10%.

Not in this PR

• A new entity is still inserted, then updated to set its reference instance (1 extra query per new patient).
• A new instance still goes through get_or_create + UPDATE instead of a single INSERT.

How to test

Print screen / video

IA-5486 (instance, org unit and entity cache, bulk audit rows) against develop

Measured locally on the same scenarios as before: averages of 3 runs, 2 runs at 2,000.

All new (registration plus follow-up per patient):

┌─────────────┬────────────────────────┬───────────────────────┐
│ Submissions │      SQL queries       │      Total time       │
├─────────────┼────────────────────────┼───────────────────────┤
│ 10          │ 163 → 126 (−23%)       │ 0.137 → 0.075s (−45%) │
├─────────────┼────────────────────────┼───────────────────────┤
│ 50          │ 643 → 446 (−31%)       │ 0.511 → 0.270s (−47%) │
├─────────────┼────────────────────────┼───────────────────────┤
│ 500         │ 6,043 → 4,046 (−33%)   │ 4.90 → 2.59s (−47%)   │
├─────────────┼────────────────────────┼───────────────────────┤
│ 2,000       │ 24,043 → 16,049 (−33%) │ 17.2 → 11.1s (−36%)   │
└─────────────┴────────────────────────┴───────────────────────┘

Campaign (registrations sent again, updated, plus new follow-ups):

┌─────────────┬────────────────────────┬───────────────────────┐
│ Submissions │      SQL queries       │      Total time       │
├─────────────┼────────────────────────┼───────────────────────┤
│ 10          │ 152 → 105 (−31%)       │ 0.111 → 0.071s (−36%) │
├─────────────┼────────────────────────┼───────────────────────┤
│ 50          │ 612 → 365 (−40%)       │ 0.478 → 0.284s (−41%) │
├─────────────┼────────────────────────┼───────────────────────┤
│ 500         │ 5,787 → 3,290 (−43%)   │ 4.88 → 2.65s (−46%)   │
├─────────────┼────────────────────────┼───────────────────────┤
│ 2,000       │ 23,037 → 13,043 (−43%) │ 23.5 → 11.4s (−52%)   │
└─────────────┴────────────────────────┴───────────────────────┘

Per submission: 4 fewer queries for a new one (its uuid lookup, the audit INSERT, the org unit, the entity). For a campaign batch it's about 5 fewer on average: each re-sent registration also skips 3 uuid lookups and the load of its entity.

S3 calls don't change: the same number of exists() and saves before and after.

On staging (estimate)

Locally the database answers almost instantly, so these percentages overstate what you'll see on staging. There, most of the ~85ms per submission is S3, which this work doesn't touch. At about 1–2ms per query:

┌───────────────────────┬───────────────┬─────────────────────┐
│   2,000 submissions   │ Queries saved │    Expected gain    │
├───────────────────────┼───────────────┼─────────────────────┤
│ All new (~174s today) │ ~8,000        │ ~8–16s, 5–9%        │
├───────────────────────┼───────────────┼─────────────────────┤
│ Campaign              │ ~10,000       │ ~10–20s, around 10% │
└───────────────────────┴───────────────┴─────────────────────┘

Notes

Doc

[mestachs]

alea iacta est