Update relational schema: RI model, LocationGroup, step functions, ETL fixes - #280
Open
benhayes21 wants to merge 9 commits into
Open
benhayes21 wants to merge 9 commits into
benhayes21 wants to merge 9 commits into
Conversation
- Fix 134 column case mismatches across _import_account (82), _import_location (39), PolicyDetails (1), and LocationDetail (12) to match OED spec exactly - Add 9 PV (solar panel) fields to _import_location and LocationDetail - Add _staging_riinfo, _staging_riscope, _import_riinfo, _import_riscope tables with full column sets from the ReinsInfo/ReinsScope spec; wire into usp_Database_Load - Add AccountCoverageTerm and AccountPDTerm junction tables; implement account-level term views (vw_account_terms_1Building through _6All, vw_level_11/12/13_term) and replace the TO DO comment in usp_Terms_Load with working account-term load logic - Add StepPolicy table (FK to Policy) and usp_StepPolicy_Load procedure for step-function parametric policy fields, called from usp_Database_Load - load_sql.py: remove .head(10) on location CSV, uncomment usp_Database_Load exec, RI staging table names already match schema Closes #279 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New StepPolicyTerm table (FK to Term + FK to StepPolicy) linking step-function parametric policies into the standard term hierarchy - vw_step_policy_terms unpivots StepPolicy into per-coverage-type rows (Building/Contents/BuildingContents → CoverageTypeId 1/3/6) carrying DeductibleBuilding/Contents/BuildingContents and PayOutLimit* as Ded/Limit - vw_level_14_term wraps the view with tmpTermId ordering - usp_StepPolicyTerm_Load creates Term records (level 14), StepPolicyTerm links, and fans down to ItemTerm via Policy→Account→Location→Coverage→Item - usp_Database_Load now calls usp_StepPolicyTerm_Load after usp_StepPolicy_Load - Diagram updated: StepPolicyTerm junction added, term levels bumped to 14 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three new tables normalise reinsurance data from OED flat CSVs: - ReinsInfo: treaty master (type, risk/programme terms, inuring priority, attachment basis, dates, CededPercent at treaty level) - ReinsScope: per-scope filter rows (hierarchical fields + FILTER_LEVEL_EXTRA_FIELDS + SS CededPercent scope override) - ReinsScopeLink: pre-resolved entity PKs (Portfolio/Account/Policy/Location) determined by RiskLevel (SEL/ACC/POL/LOC/LGR) RI staging tables (_staging_riinfo, _staging_riscope, _import_riinfo, _import_riscope) added; usp_ReinsInfo_Load and usp_ReinsScope_Load wire into usp_Database_Load. LGR risk level fans out to multiple Location rows. Schema diagram updated with RI tables, connections, legend, and gap analysis resolved card. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Schema fixes: - Move StepPolicyTerm CREATE TABLE to after Term (FK dependency order) - Move orphaned account-terms code (levels 11-13) into usp_Terms_Load body where it belongs; it was floating between view definitions causing CREATE VIEW failures - Fix usp_ReinsScope_Load LGR JOIN: LocGroup is on LocationDetail not Location; add LocationDetail→Location join - Add DELETE cleanup block to usp_Database_Load so re-runs are idempotent load_sql.py: - Replace engine.connect() with engine.begin() so EXEC auto-commits - Add fast_executemany=True to bypass SQL Server 2100-parameter limit (method='multi' sent all rows in one INSERT, hitting the limit) - Read DB password from OED_DB_PASSWORD env var instead of hardcoding .gitignore: add .env / *.env patterns Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New table LocationGroup(LocationGroupId, LocGroup) normalises the LocGroup string into a proper entity. Location gains a LocationGroupId FK so group membership is a structural relationship rather than a searchable string in LocationDetail. LocGroup removed from LocationDetail. ReinsScopeLink gains a fifth nullable FK (LocationGroupId) for LGR risk level, replacing the previous per-Location fan-out. usp_ReinsScope_Load LGR case now inserts one ReinsScopeLink row per matched LocationGroup; Location.LocationGroupId is the join path to individual locations at query time. New usp_LocationGroup_Load extracts distinct LocGroup values from _import_location and runs before usp_Location_Load. usp_Location_Load updated to LEFT JOIN LocationGroup and populate Location.LocationGroupId. Cleanup block updated to delete LocationGroup after Location. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Column widths: - Condition.CondNumber/CondName/CondTag: VARCHAR(1) → VARCHAR(20/100/20) - LocationDetail+_import_location.LocName: NVARCHAR(20) → NVARCHAR(200) - _import_riinfo.ReinsName: VARCHAR(30) → VARCHAR(200) - #tmpterm_denormalised.TermPeril: VARCHAR(50) → VARCHAR(250) (insurance_conditions has 67-char peril strings) Logic bugs: - usp_Database_Load: TRUNCATE _import_* tables at start so re-runs don't accumulate rows from prior test cases - usp_Condition_Load: join _import_account on CondNumber (was missing) and use CROSS APPLY TOP 1 so multiple layers/rows per condition don't produce duplicate ConditionIds - ConditionCoverageTerm/ConditionPDTerm/ConditionTerm inserts: add WHERE EXISTS ConditionLocation guard so we don't reference TermIds that were filtered out during the Term insert (FK violation fix) - ItemTerm insert in usp_Terms_Load: add MAX(ItemTermId) offset so it doesn't collide with rows already written by usp_StepPolicyTerm_Load Add test suite (SQL/tests/) and export script (SQL/export_sql.py) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Build PreviewYou can find files attached to the below linked Workflow Run URL (Logs).
|
Folds in the fixes from #294 on top of this branch's relational- schema rework: - load_sql.py: batch to_sql() writes with chunksize=10000 (works with fast_executemany to scale past SQL Server's 2100-param limit on large OED input files) and normalize PolInceptionDate/PolExpiryDate to ISO 8601 before staging, since the OED spec requires it and source files may use a locale-specific date format. - LocationDetail was defined in the schema (and even cleared by usp_Database_Load's idempotency reset) but never populated: no procedure existed to load its ~160 detail attributes from _import_location. Added usp_LocationDetail_Load, joining _businesskeys_location to _import_location on the natural key (matching usp_Location_Load's existing join pattern) and wired it into usp_Database_Load right after usp_Location_Load. LocGroup is intentionally excluded from the column list since this branch already extracts it into the separate LocationGroup table. Verified against a local SQL Server instance: full 12598-row location.csv loads end-to-end into Location and LocationDetail with no orphaned or duplicate rows, and correct attribute values. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3 tasks done
Build PreviewYou can find files attached to the below linked Workflow Run URL (Logs).
|
Same gap as LocationDetail: PolicyDetails was defined in the schema (and cleared by usp_Database_Load's idempotency reset) but never loaded. Added usp_PolicyDetails_Load, joining _businesskeys_policy to _import_account on the natural key (PortNumber, AccNumber, PolNumber), and wired it into usp_Database_Load after usp_Policy_Load. Column choice cross-checked against OEDInputFields.csv: the existing 15 PolicyDetails columns (PolStatus, PolInceptionDate/ExpiryDate, ProducerName, Underwriter, BranchName, LOB, ExpiringPolNumber, PolGrossPremium/Tax/Brokerage/NetPremium, HoursClause, PolUserDef1-5) are the complete set of scalar, Property-relevant, policy-level fields from the spec. Everything else policy-related is out of scope here: FlexiPolZZZ belongs in a not-yet-built FlexiPol table (same pattern as FlexiLocZZZ, also absent from LocationDetail); the Ded/ Limit/PayOut fields are handled via the Term tables/views; and fields like PolSIR, PolCoverage, OSVendor, IsPackage etc. are marked "n/a" for Property in the spec (Cyber/Liability-only). Also noted, out of scope for this change: Layer table is separately missing LayerParticipation/LayerLimit/LayerAttachment/LayerAgg*, which are populated in _import_account but never loaded into Layer. Verified against a local SQL Server instance: Policy and PolicyDetails both load with matching row counts and no orphans. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Build PreviewYou can find files attached to the below linked Workflow Run URL (Logs).
|
The PR #280 test suite (SQL/tests/, 77 pytest cases) caught a bug in usp_LocationDetail_Load introduced by the previous commit: SELECT DISTINCT across all detail columns doesn't collapse duplicate _import_location rows when CondTag differs between them. _import_location legitimately carries multiple rows per (PortNumber, AccNumber, LocNumber) -- one per CondTag -- since condition-location membership is a many-valued relationship loaded separately via usp_ConditionLocation_Load. The insurance_conditions validation case exercises this, so its location.csv has two rows per location with different CondTag values; DISTINCT saw them as genuinely different rows and the INSERT hit the LocationId primary key twice. Replaced the DISTINCT with a ROW_NUMBER()-ranked CTE that picks exactly one _import_location row per LocationId, ordered by CondTag for determinism. Verified account.csv has no equivalent duplication for the PolicyDetails columns across all 15 validation cases, so usp_PolicyDetails_Load doesn't need the same treatment. All 77 tests in SQL/tests/ now pass, including insurance_conditions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Build PreviewYou can find files attached to the below linked Workflow Run URL (Logs).
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #279
Summary
ReinsInfo,ReinsScope, andReinsScopeLinktables with full load procedures covering FAC/QS/SS/PR/CXL/AXL contract types and SEL/ACC/POL/LOC/LGR risk levelsLocGroupinto its own table;Location.LocationGroupIdFK enables clean LGR reinsurance scoping without denormalising the Location rowSQL/tests/): 77 parametrised pytest tests load all 15 OasisLMF validation cases and assert round-trip fidelity for locations, accounts, RI, step policies, and idempotencySQL/export_sql.py): Exports normalised data back to flat OED CSVs from the_import_*tablesload_sql.py: SA password moved toOED_DB_PASSWORDenv var;EXEC usp_Database_Loadnow runs correctly viaengine.begin(); staged inserts now batch withchunksize=10000(scales past SQL Server's 2100-param limit on large files); accountPolInceptionDate/PolExpiryDatenormalized to ISO 8601 before stagingLocationDetail/PolicyDetailswere never populated: both tables were defined in the schema (and even cleared byusp_Database_Load's idempotency reset) but had no load procedure, so every location/policy detail attribute was silently dropped. Addedusp_LocationDetail_Load(164 columns, joined via_businesskeys_location↔_import_location,LocGroupexcluded since it now lives inLocationGroup) andusp_PolicyDetails_Load(joined via_businesskeys_policy↔_import_account), both wired intousp_Database_Loadright after their respective skeleton loads. Column sets cross-checked againstOpenExposureData/OEDInputFields.csvto confirm nothing Property-relevant was left out.Test plan
OED_DB_PASSWORD=<pw> pytest SQL/tests/ -v)usp_Database_Loadtwice produces identical row countsLocationDetail/PolicyDetailsmanually verified against a local SQL Server instance: full 12598-rowlocation.csvloads withLocation/LocationDetailrow counts matching exactly and no orphaned or duplicate rows;Policy/PolicyDetailslike-for-like🤖 Generated with Claude Code