Alloc rule 2 bug

docs/md/fmprofiles.md

@@ -121,6 +121,8 @@ The choices are as follows;
 | Back-allocate losses to items in proportion to prior level loss              | 2           |
 | Back-allocate losses to items in proportion to prior level loss (reinsurance)| 3           |
 
+Allocrules 2 and 3 perform the same calculation and give the same answer. Rule 3 works out the item proportions at every level and layer as the calculation proceeds, while rule 2 only works them out when they are needed at the final level. Where there is more than one layer, each layer is allocated using its own prior level losses under both rules.
+
 ## Effective deductibles
 
 Often there are more than one hierarchal levels with deductibles, and there is a choice of methods of accumulation of deductibles through the hierarchy. Whenever a rule with a deductible is used in the loss calculation then it is accumulated through the calculation in an **effective_deductible** variable. The effective deductible is the smaller of the deductible amount and the loss. 

examples/.gitignore

@@ -21,3 +21,4 @@ getmodel.bin
 gul_mode2_items2.bin
 log
 !static/validation/*
+!fm_alloc/*/*.csv

examples/fm_alloc/README.md

@@ -0,0 +1,108 @@
+# fmcalc back-allocation test cases
+
+Five small financial structures for testing the fmcalc back-allocation rules (`-a1`, `-a2`,
+`-a3`). All have more than one policy layer and are run by `ktest/runtests.sh`.
+
+## case1 - layers with different loss distributions
+
+Four items, two per location, two coverages per location, two policy layers:
+
+| level | aggregation                    | terms                                                    |
+|-------|--------------------------------|----------------------------------------------------------|
+| 1     | one agg per item (coverage)    | layer 1 limits items 1 and 3 to 500; layer 2 limits items 2 and 4 to 500 |
+| 2     | items 1,2 -> agg 1; 3,4 -> agg 2 (location) | none                                         |
+| 3     | all -> agg 1 (account)         | both layers limited to 4000                              |
+
+Ground up losses are 1000, 3000, 3000, 1000 for items 1 to 4.
+
+`fm_xref` output ids are assigned in item then layer order, so output ids 1, 3, 5, 7 are
+layer 1 for items 1 to 4 and output ids 2, 4, 6, 8 are layer 2 for items 1 to 4.
+
+Each layer's level 2 losses are:
+
+* layer 1 - location 1 = 500 + 3000 = 3500, location 2 = 500 + 1000 = 1500
+* layer 2 - location 1 = 1000 + 500 = 1500, location 2 = 3000 + 500 = 3500
+
+so the two layers distribute their losses over the locations in opposite proportions. The
+level 3 limit of 4000 applies to a 5000 loss in both layers, and 4000 is back-allocated:
+
+| output_id | item | layer | -a1  | -a2 and -a3 |
+|-----------|------|-------|------|-------------|
+| 1         | 1    | 1     | 500  | 400         |
+| 2         | 1    | 2     | 500  | 800         |
+| 3         | 2    | 1     | 1500 | 2400        |
+| 4         | 2    | 2     | 1500 | 400         |
+| 5         | 3    | 1     | 1500 | 400         |
+| 6         | 3    | 2     | 1500 | 2400        |
+| 7         | 4    | 1     | 500  | 800         |
+| 8         | 4    | 2     | 500  | 400         |
+
+Alloc rule 1 allocates in proportion to the ground up losses, so both layers give the same
+answer. Alloc rules 2 and 3 allocate in proportion to the prior level losses, so each layer
+follows its own distribution and the two layers are mirror images of each other.
+
+This is a regression test for [#2055](https://github.com/OasisLMF/OasisLMF/issues/2055),
+where alloc rule 2 back-allocated every layer using layer 1's proportions and gave
+171.43, 1028.57, 933.33 and 1866.67 for the layer 2 items above. Because layer ids follow
+the row order of the account file, that made per item results depend on the order of the
+input.
+
+## case2 - a layer with no loss below the top level
+
+Five items, three layers, two levels, with deductibles that leave some level 1
+aggregations with no loss in layer 1 but a loss in a later layer. Alloc rule 2 only fills
+in the item proportions for layer 1, and skips the aggregations that had no loss, so the
+later layers were left with no proportions to allocate with and fmcalc segmentation
+faulted. The expected output is pinned for all three allocation rules.
+
+## case3 - a layer with no loss at the top level, with layers below it
+
+Two items with ground up losses of 1000 and 3000, three levels, a single layer at level 1 and
+two layers at levels 2 and 3. Layer 1 is wiped out at the top level by a deductible larger than
+the loss; layer 2 is limited to 4000.
+
+`compute_item_proportions` has a second branch for the case where the level below does not carry
+the current layer, and it copied layer 1's proportions *at the same level* rather than computing
+them. The walk down the levels that alloc rule 2 performs terminates in that branch, and where
+layer 1 has no loss there its proportions are null, so fmcalc dereferenced a null pointer and
+segmentation faulted. This case is the regression test for that crash, which showed up under
+`-a2` only.
+
+## case4 - a single layer on the level below a layered level
+
+The same structure as case3 with a single layer at level 2, so the top level takes the copy
+branch directly rather than through the recursion. Layer 1's proportions there are all zeros
+rather than null, so every other layer copied the zeros, no item received a share, and layer 2's
+whole 4000 loss was dropped instead of being allocated - back-allocated losses summed to zero
+against a non-zero layer loss. This affected `-a2` and `-a3` alike, so this case is the
+regression test for alloc rule 3 as well.
+
+In both cases layer 1 back-allocates 0 and layer 2 back-allocates its 4000 as 1000 and 3000, in
+ground up proportion, and all three allocation rules agree:
+
+| output_id | item | layer | -a1, -a2 and -a3 |
+|-----------|------|-------|------------------|
+| 1         | 1    | 1     | 0                |
+| 2         | 1    | 2     | 1000             |
+| 3         | 2    | 1     | 0                |
+| 4         | 2    | 2     | 3000             |
+
+## case5 - an item with no aggregation at the previous level
+
+Two items, four levels, four layers, reduced from the client structure in
+[#2055](https://github.com/OasisLMF/OasisLMF/issues/2055) by dropping items until the fault
+stopped reproducing. The aggregation ids at level 1 are not contiguous - aggs 1 and 4 carry no
+items - so the level 1 loss vector holds entries that no item maps to, and the item to
+previous-level-index lookup yields -1 for the items that do carry loss. That -1 was then used to
+index the previous level's vector, both when totalling the previous level's loss and when reading
+back `item_idx`, so fmcalc read out of bounds and segmentation faulted on a null `item_idx`.
+
+Released 3.12.4 faults here under `-a3`, and the per-layer recursion added earlier on this branch
+made it fault under `-a2` as well. Both are fixed by skipping items that have no entry at the
+previous level.
+
+`-a2` and `-a3` back-allocate the whole of each layer's loss - 520.00 for layer 1 and 92.50 for
+layer 2, matching the `-a0` gross. `-a1` allocates only 54.41 to layer 2 on this structure, which
+does not match its gross; that is a separate pre-existing problem in the rule 1 path and is out of
+scope here. The `-a1` expected output is pinned so that a change in it is noticed, not because it
+is correct.

examples/fm_alloc/case1/fm_policytc.csv

@@ -0,0 +1,15 @@
+layer_id,level_id,agg_id,profile_id
+1,1,1,2
+1,1,2,1
+1,1,3,2
+1,1,4,1
+2,1,1,1
+2,1,2,2
+2,1,3,1
+2,1,4,2
+1,2,1,1
+1,2,2,1
+2,2,1,1
+2,2,2,1
+1,3,1,3
+2,3,1,3

examples/fm_alloc/case1/fm_profile.csv

@@ -0,0 +1,4 @@
+profile_id,calcrule_id,deductible1,deductible2,deductible3,attachment1,limit1,share1,share2,share3
+1,100,0,0,0,0,0,0,0,0
+2,14,0,0,0,0,500,0,0,0
+3,14,0,0,0,0,4000,0,0,0

examples/fm_alloc/case1/fm_programme.csv

@@ -0,0 +1,11 @@
+from_agg_id,level_id,to_agg_id
+1,1,1
+2,1,2
+3,1,3
+4,1,4
+1,2,1
+2,2,1
+3,2,2
+4,2,2
+1,3,1
+2,3,1

examples/fm_alloc/case1/fm_xref.csv

@@ -0,0 +1,9 @@
+output_id,agg_id,layer_id
+1,1,1
+2,1,2
+3,2,1
+4,2,2
+5,3,1
+6,3,2
+7,4,1
+8,4,2

examples/fm_alloc/case1/guls.csv

@@ -0,0 +1,17 @@
+event_id,item_id,sidx,loss
+1,1,-3,1000
+1,1,-2,0
+1,1,-1,0
+1,1,1,1000
+1,2,-3,3000
+1,2,-2,0
+1,2,-1,0
+1,2,1,3000
+1,3,-3,3000
+1,3,-2,0
+1,3,-1,0
+1,3,1,3000
+1,4,-3,1000
+1,4,-2,0
+1,4,-1,0
+1,4,1,1000

examples/fm_alloc/case2/fm_policytc.csv

@@ -0,0 +1,19 @@
+layer_id,level_id,agg_id,profile_id
+1,1,1,10
+2,1,1,1
+3,1,1,10
+1,1,2,5
+2,1,2,4
+3,1,2,9
+1,1,3,4
+2,1,3,2
+3,1,3,1
+1,1,4,8
+2,1,4,7
+3,1,4,4
+1,1,5,5
+2,1,5,3
+3,1,5,9
+1,2,1,7
+2,2,1,6
+3,2,1,9

examples/fm_alloc/case2/fm_profile.csv

@@ -0,0 +1,12 @@
+profile_id,calcrule_id,deductible1,deductible2,deductible3,attachment1,limit1,share1,share2,share3
+1,100,0,0,0,0,0,0,0,0
+2,1,1000,0,0,0,8000,0,0,0
+3,100,0,0,0,0,0,0,0,0
+4,12,200,0,0,0,0,0,0,0
+5,12,1000,0,0,0,0,0,0,0
+6,12,0,0,0,0,0,0,0,0
+7,1,200,0,0,0,500,0,0,0
+8,100,0,0,0,0,0,0,0,0
+9,100,0,0,0,0,0,0,0,0
+10,1,1000,0,0,0,500,0,0,0
+11,100,0,0,0,0,0,0,0,0

examples/fm_alloc/case2/fm_programme.csv

@@ -0,0 +1,11 @@
+from_agg_id,level_id,to_agg_id
+1,1,4
+2,1,2
+3,1,3
+4,1,5
+5,1,1
+1,2,1
+2,2,1
+3,2,1
+4,2,1
+5,2,1

examples/fm_alloc/case2/fm_xref.csv

@@ -0,0 +1,16 @@
+output_id,agg_id,layer_id
+1,1,1
+2,1,2
+3,1,3
+4,2,1
+5,2,2
+6,2,3
+7,3,1
+8,3,2
+9,3,3
+10,4,1
+11,4,2
+12,4,3
+13,5,1
+14,5,2
+15,5,3

examples/fm_alloc/case2/guls.csv

@@ -0,0 +1,26 @@
+event_id,item_id,sidx,loss
+1,1,-3,1000
+1,1,-2,0
+1,1,-1,0
+1,1,1,1000
+1,1,2,1000
+1,2,-3,3000
+1,2,-2,0
+1,2,-1,0
+1,2,1,3000
+1,2,2,3000
+1,3,-3,5000
+1,3,-2,0
+1,3,-1,0
+1,3,1,5000
+1,3,2,5000
+1,4,-3,500
+1,4,-2,0
+1,4,-1,0
+1,4,1,500
+1,4,2,500
+1,5,-3,100
+1,5,-2,0
+1,5,-1,0
+1,5,1,100
+1,5,2,100

examples/fm_alloc/case3/fm_policytc.csv

@@ -0,0 +1,7 @@
+layer_id,level_id,agg_id,profile_id
+1,1,1,1
+1,1,2,1
+1,2,1,1
+2,2,1,1
+1,3,1,2
+2,3,1,3

examples/fm_alloc/case3/fm_profile.csv

@@ -0,0 +1,4 @@
+profile_id,calcrule_id,deductible1,deductible2,deductible3,attachment1,limit1,share1,share2,share3
+1,100,0,0,0,0,0,0,0,0
+2,12,100000,0,0,0,0,0,0,0
+3,14,0,0,0,0,4000,0,0,0

examples/fm_alloc/case3/fm_programme.csv

@@ -0,0 +1,6 @@
+from_agg_id,level_id,to_agg_id
+1,1,1
+2,1,2
+1,2,1
+2,2,1
+1,3,1

examples/fm_alloc/case3/fm_xref.csv

@@ -0,0 +1,5 @@
+output_id,agg_id,layer_id
+1,1,1
+2,1,2
+3,2,1
+4,2,2

examples/fm_alloc/case3/guls.csv

@@ -0,0 +1,11 @@
+event_id,item_id,sidx,loss
+1,1,-3,1000
+1,1,-2,0
+1,1,-1,0
+1,1,1,1000
+1,1,2,1000
+1,2,-3,3000
+1,2,-2,0
+1,2,-1,0
+1,2,1,3000
+1,2,2,3000

examples/fm_alloc/case4/fm_policytc.csv

@@ -0,0 +1,6 @@
+layer_id,level_id,agg_id,profile_id
+1,1,1,1
+1,1,2,1
+1,2,1,1
+1,3,1,2
+2,3,1,3

examples/fm_alloc/case4/fm_profile.csv

@@ -0,0 +1,4 @@
+profile_id,calcrule_id,deductible1,deductible2,deductible3,attachment1,limit1,share1,share2,share3
+1,100,0,0,0,0,0,0,0,0
+2,12,100000,0,0,0,0,0,0,0
+3,14,0,0,0,0,4000,0,0,0

examples/fm_alloc/case4/fm_programme.csv

@@ -0,0 +1,6 @@
+from_agg_id,level_id,to_agg_id
+1,1,1
+2,1,2
+1,2,1
+2,2,1
+1,3,1

examples/fm_alloc/case4/fm_xref.csv

@@ -0,0 +1,5 @@
+output_id,agg_id,layer_id
+1,1,1
+2,1,2
+3,2,1
+4,2,2

examples/fm_alloc/case4/guls.csv

@@ -0,0 +1,11 @@
+event_id,item_id,sidx,loss
+1,1,-3,1000
+1,1,-2,0
+1,1,-1,0
+1,1,1,1000
+1,1,2,1000
+1,2,-3,3000
+1,2,-2,0
+1,2,-1,0
+1,2,1,3000
+1,2,2,3000

examples/fm_alloc/case5/fm_policytc.csv

@@ -0,0 +1,22 @@
+layer_id,level_id,agg_id,profile_id
+1,1,1,2
+2,1,1,3
+1,1,2,3
+2,1,2,2
+1,1,3,3
+3,1,3,2
+1,1,4,3
+4,1,4,2
+1,2,1,4
+2,2,1,5
+3,2,1,5
+1,2,2,1
+4,2,2,1
+1,3,1,6
+2,3,1,7
+3,3,1,7
+4,3,1,7
+1,4,1,8
+2,4,1,9
+3,4,1,10
+4,4,1,11

examples/fm_alloc/case5/fm_profile.csv

@@ -0,0 +1,12 @@
+profile_id,calcrule_id,deductible1,deductible2,deductible3,attachment1,limit1,share1,share2,share3
+1,100,0,0,0,0,0,0,0,0
+2,14,0,0,0,0,1000000000,0,0,0
+3,14,0,0,0,0,100000,0,0,0
+4,12,25000,0,0,0,0,0,0,0
+5,100,0,0,0,0,0,0,0,0
+6,100,0,0,0,0,0,0,0,0
+7,12,50000,0,0,0,0,0,0,0
+8,2,0,0,0,0,1000000000,0.02,0,0
+9,2,0,0,0,0,1000000000,0.0925,0,0
+10,2,0,0,0,0,1000000000,0.1675,0,0
+11,2,0,0,0,0,1000000000,0.044,0,0

examples/fm_alloc/case5/fm_programme.csv

@@ -0,0 +1,8 @@
+from_agg_id,level_id,to_agg_id
+1,1,2
+2,1,3
+2,2,1
+3,2,1
+1,3,1
+2,3,1
+1,4,1

examples/fm_alloc/case5/fm_xref.csv

@@ -0,0 +1,5 @@
+output_id,agg_id,layer_id
+1,1,1
+2,1,2
+3,2,1
+4,2,3