IA-5375 allow import/export of mappings by mestachs · Pull Request #3369 · BLSQ/iaso

mestachs · 2026-10-01T06:39:38Z

What problem is this PR solving?

Allow to export/import/copy dhis2 mappings from another form version (or form or server)

Related JIRA tickets

IA-5375

Changes

Mainly client side wizard
- when importing you need to review questions that might be mapped/nevermapped in the current version and that the mapping to import says its
- added some validations on the json to not accept everything but clearly it just prevent "simple accident" (ex uploading a geojson instead of a mapping file or importing a tracker mapping into an aggregate one)
- none of the dhis2 ids is "revalidated", we trust, export error might show up because of that
Improved just a bit the server side patch and validation
Add audit logs to patch and link to display them (just the api output)
- since now a single "mass import" can mess up everything, let's keep a copy before modifications

How to test

the seed command is recommended (much more complete for dhis2 mappings)
- as the name suggest it's all around dhis2.
- dhis2 has play servers that can be used, since they regularly release/patches the url change over time
- you can verify the version : via https://play.dhis2.org (prefer stable version, and sometimes it's public instances, they can be vandalized by people, if you can login with user:admin password:district then the seed command should work
- you can seed via : ``docker compose run --rm iaso manage seed_test_data --mode=seed --dhis2version=2.40.12`
- when the command completes you'll get credentials to log in your dev environnement
then you can go in the menu :
- Forms
  - so you can pick probably a "Quantity PCA form stable-xxxx"
  - download the xlsform (the file with survey / choices / settings sheet)

- re-upload it as is with "create version" - final list will look like this

- go back to the form, click on the dhis2 icon

- pick the last version (2026...)

- the general concept is a question is mapped to one data element in dhis2 (or several if tracker, this adds a bit of complexity everywhere), - note this screen is old, was initially targeted to blsq people, a redesign might come afterwards, and we some legacy too (typing, not standard components, weird diff in mappings since I was young and in a hurry) - then you have access to the 2 new buttons : one for exporting, one for importing the mappings where the wizard/dialog lies.

The rest of test is playing with same features

creating form versions that will copy the mapping and remove questions from the mappings that are no more in the forms
importing a json or comparing/importing with a previous version
target mainly event/aggregate mappings (tracker is no more used and would definitively need some cleanup)
create new version of the same form (add/remove/rename questions)
then you can see impacts on the wizard
you can also manipulate the mapping in normal edit session
then replay with the wizard (reimport form another version)
rince and repeat

Print screen / video

2 new buttons at the top

then a "big" wizard/dialog to pick from another version or import from a file

a first step to pick the mapping

then decide for each questions, if you want to take it or not (especially if a question was mapped in current and to import mapping)

Notes

re-validating the dhis2 ids hidden in mapping is out of scope (the export will fail that will the sign)
since we now have the audit logs we might be able to restore an old/working mapping if necessary

Doc

Tell us where the doc can be found (docs folder, wiki, in the code...).

…o the api for the moment

…m in multiple projects

tdethier

general comments:

i made a few comments to improve readability, otherwise backend looks good
i don't really understand what these mappings are, so some comments i made below might be dumb

testing:

the "how to test" section might need a bit more explanations, because i was lost
comparing a mapping with itself shows that all quesitons are dropped, i would expect everything to be shown in "identical"

when checking a mapping and clicking on a question on the left side, i get an error screen

it looks like each click calls our API, which calls the DHIS2 API and there's a configuration issue on my account - we should probably get a better error message in such cases

ERROR    2026-10-07 10:56:55,656 django.request -- Internal Server Error: /api/datasources/78/dataElements.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://play.im.dhis2.org/stable-2-41-9/api/dataElements.json?fields=id%2Cname%2CvalueType%2CdomainType%2CoptionSet%5Boptions%5Bid%2Cname%2Ccode%5D%5D%2CcategoryCombo%5Bid%2Cname%2CcategoryOptionCombos%5Bid%2Cname%5D%5D%2CdataSetElements%5BdataSet%5Bid%2Cname%2CperiodType%5D%5D&pageSize=10&filter=code%3Ailike%3Aimei

i'll let somebody else review the frontend

tdethier · 2026-10-06T13:20:01Z



+class MappingVersionQuerySet(models.QuerySet):
+    def filter_for_user(self, user: User):


i would probably add a unit test at the model level to make sure you get the expected results from this

tdethier · 2026-10-06T13:48:41Z

+
+
+def visit_mappable(node, form_descriptor, mappable_questions):
+    if node.get("type") not in NOT_MAPPABLE_TYPES and "name" in node:


i would make local variables for node_type and node_name

tdethier · 2026-10-06T15:14:48Z

        mapping_version.form_version = form_version

        questions_by_name = form_version.questions_by_name()
+        # unlike questions_by_name, also has the questions inside repeat groups


do you still need questions_by_name then?

tdethier · 2026-10-07T09:33:33Z

+SELECT_MULTIPLE_TYPE = "select all that apply"
+
+
+def _choices(node, form_descriptor):


i would also rename this, i don't get why it's called _choices but returns children nodes

tdethier · 2026-10-07T10:08:35Z

 from .common import HasPermission, ModelViewSet, TimestampField


+def get_question_mapping_shape_error(mapping_type, data_element):


nitpick: i would probably add unit tests for this function and test it without any API call, but tests with API calls work too

tdethier · 2026-10-07T10:15:25Z

+                if data_element is not None and not is_unmap:
+                    shape_error = get_question_mapping_shape_error(instance.mapping.mapping_type, data_element)
+                    if shape_error:
+                        raise serializers.ValidationError({path: shape_error})


maybe we should store errors in a list, then check outside the for question_name, data_element loop if the list has errors, in case there are multiple errors in the same question_mappings? i'm not sure if that scenario is possible

tdethier · 2026-10-07T10:16:55Z

+        self.assertJSONResponse(resp, status.HTTP_200_OK)
+        self.assertEqual(len(resp.json()["mapping_versions"]), 0)


same suggestion for all calls

Suggested change

self.assertJSONResponse(resp, status.HTTP_200_OK)

self.assertEqual(len(resp.json()["mapping_versions"]), 0)

result = self.assertJSONResponse(resp, status.HTTP_200_OK)

self.assertEqual(len(result["mapping_versions"]), 0)

went a step further

def assertMappingVersionsCount(self, response, expected_count): result = self.assertJSONResponse(response, status.HTTP_200_OK) self.assertEqual(len(result["mapping_versions"]), expected_count)

tdethier · 2026-10-07T10:25:26Z

+        self.assertEqual(self.get_question_mappings(mapping_version_id), original)
+        self.assertEqual(Modification.objects.filter(object_id=mapping_version_id).count(), 1)
+
+    def test_mappingversions_patch_logs_modification(self):


mestachs · 2026-10-08T06:46:02Z

the 500 / http error looks more the datasource or dhis2 wasn't available anymore (perhaps play dhis2 url version increment ?) or you seeded the account right before testing ?

thanks, I'll have a look at the comments

… first) and it's coverage

…mapping, but that's ok

…eally performant

mestachs · 2026-10-08T10:33:59Z

so from your bad experience during testing

removed the show other mappings using the same dataset/program (not the query was triggering too much remote n+1)
- advanced user can still export and reimport the json (the wizard will still review and allow safe import only)
fixed a bit the seed command to have more valid xlsform and mappings ~~(quality form is still not matching its mapping but it's a lot more complex) ~~(DONE)~~~~
I noticed a n+1 when creating more mapping versions so fixed it and wrote test for it

…rt-export-of-mappings

Bewi

Globally it looks good, I would just give a bit more love to the dialog:

Fix table overflow (first column should not grow like it does now)
Empty tables => improve layout, either hide it and show a message or keep headings and show a message under it

Bewi · 2026-10-09T09:14:25Z

+        });
+        return result;
+    }, [rows]);
+    const bucketRows = rows.filter(row => row.kind === bucket);


Bewi · 2026-10-09T09:33:53Z

+            mappingVersionId: mappingVersion.id,
+            questionMappings: plan.changes,
+        });
+        openSnackBar(


Why not using the bulkUpdate snack query for this?

Ah, I get it, it doesn't allow to use replacements.
Could we log a task for this ?

Bewi · 2026-10-09T09:46:42Z


            {currentMappingVersion && (
                <Box className={classes.containerFullHeightNoTabPadded}>
+                    <Box className={classes.actions}>


I see we still use className with classes here.
If not to much of work, I would change it to use SX in this file.

Bewi · 2026-10-09T09:52:36Z

+    };
+    rows.forEach(row => {
+        const decision = decisions[row.questionKey];
+        if (row.kind === 'dropped') {


Would be nice to have all kinds in an enum

Bewi · 2026-10-09T09:53:39Z

+export class MappingImportError extends Error {
+    constructor(
+        public reason:
+            | 'invalidJson'


Same here, to prevent typos and improve reusability of this list of invalid errors

mestachs added 7 commits October 1, 2026 08:32

IA-5375 allow import/export of mappings

35152a1

conside neverMapped as a mapped question, shows it in the conflicts

5c8f2bd

stronger validation for mappings json

e5bed48

remove undo, users had the time thinker/review before clicking apply

6c274b0

add auditlog on patch (not really needed for creation), show a link t…

adbf106

…o the api for the moment

small explaination on why mappable questions are needed

7204737

Centralize for_user scope for MappingVersion, prevent issues when for…

e5a42ce

…m in multiple projects

tdethier self-requested a review October 6, 2026 08:27

tdethier reviewed Oct 7, 2026

View reviewed changes

mestachs added 7 commits October 8, 2026 08:58

add test for QuerySet.filter_for_user

219d0f0

remove unncessary code/params, align with current pyxform behaviour

ecd000a

improve tests readability, improve shape error (all question, not the…

6687d3f

… first) and it's coverage

add a test to reproduce export/import roundtrip behaviour

4e0f7f6

make seed command upload valid xlsx, quality still doesn't match its …

68ea513

…mapping, but that's ok

don't show mapping versions from other forms, its confusing and not r…

65c0e72

…eally performant

fix n+1

4972055

mestachs added 2 commits October 8, 2026 12:48

better mapping for quality

87df09c

Merge remote-tracking branch 'origin/develop' into IA-5375-allow-impo…

cc9916e

…rt-export-of-mappings

Bewi approved these changes Oct 9, 2026

View reviewed changes

mestachs added 5 commits October 9, 2026 16:52

IA-5375 use sx instead of makeStyles in mapping details

e4fb849

IA-5375 use enums for diff kinds, decisions and import error reasons

31fa1eb

IA-5375 memoize the rows of the selected import tab

02113b1

IA-5375 keep import table columns fixed and show a message in empty tabs

386a4fb

let's hide completely empty tabs

f699d80



		class MappingVersionQuerySet(models.QuerySet):
		def filter_for_user(self, user: User):



		def visit_mappable(node, form_descriptor, mappable_questions):
		if node.get("type") not in NOT_MAPPABLE_TYPES and "name" in node:

		SELECT_MULTIPLE_TYPE = "select all that apply"


		def _choices(node, form_descriptor):

		from .common import HasPermission, ModelViewSet, TimestampField


		def get_question_mapping_shape_error(mapping_type, data_element):

		self.assertJSONResponse(resp, status.HTTP_200_OK)
		self.assertEqual(len(resp.json()["mapping_versions"]), 0)

Conversation

mestachs commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem is this PR solving?

Related JIRA tickets

Changes

How to test

Print screen / video

Notes

Doc

Uh oh!

tdethier left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mestachs commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mestachs commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Bewi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mestachs commented Oct 1, 2026 •

edited

Loading

tdethier left a comment •

edited

Loading

mestachs commented Oct 8, 2026 •

edited

Loading

mestachs commented Oct 8, 2026 •

edited

Loading