switch Nan values to None

[pooleycodes]

What type of PR is this? (check all applicable)

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update

Description

When adding the severity column in log.py, Pandas represents missing values as NaN. Serialising these to JSON produces NaN tokens, which PostgREST rejects for a jsonb column ("NaN is invalid"), causing add_data to fail. This converts Pandas NaN values to None so they serialise as null/blank.

Can see in digital_land/check.py‎, entity is now added to the issue log, this will be added as a NaN value here, (is it worth adding to the issue log here?) and this is NaN causes json write issues.

Related Tickets & Documents

• Ticket Link
• Related Issue #
• Closes #

QA Instructions, Screenshots, Recordings

Please replace this line with instructions on how to test your changes, a note
on the devices and browsers this has been tested on, as well as any relevant
images for UI changes.

Added/updated tests?

We encourage you to keep the code coverage percentage at 80% and above. Please refer to the Digital Land Testing Guidance for more information.

• Yes
• No, and this is why: please replace this line with details on why tests
  have not been included
• I need help with writing tests

[optional] Are there any post deployment tasks we need to perform?

[optional] Are there any dependencies on other PRs or Work?

Summary by CodeRabbit

Summary by CodeRabbit

• Bug Fixes
  • Improved handling of missing values in issue log records by consistently normalising merged record fields so that missing data is represented as blank/null values rather than NaN.

[coderabbitai]

Walkthrough

IssueLog.add_severity_column() now converts Pandas missing values to None when storing merged dataframe records in self.rows.

Changes

Issue log null normalisation

Layer / File(s)	Summary
Normalise merged log rows digital_land/log.py	Merged dataframe records are post-processed so Pandas missing values become None before assignment to self.rows.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the bug fix of converting Pandas NaN values to None.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 2826-bug-add_data-fails-with-postgrest-jsonb-nan-is-invalid---multiple-organisation-change

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

digital_land/log.py (1)

168-175: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Use pd.isna() here.

v != v only catches float NaN, so pd.NA and pd.NaT can slip through and be stringified in the CSV output as <NA>/NaT. pd.isna(v) handles all Pandas null scalars and is clearer.

♻️ Proposed refactor

             # Switch Pandas NaN to None to avoid issues with JSON
             self.rows = [
                 {
-                    k: (None if isinstance(v, float) and v != v else v)
+                    k: (None if pd.isna(v) else v)
                     for k, v in row.items()
                 }
                 for row in merged_df.to_dict(orient="records")
             ]

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@digital_land/log.py` around lines 168 - 175, Update the row-value
normalization in the list comprehension building self.rows to use pandas’
pd.isna(v) for null detection instead of the float-specific v != v check, while
preserving non-null values unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@digital_land/log.py`:
- Around line 168-175: Update the row-value normalization in the list
comprehension building self.rows to use pandas’ pd.isna(v) for null detection
instead of the float-specific v != v check, while preserving non-null values
unchanged.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 96a22dc6-6537-42c9-8db9-5b06849db6a9

📥 Commits

Reviewing files that changed from the base of the PR and between 9240853 and e304699.

📒 Files selected for processing (1)

• digital_land/log.py

[gibahjoe]

LGTM