Skip to content

Return clear error for non-finite (NaN/Inf) floats in KServe REST output#4337

Open
exzile wants to merge 3 commits into
openvinotoolkit:mainfrom
exzile:fix/rest-nonfinite-json
Open

Return clear error for non-finite (NaN/Inf) floats in KServe REST output#4337
exzile wants to merge 3 commits into
openvinotoolkit:mainfrom
exzile:fix/rest-nonfinite-json

Conversation

@exzile

@exzile exzile commented Jun 28, 2026

Copy link
Copy Markdown

Summary

The REST output serializers silently produced malformed JSON when a model output contained non-finite floats (NaN/±Inf). Floats were written via rapidjson Writer::Double() while ignoring its return value — rapidjson writes nothing for non-finite values (they are invalid per the JSON spec and the default writer has no kWriteNanAndInfFlag), so the affected array element was dropped, corrupting the response. There was no isfinite guard anywhere in the REST output path.

This now returns a clear JSON_SERIALIZATION_ERROR with a descriptive message instead of emitting broken JSON, on both the KServe v2 and the TFS REST paths.

Change

  • Added an isJsonRepresentable helper (overloaded: integral types always true; float/double use std::isfinite).
  • KServe v2 path: guarded both write paths of the PARSE_OUTPUT_DATA macro (val-field and raw-output) to return JSON_SERIALIZATION_ERROR with Output "<name>" contains a non-finite (NaN/Inf) value that cannot be serialized to JSON.
  • TFS path: applied the same finiteness check to makeJsonFromPredictResponse (PredictResponse), validating float/double outputs before MakeJsonFromTensors and returning the same error.
  • Integer outputs are unaffected (helper resolves to the always-true template; negligible cost).
  • Added unit tests for Inf, NaN, and the unaffected-integer case on both paths.

Validation

Built ovms_test (Windows, py-off) and ran the serialization suite — all pass:

  • Full *MakeJsonFromPredictResponse* suite: 85/85 (KFS + TFS, raw + val-field) — no regression.
  • New non-finite tests: 3 KFS + 4 TFS (Inf/NaN float, double, int-unaffected) all pass.

Context

Found while triaging #2337 (a model emitting -Inf/NaN on GPU). That issue's root cause is upstream/model-side and appears resolved on current OpenVINO; this PR is the independent OVMS-side hardening so that any model producing non-finite values fails cleanly over REST instead of returning corrupt JSON.

Relates to #2337.

exzile and others added 2 commits June 27, 2026 23:27
The KServe v2 REST output serializer wrote floats via rapidjson Writer::Double()
and ignored its return value. rapidjson writes nothing for NaN/Inf (invalid in
JSON), so a model output containing non-finite values silently produced malformed
JSON. Add an isJsonRepresentable finite-check in both PARSE_OUTPUT_DATA write
paths and return JSON_SERIALIZATION_ERROR with a clear message instead. Integer
outputs are unaffected. Adds unit tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply the same finiteness check used on the KServe path to the
TF Serving makeJsonFromPredictResponse: float/double outputs are
validated before serialization and a clear JSON_SERIALIZATION_ERROR
is returned for NaN/Inf instead of producing malformed JSON.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@exzile

exzile commented Jun 28, 2026

Copy link
Copy Markdown
Author

Update: extended the same non-finite (NaN/Inf) guard to the TFS REST serialization path as well (it was originally flagged as a possible follow-up). Both REST paths are now covered, and the description/tests have been updated accordingly — full *MakeJsonFromPredictResponse* suite now 85/85 (KFS + TFS).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant