Support JSON/JSONB metadata discovery from ingestion data

[sanphy-citrus]

Problem

Currently, JSON/JSONB metadata discovery requires reading JSON/JSONB values separately from the database after ingestion.

This can result in an additional database query even though the JSON/JSONB data may already be available during the SLayer ingestion process.

Requirement

Support discovering metadata from JSON/JSONB values that are already available during ingestion.

The JSON/JSONB structure is not known in advance and may vary between rows.

For example:

• A table may contain 2–3 JSONB columns.
• Each JSONB column may have a different structure.
• Different rows within the same JSONB column may also contain different structures.
• JSON objects may be nested to multiple levels.

Expected Behavior

During ingestion, when a JSON/JSONB value is available, the metadata discovery process should be able to:

1. Traverse the JSON/JSONB structure recursively.
2. Discover nested field paths.
3. Infer the appropriate data type for scalar values.
4. Handle different structures across multiple rows.
5. Merge discovered fields/types for the corresponding JSONB column.
6. Generate the appropriate SLayer metadata.
7. Avoid an additional database query solely for JSON/JSONB metadata discovery.

Example

Given:

{
  "name": "John",
  "address": {
    "city": "Kochi",
    "state": "Kerala"
  },
  "orders": [
    {
      "id": 101,
      "amount": 500
    }
  ]
}

The discovery process should be able to identify paths such as:

name
address
address.city
address.state
orders
orders.id
orders.amount

Current Implementation

There is already a separate JSON processor that performs recursive JSON traversal, type inference, type merging, sensitive-field filtering, and SLayer metadata generation.

The proposed change is to reuse this existing discovery logic and provide it with JSON/JSONB values already obtained during SLayer ingestion instead of querying the database again.

Benefit

This would:

• Avoid unnecessary database reads.
• Reduce database load.
• Reuse JSON/JSONB data already available during ingestion.
• Support arbitrary and changing JSON structures.
• Keep JSON/JSONB metadata generation integrated with the ingestion flow.

[ZmeiGorynych]

Thanks for the issue!
To help us address it, can you please give some details:

1. "There is already a separate JSON processor that performs recursive JSON traversal, type inference, type merging, sensitive-field filtering, and SLayer metadata generation" - could you please tell me what file/functions/line numbers this refers to.

2. Where do you see that metadata then being

   • stored?
   • exposed?
   • used?

3. Do you mean this to work when each record in a column follows the same schema, or for a json column that contains arbitrary records ("Different rows within the same JSONB column may also contain different structures." seems to sugges the latter, but I have a hard time imagining what could be pre-extracted in that case)

[sanphy-citrus]

Thanks for the response.
question 1,2 answer : The Json Processor has been added by me as a seperate python script to get the functionality what i described in the expected behaviour section. The Slayer current behaviour it self takes the json data as such and not seperating keys , values.
my requirement of Slayer I have clearly described in the "Expected Behaviour section".
question 3 answer : I mean this to work when each record in a column follows the same schema. The optional thing is "Different rows within the same JSONB column may also contain different structures" .