Decide and document how to handle versioning, extensions and validation

[tbleher]

Historically, OpenCRG files have no version number, and there is no checking of allowed data sections andy keys (at least in the Matlab API, which is the reference implementation) - unknown data sections and keys are silently ignored.

This has the advantage that new data sections and keys can be introduced, and old versions can still read the files (but will ignore any directives in the new keys or data sections). It also means that users can add their own keys and data sections (e.g. to reference 3D visualization to be used by the simulation).

This feature is also used to comment out blocks of a CRG file: by using a data section name with a name not interpreted by the reader (e.g. something like $ROAD_CRG_TMP), it becomes possible to effectively "comment out" blocks of the CRG file.

This scheme also has some important downsides, however:

• Since there is no version number, it is not possible to programatically know to which standard the file should conform
• Since users can add arbitrary keys, it is in general not possible to detect invalid keys (e.g. if they are wrongly spelled) - they will be silently ignored, and users are left scratching their heads why the key has no effect
• It becomes effectively impossible to introduce new keys into the standard, without potentially breaking some user, since any key might already be in use by a downstream user, with a different semantics

To solve this, this issue proposes:

• Adding a version number to CRG files
• Creating a policy on the naming of data blocks and keys for extensions
• Creating a policy on the naming of data blocks for extensions

The goal is to increase interoperability between implementations, allow stricter checking of CRG files (so errors are caught earlier), and to make future extensions of the standard easier.

Adding a version number to CRG files

By adding a version number to CRG files, it becomes possible to see to which standards version a file should conform to. This means the file can be programmatically checked for conformance against the standard.

Proposal:
Add a version as the first line of the $CT block.

This could look like the following:

$CT
OpenCRG Version 2.1
Normal coments here
$

Alternatively, it would be possible to add a "magic comment" to $CT:

$CT ! OpenCRG Version 2.1
Normal coments here
$

The second version has the advantage that an OpenCRG file could in the future be easily identified by searching for the string $CT ! OpenCRG at the start of the file (we could mandate that there must be no extra spaces there, so that one can search for this exact string).

Both alternatives are backward and forward compatible: old parsers can still read the new files, and new parsers could treat the lack of the version as indication that the OpenCRG file is 2.0 or older.

Rules (to be added to the standard):

• Starting with OpenCRG 2.1, each OpenCRG file MUSt have a version number identifying the version of the standard the file conforms to.
• OpenCRG readers SHOULD treat the lack of a version number as an indication that the file conforms to OpenCRG 2.0 or older.

Creating a policy on the naming of data blocks and keys for extensions

It is desirable to allow users of the file to add additional information. However, this should be done in a controlled way, so that:

• it is possible to check whether a file conforms to a standard, and no key is misspelled
• it is possible to extend the standard (add new keys or data blocks), so that all valid files continue to work

To achieve this, I propose adding rules like the following to the standard:

• Users may add new data sections to an OpenCRG file. These data sections MUST follow this naming scheme: $EXT_<domain-name>_<section-name>, where <domain-name> is the domain name of the organization adding the data section (e.g. bmw.de), and <section-name> is a user defined section name
• Future standards SHALL NOT add any data section starting with $EXT_
• Readers of OpenCRG files SHALL ignore any $EXT_ section (with all contents therein) they do not know about, and not try to parse it further (it might not contain key-value pairs, for example, but other content). The reader may notify the reader about the presence of such a section.
• Code which reads and then writes an OpenCRG file SHOULD reproduce any $EXT_ section it does not know about verbatim
• Users may add custom keys to data sections described by the standard. These keys MUST follow the naming scheme EXT_<domain-name>_<section-name> (with <domain-name> and <section-name> having the same meaning es for data sections
• Readers of OpenCRG files SHALL ignore any EXT_ key they do not know about
• Code which reads and then writes an OpenCRG file SHOULD reproduce any EXT_ key it does not know about verbatim

These rules mean that all files that only used data sections and keys described by the standard continue to be valid OpenCRG files. Files which used custom data sections and keys will need to be adjusted.

Creating a policy on the naming of data blocks for comments

OpenCRG does not offer a built-in block-comment mechanism (like C with /* ... */ or XML with <!-- ... -->).
Users historically renamed data sections for this. OpenCRG should offer a standard way to do this.

Proposal:
Any data section starting with $COMMENT is treated as a comment.

Rules to be added to the standard:

• Any data section starting with $COMMENT SHOULD be ignored by readers, and not be further parsed.
• Code which reads and then writes an OpenCRG file SHOULD reproduce any $COMMENT section verbatim

Impact on tools

Once these rules are implemented, it becomes possible to warn on invalid keys and data sections and check files for conformity to the standard.

[tbleher]

@jorauh As discussed, I have created a proposal how a version could be added, and how a naming scheme for extensions and comments could look like. I would appreciate your comments on this, since you know both the standard and relevant implementations much better than me.