Formalize AST format

.github/workflows/ci.yml

@@ -18,5 +18,9 @@ jobs:
           python-version: "3.11"
       - name: Install test dependency
         run: pip install pytest
+      - name: Validate JSON schemas
+        run: |
+          #pip install jsonschema
+          #python test/validate_schemas.py
       - name: Run harness self-tests
         run: python -m pytest test/.self-test -v

.github/workflows/tests.yml

@@ -18,6 +18,11 @@ jobs:
         with:
           python-version: "3.11"
 
+      - name: Validate JSON schemas
+        run: |
+          #pip install jsonschema
+          #python test/validate_schemas.py
+
       - name: Install test harness dependency
         run: pip install pytest
 
@@ -33,4 +38,4 @@ jobs:
           exit 1
 
       - name: Run tests for grammar ${{ matrix.grammar }}
-        run: python3 test/run_tests.py test --grammar ${{ matrix.grammar }} --no-build
+        run: python3 test/run_tests.py test --grammar ${{ matrix.grammar }} --no-build

doc/grammar1.md

@@ -95,3 +95,138 @@ SysProLang uses C-style comments:
 - **Comments and whitespace** are ignored by the parser (handled by the lexer).
 - **Error recovery**: see [`doc/error-handling.md`](error-handling.md) for the
   recommended error recovery strategy.
+
+## Token kinds
+
+These are the `"kind"` values that appear in `tokens.json` golden files:
+
+| Token kind | Grammar source | Notes |
+|---|---|---|
+| `IDENT` | `IDENT` (any identifier) | |
+| `INT` | `INTEGER_LITERAL` | |
+| `RETURN` | `return` | Keyword |
+| `VAL` | `val` | Keyword |
+| `VAR` | `var` | Keyword |
+| `PLUS` | `+` | |
+| `MINUS` | `-` | |
+| `MULT` | `*` | |
+| `DIV` | `/` | |
+| `ASSIGN` | `=` | Assignment |
+| `LPAREN` | `(` | |
+| `RPAREN` | `)` | |
+| `SEMI` | `;` | |
+| `EOF` | (end of file) | Always the last token |
+
+> Comments are filtered out by the lexer before the parser sees them,
+> so comment token kinds never appear in golden token files.
+
+### Example
+
+Source:
+
+```
+var x = 10;
+return x / 2;
+```
+
+```json
+[
+{"kind": "VAR", "value": "var", "line": 1, "column": 1},
+{"kind": "IDENT", "value": "x", "line": 1, "column": 5},
+{"kind": "ASSIGN", "value": "=", "line": 1, "column": 7},
+{"kind": "INT", "value": "10", "line": 1, "column": 9},
+{"kind": "SEMI", "value": ";", "line": 1, "column": 11},
+{"kind": "RETURN", "value": "return", "line": 2, "column": 1},
+{"kind": "IDENT", "value": "x", "line": 2, "column": 8},
+{"kind": "DIV", "value": "/", "line": 2, "column": 10},
+{"kind": "INT", "value": "2", "line": 2, "column": 12},
+{"kind": "SEMI", "value": ";", "line": 2, "column": 13},
+{"kind": "EOF", "value": "", "line": 2, "column": 14}
+]
+```
+
+## AST node kinds
+
+These are the `"kind"` values that appear in `ast.json` golden files:
+
+| AST kind | `elems[]` children | Notes |
+|---|---|---|
+| `Program` | `[statements...]` | Wraps all top-level statements (implicit `main()`) |
+| `Return` | `[expr]` | Return value is the single child |
+| `Declare` | `[name, expr]` | `var`/`val` declaration with name as `Ident` node |
+| `Assign` | `[name, expr]` | Name is an `Ident` node |
+| `IntLiteral` | `[]` | Literal integer value |
+| `Ident` | `[]` | Identifier name reference |
+| `BinOp` | `[left, right]` | Binary arithmetic operator |
+| `Unary` | `[operand]` | Unary minus |
+| `Error` | `[]` | Produced during error recovery |
+
+### Example
+
+Source:
+
+```scala
+var x = 10;
+return x / 2;
+```
+
+```json
+{
+  "line": 1,
+  "column": 1,
+  "kind": "Program",
+  "elems": [
+    {
+      "line": 1,
+      "column": 1,
+      "kind": "Declare",
+      "mut": "var",
+      "elems": [
+        {
+          "line": 1,
+          "column": 5,
+          "kind": "Ident",
+          "value": "x",
+          "elems": [],
+        },
+        {
+          "line": 1,
+          "column": 9,
+          "kind": "IntLiteral",
+          "value": 10,
+          "elems": []
+        }
+      ],
+    },
+    {
+      "line": 2,
+      "column": 1,
+      "kind": "Return",
+      "elems": [
+        {
+          "line": 2,
+          "column": 10,
+          "kind": "BinOp",
+          "value": "/",
+          "elems": [
+            {
+              "line": 2,
+              "column": 8,
+              "kind": "Ident",
+              "value": "x",
+              "elems": [],
+            },
+            {
+              "line": 2,
+              "column": 12,
+              "kind": "IntLiteral",
+              "value": 2,
+              "elems": []
+            }
+          ],
+        }
+      ]
+    }
+  ],
+}
+```

doc/grammar2.md

@@ -137,6 +137,69 @@ All semantic rules from grammar 1 apply, plus:
 - **Logical operators** `&&` and `||` are short-circuit: `&&` evaluates the right
   operand only if the left is non-zero; `||` evaluates the right operand only if
   the left is zero. `!` negates (non-zero becomes `0`, zero becomes `1`).
-- **All values remain `Int64`** — no type system yet.
+- **All values remain `Int64`** --- no type system yet.
 - **Error recovery**: see [`doc/error-handling.md`](error-handling.md) for the
-  recommended error recovery strategy.
+  recommended error recovery strategy.
+
+## Token kinds (new in grammar 2)
+
+These `"kind"` values appear in `tokens.json` golden files, in addition to those from grammar 1:
+
+| Token kind | Grammar source | Notes |
+|---|---|---|
+| `IF` | `if` | Keyword |
+| `ELSE` | `else` | Keyword |
+| `WHILE` | `while` | Keyword |
+| `BREAK` | `break` | Keyword |
+| `CONTINUE` | `continue` | Keyword |
+| `TRUE` | `true` | Boolean literal |
+| `FALSE` | `false` | Boolean literal |
+| `LBRACE` | `{` | Block opening |
+| `RBRACE` | `}` | Block closing |
+| `EQ` | `==` | Equality |
+| `NE` | `!=` | Inequality |
+| `LT` | `<` | Less-than |
+| `GT` | `>` | Greater-than |
+| `LE` | `<=` | Less-or-equal |
+| `GE` | `>=` | Greater-or-equal |
+| `AND` | `&&` | Logical AND (short-circuit) |
+| `OR` | `\|\|` | Logical OR (short-circuit) |
+| `NOT` | `!` | Logical NOT |
+
+All tokens from grammar 1 also apply.
+
+### Example
+
+> TODO
+
+## AST node kinds (new in grammar 2)
+
+These `"kind"` values appear in `ast.json` golden files, in addition to those from grammar 1:
+
+| AST kind | `elems[]` children | Notes |
+|---|---|---|
+| `If` | `[cond, thenBody, elseBody?]` | `elseBody` is omitted if absent (2 or 3 children) |
+| `While` | `[cond, body]` | |
+| `Break` | `[]` | |
+| `Continue` | `[]` | |
+| `Block` | `[statements..]` | Wraps a `{ ... }` block |
+| `BoolLiteral` | `[]` | Boolean literal |
+| `BinOp` | `[left, right]` | New operators: `"=="`, `"!="`, `"<"`, `">"`, `"<="`, `">="`, `"&&"`, `"\|\|"` |
+| `Unary` | `[operand]` | New operator: `"!"` |
+
+All AST kinds from grammar 1 also apply.
+
+### Notes on expression AST
+
+- `==`, `!=`, `<`, `>`, `<=`, `>=` comparisons are represented as `BinOp` nodes.
+- `&&`, `||` are also `BinOp` nodes.
+- `!` (logical NOT) is a `Unary` node.
+- `true` and `false` literals are `BoolLiteral` nodes. In grammar 2 (no type system)
+  semantically they are treated as `Int64` 1 or 0.
+- `If` nodes have `elems[0]` = condition, `elems[1]` = then-body, and optionally
+  `elems[2]` = else-body. The presence of exactly 2 or 3 children distinguishes
+  `if (cond) stmt` from `if (cond) stmt else stmt`.
+
+### Example
+
+> TODO

doc/grammar3.g4

@@ -13,7 +13,21 @@ grammar grammar3;
 // lexer runs a single-line comment to end-of-line, a generated line comment
 // must always end with a newline (whitespace.py guarantees this).
 
-program : externDeclaration* funcDeclaration* EOF;
+program : topDeclaration* EOF;
+
+topDeclaration
+    : externDeclaration
+    | funcDeclaration
+
+// extern declaration for C interop.
+// Example: extern def foo(a, b);
+externDeclaration : 'extern' 'def' IDENT '(' paramList? ')' ';';
+
+// Function declaration with optional parameters.
+// Example: def add(a, b) { return a + b; }
+funcDeclaration : 'def' IDENT '(' paramList? ')' block;
+
+paramList : IDENT (',' IDENT)*;
 
 statement
     : returnStatement
@@ -50,16 +64,6 @@ breakStatement : 'break' ';';
 
 continueStatement : 'continue' ';';
 
-// extern declaration for C interop.
-// Example: extern def foo(a, b);
-externDeclaration : 'extern' 'def' IDENT '(' paramList? ')' ';';
-
-// Function declaration with optional parameters.
-// Example: def add(a, b) { return a + b; }
-funcDeclaration : 'def' IDENT '(' paramList? ')' block;
-
-paramList : IDENT (',' IDENT)*;
-
 // Expression definitions go from lowest operator precedence
 // to the highest, allowing for straightforward expression parsing.
 // Comparison operators produce integer values (0 or 1).
@@ -126,4 +130,4 @@ IDENT : [a-zA-Z_] [a-zA-Z0-9_]*;
 // parser rule references them.
 WS : [ \t\n\r]+;
 LINE_COMMENT : '//' ~[\r\n]*;
-BLOCK_COMMENT : '/*' .*? '*/';
+BLOCK_COMMENT : '/*' .*? '*/';

doc/grammar3.md

@@ -5,7 +5,16 @@ Version: 3
 ## Grammar
 
 ```bnf
-program ::= { externDeclaration } { funcDeclaration } { statement } EOF
+program ::= { topDeclaration } EOF
+
+topDeclaration ::= externDeclaration
+                 | funcDeclaration
+
+externDeclaration ::= "extern" "def" IDENT "(" [ paramList ] ")" ";"
+
+funcDeclaration ::= "def" IDENT "(" [ paramList ] ")" block
+
+paramList ::= IDENT { "," IDENT }
 
 statement ::=
     returnStatement
@@ -39,12 +48,6 @@ breakStatement ::= "break" ";"
 
 continueStatement ::= "continue" ";"
 
-externDeclaration ::= "extern" "def" IDENT "(" [ paramList ] ")" ";"
-
-funcDeclaration ::= "def" IDENT "(" [ paramList ] ")" block
-
-paramList ::= IDENT { "," IDENT }
-
 
 ; Expression definitions go from lowest operator precedence
 ; to the highest, allowing for straightforward expression parsing.
@@ -141,17 +144,48 @@ All semantic rules from grammar 2 apply, plus:
 - **Function definitions** use `def name(params) { body }`. Parameters are passed
   by value. A function must have a `return` statement if it returns a value.
 - **Functions are visible anywhere** in the program after they are defined (forward
-  references are allowed — the compiler can collect all function definitions before
+  references are allowed --- the compiler can collect all function definitions before
   codegen).
 - **`extern` declarations** declare a C function with no body. These are linked
   with the compiled program at the end. The calling convention is the C ABI.
 - **Function calls** are expressions. A call evaluates all arguments, then transfers
   control to the function. The function's return value is the result of the call
   expression.
-- **Top-level statements** (outside any function) form the body of the implicit
-  `main()` function. Top-level statements are executed in order when the program
-  starts. A `return` at the top level returns from `main()`.
-- **All values remain `Int64`** — no type system yet. Functions take `Int64`
+- **All values remain `Int64`** --- no type system yet. Functions take `Int64`
   parameters and return `Int64`.
+- **A `main` function must be explicitly defined** Top-level statements are not allowed.
 - **Error recovery**: see [`doc/error-handling.md`](error-handling.md) for the
-  recommended error recovery strategy.
+  recommended error recovery strategy.
+
+## Token kinds (new in grammar 3)
+
+These `"kind"` values appear in `tokens.json` golden files, in addition to those from grammar 1–2:
+
+| Token kind | Grammar source | Notes |
+|---|---|---|
+| `DEF` | `def` | Keyword |
+| `EXTERN` | `extern` | Keyword |
+| `COMMA` | `,` | Parameter/argument separator |
+
+All tokens from grammar 1–2 also apply.
+
+### Example
+
+> TODO
+
+## AST node kinds (new in grammar 3)
+
+These `"kind"` values appear in `ast.json` golden files, in addition to those from grammar 1–2:
+
+| AST kind  | `elems[]` children | Notes |
+|---|---|---|
+| `ExternDecl` | `[name, params...]` | `external def` function declaration |
+| `FuncDecl` | `[name, params..., body]` | `def` function declaration with body as `Block` node |
+| `Param` | `[name]` | Name is an `Ident` node |
+| `Call` | `[callee, args...]` | Callee is an `Ident` node followed by argument expressions |
+
+All AST kinds from grammar 1–2 also apply.
+
+### Example
+
+> TODO