Documentation additions and other fixes

docs/dm/accel-boot.md

@@ -0,0 +1,27 @@
+# Accelerator Boot Flow
+
+This section documents the boot flow of the different accelerators.
+
+## Spatz Cluster
+
+On bootup, the Spatz domain needs to be enabled using `car_enable_domain(CAR_SPATZ_RST)`, which takes
+the cluster out of reset. The bootloader of the cluster then waits for an interrupt to continue booting.
+
+At this point, the accelerator code needs to be copied by the host domain into an area of memory accessible
+by the cluster. Once the code is in place, the address to the code needs to be communicated to the accelerator
+at `SPATZ_CLUSTER_BOOT_ADDR`. Make sure that the code is accessible to the accelerator and not in a cache:
+currently, this is can be accomplished using a fence instruction.
+
+To signal the Spatz bootloader that it can continue booting, assert an interrupt using a mailbox. If
+executing from the host domain, core 0, the mailbox IDs are 2 and 3, otherwise refer to the mailbox
+ID table in the [corresponding section of the user manual](../um/arch.md#mailbox-unit).
+
+Once the main function is done executing, the accelerator runtime writes the exit status into the
+`SPATZ_CLUSTER_EOC` register as `exit_status << 1 | 1` and returns to the bootloader. The bootloader
+then is in an infinite loop and is not able to restart execution of any program. To execute another
+program, disable and re-enable the domain and re-do the steps above. Alternatively, upload a program
+to the cluster that acts as runtime environment for program execution and implements chainloading
+different programs / functions (which is what the OpenMP implementation does).
+
+The bootloader can be found at `spatz/hw/system/spatz_cluster/test`, and does not pull in the Spatz runtime
+environment in `spatz/sw/snRuntime/sw` (which is where the startup / exit code of the application lives).

docs/dm/index.md

@@ -0,0 +1,5 @@
+# Developer Manual
+
+The *developer manual* provides information on the internals of Carfield that are abstracted to the user.
+
+- [Accelerator boot flow](accel-boot.md)

docs/um/arch.md

@@ -538,6 +538,42 @@ Assuming each mailbox is identified with id `i`, the register file map reads:
 The above register map can be found in the dedicated
 [repository](https://github.com/pulp-platform/mailbox_uni) and is reported here for convenience.
 
+Note that the `INT_RCV` interrupts are not connected to interrupt lines and can thus only be used with polling.
+
+The current ID assignment for mailbox IDs is the following:
+
+| **id** | **From**            | **To**               |
+|--------|---------------------|----------------------|
+| 0      | Safe domain         | Spatz domain, core 0 |
+| 1      | Safe domain         | Spatz domain, core 1 |
+| 2      | Host domain, core 0 | Spatz domain, core 0 |
+| 3      | Host domain, core 0 | Spatz domain, core 1 |
+| 4      | Host domain, core 1 | Spatz domain, core 0 |
+| 5      | Host domain, core 1 | Spatz domain, core 1 |
+| 6      | Host domain, core 0 | Pulp domain          |
+| 7      | Host domain, core 1 | Pulp domain          |
+| 8      | Host domain, core 0 | Secure domain        |
+| 9      | Host domain, core 1 | Secure domain        |
+| 10     | Host domain, core 0 | Safe domain          |
+| 11     | Host domain, core 1 | Safe domain          |
+| 12     | Spatz domain        | Host domain, core 0  |
+| 13     | Spatz domain        | Host domain, core 1  |
+| 14     | Spatz domain        | Safe domain          |
+| 15     | Pulp domain         | Host domain, core 0  |
+| 16     | Pulp domain         | Host domain, core 1  |
+| 17     | Pulp domain         | Safe domain          |
+| 18     | Secure domain       | Host domain, core 0  |
+| 19     | Secure domain       | Host domain, core 1  |
+| 20     | Secure domain       | Safe domain          |
+| 21     | Safe domain         | Host domain, core 0  |
+| 22     | Safe domain         | Host domain, core 1  |
+| 23     | Safe domain         | Secure domain        |
+| 24     | Safe domain         | Pulp domain          |
+
+The `INT_SND` is raised on the **To** side of the table. Note that there is no enforcement
+as to which domain is able to read or write which mailbox, all domains can read and write
+to any mailbox.
+
 ## Platform control registers
 
 PCRs provide basic system information, and control clock, reset and other functionalities of

docs/um/sw.md

@@ -78,7 +78,7 @@ BMPs for all domains can be built from the root of Carfield through a portable *
 To simplify each domain SW build as much as possible, we provide a make fragment located at
 `sw/tests/bare-metal/<domain>/sw.mk`, included in the main `sw.mk`.
 
-BMPs for each domain are compiled *in situ* in the domain repository, since each IP was design for,
+BMPs for each domain are compiled *in situ* in the domain repository, since each IP was designed for,
 or supports also, standalone execution and has its own build flow.
 
 The global command
@@ -172,7 +172,7 @@ launching its execution.
 
 Upon target device completion, the offloader:
 
-* Is asynchronously notified of the event via a mailboxe interrupt; BMOs of this kind are called
+* Is asynchronously notified of the event via a mailbox interrupt; BMOs of this kind are called
   *non-blocking*
 * Sychronously polls a specific register to catch the completion; BMOs of this kind are called
   *blocking*

docs/um/troubleshooting.md

@@ -0,0 +1,13 @@
+# Troubleshooting
+
+This is a collection of issues and their fixes.
+
+## Bare Metal: Accelerator domain never returns an exit code
+
+Double-check that the accelerator binaries are compiled for Carfield:
+In stand-alone mode the exit function does not write to the `EOC` register.
+
+## Bare Metal: `printf` does not work on accelerators
+
+Double-check that the accelerator binaries are compiled for Carfield:
+In stand-alone mode the peripheral addresses will not match for the UART.

hw/carfield.sv

@@ -1158,7 +1158,7 @@ if (CarfieldIslandsCfg.safed.enable) begin : gen_safety_island
   end
 
   for (genvar i = 0; i < CheshireNumIntHarts; i++ ) begin : gen_safed_mbox_intr
-    assign safed_hostd_mbox_intr [i] = snd_mbox_intrs[SafedMboxOffset + CheshireNumIntHarts + 1];
+    assign safed_hostd_mbox_intr [i] = snd_mbox_intrs[SafedMboxOffset + i];
   end
   assign safed_secd_mbox_intr = snd_mbox_intrs[SafedMboxOffset + CheshireNumIntHarts + 0];
 

mkdocs.yml

@@ -46,3 +46,7 @@ nav:
   - um/index.md
   - Architecture: um/arch.md
   - Software Stack: um/sw.md
+  - Troubleshooting: um/troubleshooting.md
+- Developer Manual:
+  - dm/index.md
+  - Accelerator Boot Flow: dm/accel-boot.md

scripts/elf2header.py

@@ -150,8 +150,10 @@ def __gen_stim_header(self, filename, width):
         file.write('#define ELF_BOOT_ADDR %s \n' %(hex(self.start_addr)))
       file.write('int load_binary() {\n' )
       file.write(c_function)
+      sorted_keys = sorted(self.mem.keys())
+      assert (int(sorted_keys[1]) - int(sorted_keys[0])) == 4
       for key in sorted(self.mem.keys()):
-        file.write('  (*(volatile unsigned int *)(uint64_t)(0x%X)) = 0x%0*X ;\n' % (int(key), width*2, self.mem.get(key)))
+        file.write('  (*(volatile uint32_t *)(uint64_t)(0x%X)) = 0x%0*X ;\n' % (int(key), width*2, self.mem.get(key)))
       file.write('#endif\n')
       file.write('return 0; \n }\n')