Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# amr/Makefile — canonical entry points for local development.
# Mirror of what CI will run; designed so wiring CI later is one job per target.
# See docs/CI_FUTURE.md for the GitHub Actions / GitLab CI mapping.
# Mirrors what CI runs; each target maps onto a job in .github/workflows/ci.yml.

SHELL := /usr/bin/env bash
BUILD_DIR ?= build
BUILD_TYPE ?= Debug
CMAKE_ARGS ?=
JOBS ?= $(shell nproc)

# Optional features (default off so a bare `make build` works)
# Optional features. MPI + sanitizers default ON (mirrors the Debug CI job);
# CUDA defaults OFF (needs nvcc). Override per-invocation, e.g. `make build WITH_MPI=OFF`.
WITH_MPI ?= ON
WITH_CUDA ?= OFF
WITH_SANITIZE ?= ON
Expand Down
29 changes: 17 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ in its own directory:

| Dir | Backend | Parallelism | Build | Tests |
|-----------|--------------------|------------------------|------------------|--------------------------------|
| `omp/` | shared-memory C++ | OpenMP | CMake or `g++` | Catch2 (`omp/tests.cpp`) |
| `mpi/` | distributed C++ | MPI (+OpenMP per rank) | `mpi/Makefile` | Catch2 (`mpi/tests.cpp`) |
| `cuda/` | single-GPU | CUDA + Thrust | `cuda/Makefile` | custom harness (`cuda/tests.cu`)|
| `omp/` | shared-memory C++ | OpenMP | CMake or `g++` | Catch2 (`omp/tests.cpp`) |
| `mpi/` | distributed C++ | MPI (+OpenMP per rank) | CMake (`-DAMR_BUILD_MPI=ON`) | Catch2 (`mpi/tests.cpp`) |
| `cuda/` | single-GPU | CUDA + Thrust | CMake (`-DAMR_BUILD_CUDA=ON`) | custom harness (`cuda/tests.cu`)|

Each C++ backend has the same module layout: `core` (Morton + strong types),
`tree` (refine/balance/coarsen), `physics` (oracles), `viz` (SVG/VTK), plus
Expand Down Expand Up @@ -51,24 +51,25 @@ OMP_NUM_THREADS=8 g++ -std=c++20 -O2 -fopenmp -Iomp omp/main.cpp -o omp/amr && o
### MPI (`mpi/`)

```bash
cd mpi && make # needs mpicxx + system Catch2
mpirun -np 4 ./test
mpirun -np 4 ./amr
cmake -S . -B build -DAMR_BUILD_MPI=ON && cmake --build build
mpirun -np 4 ./build/amr_mpi_tests
mpirun -np 4 ./build/amr_mpi
```

> In containers / sandboxes where the shared-memory transport stalls, force
> TCP: `mpirun --mca btl tcp,self -np 4 ./test`.
> TCP: `mpirun --mca btl tcp,self -np 4 ./build/amr_mpi_tests`.

### CUDA (`cuda/`)

```bash
cd cuda && make ARCH=sm_80 # set ARCH to your GPU; default sm_70
./amr # needs a working CUDA driver matching the runtime
# set the arch to your GPU (default sm_70 → AMR_CUDA_ARCHITECTURES=70):
cmake -S . -B build -DAMR_BUILD_CUDA=ON -DAMR_CUDA_ARCHITECTURES=80 && cmake --build build
./build/amr_cuda # needs a working CUDA driver matching the runtime
```

> The Thrust device lambdas require nvcc's `--extended-lambda` (already in
> `cuda/Makefile`). The code compiles and links without a GPU; running needs a
> driver whose version matches the CUDA runtime.
> The Thrust device lambdas require nvcc's `--extended-lambda`, which CMake sets
> automatically for the CUDA targets. The code compiles and links without a GPU;
> running needs a driver whose version matches the CUDA runtime.

## Benchmarks & docs

Expand Down Expand Up @@ -103,3 +104,7 @@ count (parallel determinism). Full data: `benchmarks/results/omp_scaling_restruc

Proper strong/weak-scaling sweeps (parametrized sizes, MPI ranks, GPU) are
produced by the Stage-2 SLURM harness, not this fixed demo.

## License

BSD 3-Clause — see [LICENSE](LICENSE).
13 changes: 6 additions & 7 deletions REFERENCES.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ The canonical parallel construction + **minimal 2:1 balance** of linear
Morton octrees via the *insulation-layer* property (no octant outside the 3^d
envelope can force a split). Output is the *coarsest* (minimal) balanced
octree; execution is iterative but bounded. Reference implementation = Dendro
(`github.com/paralab/Dendro-5.01`). See `docs/gpu-balance-notes.md` for how the
balance kernels here relate to it.
(`github.com/paralab/Dendro-5.01`). See `docs/design/gpu-balance-notes.md` for how
the balance kernels here relate to it.

**[IBG2012]** T. Isaac, C. Burstedde, O. Ghattas.
*Low-Cost Parallel Algorithms for 2:1 Octree Balance.*
Expand All @@ -37,7 +37,7 @@ The **GPU-amenable balance algorithm** we target: *octant preclusion* + the
balance reduces to sort + binary-search + compaction + `Complete` instead of a
per-pass ripple. `Reduce` (Fig. 8), generate-coarse-neighbourhood + `Linearize`
(Fig. 6), and the preclusion variant (Fig. 7) are transcribed in
`docs/gpu-balance-notes.md`.
`docs/design/gpu-balance-notes.md`.

**[IBWG2015]** T. Isaac, C. Burstedde, L. C. Wilcox, O. Ghattas.
*Recursive Algorithms for Distributed Forests of Octrees.*
Expand All @@ -51,7 +51,7 @@ exchange (Stage 2 tuning) and the multi-GPU halo path.
**[Holke2018]** J. Holke. *Scalable algorithms for parallel tree-based adaptive
mesh refinement with general element types.* PhD thesis, Univ. of Bonn (2018).
Later: J. Holke et al., *t8code v1.0*, J. Open Source Softw. (2024).
Vendored at `~/Documents/ic/code/t8code/`.
Vendored in the adaptive-dic repo at `t8code/`.

Generalization of p4est to mixed element types via a "scheme" abstraction.
Reference implementation for the cross-backend parity tests; the algorithm
Expand All @@ -60,8 +60,7 @@ API. Used for validating correctness of new partitioning strategies.

**[CDK2019]** J. Červený, V. Dobrev, T. Kolev.
*Non-Conforming Mesh Refinement for High-Order Finite Elements.*
arXiv:1905.04033 (2019). LLNL-JRNL-751849.
PDF: `~/Documents/ic/refs/Non-Conforming Mesh Refinement For High-Order Finite Elements.pdf`.
arXiv:[1905.04033](https://arxiv.org/abs/1905.04033) (2019). LLNL-JRNL-751849.

Algebraic-constraint approach (variational restriction) for non-conforming
elements with hanging nodes — the foundation for the upcoming `common/oracle_scalar_field.hpp`
Expand Down Expand Up @@ -112,7 +111,7 @@ For `cuda/*` kernels (cooperative groups, async copy, persistent kernels).

## DIC-side cross-reference

See `code/REFERENCES.md` (`~/Documents/ic/code/REFERENCES.md`) for the parallel
See the adaptive-dic `REFERENCES.md` for the parallel
DIC references: `[Sciuti2021]` (the MATLAB driver), `[ZZ1987]` (the error
estimator wrapped by the `ScalarFieldOracle`), `[HildRoux2006]`, `[Mathieu2015]`.

Expand Down
6 changes: 3 additions & 3 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ No network? Compile directly against system Catch2 — see the [README](../READM
## Run a demo

```bash
OMP_NUM_THREADS=8 ./build/omp/amr # refine + 2:1-balance a sphere, write SVG
OMP_NUM_THREADS=8 ./build/amr # refine + 2:1-balance a sphere, write SVG
```

The binary refines a geometric oracle, balances the tree, and writes an SVG of
Expand All @@ -25,8 +25,8 @@ the mesh. Tweak `dim`, `max_level`, and the oracle in `omp/main.cpp`.
cmake -S . -B build -DAMR_BUILD_MPI=ON -DAMR_BUILD_CUDA=ON # enable as needed
```

- **MPI:** `mpirun -np 4 ./build/mpi/amr`
- **CUDA:** needs a GPU + matching driver; `./build/cuda/amr`
- **MPI:** `mpirun -np 4 ./build/amr_mpi`
- **CUDA:** needs a GPU + matching driver; `./build/amr_cuda`

See the [README](../README.md) for the full build matrix and the
[Makefile](../Makefile) targets (`make build`, `make test`, `make ci-local`).
Expand Down
6 changes: 5 additions & 1 deletion docs/how-to-add-an-oracle.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,13 @@ class, no registration. See `omp/physics.hpp` for the built-in `CircleOracle`
2. **Use it** where the tree is built (`omp/main.cpp`):

```cpp
tree.refine(MyOracle{cfg}, max_level);
tree.refine(MyOracle{cfg}); // one pass; the tree passes max_level to the oracle
```

`refine()` runs a single level-pass and returns `true` while it still split
something, so drive it in a loop (`while (tree.refine(MyOracle{cfg})) {}`) to
refine to convergence.

3. **Test it** — add a Catch2 case in `omp/tests.cpp` asserting the expected
leaf set / invariants on a small fixture.

Expand Down
Loading