diff --git a/deploy/install/README.md b/deploy/install/README.md index e39e01bdc..d10383019 100644 --- a/deploy/install/README.md +++ b/deploy/install/README.md @@ -5,7 +5,7 @@ This directory holds the Core/Web installer, the `oac` command and the node inst | Module | Role | | --- | --- | | `install.sh`, `install.py` | Core/Web installer: host checks, fresh installation and same-bundle repair | -| `oac_cli.py` | The `oac` command (`status`, `start`, `stop`, `apply`, `domain`, `rotate-core-key`), packaged as `oac.pyz` | +| `oac_cli.py` | The `oac` command (`status`, `start`, `stop`, `apply`, `domain`, `rotate-core-key`, `uninstall`), packaged as `oac.pyz` | | `config.schema.json`, `config_model.py` | The `config.json` schema, its defaults and the subset validator | | `configuration.py` | Everything under `generated/` and the service input digests | | `ingress.py`, `ingress_config.py` | Managed HTTPS gateway and the domain operation | @@ -24,7 +24,7 @@ This directory holds the Core/Web installer, the `oac` command and the node inst - The Core/Web installer never adds its own host as a node, imports no Runtime image and gives Core neither the Docker socket nor host devices. Nodes are added afterwards from Web, this host included. - Ingress is an installation concern, independent of Runtime and Sandbox Provider selection. - A repeated installation or repair never changes provider identity, the backend namespace or native history. The database, Projects and their keys, provider identity and the credential encryption key survive repair. -- Recovery never deletes data and never prunes containers, volumes or images. The one removal is a [new installation's own cleanup](#new-installations). +- Recovery never deletes data and never prunes containers, volumes or images. Only a [new installation's own cleanup](#new-installations) and [`oac uninstall`](#uninstall) remove anything. - Do not add another launcher, scheduler, supervisor or recovery path. ## Configuration and apply @@ -42,17 +42,23 @@ This directory holds the Core/Web installer, the `oac` command and the node inst - Install only into an empty directory or over an [incomplete installation](#new-installations), or repair a complete installation of the same source revision. Refuse older formats and different revisions before changing anything; keep their data and direct the operator to install separately. Distributions carry only current installation code: no conversion, migration or binary replacement. Keep the refusal checks and their tests. - The packaged `oac.pyz` embeds its build revision and refuses a `state.json` whose `source_commit` differs. -- The installer and every mutating `oac` command share `.oac.lock`. The installer holds it across creation, payload, native service and launcher repair, and apply, calling the already-locked apply implementation without locking again. Never replace the lock file; its inode must stay stable. Only the cleanup of a new installation unlinks it, with the directory the installer created, while holding it, and `locked` refuses a lock whose path no longer names the file it locked. +- The installer and every mutating `oac` command share `.oac.lock`. The installer holds it across creation, payload, native service and launcher repair, and apply, calling the already-locked apply implementation without locking again. Never replace the lock file; its inode must stay stable. Only the cleanup of a new installation, with the directory the installer created, and `oac uninstall` unlink it, last and while holding it. `locked` refuses a lock whose path no longer names the file it locked. ## New installations - An installation is complete after its first start: apply converged, the services are healthy and, for Web-only, its Core accepts the Core key. `create()` writes `state.json` before anything else, atomically, with the project name and `"complete": false`, and the installer sets it to `true` at that moment. A later failure, such as a refused sandbox selection, keeps the installation. -- Until then, `oac apply`, `start`, `domain` and `rotate-core-key` and Web's domain setup refuse the installation and point to the installer; `oac status` reports it. Only the installer starts or removes an incomplete installation. +- Until then, `oac apply`, `start`, `domain` and `rotate-core-key` and Web's domain setup refuse the installation and point to the installer and `oac uninstall`; `oac status` reports it. Only the installer starts an incomplete installation; the installer or `oac uninstall` removes it. - Any failure or interrupt (Ctrl-C, SIGTERM, SIGHUP) of an installation that is not complete runs `oac_cli.remove` before anything is printed, so a closed terminal can't stop it. The original error then goes on, and the installer prints it followed by `Nothing was kept; fix the problem and rerun the same command.` `remove` disables native Core's unit, runs `docker compose -p down --volumes --remove-orphans` from `/` without `COMPOSE_*` variables and with its output discarded, and deletes every file in the installation directory. The directory goes too when the installer created it; otherwise it stays with its lock. Loaded images stay. - The release downloader execs the installer, so the installer itself receives Ctrl-C, SIGTERM and SIGHUP and nothing stops it while it removes a failed installation. - `remove` touches only the `oac-<10 hex digits>` project and its `-core.service` named in `state.json`, and never follows a link. `state.json`, then the `oac` command, then `.oac.lock` are removed last, and the files stay when a service can't be removed, so the next run still recognizes the installation and no other command locks it afresh mid-removal. Those final unlinks ignore SIGINT, SIGTERM and SIGHUP; if the command cannot be unlinked, cleanup restores `state.json` for a retry. SIGKILL or power loss during those final unlinks can leave files that need manual removal. The error lists what is left and the commands that remove it; the printed Compose command uses the same directory and environment isolation as automatic cleanup. - A rerun over an incomplete installation, a `state.json` without `"complete": true`, removes it the same way under the lock and then checks the settings and ports and installs with the flags given now. Only that explicit marker proves completion. A directory without `state.json` is refused whatever it holds, and nothing in it is removed. +## Uninstall + +- `oac uninstall` is the only uninstall entry point; only the installed `oac` matches its release's layout, so there is no uninstall script. It needs only `state.json`, accepts complete and incomplete installations alike, and calls `remove` with image removal and without `keep_root`. Without `state.json` it removes nothing and says so. The operator guide is [Uninstall](../../docs/getting-started/operations.md#uninstall). +- It removes each image `state.json` records unless a tag names it or a container of any project uses it, and reports those it keeps. The installer loads images untagged, so either means something outside the installation uses the image. It never forces an image removal, and runs Docker as `remove` runs Compose. +- Before confirmation it prints what it removes and, when Core answers, the nodes and the number of sandboxes in use that Core lists, with a warning that uninstall stops no sandbox; E2B sandboxes keep running and billing. It then needs the typed directory or `--yes`; without a terminal only `--yes` confirms. It never contacts a node or a sandbox provider; afterwards it prints the nodes' `--force` uninstall command with the installation ID. + ## Install-time sandbox selection `--sandbox docker|microsandbox|e2b|none` (default `microsandbox`; only `none` with `--web-only`) is a one-time action. After the services are healthy, the installer posts `/core/v1/sandbox/deployment` once, as Web's setup would, and never on a repair. The choice is not written to `config.json`; PostgreSQL owns it, and an existing database selection is never overwritten. diff --git a/deploy/install/install_output.py b/deploy/install/install_output.py index 9c96a1ff3..6f192d5c7 100644 --- a/deploy/install/install_output.py +++ b/deploy/install/install_output.py @@ -69,6 +69,6 @@ def summary(root, config, addresses, fresh, selection, deployment, reachable, in heading("Manage") print(f" Settings: {root / 'config.json'}") command = shlex.quote(str(root / "oac")) - for label, action in (("Apply settings", "apply"), ("Status", "status"), ("Start", "start"), ("Stop", "stop")): + for label, action in (("Apply settings", "apply"), ("Status", "status"), ("Start", "start"), ("Stop", "stop"), ("Uninstall", "uninstall")): print(f" {label}: {command} {action}") print("\nNo model request was made. Quickstart: docs/getting-started/quickstart.md", flush=True) diff --git a/deploy/install/installer_fakes.py b/deploy/install/installer_fakes.py index 2160a9d9c..ff53ff458 100644 --- a/deploy/install/installer_fakes.py +++ b/deploy/install/installer_fakes.py @@ -41,6 +41,7 @@ def __init__(self, test): self.web_port = None self.native_root = None # the installation whose native unit systemctl manages self.missing_images = set() + self.other_containers = {} # image ID -> containers of other installations that use it self.core_installation_id = "11111111-2222-4333-8444-555555555555" self.bindings = {"nodes": 0, "nodes_on_other_address": 0, "hosted_sandboxes": 0, "self_hosted_executors": 0} self.nodes = [] @@ -99,7 +100,13 @@ def run(self, args, check=False, **kwargs): stdout = "2.30.0" elif args[:3] == ["docker", "image", "inspect"]: code = 1 if args[3] in self.missing_images else 0 - stdout = "" if code else args[3] + " linux/amd64" + stdout = "" if code else "[]" if "{{json .RepoTags}}" in args else args[3] + " linux/amd64" + elif args[:3] == ["docker", "image", "rm"]: + self.missing_images.add(args[3]) + elif args[:3] == ["docker", "image", "ls"]: + stdout = "\n".join(sorted(set(MANIFEST["images"].values()) - self.missing_images)) + elif args[:2] == ["docker", "ps"] and args[-1].startswith("ancestor="): + stdout = "\n".join(self.other_containers.get(args[-1].removeprefix("ancestor="), [])) elif args[:2] == ["docker", "ps"]: stdout = "\n".join(name for name, item in self.containers.items() if item["running"]) if "--format" in args else "\n".join("id-" + name for name in self.containers) elif args[:2] == ["docker", "inspect"]: diff --git a/deploy/install/oac_cli.py b/deploy/install/oac_cli.py index c6c3821e6..830afdbca 100644 --- a/deploy/install/oac_cli.py +++ b/deploy/install/oac_cli.py @@ -1,4 +1,4 @@ -"""oac: status, start, stop, apply and rotate-core-key for one installation. +"""oac: status, start, stop, apply, rotate-core-key and uninstall for one installation. The installation directory is the directory that holds the command. The bundle it was installed from is never needed. config.json is the only file an operator edits; @@ -12,6 +12,7 @@ import datetime import errno import fcntl +from http.client import HTTPException import ipaddress import json import os @@ -40,8 +41,8 @@ SOURCE_COMMIT = None # Set by the packaged entrypoint from its build revision. UNSUPPORTED_VERSION = ("This installation version or historical conversion is not supported; " "preserve its data and reinstall into a new empty directory. Nothing was changed.") -INCOMPLETE = ("This installation did not finish installing. Rerun the installer command; it removes what is left " - "and installs again.") +INCOMPLETE = ("This installation did not finish installing. Rerun the installer command, which removes what is " + "left and installs again, or remove it with oac uninstall.") class OacError(Exception): @@ -800,14 +801,16 @@ def _remove_last_files(root, state, keep_root): signal.signal(signum, handler) -def remove(root, state, keep_root=False): +def remove(root, state, keep_root=False, images=False): """Remove one installation: native Core's unit, its Compose project with its volumes, then its files. state is the loaded state.json. Only this installation's own project and unit are - touched; loaded images are kept. keep_root keeps the directory and its .oac.lock, which - the caller holds, and removes everything else in it. The files stay while a service is - left, so state.json still names it. Nothing is printed, so a closed terminal can't stop - the removal. Raises OacError naming what is left and the commands that remove it. + touched. Loaded images are kept unless images is set; see remove_images. keep_root keeps + the directory and its .oac.lock, which the caller holds, and removes everything else in + it. The files stay while a service is left or an image removal fails, so state.json still + names them. Nothing is printed, so a closed terminal can't stop the removal. Returns a + note for each image kept; raises OacError naming what is left and the commands that + remove it. """ root = Path(root) if not root.is_absolute() or root.is_symlink() or root.resolve() != root: @@ -815,7 +818,7 @@ def remove(root, state, keep_root=False): project = state.get("project") if not isinstance(project, str) or not re.fullmatch(r"oac-[0-9a-f]{10}", project): raise OacError("state.json names no Compose project of this installation; nothing was removed") - left = [] + left, kept = [], [] if native_service.is_native(state): unit = native_service.unit_name(state) try: @@ -833,6 +836,11 @@ def remove(root, state, keep_root=False): except (subprocess.CalledProcessError, OSError, KeyboardInterrupt): manual = shlex.join(["env", *(arg for name in compose_names for arg in ("-u", name)), *down]) left.append((f"Compose project {project} and its volumes", f"(cd / && {manual})")) + if images and not left: + try: + kept = remove_images(state, environment) + except (subprocess.CalledProcessError, OSError, ValueError, KeyboardInterrupt): + left.append(("the images of this installation", f"{shlex.quote(str(root / 'oac'))} uninstall")) target = shlex.quote(str(root)) files = (f"the files in {root}", f"find {target} -mindepth 1 -delete" if keep_root else f"rm -rf {target}") if left: @@ -854,6 +862,36 @@ def remove(root, state, keep_root=False): if left: raise OacError("Removal did not finish. Left: " + "; ".join(what for what, _ in left) + ". Remove them with:\n" + "\n".join(" " + command for _, command in left)) + return kept + + +def remove_images(state, environment): + """Remove the images state.json records; returns a note for each one kept. + + The installer loads images untagged, so a tag, or a container of any project, means + something outside this installation uses the image, and it stays. Docker runs as remove + runs Compose, and prints nothing. + """ + kept = [] + quiet = {"cwd": "/", "env": environment, "stdin": subprocess.DEVNULL, "stderr": subprocess.DEVNULL} + for name, image in sorted(state["images"].items()): + found = run(["docker", "image", "inspect", image, "--format", "{{json .RepoTags}}"], check=False, + stdout=subprocess.PIPE, text=True, **quiet) + if found.returncode: + present = run(["docker", "image", "ls", "--all", "--quiet", "--no-trunc"], + stdout=subprocess.PIPE, text=True, **quiet).stdout.split() + if image in present: + raise subprocess.CalledProcessError(found.returncode, found.args) + continue # Docker confirmed that the image is absent. + tags = json.loads(found.stdout) or [] + users = run(["docker", "ps", "-aq", "--filter", "ancestor=" + image], stdout=subprocess.PIPE, text=True, + **quiet).stdout.split() + if users or tags: + kept.append(f"Kept the {name} image {image}: " + ("another container uses it." if users else + f"it is tagged {', '.join(tags)}.")) + else: + run(["docker", "image", "rm", image], stdout=subprocess.DEVNULL, **quiet) + return kept # Commands -------------------------------------------------------------------- @@ -1048,6 +1086,87 @@ def rotate_core_key(root, yes=False, interactive=None, out=print): out("A separate Web-only installation keeps its own copy: copy secrets/core.key to that host and run its oac apply.") +def core_sandboxes(root, state): + """Core's (nodes, sandbox deployment), or None when Core does not answer. Any part of the installation may be missing.""" + try: + if not root.is_absolute() or root.resolve() != root: + return None + check_directories(root) + check_private(root / "secrets/core.key", "secrets/core.key") + try: + config = load_config(root) + except (OacError, config_model.ConfigError): + config = None + # The address the running Core was started with. + base, key = core_base(written_view(root, state, config)), bearer(configuration.read_core_key(root)) + answers = [http(base + "/core/v1/sandbox/" + name, key) for name in ("nodes", "deployment")] + if any(status != 200 for status, _ in answers): + return None + return json.loads(answers[0][1])["data"], json.loads(answers[1][1]) + except (OacError, RuntimeError, OSError, ValueError, KeyError, TypeError, HTTPException): + return None + + +def uninstall(root, yes=False, interactive=None, out=print): + """Remove this installation and all its data from this host, finished or not. Nodes and sandboxes are only listed.""" + interactive = sys.stdin.isatty() if interactive is None else interactive + if not (root / "state.json").exists(): + # Without state.json nothing here is known to be the installation's, so nothing is removed. + out(f"No installation is left in {root}: it has no state.json. Nothing was removed.") + return + with locked(root): + state = load_state(root) + project, core = state["project"], state["mode"] != "web-only" + out(f"This removes the installation in {root} from this host:") + out(f" Compose project {project}: its containers and networks" + + (f", and the database volume {project}_database" if core else "")) + if native_service.is_native(state): + out(f" Native Core service {native_service.unit_name(state)}") + for name, image in sorted(state["images"].items()): + out(f" The {name} image {image}, unless another container or a tag uses it") + out(f" The directory {root}") + nodes = [] + if not core: + out("The paired Core and its data are not touched.") + else: + out("All data is deleted: the database with every Project, API key, Session and stored credential, and " + "the Core key. To keep it, back it up first: docs/getting-started/operations.md#back-up") + found = core_sandboxes(root, state) + release = ("While Core is still up, archive their Sessions, or choose Reset deployment in Web " + "(System → Manage sandbox configuration) and let it complete.") + if found is None: + nodes = None + out("Core did not answer, so its nodes and sandboxes can't be listed. Nodes stay on their hosts.") + out("Uninstall stops no sandbox: node sandboxes keep running on their nodes, and E2B keeps running, " + "and billing for, its sandboxes. " + release) + else: + nodes, deployment = found + if nodes: + out("Nodes registered with this Core, which stay on their hosts: " + ", ".join( + f'{node.get("name")} ({"online" if node.get("online") else "offline"})' for node in nodes)) + count = (deployment.get("resources") or {}).get("allocations") or 0 + if count: + where = ("E2B keeps running them, and billing for them" if deployment.get("provider") == "e2b" + else "they keep running on their nodes") + out(f"Core has {count} sandbox(es) in use. Uninstall does not stop them: {where}. {release}") + if not yes: + if not interactive: + raise OacError("Confirm the uninstall with --yes, or run it in a terminal; nothing was removed") + if input(f"Type the installation directory, {root}, to remove it: ").strip() != str(root): + raise OacError("The uninstall was not confirmed; nothing was removed") + kept = remove(root, state, images=True) + for note in kept: + out(note) + out(f"Removed the installation in {root}.") + if nodes != []: + hosts = (f'each node host ({", ".join(str(node.get("name")) for node in nodes)})' if nodes else + "each node host that served this installation") + out(f"This Core is gone, so on {hosts}, uninstall the node with --force using node-install.pyz from this " + "release's bundle:") + out(f' sudo python3 node-install.pyz --uninstall --installation-id {state["installation_id"]} --force') + out("See docs/getting-started/nodes.md#remove-a-node") + + def main(argv=None, root=None, out=print): root = Path(root) if root else Path(sys.argv[0]).resolve().parent if Path(sys.argv[0]).name == "parsar": @@ -1070,7 +1189,11 @@ def main(argv=None, root=None, out=print): commands.add_parser("domain-server", help=argparse.SUPPRESS) rotate = commands.add_parser("rotate-core-key", help="Replace the Core key; the old key stops working") rotate.add_argument("--yes", action="store_true", help="Do not ask for confirmation") + remove_parser = commands.add_parser("uninstall", help="Remove this installation and all its data from this host") + remove_parser.add_argument("--yes", action="store_true", help="Do not ask for confirmation") args = parser.parse_args(argv) + if args.command == "uninstall": + return uninstall(root, yes=args.yes, out=out) if not (root / "state.json").exists(): raise OacError(f"{root} is not an installation directory; run the oac command inside it") state = load_state(root) diff --git a/deploy/install/test_install_output.py b/deploy/install/test_install_output.py index 34150edad..a97c159d9 100644 --- a/deploy/install/test_install_output.py +++ b/deploy/install/test_install_output.py @@ -51,6 +51,7 @@ def test_summary_groups_details_and_keeps_commands_copyable(self): self.assertEqual(positions, sorted(positions)) self.assertIn(" Core key file: /tmp/install with spaces/secrets/core.key\n", text) self.assertIn(" Apply settings: '/tmp/install with spaces/oac' apply\n", text) + self.assertIn(" Uninstall: '/tmp/install with spaces/oac' uninstall\n", text) self.assertNotIn("\033[", text) diff --git a/deploy/install/test_oac.py b/deploy/install/test_oac.py index e9a182276..c1c23ec5d 100644 --- a/deploy/install/test_oac.py +++ b/deploy/install/test_oac.py @@ -1,8 +1,10 @@ """Acceptance behavior of the oac command against a fake Docker/systemd host.""" import hashlib +import http.client import json from pathlib import Path import stat +import subprocess import tempfile from types import SimpleNamespace import unittest @@ -413,5 +415,124 @@ def test_the_next_apply_finishes_any_interrupted_apply_rotation_or_rollback(self self.apply() self.assertConverged() + def test_uninstall_removes_the_services_volumes_unused_images_and_directory(self): + self.install() + project = oac_cli.load_state(self.root)["project"] + self.host.other_containers[IMAGES["database"]] = ["id-of-another-installation"] + self.host.deployment.update(provider="e2b", resources={"allocations": 2, "pending": 0}) + oac_cli.main(["uninstall", "--yes"], root=self.root, out=self.output.append) + self.assertTrue(any(line.startswith("Core has 2 sandbox(es) in use. Uninstall does not stop them: E2B keeps " + "running them, and billing for them.") for line in self.output)) + self.assertIn(["docker", "compose", "-p", project, "down", "--volumes", "--remove-orphans"], self.host.commands) + self.assertEqual(self.host.containers, {}) + self.assertEqual(self.host.missing_images, {IMAGES["core"], IMAGES["web"]}) + self.assertIn(f'Kept the database image {IMAGES["database"]}: another container uses it.', self.output) + self.assertFalse(self.root.exists()) + + def test_uninstall_changes_nothing_unless_confirmed(self): + self.install() + before = {str(p.relative_to(self.root)): p.read_bytes() for p in self.root.rglob("*") if p.is_file()} + with self.assertRaisesRegex(oac_cli.OacError, "--yes"): + oac_cli.uninstall(self.root, interactive=False, out=self.output.append) + with mock.patch("builtins.input", return_value=str(self.work)), \ + self.assertRaisesRegex(oac_cli.OacError, "not confirmed; nothing was removed"): + oac_cli.uninstall(self.root, interactive=True, out=self.output.append) + self.assertEqual(before, {str(p.relative_to(self.root)): p.read_bytes() for p in self.root.rglob("*") if p.is_file()}) + self.assertEqual(self.host.running(), {"database", "core", "web"}) + self.assertFalse(any("down" in command or "rm" in command for command in self.host.commands)) + + def test_uninstall_removes_an_installation_that_never_finished(self): + install.create(self.root, SimpleNamespace(core_key_file=None), config_model.initial("all", False), + {"source_commit": "a" * 40}, IMAGES) + (self.root / "config.json").unlink() + installation = oac_cli.load_state(self.root)["installation_id"] + # create() marks it incomplete, which other mutating commands refuse. + with self.assertRaisesRegex(oac_cli.OacError, "or remove it with oac uninstall"): + oac_cli.start(self.root, out=self.output.append) + oac_cli.uninstall(self.root, yes=True, out=self.output.append) + self.assertFalse(self.root.exists()) + self.assertIn("Core did not answer, so its nodes and sandboxes can't be listed. Nodes stay on their hosts.", + self.output) + self.assertIn(f" sudo python3 node-install.pyz --uninstall --installation-id {installation} --force", self.output) + + def test_a_rerun_finishes_an_interrupted_uninstall(self): + self.install() + (self.root / "oac").write_text("the installed command") + with interrupt(oac_cli.shutil, "rmtree"), self.assertRaisesRegex(oac_cli.OacError, "Removal did not finish"): + oac_cli.uninstall(self.root, yes=True, out=self.output.append) + self.assertEqual(self.host.containers, {}) + self.assertTrue((self.root / "state.json").exists() and (self.root / "oac").exists()) + oac_cli.uninstall(self.root, yes=True, out=self.output.append) + self.assertFalse(self.root.exists()) + # Once state.json is gone, a rerun only says so. + self.root.mkdir() + (self.root / "oac").write_text("the installed command") + oac_cli.uninstall(self.root, yes=True, out=self.output.append) + self.assertIn(f"No installation is left in {self.root}: it has no state.json. Nothing was removed.", self.output) + self.assertEqual([path.name for path in self.root.iterdir()], ["oac"]) + + def test_uninstall_lists_registered_nodes_and_how_to_uninstall_them(self): + self.install() + self.host.nodes = [{"name": "node-a", "online": True}, {"name": "node-b", "online": False}] + installation = oac_cli.load_state(self.root)["installation_id"] + oac_cli.uninstall(self.root, yes=True, out=self.output.append) + listed = self.output.index("Nodes registered with this Core, which stay on their hosts: node-a (online), node-b (offline)") + self.assertLess(listed, self.output.index(f"Removed the installation in {self.root}.")) + self.assertIn(f" sudo python3 node-install.pyz --uninstall --installation-id {installation} --force", self.output) + + def test_uninstall_preserves_state_when_docker_cannot_inspect_images(self): + self.install() + original = oac_cli.run + for daemon_unavailable in (False, True): + def fail_inspection(args, **kwargs): + if args[:3] == ["docker", "image", "inspect"]: + return subprocess.CompletedProcess(args, 1, stdout="") + if daemon_unavailable and args[:3] == ["docker", "image", "ls"]: + raise subprocess.CalledProcessError(1, args) + return original(args, **kwargs) + + with self.subTest(daemon_unavailable=daemon_unavailable), \ + mock.patch.object(oac_cli, "run", side_effect=fail_inspection), \ + self.assertRaisesRegex(oac_cli.OacError, "the images of this installation"): + oac_cli.uninstall(self.root, yes=True, out=self.output.append) + self.assertTrue((self.root / "state.json").is_file()) + self.assertEqual(self.host.missing_images, set()) + self.host.missing_images.add(IMAGES["core"]) + oac_cli.uninstall(self.root, yes=True, out=self.output.append) + self.assertFalse(self.root.exists()) + self.assertEqual(self.host.missing_images, set(IMAGES.values())) + + def test_uninstall_completes_when_core_returns_a_truncated_response(self): + self.install() + with mock.patch.object(oac_cli, "http", side_effect=http.client.IncompleteRead(b"{")): + oac_cli.uninstall(self.root, yes=True, out=self.output.append) + self.assertFalse(self.root.exists()) + self.assertIn("Core did not answer, so its nodes and sandboxes can't be listed. Nodes stay on their hosts.", + self.output) + self.assertTrue(any("node-install.pyz --uninstall" in line and "--force" in line for line in self.output)) + + def test_uninstall_does_not_probe_core_through_private_file_or_directory_links(self): + self.install() + for name in ("secrets/core.key", "generated", "secrets"): + path = self.root / name + outside = self.work / path.name + path.rename(outside) + path.symlink_to(outside, target_is_directory=outside.is_dir()) + with self.subTest(path=name), mock.patch.object(oac_cli, "http") as request, \ + self.assertRaisesRegex(oac_cli.OacError, "--yes"): + oac_cli.uninstall(self.root, interactive=False, out=self.output.append) + request.assert_not_called() + path.unlink() + outside.rename(path) + # An unsafe probe does not prevent removing the installation's own files. + path.rename(outside) + path.symlink_to(outside, target_is_directory=True) + with mock.patch.object(oac_cli, "http") as request: + oac_cli.uninstall(self.root, yes=True, out=self.output.append) + request.assert_not_called() + self.assertFalse(self.root.exists()) + self.assertTrue((outside / "core.key").is_file()) + + if __name__ == "__main__": unittest.main() diff --git a/docs/getting-started/README.md b/docs/getting-started/README.md index 1c1b4cf52..833d295aa 100644 --- a/docs/getting-started/README.md +++ b/docs/getting-started/README.md @@ -11,7 +11,7 @@ Operators install Core and Web, add sandbox capacity and issue Project API keys. | [Install Core and Web](install.md) | The default installation, from an empty host to the first Project API key | | [Installation options](install-options.md) | Installer flags, existing reverse proxies, split and native deployments, offline hosts | | [Nodes](nodes.md) | Adding, changing, removing and troubleshooting sandbox nodes | -| [Operations](operations.md) | The `oac` command, the Core key, backups, upgrades and troubleshooting | +| [Operations](operations.md) | The `oac` command, the Core key, backups, uninstall, upgrades and troubleshooting | | [Configuration reference](../configuration.md) | Every setting, file and environment variable | | [Web console](../web/README.md) | What the console shows and manages | diff --git a/docs/getting-started/operations.md b/docs/getting-started/operations.md index b490ad99e..06a9bac63 100644 --- a/docs/getting-started/operations.md +++ b/docs/getting-started/operations.md @@ -22,6 +22,7 @@ Each installation has its own management command in its directory. It needs neit | `oac apply --yes` | Web-only: pairs Web with a different Core without asking | | `oac domain HOSTNAME [--confirm-public-url-change URL]` | Managed ingress: sets the public URL to `https://HOSTNAME`, as **Configure domain and HTTPS** in Web does; see [Configure the domain and HTTPS](install.md#configure-the-domain-and-https) | | `oac rotate-core-key [--yes]` | Replaces the Core key; see [Rotate the Core key](#rotate-the-core-key) | +| `oac uninstall [--yes]` | Removes the installation and all its data from this host; see [Uninstall](#uninstall) | For a second installation, use its own command, such as `~/.oac/web/oac status`. @@ -136,6 +137,22 @@ Back up these together; a restore needs all of them: Never prune Docker volumes or delete native harness history to make a retry pass. A deleted Session does not prove that all provider resources were reclaimed. +## Uninstall + +```sh +~/.oac/core/oac uninstall +``` + +It removes the installation from this host: its Compose project with the containers, networks and database volume, native Core's service, the images the installer loaded, and the installation directory, including `secrets/` and the `oac` command itself. It keeps an image that has a tag or that another container uses, such as one of another installation of the same release, and says so. + +All data goes with it: Projects and API keys, Session history, stored credentials and the Core key. The database volume is useless without `secrets/`, so it is never kept on its own. To keep the data, stop the installation with `oac stop` instead, or [back it up](#back-up) first. + +The command lists what it removes and, when Core answers, the registered nodes. Confirm by typing the installation directory, or pass `--yes`, which a run without a terminal requires. It holds the installation lock and needs only `state.json`, so it also removes an installation that did not finish installing or lost `config.json`. It removes the directory last; if it stops part way, run it again. + +Uninstall stops no sandbox: node sandboxes keep running on their nodes, and E2B sandboxes keep running, and billing, at E2B. While Core is still up, archive their Sessions or [reset the deployment](nodes.md#change-the-sandbox-configuration) and let it complete; the command shows how many sandboxes Core has in use. + +Nodes on other hosts keep running. To uninstall them the usual way, remove them in Web first, as in [Remove a node](nodes.md#remove-a-node). After `oac uninstall` their Core is gone: on each node host, run the node uninstall command with `--force`, using `node-install.pyz` from the [bundle you installed from](#installation-version-policy). `oac uninstall` prints that command with the installation ID. A Web-only installation removes only Web; its Core keeps its data and nodes. + ## Installation version policy An installation runs one release for its whole life. In-place version upgrades, downgrades and historical conversions are not supported. Nothing migrates data between releases. @@ -157,15 +174,15 @@ The installer and mutating `oac` commands hold the same installation lock, `.oac | `ADDRESS (…) is not an address of this machine …` | Set `--host`, or `host` in `config.json`, to one of the machine's IP addresses or a wildcard such as `0.0.0.0` | | `Automatic HTTPS needs ports 80 and 443 …` | Free the port the message names, install without `--public-url` and set up the domain later, or install with `--ingress external` and use your own [reverse proxy](install-options.md#https-and-the-reverse-proxy) | | `Installation directory is not empty …` | Use an empty `--install-dir` | -| `This installation is configured by …/config.json …` | Flags only seed a new installation: edit `config.json` and run `oac apply` | +| `This installation is configured by …/config.json …` | Flags only seed a new installation: edit `config.json` and run `oac apply`. To start over with other flags, [uninstall](#uninstall) it first | | `This installation version or historical conversion is not supported …` | The target directory holds an installation of another release, or a default install found one at `~/.parsar/core`. Keep it, and install into another empty `--install-dir` ([version policy](#installation-version-policy)) | | `generated/ was edited by hand` | Put the change in `config.json`, then `oac apply --discard-edits` | | `config.json has changes that are not applied` | Run `oac apply` | | `Core rejects secrets/core.key …` | Run `oac apply`, which restarts Core with the key's digest | | `config.json not applied: …` | `oac apply` printed Core's startup error above; fix `config.json` and apply again | | `The services did not start: …` | A new installation's first start failed, and the installer [removed what it created](install.md#install). Compose's or Core's error is printed above it; fix the cause and run the same command again | -| `Removal did not finish. Left: …` | The installer could not remove everything a failed new installation created. Run the printed commands to remove what is left, then run the same command again | -| `This installation did not finish installing …` | The installer stopped before reporting that the services were running. Rerun the installer command, which [removes what is left](install.md#install) and installs again | +| `Removal did not finish. Left: …` | The installer, cleaning up a failed new installation, or `oac uninstall` could not remove everything. Run the printed commands to remove what is left, or fix the cause and run the same command again | +| `This installation did not finish installing …` | The installer stopped before reporting that the services were running. Rerun the installer command, which [removes what is left](install.md#install) and installs again, or [uninstall](#uninstall) it | | `… already in use on this server. Automatic HTTPS cannot run beside another program …` during domain setup | Another program holds port 80 or 443. Stop it, using the printed `ss` command to find it, and retry; automatic HTTPS cannot share [these ports](install-options.md#ports) | | `HTTPS verification failed …` during domain setup | DNS points elsewhere, a firewall or NAT blocks inbound ports 80 and 443, or the certificate request failed; see [Configure the domain and HTTPS](install.md#configure-the-domain-and-https) | | Web answers 403 `Forbidden` | Open exactly the console address `oac status` prints; a reverse proxy must pass the original Host |