Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -3804,7 +3804,18 @@ Existing, rotated or revoked credentials are never replaced by onboarding. Machi
bootstrap routes use this grant, not an Environment ID as authentication. Public
artifact routes contain no credentials. Native bundles must match the Core source
revision and Runtime wire version. Core release qualification consumes the same
three-platform native CI artifacts and includes them in its distribution.
three-platform native CI artifacts and publishes them as independent, source-qualified
Release assets. Core images and default control-service archives carry only their
small catalog (build, protocol, platform, checksum and versioned HTTPS URL), never
native execution archives. The public artifact route redirects a requested platform
to its catalog URL without proxying or caching it; clients verify the Core-provided
checksum before extraction. Installation grants are sent only to Core, never to
artifact hosts. Explicit offline distributions include one copy of each native
archive outside the Core image. The Core installer retains this directory privately
and mounts it read-only for container Core, or points native Core at the same files.
Core verifies local archives before serving; missing online archives redirect, while
corrupt local content fails closed. Neither installation nor repair downloads native
execution payloads; Session bootstrap requests only the current machine's platform.
`make check-distribution` exercises catalog assembly with manifests larger than
Node's default subprocess output buffer; catalog reads allow up to 64 MiB.
Bootstrap scripts own platform download/extraction only; installation, startup,
Expand Down
21 changes: 15 additions & 6 deletions apps/docs/content/docs/self-hosted-native.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -81,12 +81,21 @@ to resume; completed components and credentials are retained and an existing
daemon is reused. After authentication failures, check the Environment credential
in Core. Do not remove the workspace or Session history to retry.

Qualified release distributions contain Linux amd64, macOS arm64 and Windows
amd64 installers. Core serves these matched artifacts directly, including in a
private repository deployment. Unsupported platforms fail explicitly. Operators
running a standalone Core binary can set `OAC_NATIVE_INSTALLER_DIR` to its matched
`native-installers` directory. Without qualified artifacts, Session responses
report installation unavailable instead of selecting another version.
Qualified releases publish separate Linux amd64, macOS arm64 and Windows amd64
installers. Default Core installation carries only their version and checksum
catalog. The one-command bootstrap downloads only the current platform from the
fixed Release URL and verifies its checksum before extraction. It requires access
to that public download host; installation credentials stay on Core. Unsupported
platforms fail explicitly, and download failure never selects a different version.

The explicit offline Core archive carries one copy of each installer outside the
Core image. Installing it retains the files in the installation's private
`native-installers` directory and makes them available through Core's same public
artifact endpoint. No external download is needed for these archives. Standalone
Core operators can set `OAC_NATIVE_INSTALLER_DIR` to the matched catalog directory,
with optional locally supplied archives. A corrupt local archive refuses startup;
missing catalog metadata disables one-command installation. Model access and
capability dependencies may still require networking.

## Manual distribution installation

Expand Down
4 changes: 2 additions & 2 deletions apps/docs/content/guide-sources.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"contracts/agents-api/environments.md": "4c4b9423b22919d6aa6bd98d96422e1a8d610d3442d4cf7731a575bd31fd43f0",
"docs/getting-started/nodes.md": "13e1f1e867d3e361ff860960ceacc2dd5806e64f4176cd33f5e49f9a9f4d57e4",
"docs/getting-started/self-hosted.md": "4591e05bfa7657f647bedecd9db07b7086acd94b5548511c88f7e5e3567fdb11",
"docs/self-hosted-native.md": "a3b470461285f0aa9507b262ec8edf11e02f6683328d05206849debe1360b77f",
"docs/self-hosted-native.md": "1d08e46b8406411f0efeab5940cbda907cd55afb8aec159f5233d2c0ef9d3d54",
"apps/web/public/onboarding/monitor-en.webp": "29dc220cb1250c7016b7c4bf7f30e9510b07c07b814a48c3aa9303b2d76f20b2",
"docs/web/README.md": "47159689b2488f0a94a1af8488bcf98b465e0cca4003d64a4699d7b4db24e086",
"docs/api/web-management.md": "ec9a583c0a94845082ec31c7220ddcfb9aff7147c63fb36fa4c87463570f1b8c",
Expand Down Expand Up @@ -45,7 +45,7 @@
"content/docs/environments-and-files.mdx": "fa7062118cc8f13ddbd627f9990744a2002b7ca988e0b203df7ef3fd3996d9e4",
"content/docs/hosted-providers.mdx": "06f0b13b053965bbdd1bfb0a7e85d19b50987841e0512fa8331fc9a67ff71fe0",
"content/docs/self-hosted-execution.mdx": "161a9000634b4075a55960556322a1b535d8157ba0a4abbeaa84c85c7292ad33",
"content/docs/self-hosted-native.mdx": "4893e7effe8a395856611da2216e7b7a9f27b8f7de6812fbc9c5ebea0326de34",
"content/docs/self-hosted-native.mdx": "b2c5b2bb2080b5338bbaa3469f196555be31089ee713d2b9f1be8d47fa7e29cc",
"public/images/source/apps/web/public/onboarding/monitor-en.webp": "29dc220cb1250c7016b7c4bf7f30e9510b07c07b814a48c3aa9303b2d76f20b2",
"content/docs/console.mdx": "a930ce36035de74f0c2bef71bed067fc2287ba1c70a485758bf100d484f6adfd",
"content/docs/admin-api.mdx": "39f5e5373145f8d7e24e2880216fe59ac1c7fc3e43a768be0504936c0b4221fb",
Expand Down
1 change: 1 addition & 0 deletions deploy/distribution/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates
COPY --chmod=0555 bin/oac-core bin/oac-core-migrate bin/oac-core-device bin/oac-core-environment-key /usr/local/bin/

COPY e2b/ /opt/oac/e2b/
# Only the small version/checksum catalog, never native execution archives.
COPY native-installers/ /opt/oac/native-installers/

ENV OAC_ADDR=:8091
Expand Down
17 changes: 15 additions & 2 deletions deploy/install/configuration.py
Original file line number Diff line number Diff line change
Expand Up @@ -183,8 +183,8 @@ def core_environment(root, config, state):
}
if native:
result["OAC_E2B_PROVIDER_BIN"] = str(root / "native/e2b/oac-e2b-provider")
if (root / "native/native-installers/catalog.json").is_file():
result["OAC_NATIVE_INSTALLER_DIR"] = str(root / "native/native-installers")
if (root / "native-installers/catalog.json").is_file():
result["OAC_NATIVE_INSTALLER_DIR"] = str(root / "native-installers") if native else "/opt/oac/native-installers"
if core["oauth_trusted_origins"]:
result["OAC_OAUTH_TRUSTED_ORIGINS"] = ",".join(core["oauth_trusted_origins"])
if core["runtime_history"] is not None:
Expand Down Expand Up @@ -234,6 +234,8 @@ def compose_config(root, config, state):
services["database"]["ports"] = [f'127.0.0.1:{config["ports"]["database"]}:5432']
else:
mounts = [bind(root / "secrets" / name, f"{RUN}/{name}") for name in ("credential.key", "database.password")]
if (root / "native-installers/catalog.json").is_file():
mounts.append(bind(root / "native-installers", "/opt/oac/native-installers"))
mounts += [bind(root / "generated" / name, f"{RUN}/{name}") for name in ("core-key-digests.json", "settings.json")]
if config["core"]["runtime_history"] is not None:
mounts.append(bind(root / "generated/runtime-history.json", f"{RUN}/runtime-history.json"))
Expand Down Expand Up @@ -287,6 +289,16 @@ def __init__(self, files, services, unit):
self.files, self.services, self.unit = files, services, unit


def native_installer_inputs(root):
directory = root / "native-installers"
catalog = directory / "catalog.json"
if not catalog.is_file():
return None
# Archives are immutable and verified on installation/startup. Adding an
# offline archive must restart Core so its local availability map refreshes.
return [sha256(catalog.read_bytes()), sorted(p.name for p in directory.glob("*.tar.gz") if p.is_file())]


def render(root, config, state, applied_at):
root = Path(root)
mode, native = config["mode"], config.get("native_core", False)
Expand All @@ -307,6 +319,7 @@ def render(root, config, state, applied_at):
core_settings = [item for item in settings["settings"] if "core" in item["restarts"]]
external["core"] = json.dumps({
"core-key-digests.json": sha256(files["core-key-digests.json"]),
"native-installers": native_installer_inputs(root),
"settings": sha256(json.dumps([settings["path"], settings["apply_command"], core_settings], sort_keys=True)),
"runtime-history.json": sha256(files.get("runtime-history.json", "")),
"credential.key": secrets["credential.key"], "database.password": secrets["database.password"],
Expand Down
2 changes: 2 additions & 0 deletions deploy/install/install.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import configuration
from configuration import valid_core_origin
import rename
import native_installers
import native_service
import oac_cli
import sandbox_setup
Expand Down Expand Up @@ -528,6 +529,7 @@ def finish(root, bundle, manifest, fresh=False, selection=None):
step("Preparing service files")
prepare_node_payload(root, state, bundle)
native_service.prepare(root, state, bundle)
native_installers.prepare(root, state, bundle)
install_oac(root, bundle)
retry = f"rerun ./install.sh --install-dir {root}"
try:
Expand Down
56 changes: 56 additions & 0 deletions deploy/install/native_installers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
"""Retain matched catalog metadata and explicitly supplied offline installers."""

import json
import os
from pathlib import Path
import re
import shutil
import tempfile


from distribution import digest


def prepare(root, state, bundle):
if state["mode"] == "web-only":
return
source, target = Path(bundle) / "native-installers", Path(root) / "native-installers"
if not source.exists():
return
if source.is_symlink() or not source.is_dir() or target.is_symlink():
raise RuntimeError("Invalid native installer directory")
catalog_path = source / "catalog.json"
if catalog_path.is_symlink():
raise RuntimeError("Invalid native installer catalog")
catalog = json.loads(catalog_path.read_text())
if catalog["version"] != state["source_commit"]:
raise RuntimeError("Native installer catalog does not match Core")
expected = {"catalog.json": digest(catalog_path)}
for platform, artifact in catalog["artifacts"].items():
if not re.fullmatch(r"(linux|darwin|windows)-(amd64|arm64)", platform):
raise RuntimeError("Invalid native installer platform")
expected[platform + ".tar.gz"] = artifact["sha256"]
# Validate both directories before filling any missing files. Offline content
# already installed stays available when repairing with a thin bundle.
for directory in (source, target):
if not directory.exists():
continue
for path in directory.iterdir():
if (path.name not in expected or path.is_symlink() or not path.is_file()
or digest(path) != expected[path.name]):
raise RuntimeError("Native installer files differ; preserve them and inspect the distribution")
target.mkdir(mode=0o700, exist_ok=True)
for name in expected:
incoming, installed = source / name, target / name
if not incoming.exists() or installed.exists():
continue
descriptor, temporary = tempfile.mkstemp(prefix=".native-installer-", dir=root)
try:
with os.fdopen(descriptor, "wb") as outgoing, incoming.open("rb") as stream:
shutil.copyfileobj(stream, outgoing)
outgoing.flush()
os.fsync(outgoing.fileno())
os.replace(temporary, installed)
finally:
if os.path.exists(temporary):
os.unlink(temporary)
33 changes: 33 additions & 0 deletions deploy/install/test_install.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,39 @@ def key_file(self, contents="synthetic-existing-core-key-0123456789", mode=0o600
path.chmod(mode)
return path

def test_catalog_installed_for_container_and_native_core(self):
import hashlib
native = self.bundle / "native-installers"
native.mkdir()
raw = b"offline native archive"
catalog = {"version": self.manifest["source_commit"], "artifacts": {
"linux-amd64": {"sha256": hashlib.sha256(raw).hexdigest()}}}
(native / "catalog.json").write_text(json.dumps(catalog))
# The installer checks the outer checksum inventory before copying.
with (self.bundle / "SHA256SUMS").open("a") as sums:
for path in native.iterdir():
sums.write(distribution.digest(path) + " " + str(path.relative_to(self.bundle)) + "\n")
self.install()
first = self.document("generated/compose.json")["services"]["core"]["labels"]
(native / "linux-amd64.tar.gz").write_bytes(raw)
self.install()
second = self.document("generated/compose.json")["services"]["core"]["labels"]
self.assertNotEqual(first, second)
(native / "linux-amd64.tar.gz").unlink()
self.install()
self.assertEqual(second, self.document("generated/compose.json")["services"]["core"]["labels"])
installed = self.root / "native-installers"
self.assertEqual((installed / "linux-amd64.tar.gz").read_bytes(), raw)
compose = self.document("generated/compose.json")
mount = next(m for m in compose["services"]["core"]["volumes"] if m["target"] == "/opt/oac/native-installers")
self.assertTrue(mount["read_only"])
self.assertEqual(mount["source"], str(installed))
config = self.document("config.json")
config["native_core"] = True
config["ports"]["database"] = 5432
environment = install.configuration.core_environment(self.root, config, self.document("state.json"))
self.assertEqual(environment["OAC_NATIVE_INSTALLER_DIR"], str(installed))

def test_host_check_accepts_current_account_including_root(self):
for uid in (0, 1000):
with self.subTest(uid=uid), mock.patch.object(install.os, "getuid", return_value=uid), \
Expand Down
49 changes: 49 additions & 0 deletions deploy/install/test_native_installers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
"""Online/offline catalog retention shared by container and native Core."""
import json
from pathlib import Path
import tempfile
import unittest

import native_installers
from distribution import digest


class NativeInstallersTests(unittest.TestCase):
def test_online_install_offline_addition_and_repair_preserve_verified_content(self):
with tempfile.TemporaryDirectory() as temporary:
base = Path(temporary)
root, bundle = base / "root", base / "bundle"
root.mkdir()
source = bundle / "native-installers"
source.mkdir(parents=True)
archive = source / "linux-amd64.tar.gz"
archive.write_bytes(b"qualified")
payload_hash = digest(archive)
archive.unlink()
catalog = {"version": "a" * 40, "artifacts": {"linux-amd64": {"sha256": payload_hash}}}
(source / "catalog.json").write_text(json.dumps(catalog))
state = {"mode": "core-only", "source_commit": "a" * 40}
native_installers.prepare(root, state, bundle)
self.assertEqual([p.name for p in (root / "native-installers").iterdir()], ["catalog.json"])
archive.write_bytes(b"qualified")
native_installers.prepare(root, state, bundle)
installed = root / "native-installers/linux-amd64.tar.gz"
self.assertEqual(installed.read_bytes(), b"qualified")
self.assertEqual(installed.stat().st_mode & 0o777, 0o600)
archive.unlink()
native_installers.prepare(root, state, bundle)
self.assertEqual(installed.read_bytes(), b"qualified")
installed.write_bytes(b"corrupt")
with self.assertRaisesRegex(RuntimeError, "files differ"):
native_installers.prepare(root, state, bundle)
self.assertEqual(installed.read_bytes(), b"corrupt")

def test_foreign_catalog_refuses_writes(self):
with tempfile.TemporaryDirectory() as temporary:
base = Path(temporary)
source = base / "bundle/native-installers"
source.mkdir(parents=True)
(source / "catalog.json").write_text(json.dumps({"version": "foreign", "artifacts": {}}))
with self.assertRaisesRegex(RuntimeError, "does not match"):
native_installers.prepare(base / "root", {"mode": "all", "source_commit": "a" * 40}, base / "bundle")
self.assertFalse((base / "root").exists())
10 changes: 10 additions & 0 deletions docs/maintainers.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,16 @@ release base must select offline mode. Nodes obtain bootstrap metadata from the
console that generated their command. The console serves local artifacts or redirects
missing ones to the pinned HTTPS release URL. Published assets download anonymously.

Native daemon/Harness installers are separate `oac-native-<source>-<platform>.tar.gz`
Release assets with checksum files. The default Core archive and image carry only
`native-installers/catalog.json`; Session bootstrap downloads the machine's platform
on demand. The explicit offline archive includes these installers once, outside the
Core image. Core installation retains them locally for the same bootstrap endpoint.
The tag workflow assembles the catalog from matching native CI outputs. For a local
build with native onboarding, first run `scripts/build-native-catalog.mjs INPUT OUTPUT`
and set `OAC_NATIVE_INSTALLER_BUILD_DIR=OUTPUT`; missing or foreign native assets
prevent release publication. No Release or registry download occurs when Core starts.

A bundle carries a fixed set of docs (the build lists them). Links between them stay
relative; every other relative link is rewritten to the same file on GitHub at the
bundle's commit, and the build fails if a link or anchor does not resolve.
Expand Down
21 changes: 15 additions & 6 deletions docs/self-hosted-native.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,12 +78,21 @@ to resume; completed components and credentials are retained and an existing
daemon is reused. After authentication failures, check the Environment credential
in Core. Do not remove the workspace or Session history to retry.

Qualified release distributions contain Linux amd64, macOS arm64 and Windows
amd64 installers. Core serves these matched artifacts directly, including in a
private repository deployment. Unsupported platforms fail explicitly. Operators
running a standalone Core binary can set `OAC_NATIVE_INSTALLER_DIR` to its matched
`native-installers` directory. Without qualified artifacts, Session responses
report installation unavailable instead of selecting another version.
Qualified releases publish separate Linux amd64, macOS arm64 and Windows amd64
installers. Default Core installation carries only their version and checksum
catalog. The one-command bootstrap downloads only the current platform from the
fixed Release URL and verifies its checksum before extraction. It requires access
to that public download host; installation credentials stay on Core. Unsupported
platforms fail explicitly, and download failure never selects a different version.

The explicit offline Core archive carries one copy of each installer outside the
Core image. Installing it retains the files in the installation's private
`native-installers` directory and makes them available through Core's same public
artifact endpoint. No external download is needed for these archives. Standalone
Core operators can set `OAC_NATIVE_INSTALLER_DIR` to the matched catalog directory,
with optional locally supplied archives. A corrupt local archive refuses startup;
missing catalog metadata disables one-command installation. Model access and
capability dependencies may still require networking.

## Manual distribution installation

Expand Down
Loading