fix(service-settings,cli): pick the data key file that opens the stored secrets, or refuse in production

.changeset/30042-boot-picks-the-key-file-that-opens-its-secrets.md

@@ -0,0 +1,20 @@
+---
+'@objectstack/cli': patch
+'@objectstack/service-settings': patch
+---
+
+fix(cli): when two data-key files hold different keys, `os start` and `os dev` read with the one that opens the project's stored secrets, or a production boot refuses
+
+Clause-②: yes (narrowing) — a production `os start` / `os serve` boot whose two candidate key files (the shared `$HOME/.objectstack/dev-crypto-key` or `$OS_HOME`'s, and the project's `.objectstack/dev-crypto-key`) both fail to open the database's newest stored secret now refuses to start, naming both files; it used to start and serve every stored secret as unset. The widening half: `@objectstack/service-settings` gains new exports (`chooseKeyFileByTrialOpen`, `settleKeyFileChannel`, `storedSecretReaderOverEngine`, `deferredLocalCryptoProvider`, `DataKeyOpensNothingError`, `KEY_FILE_CANDIDATE_CHANNEL_ENV` and their types).
+Metadata-protocol: none
+
+<!-- adr-0087: not-required (no-migration-prescription) No key of any schema, export or stored row changes shape: the refusal is a boot posture over which key file opens existing ciphertext. No ciphertext is rewritten, so there is nothing at rest for a migration to move. -->
+
+**BREAKING** boot narrowing (`@objectstack/cli`, `@objectstack/service-settings`): a production boot whose candidate data keys open none of the stored secrets refuses to start.
+
+- **`@objectstack/cli`.** When the shared key file and the project's own key file (the one an earlier `os start` minted) hold different keys, the CLI no longer picks one. It hands the `serve` child both, and the child decides once the database is open: it opens the newest stored `sys_secret` value under each key and reads with the key that opens it, and the boot says which file it took. A project that only ever ran `os start`, on a machine whose home holds a key from `os dev` in another project, reads its secrets back again. With no stored secret yet, the shared file is used, as before. When neither key opens the stored secrets, a production boot (`os start`, `os serve`) refuses to start and names both files; a development boot (`os dev`) starts and prints a boot error naming both files. The one-shot commands (`os migrate`, `os secret rewrap`, `os secret orphans`, `os storage orphans`) now read the same file by the same rules: the project's own file when it alone holds a key, and the key that opens a stored secret when both do.
+- **`@objectstack/service-settings`.** The settings service makes that choice in its `start()`, before any reader decrypts, when the CLI names a second candidate on the internal `OS_INTERNAL_CRYPTO_KEY_FILE_CANDIDATE` channel. A leading `~` in `OS_HOME` (or `OBJECTSTACK_HOME`) is now the home directory when the provider finds its key file, as `os start` already read it; it used to name a directory called `~` under the working directory.
+
+**The remedy.** If a production boot refuses with "neither data key opens this database's stored secrets", find the key those secrets were sealed under (a backup of either file, or the `OS_SECRET_KEY` the deployment once ran with), keep a copy of both files, and put that key at the shared file the message names, or set `OS_SECRET_KEY` to it.
+
+Supersedes, in the same release: the note for #29934 says that when the shared file and the project's file hold different keys, the shared one is used and a warning names both paths. After this change the boot uses whichever of the two opens the stored secrets, and refuses in production when neither does.

packages/cli/src/commands/serve.ts

@@ -5325,13 +5325,17 @@ export default class Serve extends Command {
           // cleartext (by design).
           let secrets: any = undefined;
           try {
-            const { LocalCryptoProvider } = await import(
+            const { deferredLocalCryptoProvider } = await import(
               /* webpackIgnore: true */ '@objectstack/service-settings'
             );
             // First block to touch `sharedCryptoProvider` (still undefined
-            // here), so create it directly; the secret-field wiring below
-            // reuses this instance so every sys_secret shares one key.
-            sharedCryptoProvider = new LocalCryptoProvider();
+            // here); the secret-field wiring below reuses this instance so
+            // every sys_secret shares one key. Built at its FIRST USE, not
+            // here: the database is not open yet, and when two key files hold
+            // different keys the boot chooses between them only once it is
+            // (the settings service's `start()`). The binder first decrypts at
+            // `kernel:ready`, after that choice.
+            sharedCryptoProvider = deferredLocalCryptoProvider();
             secrets = createDatasourceSecretBinder({
               engine: lazySecretEngine,
               cryptoProvider: sharedCryptoProvider,
@@ -5631,6 +5635,11 @@ export default class Serve extends Command {
             // every sys_secret value after a restart. Let that error be loud:
             // secret writes must not proceed under an unstable key.
             sharedCryptoProvider = new LocalCryptoProvider();
+          } else if (typeof sharedCryptoProvider.resolve === 'function') {
+            // The deferred instance the datasource binder shares: build it now
+            // (the key file was chosen during `runtime.start()`), so the same
+            // production refusal stays at boot rather than at the first write.
+            sharedCryptoProvider.resolve();
           }
           dataEngine.setCryptoProvider(sharedCryptoProvider);
           if (isDev) {

packages/cli/src/utils/internal-crypto-key-file-channel.test.ts

@@ -11,15 +11,17 @@
  * boot, one settings secret) is `test/dev-start-shared-data-key.integration.test.ts`.
  */
 
-import { afterEach, beforeEach, describe, expect, it } from 'vitest';
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
 import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
 import { tmpdir } from 'node:os';
 import { join } from 'node:path';
 import { randomBytes } from 'node:crypto';
-import { LocalCryptoProvider } from '@objectstack/service-settings';
+import { LocalCryptoProvider, resolvePersistedKeyFile, settleKeyFileChannel } from '@objectstack/service-settings';
 import { applyCryptoKeyFileChannel } from './internal-crypto-key-file-channel.js';
+import { resolveStartHome } from './start-home.js';
 
 const CHANNEL = 'OS_INTERNAL_CRYPTO_KEY_FILE';
+const CANDIDATE = 'OS_INTERNAL_CRYPTO_KEY_FILE_CANDIDATE';
 const ctx = { scope: 'settings', namespace: 'ai', key: 'openai_api_key' } as const;
 
 let home: string;
@@ -102,16 +104,42 @@ describe('#29934 — the key file an earlier `os start` kept in the project', ()
     expect(existsSync(canonicalFile())).toBe(false);
   });
 
-  it('both files hold different keys: the canonical one is used, and the warning names both', async () => {
+  it('both files hold different keys: the child is handed both, and the notice names both', async () => {
     writeKey(legacyFile());
-    const canonical = writeKey(canonicalFile());
-    const sealed = await new LocalCryptoProvider({ key: canonical }).encrypt('sk-canonical', ctx);
-
+    writeKey(canonicalFile());
     for (const { child, notice } of [await startChild(), await devChild()]) {
       expect(child[CHANNEL]).toBe(canonicalFile());
-      expect(notice).toContain(`${canonicalFile()} (used)`);
-      expect(notice).toContain(`${legacyFile()} (NOT used)`);
-      expect(await new LocalCryptoProvider({ env: child }).decrypt(sealed, ctx)).toBe('sk-canonical');
+      // [#30042] Only the child opens the database, so the parent rules
+      // nothing: the second file travels on the candidate channel.
+      expect(child[CANDIDATE]).toBe(legacyFile());
+      expect(notice).toContain(canonicalFile());
+      expect(notice).toContain(legacyFile());
+    }
+  });
+
+  it.each([
+    ['the case-1 shape (a start-sealed project, a foreign home key)', 'legacy'],
+    ['the dev-then-start shape', 'canonical'],
+  ] as const)('%s: the child reads its secret back with the %s key', async (_shape, sealer) => {
+    const keys = { legacy: writeKey(legacyFile()), canonical: writeKey(canonicalFile()) };
+    const handle = await new LocalCryptoProvider({ key: keys[sealer] }).encrypt('sk-30042', ctx);
+    const stored = async () => ({ namespace: ctx.namespace, key: ctx.key, ciphertext: handle.ciphertext });
+    for (const { child } of [await startChild(), await devChild()]) {
+      const quiet = vi.spyOn(console, 'warn').mockImplementation(() => {});
+      // What the child's settings service does at start(), the database open.
+      const choice = await settleKeyFileChannel({ env: child, readStoredSecret: stored });
+      quiet.mockRestore();
+      expect(choice?.path).toBe(sealer === 'legacy' ? legacyFile() : canonicalFile());
+      expect(child[CANDIDATE]).toBeUndefined();
+      expect(await new LocalCryptoProvider({ env: child }).decrypt(handle, ctx)).toBe('sk-30042');
+    }
+  });
+
+  it('an inherited candidate never survives a one-file answer', async () => {
+    writeKey(canonicalFile());
+    const env = { ...operatorEnv, [CANDIDATE]: '/inherited/elsewhere' };
+    for (const { child } of [await startChild(env), await devChild(env)]) {
+      expect(child[CANDIDATE]).toBeUndefined();
     }
   });
 
@@ -131,10 +159,19 @@ describe('#29934 — an explicit env key keeps winning, unchanged', () => {
     ['OS_DEV_CRYPTO_KEY', randomBytes(32).toString('base64')],
   ])('%s set: no file is named and an inherited channel is removed', async (name, value) => {
     writeKey(legacyFile());
-    const env = { ...operatorEnv, [name]: value, [CHANNEL]: '/inherited/elsewhere' };
+    const env = { ...operatorEnv, [name]: value, [CHANNEL]: '/inherited/elsewhere', [CANDIDATE]: '/inherited/too' };
     for (const { child, notice } of [await startChild(env), await devChild(env)]) {
       expect(child[CHANNEL]).toBeUndefined();
+      expect(child[CANDIDATE]).toBeUndefined();
       expect(notice).toBeUndefined();
     }
   });
 });
+
+describe('#30042 — a leading `~` in OS_HOME names one directory for the key file and for `os start`', () => {
+  it.each(['~/pinned-home', '~'])('OS_HOME=%s', (value) => {
+    const env = { ...operatorEnv, OS_HOME: value };
+    const startHome = resolveStartHome(undefined, { hasProjectConfig: true, cwd: project, env });
+    expect(resolvePersistedKeyFile({ env })?.canonical).toBe(join(startHome, 'dev-crypto-key'));
+  });
+});

packages/cli/src/utils/internal-crypto-key-file-channel.ts

@@ -22,8 +22,12 @@ import { resolveStartHome } from './start-home.js';
  * `resolvePersistedKeyFile` over the env the child sees, with `OS_HOME` as dev
  * reads it (`--home` counts as `OS_HOME`, being start's spelling of it). The
  * file `os start` used before — `<start home>/dev-crypto-key` — is the legacy
- * file: read when it alone holds a key, named in a warning when it and the
- * canonical file both hold different keys, never chosen silently.
+ * file: read when it alone holds a key, and never chosen silently. When it and
+ * the canonical file hold different keys, the parent cannot rule between them:
+ * it names the canonical file on the channel and the legacy file on the
+ * second channel `OS_INTERNAL_CRYPTO_KEY_FILE_CANDIDATE`, and the child, which
+ * opens the database, takes the file whose key opens a stored secret (or
+ * refuses in production when neither does; `settleKeyFileChannel`).
  *
  * An env key (`OS_SECRET_KEY`, a usable `OS_DEV_CRYPTO_KEY`) still answers
  * first, unchanged: no file is named and the channel is removed. The parent
@@ -46,7 +50,9 @@ export async function applyCryptoKeyFileChannel(
     homeFlag?: string;
   },
 ): Promise<string | undefined> {
-  const { resolvePersistedKeyFile, KEY_FILE_CHANNEL_ENV } = await import('@objectstack/service-settings');
+  const { resolvePersistedKeyFile, KEY_FILE_CHANNEL_ENV, KEY_FILE_CANDIDATE_CHANNEL_ENV } = await import(
+    '@objectstack/service-settings'
+  );
   const startHome = resolveStartHome(opts.homeFlag, {
     hasProjectConfig: opts.hasProjectConfig,
     cwd: opts.cwd,
@@ -57,6 +63,9 @@ export async function applyCryptoKeyFileChannel(
     env: keyEnv,
     legacyKeyFile: path.join(startHome, 'dev-crypto-key'),
   });
+  // Both channels are the parent's: an inherited copy never speaks for a file
+  // this run did not choose.
+  delete childEnv[KEY_FILE_CANDIDATE_CHANNEL_ENV];
   if (!answer) {
     delete childEnv[KEY_FILE_CHANNEL_ENV];
     return undefined;
@@ -68,11 +77,15 @@ export async function applyCryptoKeyFileChannel(
       `\`os dev\` and \`os start\` now share ${answer.canonical}; move the file there to converge.`
     );
   }
-  if (answer.kind === 'both-differ') {
+  if (answer.kind === 'both-differ' && answer.legacy) {
+    // Only the child opens the database, so only the child can tell which key
+    // opens its stored secrets: it rules between the two files at boot
+    // (`settleKeyFileChannel` in the settings service), names the one it
+    // took, and refuses in production when neither opens them.
+    childEnv[KEY_FILE_CANDIDATE_CHANNEL_ENV] = answer.legacy;
     return (
-      `Data key: two different key files exist, ${answer.canonical} (used) and ${answer.legacy} (NOT used). ` +
-      `Secrets saved under the unused key will not decrypt. If this project's secrets were saved under ` +
-      `\`os start\`, keep a copy of both files and move ${answer.legacy} over ${answer.canonical}.`
+      `Data key: two different key files exist, ${answer.canonical} and ${answer.legacy}. ` +
+      `The server uses the one that opens this project's stored secrets, and says which.`
     );
   }
   return undefined;

packages/cli/src/utils/one-shot-settings.pin.test.ts

@@ -34,12 +34,13 @@
  * `schema-migrate.one-shot-family.integration.test.ts`.
  */
 
-import { describe, it, expect, beforeEach, afterEach } from 'vitest';
-import { existsSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
+import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
+import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
+import { randomBytes } from 'node:crypto';
 import { tmpdir } from 'node:os';
 import { join, relative, resolve } from 'node:path';
 import { fileURLToPath } from 'node:url';
-import { LocalCryptoProvider } from '@objectstack/service-settings';
+import { LocalCryptoProvider, settleKeyFileChannel } from '@objectstack/service-settings';
 // The one code/prose separator, typed by the hand-written `.d.mts` beside it.
 import { maskComments } from '../../../../scripts/js-comment-mask.mjs';
 import { oneShotSettingsPlugin, refusingCryptoProvider, resolveExistingDataKey } from './one-shot-settings.js';
@@ -104,6 +105,7 @@ describe('[#21471] the settings service is composed in one place in this package
 const KEY_ENV = [
   'NODE_ENV', 'OS_SECRET_KEY', 'OS_DEV_CRYPTO_KEY', 'OBJECTSTACK_DEV_CRYPTO_KEY',
   'OS_HOME', 'OBJECTSTACK_HOME', 'OS_CRYPTO_AUTOKEY',
+  'OS_INTERNAL_CRYPTO_KEY_FILE', 'OS_INTERNAL_CRYPTO_KEY_FILE_CANDIDATE',
 ] as const;
 const KEY_HEX = '606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f';
 const savedEnv: Record<string, string | undefined> = {};
@@ -202,3 +204,70 @@ describe('[#21471] the one-shot helper never mints a data key', () => {
     await expect(provider.keyedDigest('x')).rejects.toThrow(/the reason/);
   });
 });
+
+// ── [#30042] The file a one-shot reads is the one `os dev` / `os start` read ──
+
+describe('[#30042] one-shot commands take the same key file, by the same resolution and trial', () => {
+  let project: string;
+  let userHome: string;
+  beforeEach(() => {
+    project = mkdtempSync(join(tmpdir(), 'os-30042-project-'));
+    userHome = mkdtempSync(join(tmpdir(), 'os-30042-user-'));
+    writeFileSync(join(project, 'objectstack.config.ts'), 'export default {};\n');
+  });
+  afterEach(() => {
+    rmSync(project, { recursive: true, force: true });
+    rmSync(userHome, { recursive: true, force: true });
+    vi.restoreAllMocks();
+  });
+
+  const canonicalFile = () => join(userHome, '.objectstack', 'dev-crypto-key');
+  const legacyFile = () => join(project, '.objectstack', 'dev-crypto-key');
+  const writeKey = (file: string): Buffer => {
+    const key = randomBytes(32);
+    mkdirSync(join(file, '..'), { recursive: true });
+    writeFileSync(file, key.toString('base64'), { mode: 0o600 });
+    return key;
+  };
+  const ctx = { scope: 'settings', namespace: 'ai', key: 'openai_api_key' } as const;
+  /** A one-shot's own environment: no key and no home pin, `HOME` is this test's. */
+  const oneShotEnv = (): NodeJS.ProcessEnv => ({ HOME: userHome, NODE_ENV: 'production' });
+
+  it('only the project file an earlier `os start` kept holds a key: the one-shot reads it', async () => {
+    const legacy = writeKey(legacyFile());
+    const handle = await new LocalCryptoProvider({ key: legacy }).encrypt('sk-legacy', ctx);
+    vi.spyOn(console, 'warn').mockImplementation(() => {});
+    const env = oneShotEnv();
+    const key = await resolveExistingDataKey({ cwd: project, env });
+    expect(await key.provider!.decrypt(handle, ctx)).toBe('sk-legacy');
+    expect(existsSync(canonicalFile())).toBe(false);
+  });
+
+  it('two different keys: the provider is the one the boot\'s trial chose', async () => {
+    writeKey(canonicalFile());
+    const legacy = writeKey(legacyFile());
+    const handle = await new LocalCryptoProvider({ key: legacy }).encrypt('sk-start', ctx);
+    vi.spyOn(console, 'warn').mockImplementation(() => {});
+    const env = oneShotEnv();
+    const key = await resolveExistingDataKey({ cwd: project, env });
+    expect(key.unavailable).toBeNull();
+    expect(env.OS_INTERNAL_CRYPTO_KEY_FILE_CANDIDATE).toBe(legacyFile());
+    // What the boot's settings service does at start(), the database open.
+    await settleKeyFileChannel({
+      env,
+      readStoredSecret: async () => ({ namespace: ctx.namespace, key: ctx.key, ciphertext: handle.ciphertext }),
+    });
+    expect(await key.provider!.decrypt(handle, ctx)).toBe('sk-start');
+    expect(key.provider!.keySource).toBe('file');
+  });
+
+  it('two different keys with no trial yet: nothing has been read, and nothing is minted', async () => {
+    writeKey(canonicalFile());
+    writeKey(legacyFile());
+    const env = oneShotEnv();
+    const before = readdirSync(join(userHome, '.objectstack'));
+    const key = await resolveExistingDataKey({ cwd: project, env });
+    expect(key.provider).not.toBeNull();
+    expect(readdirSync(join(userHome, '.objectstack'))).toEqual(before);
+  });
+});