diff --git a/.dockerignore b/.dockerignore index 8dee180..03579ac 100644 --- a/.dockerignore +++ b/.dockerignore @@ -7,6 +7,8 @@ apps/server/data tray *.log .env +# Mounted at run time by docker-compose.yml, never baked into the image. +sakuya.config.json run.sh run.bat run-hoyovault-upload.sh diff --git a/.env.example b/.env.example deleted file mode 100644 index 52d3cd2..0000000 --- a/.env.example +++ /dev/null @@ -1,70 +0,0 @@ -# Copy to .env to use. Bun loads .env automatically, so there is nothing to install -# and no flag to pass — `./run.sh`, `bun dev` and `bun src/index.ts` all pick it up. -# .env is gitignored; this file is the committed reference. - -# --- Server ------------------------------------------------------------------ - -# PORT=3777 - -# Interface to bind. Loopback by default so a fresh install isn't exposed to the -# network. Set 0.0.0.0 to reach it from other machines — preferably with auth on. -# SAKUYA_HOST=127.0.0.1 - -# Where the DB, thumbnails, uploads and models live. Unset, it is autodetected: -# ~/.sakuya if that exists, else apps/server/data. Setting it disables the -# in-app "move data dir" button in Settings > System. -# SAKUYA_DATA_DIR=/mnt/data/apps/sakuya/apps/server/data - -# Mark the auth cookie Secure. Only with HTTPS in front — a Secure cookie is -# silently dropped over plain HTTP, which is a supported LAN setup. -# SAKUYA_HTTPS=true - -# AUTH_ENABLED=true -# AUTH_SECRET=change-me - -# --- Resource limits ---------------------------------------------------------- -# -# The non-Docker counterpart to `cpus` and `mem_limit` in docker-compose.yml. -# Unset, every value below keeps the behaviour these had when they were hardcoded, -# so adding a .env with none of them set changes nothing. -# -# Read docs/SETUP.md#resource-limits before relying on the memory one: CPU can be -# capped properly from inside the process, memory cannot. - -# CPU budget, in cores. Fractions allowed, same as Docker's `cpus`. Derives job -# concurrency and the ffmpeg / onnxruntime / libvips thread pools in one go. -# The default deployment runs at 2. -# SAKUYA_MAX_CPUS=2 - -# Memory ceiling: 2g, 512m, 1.5G, or a plain byte count. -# -# NOT enforced by the app itself — nothing a process sets on itself can be, and -# most of the memory here is outside the JS heap (libvips buffers, onnxruntime -# arenas, ffmpeg children). It is enforced when you start via ./run.sh on a -# systemd host, which puts the process in a cgroup with a real MemoryMax. The -# boot log states which of the two you actually got. -# SAKUYA_MAX_MEMORY=2g - -# Set false to keep the in-app limits above but never wrap the process in a -# systemd scope. -# SAKUYA_HARD_LIMIT=false - -# --- Per-subsystem overrides -------------------------------------------------- -# Each one wins over the value SAKUYA_MAX_CPUS would derive. Reach for these when -# one subsystem is the problem, rather than lowering the whole budget. - -# Background jobs (scan / tag / thumbnail / transcode) allowed to run at once. -# SAKUYA_JOB_CONCURRENCY=2 - -# ffmpeg -threads while transcoding. Thumbnail extraction is always 1 thread: -# decoding a single frame gains nothing from more. -# SAKUYA_FFMPEG_THREADS=1 - -# onnxruntime threads within one operator, while AI tagging. The tagger's -# dominant cost, and the first knob to lower if tagging starves everything else. -# SAKUYA_ONNX_THREADS=2 - -# libvips worker pool for sharp (thumbnails, tagger preprocessing). Can only -# lower it: sharp already defaults to 1 on glibc Linux, and a limit that raised -# the pool would not be a limit. -# SAKUYA_SHARP_CONCURRENCY=1 diff --git a/.gitignore b/.gitignore index f17cc14..877820f 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,9 @@ yarn-debug.log* yarn-error.log* *.log .env +# Local configuration and Docker customisation; created on first run / by you. +sakuya.config.json +docker-compose.override.yml .vscode/* /tmp /out-tsc diff --git a/Dockerfile b/Dockerfile index 400466b..02606c0 100644 --- a/Dockerfile +++ b/Dockerfile @@ -24,14 +24,21 @@ COPY apps/server apps/server COPY packages/shared packages/shared WORKDIR /app/apps/server ENV NODE_ENV=production -# The server binds loopback by default. Inside a container that makes the published port -# unreachable, so bind all interfaces here — the container boundary is the isolation. -ENV SAKUYA_HOST=0.0.0.0 +# Settings come from sakuya.config.json, mounted at /app/sakuya.config.json by docker-compose.yml. +# SAKUYA_DOCKER is what lets SAKUYA_* environment variables override it — only inside this image. +# The three pinned here are container plumbing the file can't know about: +# - host: loopback would make the published port unreachable; the container boundary is the isolation +# - port: docker/nginx.conf proxies to server:3777 +# - data dir: wherever the host keeps it, it is mounted at /data +ENV SAKUYA_DOCKER=1 \ + SAKUYA_HOST=0.0.0.0 \ + SAKUYA_PORT=3777 \ + SAKUYA_DATA_DIR=/data EXPOSE 3777 -# /api/health is registered before requireAuth in src/index.ts, so this keeps working -# with AUTH_ENABLED=true. Shell form on purpose: $PORT is expanded at run time, not build time. +# /api/health is registered before requireAuth in src/index.ts, so this keeps working with auth on. +# Shell form on purpose: $SAKUYA_PORT is expanded at run time, not build time. HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \ - CMD bun -e "const r = await fetch('http://127.0.0.1:' + (process.env.PORT ?? 3777) + '/api/health'); process.exit(r.ok ? 0 : 1)" + CMD bun -e "const r = await fetch('http://127.0.0.1:' + (process.env.SAKUYA_PORT ?? 3777) + '/api/health'); process.exit(r.ok ? 0 : 1)" CMD ["bun", "src/index.ts"] diff --git a/README.md b/README.md index bbe9b60..0eab341 100644 --- a/README.md +++ b/README.md @@ -82,10 +82,15 @@ Open `http://localhost:5173`. No configuration required to start — everything `apps/server/data/` by default, or in `~/.sakuya/` if that folder exists (`./setup.sh` offers to create it, and Settings > System can move an existing library there). +To change ports, open Sakuya to your network, turn on the login or cap CPU and memory, edit +**`sakuya.config.json`** in the repository root — one file for the server, the web app and Docker, +created with the defaults on first run. [`sakuya.config.example.json`](sakuya.config.example.json) +lists every setting. + Prefer scripts? `./setup.sh` installs Bun for you if it's missing, `./run.sh` starts dev, and `./update.sh` pulls + reinstalls. `.bat` equivalents exist for Windows. -See **[docs/SETUP.md](docs/SETUP.md)** for the full workspace layout, environment variables, +See **[docs/SETUP.md](docs/SETUP.md)** for the full workspace layout, every configuration setting, and which folders you should never touch by hand (looking at you, `apps/server/data/`). ## API diff --git a/apps/server/.env.example b/apps/server/.env.example deleted file mode 100644 index 2d397b1..0000000 --- a/apps/server/.env.example +++ /dev/null @@ -1,12 +0,0 @@ -# Password-lock the site. Leave AUTH_ENABLED unset/false to keep it open. -AUTH_ENABLED=false -AUTH_SECRET= - -# Interface to bind. Defaults to 127.0.0.1 (this machine only). Set to 0.0.0.0 to serve other -# machines on your network — do that together with AUTH_ENABLED=true, since the server exposes -# your whole library and can open a file manager on the host. Docker sets this for you. -# SAKUYA_HOST=0.0.0.0 - -# Set to true when serving over HTTPS, so the auth cookie is marked Secure. Left off by default -# because a Secure cookie is dropped over plain HTTP, which is a supported LAN setup. -# SAKUYA_HTTPS=true diff --git a/apps/server/package.json b/apps/server/package.json index 92e6f4f..379ddb7 100644 --- a/apps/server/package.json +++ b/apps/server/package.json @@ -4,8 +4,8 @@ "private": true, "type": "module", "scripts": { - "dev": "bun --watch src/index.ts", - "start": "bun src/index.ts", + "dev": "bun src/launch.ts --watch", + "start": "bun src/launch.ts", "typecheck": "tsc --noEmit" }, "dependencies": { diff --git a/apps/server/src/auth.test.ts b/apps/server/src/auth.test.ts index e307453..eee767f 100644 --- a/apps/server/src/auth.test.ts +++ b/apps/server/src/auth.test.ts @@ -1,19 +1,14 @@ import { describe, expect, test } from 'bun:test'; import crypto from 'node:crypto'; -import fs from 'node:fs'; -import os from 'node:os'; -import path from 'node:path'; - -process.env.SAKUYA_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'sakuya-auth-')); -process.env.AUTH_ENABLED = 'true'; -process.env.AUTH_SECRET = 'correct horse battery staple'; -process.env.PORT = '38778'; +import { useTestConfig } from './testConfig'; + +const SECRET = 'correct horse battery staple'; +useTestConfig('auth', { port: 38778, auth: { enabled: true, secret: SECRET } }); const BASE = 'http://localhost:38778'; const { issueToken, verifyToken } = await import('./lib/auth'); await import('./index'); -const SECRET = process.env.AUTH_SECRET; const DAY = 86_400_000; describe('session tokens', () => { diff --git a/apps/server/src/boards.test.ts b/apps/server/src/boards.test.ts index ffd0d4d..fcb10a7 100644 --- a/apps/server/src/boards.test.ts +++ b/apps/server/src/boards.test.ts @@ -1,12 +1,9 @@ import { expect, test } from 'bun:test'; -import fs from 'node:fs'; -import os from 'node:os'; -import path from 'node:path'; import { eq } from 'drizzle-orm'; +import { useTestConfig } from './testConfig'; // Point the server at a throwaway data dir before anything opens the real database. -process.env.SAKUYA_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'sakuya-boards-')); -process.env.PORT = '38771'; +useTestConfig('boards', { port: 38771 }); const BASE = 'http://localhost:38771'; const { db, sqlite, schema } = await import('./db'); diff --git a/apps/server/src/cleanup.test.ts b/apps/server/src/cleanup.test.ts index 4ebb3b7..35497c5 100644 --- a/apps/server/src/cleanup.test.ts +++ b/apps/server/src/cleanup.test.ts @@ -1,12 +1,12 @@ import { expect, test } from 'bun:test'; import fs from 'node:fs'; -import os from 'node:os'; import path from 'node:path'; import { eq } from 'drizzle-orm'; +import { useTestConfig } from './testConfig'; // Throwaway data dir before anything opens the real database. No server is started: performCleanup // is called directly so the retention cutoff can be driven from the test. -process.env.SAKUYA_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'sakuya-cleanup-')); +useTestConfig('cleanup'); const { db, sqlite, schema } = await import('./db'); const { THUMBS_DIR } = await import('./lib/config'); diff --git a/apps/server/src/config.test.ts b/apps/server/src/config.test.ts new file mode 100644 index 0000000..c2f19ff --- /dev/null +++ b/apps/server/src/config.test.ts @@ -0,0 +1,194 @@ +import { describe, expect, test } from 'bun:test'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { ConfigError, DEFAULT_CONFIG, REPO_ROOT, loadConfig, parseMemory } from '@sakuya/shared/config'; + +/** + * loadConfig takes its environment as an argument and SAKUYA_CONFIG points it at a throwaway file, + * so none of this touches the real sakuya.config.json or process.env. + */ +function scratch(files: Record = {}): { dir: string; env: Record } { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sakuya-config-')); + for (const [name, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(dir, name)), { recursive: true }); + fs.writeFileSync(path.join(dir, name), content); + } + return { dir, env: { SAKUYA_CONFIG: path.join(dir, 'sakuya.config.json') } }; +} + +const withFile = (config: unknown) => scratch({ 'sakuya.config.json': JSON.stringify(config) }); + +describe('loading', () => { + test('a missing file reads as the defaults, and is only created when asked', () => { + const { dir, env } = scratch(); + expect(loadConfig({ env }).config).toEqual(DEFAULT_CONFIG); + expect(fs.existsSync(path.join(dir, 'sakuya.config.json'))).toBe(false); + + const created = loadConfig({ env, create: true }); + expect(created.created).toContain('with the defaults'); + const written = JSON.parse(fs.readFileSync(path.join(dir, 'sakuya.config.json'), 'utf8')); + // Every setting is spelled out, so the file doubles as the list of what can be changed. + expect(written.server).toEqual(DEFAULT_CONFIG.server); + expect(written.limits).toEqual({ cpus: null, memory: null }); + + // Second start: nothing new to announce. + expect(loadConfig({ env, create: true }).created).toBeNull(); + }); + + /** The committed example is the reference for every setting; it must not drift from a real first run. */ + test('sakuya.config.example.json is exactly what a first run writes', () => { + const { dir, env } = scratch(); + loadConfig({ env, create: true }); + expect(fs.readFileSync(path.join(dir, 'sakuya.config.json'), 'utf8')).toBe( + fs.readFileSync(path.join(REPO_ROOT, 'sakuya.config.example.json'), 'utf8'), + ); + }); + + test('a partial file keeps the defaults for everything it leaves out', () => { + const { env } = withFile({ server: { port: 4000 }, limits: { memory: '1.5g' } }); + const { config } = loadConfig({ env }); + expect(config.server.port).toBe(4000); + expect(config.server.host).toBe('127.0.0.1'); + expect(config.limits.memory).toBe(Math.floor(1.5 * 1024 ** 3)); + }); + + test('a relative dataDir is relative to the config file, not the working directory', () => { + const { dir, env } = withFile({ server: { dataDir: 'my-data' } }); + expect(loadConfig({ env }).config.server.dataDir).toBe(path.join(dir, 'my-data')); + }); + + /** One file is the only knob, so a typo must not quietly revert to a default. */ + test('malformed values are errors that name the setting', () => { + const cases: [unknown, string][] = [ + [{ server: { port: 'eighty' } }, 'server.port'], + [{ server: { port: 70000 } }, 'server.port'], + [{ auth: { enabled: 'yes' } }, 'auth.enabled'], + [{ limits: { cpus: 0 } }, 'limits.cpus'], + [{ limits: { memory: 'lots' } }, 'limits.memory'], + [{ server: 'nope' }, 'server'], + ]; + for (const [config, key] of cases) { + const { env } = withFile(config); + expect(() => loadConfig({ env })).toThrow(key); + } + }); + + test('invalid JSON is a ConfigError naming the file', () => { + const { env } = scratch({ 'sakuya.config.json': '{ "server": { "port": 3777, } }' }); + expect(() => loadConfig({ env })).toThrow(ConfigError); + expect(() => loadConfig({ env })).toThrow('sakuya.config.json'); + }); + + test('unknown keys warn instead of failing', () => { + const { env } = withFile({ $schema: 'x', server: { prot: 1 }, extra: true }); + const { warnings } = loadConfig({ env }); + expect(warnings.some((w) => w.includes('"server.prot"'))).toBe(true); + expect(warnings.some((w) => w.includes('"extra"'))).toBe(true); + expect(warnings.some((w) => w.includes('$schema'))).toBe(false); + }); +}); + +describe('environment variables', () => { + test('outside Docker they are ignored, and said to be', () => { + const { env } = withFile({ server: { port: 4000 } }); + const loaded = loadConfig({ env: { ...env, SAKUYA_PORT: '5000', PORT: '6000', SAKUYA_MAX_CPUS: '1' } }); + expect(loaded.config.server.port).toBe(4000); + expect(loaded.config.limits.cpus).toBeNull(); + expect(loaded.warnings.join()).toContain('SAKUYA_PORT'); + expect(loaded.warnings.join()).toContain('PORT'); + }); + + test('inside the Docker image they override the file', () => { + const { env } = withFile({ + server: { port: 4000, host: '127.0.0.1', dataDir: '/home/me/.sakuya' }, + auth: { secret: 'pw' }, + }); + const { config, warnings } = loadConfig({ + env: { + ...env, + SAKUYA_DOCKER: '1', + SAKUYA_PORT: '3777', + SAKUYA_HOST: '0.0.0.0', + SAKUYA_DATA_DIR: '/data', + SAKUYA_AUTH_ENABLED: 'true', + }, + }); + expect(config.server).toEqual({ port: 3777, host: '0.0.0.0', dataDir: '/data', https: false }); + expect(config.auth).toEqual({ enabled: true, secret: 'pw' }); + expect(warnings).toEqual([]); + }); + + test('a bad override gets the same validation as the file', () => { + const { env } = withFile({}); + expect(() => loadConfig({ env: { ...env, SAKUYA_DOCKER: '1', SAKUYA_MAX_MEMORY: 'lots' } })).toThrow( + 'limits.memory', + ); + }); + + test('the Docker image never writes a config file', () => { + const { dir, env } = scratch(); + loadConfig({ env: { ...env, SAKUYA_DOCKER: '1' }, create: true }); + expect(fs.existsSync(path.join(dir, 'sakuya.config.json'))).toBe(false); + }); + + /** + * The one ignored variable that can't just be a warning: a login switched on the old way must not + * upgrade into a server that is silently open. + */ + test('an old AUTH_ENABLED=true with auth off in the file refuses to start', () => { + const { env } = withFile({}); + expect(() => loadConfig({ env: { ...env, AUTH_ENABLED: 'true' } })).toThrow('auth.enabled'); + expect(() => loadConfig({ env: { ...env, SAKUYA_DOCKER: '1', AUTH_ENABLED: 'true' } })).toThrow( + 'SAKUYA_AUTH_ENABLED', + ); + // Agreeing is fine. + const agreed = withFile({ auth: { enabled: true, secret: 'pw' } }); + expect(loadConfig({ env: { ...agreed.env, AUTH_ENABLED: 'true' } }).config.auth.enabled).toBe(true); + }); +}); + +describe('migrating from .env', () => { + test('a first start carries old .env settings into the new file', () => { + const { dir, env } = scratch({ + '.env': 'PORT=4000\nSAKUYA_MAX_CPUS=2\n', + // apps/server/.env is the one Bun actually loaded, so it wins over the root one. + 'apps/server/.env': 'AUTH_ENABLED=true\nAUTH_SECRET="hunter2"\nPORT=4100\n', + }); + const loaded = loadConfig({ env, create: true }); + expect(loaded.created).toContain('.env'); + expect(loaded.config.server.port).toBe(4100); + expect(loaded.config.auth).toEqual({ enabled: true, secret: 'hunter2' }); + expect(loaded.config.limits.cpus).toBe(2); + const written = JSON.parse(fs.readFileSync(path.join(dir, 'sakuya.config.json'), 'utf8')); + expect(written.auth).toEqual({ enabled: true, secret: 'hunter2' }); + }); + + test('an existing file is never rewritten from .env', () => { + const { env } = scratch({ 'sakuya.config.json': '{}', '.env': 'PORT=4000\n' }); + const loaded = loadConfig({ env, create: true }); + expect(loaded.created).toBeNull(); + expect(loaded.config.server.port).toBe(3777); + }); +}); + +describe('parseMemory', () => { + test("accepts the suffixes Docker's mem_limit accepts", () => { + expect(parseMemory('2g')).toBe(2 * 1024 ** 3); + expect(parseMemory('2G')).toBe(2 * 1024 ** 3); + expect(parseMemory('2gb')).toBe(2 * 1024 ** 3); + expect(parseMemory('512m')).toBe(512 * 1024 ** 2); + expect(parseMemory('1.5G')).toBe(Math.floor(1.5 * 1024 ** 3)); + expect(parseMemory('1024')).toBe(1024); + expect(parseMemory(1024)).toBe(1024); + }); + + test('rejects junk', () => { + expect(parseMemory(undefined)).toBeNull(); + expect(parseMemory('')).toBeNull(); + expect(parseMemory('lots')).toBeNull(); + expect(parseMemory('2x')).toBeNull(); + expect(parseMemory('-2g')).toBeNull(); + expect(parseMemory(0)).toBeNull(); + }); +}); diff --git a/apps/server/src/discover.test.ts b/apps/server/src/discover.test.ts index e0f52b1..efdc1f6 100644 --- a/apps/server/src/discover.test.ts +++ b/apps/server/src/discover.test.ts @@ -1,12 +1,9 @@ import { expect, test } from 'bun:test'; -import fs from 'node:fs'; -import os from 'node:os'; -import path from 'node:path'; import { eq } from 'drizzle-orm'; +import { useTestConfig } from './testConfig'; // Point the server at a throwaway data dir before anything opens the real database. -process.env.SAKUYA_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'sakuya-discover-')); -process.env.PORT = '38772'; +useTestConfig('discover', { port: 38772 }); const BASE = 'http://localhost:38772'; const { db, schema } = await import('./db'); diff --git a/apps/server/src/index.ts b/apps/server/src/index.ts index 5d68e6a..bbd4131 100644 --- a/apps/server/src/index.ts +++ b/apps/server/src/index.ts @@ -1,7 +1,7 @@ import express from 'express'; import sharp from 'sharp'; -import { PORT, HOST, DATA_DIR } from './lib/config'; -import { LIMITS, describeLimits } from './lib/limits'; +import { PORT, HOST, DATA_DIR, CONFIG_FILE, LIMITS, LIMITS_REPORT } from './lib/config'; +import { describeLimits } from './lib/limits'; import './db'; import { librariesRouter } from './routes/libraries'; import { mediaRouter } from './routes/media'; @@ -58,9 +58,10 @@ app.use((err: any, _req: express.Request, res: express.Response, _next: express. app.listen(PORT, HOST, () => { console.log(`Sakuya server listening on http://${HOST === '0.0.0.0' ? 'localhost' : HOST}:${PORT}`); - if (HOST === '127.0.0.1') console.log('Bound to loopback only. Set SAKUYA_HOST=0.0.0.0 to serve other machines.'); + if (HOST === '127.0.0.1') console.log('Bound to loopback only. Set server.host to 0.0.0.0 to serve other machines.'); + console.log(`Config: ${CONFIG_FILE}`); console.log(`Data dir: ${DATA_DIR}`); - const limits = describeLimits(); + const limits = describeLimits(LIMITS_REPORT); if (limits) console.log(limits); initScheduler(); }); diff --git a/apps/server/src/launch.ts b/apps/server/src/launch.ts new file mode 100644 index 0000000..7561afc --- /dev/null +++ b/apps/server/src/launch.ts @@ -0,0 +1,115 @@ +/** + * Starts the server under the hard resource limits from sakuya.config.json. + * + * bun src/launch.ts production-style start + * bun src/launch.ts --watch dev (what `bun dev` and run.sh / run.bat use) + * bun src/launch.ts --init just create sakuya.config.json, then exit (setup scripts) + * + * lib/limits.ts keeps the app *asking* for no more than the budget; this makes the kernel hold it + * to that, the way Docker's cpus / mem_limit do. A process can't cgroup itself, so the ceiling has + * to be in place before the server starts — hence a launcher, wrapping only the server and not + * the Vite dev server next to it: + * + * Linux a transient systemd scope (MemoryMax, CPUQuota) — the same kernel mechanism Docker uses + * Windows a job object the launcher joins before spawning, so the server and everything it + * spawns inherits it (lib/windowsJob.ts) + * macOS nothing equivalent; in-app limits only + * Docker not used — the image runs src/index.ts directly under compose's cpus / mem_limit + * + * Everything fails soft to in-app limits: a limit that can't be applied should be reported, not be + * the reason the server won't start. The server's own boot line then says what actually binds. + */ +import os from 'node:os'; +import path from 'node:path'; +import { ConfigError, formatBytes, loadConfig, type LoadedConfig } from '@sakuya/shared/config'; +import { cpuLabel } from './lib/limits'; +import { joinLimitedJob } from './lib/windowsJob'; + +const serverRoot = path.resolve(import.meta.dir, '..'); + +let loaded: LoadedConfig; +try { + loaded = loadConfig({ create: true }); +} catch (err) { + // A typo in the file is the common case; the message names the file and the key, a stack doesn't help. + if (!(err instanceof ConfigError)) throw err; + console.error(`[sakuya] ${err.message}`); + process.exit(1); +} +if (loaded.created) console.log(`[sakuya] ${loaded.created}`); +if (process.argv.includes('--init')) { + if (!loaded.created) console.log(`[sakuya] Using ${loaded.file}`); + process.exit(0); +} + +const { cpus, memory } = loaded.config.limits; +const wanted = [memory !== null && `memory ${formatBytes(memory)}`, cpus !== null && cpuLabel(cpus)].filter(Boolean); + +/** + * Probe each property before committing to it. The cpu controller is commonly not delegated to user + * slices even where memory is, and one refused property shouldn't cost the other — so try both, + * then each alone. The probe is the only way to find out without masking the server's own exit code. + */ +function systemdScope(): { prefix: string[]; applied: string[] } | null { + if (!Bun.which('systemd-run')) return null; + const base = ['systemd-run', '--user', '--scope', '--collect', '--quiet']; + const props: [string, string][] = []; + if (memory !== null) props.push([`MemoryMax=${memory}`, `memory ${formatBytes(memory)}`]); + // systemd counts a core as 100%, so 1.5 CPUs is CPUQuota=150%. + if (cpus !== null) props.push([`CPUQuota=${Math.round(cpus * 100)}%`, cpuLabel(cpus)]); + const attempts = props.length > 1 ? [props, ...props.map((p) => [p])] : [props]; + for (const attempt of attempts) { + const args = [...base, ...attempt.flatMap(([prop]) => ['-p', prop])]; + if (Bun.spawnSync([...args, 'true'], { stdout: 'ignore', stderr: 'ignore' }).exitCode === 0) { + return { prefix: args, applied: attempt.map(([, label]) => label) }; + } + } + return null; +} + +let prefix: string[] = []; +if (wanted.length && !loaded.docker) { + if (process.platform === 'linux') { + const scope = systemdScope(); + if (scope) { + prefix = scope.prefix; + const missed = wanted.filter((w) => !scope.applied.includes(w as string)); + console.log( + `[sakuya] Hard limits via systemd: ${scope.applied.join(', ')}` + + (missed.length ? ` (refused: ${missed.join(', ')} — that controller isn't delegated to your user)` : ''), + ); + } else { + console.log( + `[sakuya] Could not apply ${wanted.join(', ')} as a hard limit (needs systemd-run with the controller ` + + 'delegated to your user). In-app limits still apply.', + ); + } + } else if (process.platform === 'win32') { + const job = joinLimitedJob(memory, cpus, os.cpus().length); + const applied = [job.memory && `memory ${formatBytes(memory!)}`, job.cpu && cpuLabel(cpus!)].filter(Boolean); + console.log( + applied.length + ? `[sakuya] Hard limits via Windows job object: ${applied.join(', ')}` + : `[sakuya] Could not apply ${wanted.join(', ')} as a hard limit${job.error ? ` (${job.error})` : ''}. In-app limits still apply.`, + ); + } else { + console.log(`[sakuya] ${wanted.join(', ')} can't be enforced on ${process.platform}; in-app limits still apply.`); + } +} + +const watch = process.argv.includes('--watch'); +const child = Bun.spawn( + [...prefix, process.execPath, ...(watch ? ['--watch'] : []), path.join(import.meta.dir, 'index.ts')], + { + cwd: serverRoot, + stdio: ['inherit', 'inherit', 'inherit'], + }, +); + +// A terminal's Ctrl+C reaches the child directly; this covers being stopped by a supervisor +// (bun --filter, systemd, a service manager) that only signals us. +for (const signal of ['SIGINT', 'SIGTERM'] as const) { + process.on(signal, () => child.kill(signal)); +} + +process.exit(await child.exited); diff --git a/apps/server/src/lib/config.ts b/apps/server/src/lib/config.ts index 1d0276e..1fc13e2 100644 --- a/apps/server/src/lib/config.ts +++ b/apps/server/src/lib/config.ts @@ -1,19 +1,36 @@ import path from 'node:path'; import fs from 'node:fs'; import os from 'node:os'; +import { loadConfig } from '@sakuya/shared/config'; +import { detectCgroupMemoryLimit, detectCpuQuota, effectiveCpus, resolveLimits, type LimitsReport } from './limits'; +import { currentJobMemoryLimit } from './windowsJob'; const serverRoot = path.resolve(import.meta.dir, '..', '..'); +// Everything below comes from sakuya.config.json (see packages/shared/src/config.ts). Created with +// the defaults on first start, migrating any old .env, so a fresh checkout still needs no setup. +const loaded = loadConfig({ create: true }); +if (loaded.created) console.log(`[sakuya] ${loaded.created}`); +for (const warning of loaded.warnings) console.warn(`[sakuya] ${warning}`); + +export const CONFIG = loaded.config; +export const CONFIG_FILE = loaded.file; +export const IN_DOCKER = loaded.docker; + export const HOME_DATA_DIR = path.join(os.homedir(), '.sakuya'); export const LOCAL_DATA_DIR = path.join(serverRoot, 'data'); +/** server.dataDir is set (or Docker pinned it), so the in-app move is disabled. */ +export const DATA_DIR_PINNED = CONFIG.server.dataDir !== null; + /** - * Where the data lives is not a stored setting — it can't be, since the settings table lives in the - * DB we're trying to find. The folder's existence *is* the state: setup.sh or the Settings > System - * migrate button creates ~/.sakuya, and from then on every start finds it. + * Unless server.dataDir pins it, where the data lives is not a stored setting — it can't be, since + * the settings table lives in the DB we're trying to find. The folder's existence *is* the state: + * setup.sh or the Settings > System migrate button creates ~/.sakuya, and from then on every start + * finds it. */ export function resolveDataDir(): string { - if (process.env.SAKUYA_DATA_DIR) return process.env.SAKUYA_DATA_DIR; + if (CONFIG.server.dataDir) return CONFIG.server.dataDir; return fs.existsSync(HOME_DATA_DIR) ? HOME_DATA_DIR : LOCAL_DATA_DIR; } @@ -29,24 +46,23 @@ export const DOWNLOADER_DIR = path.join(DATA_DIR, 'downloader'); export const DOWNLOADER_BIN_DIR = path.join(DOWNLOADER_DIR, 'bin'); export const DOWNLOADER_COOKIES_DIR = path.join(DOWNLOADER_DIR, 'cookies'); -export const PORT = Number(process.env.PORT ?? 3777); +export const PORT = CONFIG.server.port; /** * Interface to bind. Defaults to loopback: the server exposes the whole library and - * POST /api/media/:id/reveal, which spawns a file manager on the host, and AUTH_ENABLED is off by + * POST /api/media/:id/reveal, which spawns a file manager on the host, and auth is off by * default — none of that should be reachable from the LAN because someone started the server. * - * Containers must bind 0.0.0.0 or published ports never reach the process, so the Dockerfile and - * docker-compose.yml both set SAKUYA_HOST=0.0.0.0. Set it yourself to serve other machines - * directly, ideally with AUTH_ENABLED=true. + * Containers must bind 0.0.0.0 or published ports never reach the process, so the Docker image + * pins it. Set server.host yourself to serve other machines directly, ideally with auth on. */ -export const HOST = process.env.SAKUYA_HOST ?? '127.0.0.1'; +export const HOST = CONFIG.server.host; /** * Marks the auth cookie Secure. Off by default because serving plain HTTP over a LAN is a * supported setup, and a Secure cookie is silently dropped there, which would lock users out. */ -export const AUTH_COOKIE_SECURE = process.env.SAKUYA_HTTPS === 'true'; +export const AUTH_COOKIE_SECURE = CONFIG.server.https; /** * The root package.json is the one source of truth for the app version: Settings > System reads * this, and the bundled release notes in apps/web/src/releases drive the update toast. They used @@ -69,12 +85,24 @@ function readAppVersion(): string { export const APP_VERSION = readAppVersion(); -export const AUTH_ENABLED = process.env.AUTH_ENABLED === 'true'; -export const AUTH_SECRET = process.env.AUTH_SECRET ?? ''; +export const AUTH_ENABLED = CONFIG.auth.enabled; +export const AUTH_SECRET = CONFIG.auth.secret; if (AUTH_ENABLED && !AUTH_SECRET) { - throw new Error('AUTH_SECRET is not set. Set it before enabling AUTH_ENABLED.'); + throw new Error(`auth.enabled is true but auth.secret is empty in ${CONFIG_FILE}. Set a password there.`); } +const cpu = effectiveCpus(CONFIG.limits.cpus, detectCpuQuota()); + +/** Job concurrency and native thread pools, sized from the CPU budget. See lib/limits.ts. */ +export const LIMITS = resolveLimits(cpu?.cpus ?? null); + +export const LIMITS_REPORT: LimitsReport = { + limits: LIMITS, + cpuSource: cpu?.source ?? null, + configuredMemory: CONFIG.limits.memory, + enforcedMemory: process.platform === 'win32' ? currentJobMemoryLimit() : detectCgroupMemoryLimit(), +}; + // Curated WD v3 taggers — all share 448px input + the same selected_tags.csv format, // so they are drop-in compatible with the existing preprocessing/inference code. export interface TaggerModelDef { @@ -101,7 +129,7 @@ export function modelRepoBase(id: string): string { export const IMAGE_EXTS = new Set(['.jpg', '.jpeg', '.png', '.webp', '.gif', '.avif', '.bmp', '.tiff']); export const VIDEO_EXTS = new Set(['.mp4', '.webm', '.mkv', '.mov', '.avi', '.m4v', '.ts', '.wmv']); -if (!process.env.SAKUYA_DATA_DIR && DATA_DIR !== LOCAL_DATA_DIR && fs.existsSync(path.join(LOCAL_DATA_DIR, 'tbge.db'))) { +if (!DATA_DIR_PINNED && DATA_DIR !== LOCAL_DATA_DIR && fs.existsSync(path.join(LOCAL_DATA_DIR, 'tbge.db'))) { console.warn(`[sakuya] using ${DATA_DIR} — the database still sitting in ${LOCAL_DATA_DIR} is ignored`); } diff --git a/apps/server/src/lib/limits.ts b/apps/server/src/lib/limits.ts index b1183e7..71d9524 100644 --- a/apps/server/src/lib/limits.ts +++ b/apps/server/src/lib/limits.ts @@ -1,26 +1,24 @@ /** - * Resource limits for running outside Docker. + * In-app resource limits: how many jobs run at once, and how many threads each native library + * underneath them may spawn. * - * `docker-compose.yml` caps this app with `cpus` and `mem_limit`, which are kernel cgroup limits: - * the kernel itself refuses to schedule past the quota and OOM-kills past the memory bound. A - * process cannot impose that on itself, so what this module does instead is cap the *parallelism - * that drives* the consumption — how many jobs run at once, and how many threads each of the - * native libraries underneath them is allowed to spawn. + * These are the *soft* half. The hard ceiling — the kernel refusing CPU time past a quota and + * allocations past a memory bound — is applied from outside the process: by the launcher + * (src/launch.ts, systemd scope on Linux, job object on Windows) or by Docker's cpus / mem_limit. + * A ceiling alone isn't enough, though: libvips, libx264 and onnxruntime size their thread pools + * from the core count they can see, and inside a quota they'd still spawn one per *host* core and + * then fight over the share they actually get. So the budget drives both. * - * For CPU that lands very close to `cpus: "2.0"`. For memory it is an influence rather than a - * ceiling: peak RSS is roughly concurrency x per-task peak, and most of it lives outside the JS - * heap anyway (libvips buffers, onnxruntime arenas, ffmpeg children), which is exactly why - * SAKUYA_MAX_MEMORY is *not* read here. It is consumed by run.sh, which can hand it to systemd as - * a real MemoryMax. Enforcing it from inside the process would be theatre. - * - * Unset, every value below is what it was when these numbers were hardcoded across the services — - * upgrading without a .env changes nothing. Note that at SAKUYA_MAX_CPUS=2 the derivations - * reproduce those old constants exactly; see limits.test.ts. + * With no budget, every value is what it was when these numbers were hardcoded across the services. + * At 2 CPUs the derivation reproduces those constants exactly; see limits.test.ts. */ +import fs from 'node:fs'; import os from 'node:os'; +import path from 'node:path'; +import { formatBytes } from '@sakuya/shared/config'; export interface Limits { - /** Effective CPU budget, or null when unset (no derivation happens). */ + /** Effective CPU budget, or null when there is none (no derivation happens). */ maxCpus: number | null; /** How many background jobs (scan/tag/thumbnail/transcode) may run at once. */ jobConcurrency: number; @@ -30,7 +28,7 @@ export interface Limits { onnxIntraOpThreads: number; /** onnxruntime threads across independent operators. */ onnxInterOpThreads: number; - /** libvips worker pool for sharp, or null for its default (one thread per core). */ + /** libvips worker pool for sharp, or null for its default. */ sharpConcurrency: number | null; } @@ -44,115 +42,121 @@ const UNCONFIGURED: Limits = { sharpConcurrency: null, }; -const MEMORY_UNITS: Record = { - b: 1, - k: 1024, - kb: 1024, - m: 1024 ** 2, - mb: 1024 ** 2, - g: 1024 ** 3, - gb: 1024 ** 3, - t: 1024 ** 4, - tb: 1024 ** 4, -}; +/** Derive every knob from one CPU budget. Each floors at 1: a budget of 0.5 still has to run one job. */ +export function resolveLimits(maxCpus: number | null): Limits { + if (!maxCpus) return UNCONFIGURED; + const jobConcurrency = Math.max(1, Math.floor(maxCpus)); + return { + maxCpus, + jobConcurrency, + // Split across the jobs that may run concurrently, so N transcodes at N threads each can't add + // up to more than the budget. + ffmpegThreads: Math.max(1, Math.floor(maxCpus / jobConcurrency)), + onnxIntraOpThreads: Math.max(1, Math.floor(maxCpus)), + onnxInterOpThreads: 1, + sharpConcurrency: Math.max(1, Math.floor(maxCpus)), + }; +} + +export type CpuSource = 'config' | 'quota'; + +export const cpuLabel = (cpus: number): string => `${cpus} CPU${cpus === 1 ? '' : 's'}`; /** - * Accepts what Docker accepts — "2g", "512m", "1.5G" — plus a bare byte count. Returns null for - * anything unparseable rather than throwing: a typo in a limit should not stop the server from - * booting, it should fall back to being unlimited and say so. + * The tighter of the configured budget and the quota the process is already running under — a + * Docker `cpus`, a systemd CPUQuota, an affinity mask. Taking the minimum means a container's quota + * sizes the pools without being restated in the config file. */ -export function parseMemory(raw: string | undefined): number | null { - if (!raw) return null; - const match = /^\s*(\d+(?:\.\d+)?)\s*([a-z]*)\s*$/i.exec(raw); - if (!match) return null; - const value = Number(match[1]); - const unit = match[2].toLowerCase(); - const multiplier = unit === '' ? 1 : MEMORY_UNITS[unit]; - if (!multiplier || !Number.isFinite(value) || value <= 0) return null; - return Math.floor(value * multiplier); -} - -/** Positive, finite CPU count. Fractional is allowed ("1.5") to mirror Docker's `cpus`. */ -function parseCpus(raw: string | undefined): number | null { - if (!raw) return null; - const value = Number(raw); - if (!Number.isFinite(value) || value <= 0) return null; - return value; +export function effectiveCpus( + configured: number | null, + quota: number | null, +): { cpus: number; source: CpuSource } | null { + if (configured !== null && (quota === null || configured <= quota)) return { cpus: configured, source: 'config' }; + if (quota !== null) return { cpus: quota, source: 'quota' }; + return null; } -/** A granular override: positive integer, or null to fall through to the derived value. */ -function parseCount(raw: string | undefined): number | null { - if (!raw) return null; - const value = Number(raw); - if (!Number.isInteger(value) || value <= 0) return null; - return value; +/** + * Bun's availableParallelism() honours the cgroup CPU quota and affinity, while os.cpus() lists + * every core on the host; a gap between them is a quota. (Measured: 1 inside a CPUQuota=100% scope + * on a 4-core host.) Windows job-object CPU rates don't show up here, but on Windows the budget + * comes from the config file anyway. + */ +export function detectCpuQuota(): number | null { + const available = os.availableParallelism(); + return available < os.cpus().length ? available : null; } /** - * Resolve the limits from an environment. Pure and env-injected so the derivation can be tested - * without a server or a real .env. + * The tightest cgroup v2 memory.max on the path from this process to the root — whatever actually + * binds, whether set by Docker's mem_limit or by the launcher's systemd scope. Linux only; null + * when nothing is set or the host is on cgroup v1. */ -export function resolveLimits(env: NodeJS.ProcessEnv = process.env): Limits { - const maxCpus = parseCpus(env.SAKUYA_MAX_CPUS); - - // Derive from the CPU budget, or keep the old hardcoded constants when there is no budget. - // Every derived value floors at 1: a budget of "0.5" still has to be able to run one job. - const derived: Limits = maxCpus - ? (() => { - const jobConcurrency = Math.max(1, Math.floor(maxCpus)); - return { - maxCpus, - jobConcurrency, - // Split the budget across the jobs that may run concurrently, so N transcodes at - // N threads each can't add up to more than the budget. - ffmpegThreads: Math.max(1, Math.floor(maxCpus / jobConcurrency)), - onnxIntraOpThreads: Math.max(1, Math.floor(maxCpus)), - onnxInterOpThreads: 1, - sharpConcurrency: Math.max(1, Math.floor(maxCpus)), - }; - })() - : UNCONFIGURED; - - // Granular overrides always win, so one noisy subsystem can be tuned without abandoning the dial. - return { - ...derived, - jobConcurrency: parseCount(env.SAKUYA_JOB_CONCURRENCY) ?? derived.jobConcurrency, - ffmpegThreads: parseCount(env.SAKUYA_FFMPEG_THREADS) ?? derived.ffmpegThreads, - onnxIntraOpThreads: parseCount(env.SAKUYA_ONNX_THREADS) ?? derived.onnxIntraOpThreads, - sharpConcurrency: parseCount(env.SAKUYA_SHARP_CONCURRENCY) ?? derived.sharpConcurrency, - }; +export function detectCgroupMemoryLimit(): number | null { + if (process.platform !== 'linux') return null; + let entry: string | undefined; + try { + entry = fs + .readFileSync('/proc/self/cgroup', 'utf8') + .split('\n') + .find((line) => line.startsWith('0::')); + } catch { + return null; + } + if (!entry) return null; + let tightest: number | null = null; + for (let rel = entry.slice(3).trim() || '/'; ; rel = path.posix.dirname(rel)) { + try { + const value = fs.readFileSync(path.posix.join('/sys/fs/cgroup', rel, 'memory.max'), 'utf8').trim(); + const bytes = Number(value); + if (value !== 'max' && Number.isFinite(bytes) && (tightest === null || bytes < tightest)) tightest = bytes; + } catch { + // Not every level has a memory controller file; keep walking up. + } + if (rel === '/') break; + } + return tightest; } -export const LIMITS: Limits = resolveLimits(); +export interface LimitsReport { + limits: Limits; + cpuSource: CpuSource | null; + /** limits.memory from the config file. */ + configuredMemory: number | null; + /** The ceiling the kernel actually enforces on this process, if one could be found. */ + enforcedMemory: number | null; +} /** - * One line for the boot log, or null when nothing is configured and there is nothing to say. - * - * Worth printing because the memory half of this is easy to misread: SAKUYA_MAX_MEMORY only binds - * if the launcher put the process in a cgroup, so the log states which of the two is actually in - * force rather than letting the variable's presence imply a cap that isn't there. + * One line for the boot log, or null when nothing is configured or detected. It reports what is + * *enforced*, not what was asked for, so a memory limit that isn't binding says so out loud rather + * than being implied by a clean boot. */ -export function describeLimits(env: NodeJS.ProcessEnv = process.env, limits: Limits = LIMITS): string | null { - const configured = - limits.maxCpus !== null || - Boolean(env.SAKUYA_JOB_CONCURRENCY || env.SAKUYA_FFMPEG_THREADS || env.SAKUYA_ONNX_THREADS || env.SAKUYA_SHARP_CONCURRENCY); - const memory = parseMemory(env.SAKUYA_MAX_MEMORY); - if (!configured && memory === null) return null; +export function describeLimits({ limits, cpuSource, configuredMemory, enforcedMemory }: LimitsReport): string | null { + if (limits.maxCpus === null && configuredMemory === null && enforcedMemory === null) return null; - const parts = [ - `${limits.jobConcurrency} concurrent job${limits.jobConcurrency === 1 ? '' : 's'}`, - `ffmpeg ${limits.ffmpegThreads ?? 'auto'}`, - `onnx ${limits.onnxIntraOpThreads}/${limits.onnxInterOpThreads}`, - `libvips ${limits.sharpConcurrency ?? 'auto'}`, - ]; - const cpus = limits.maxCpus === null ? `${os.cpus().length} detected cores` : `${limits.maxCpus} CPUs`; - let line = `Limits: ${cpus} -> ${parts.join(', ')}`; - if (memory !== null) { - // SAKUYA_HARD_LIMIT_APPLIED is exported by run.sh once it has re-exec'd under systemd. - line += - env.SAKUYA_HARD_LIMIT_APPLIED === 'true' - ? `. Memory capped at ${env.SAKUYA_MAX_MEMORY} by the launcher.` - : `. SAKUYA_MAX_MEMORY=${env.SAKUYA_MAX_MEMORY} is NOT enforced: start via run.sh on a systemd host, or use Docker's mem_limit.`; + const parts: string[] = []; + if (limits.maxCpus !== null) { + const knobs = [ + `${limits.jobConcurrency} concurrent job${limits.jobConcurrency === 1 ? '' : 's'}`, + `ffmpeg ${limits.ffmpegThreads ?? 'auto'}`, + `onnx ${limits.onnxIntraOpThreads}/${limits.onnxInterOpThreads}`, + `libvips ${limits.sharpConcurrency ?? 'auto'}`, + ]; + const from = cpuSource === 'quota' ? 'detected quota' : 'config'; + parts.push(`${cpuLabel(limits.maxCpus)} (${from}) -> ${knobs.join(', ')}`); } - return line; + + // The kernel rounds a ceiling *down* to a page, so an enforced limit never reads as larger than asked. + if (configuredMemory !== null && (enforcedMemory === null || enforcedMemory > configuredMemory)) { + const note = enforcedMemory === null ? '' : `; the actual ceiling is ${formatBytes(enforcedMemory)}`; + parts.push( + `memory ${formatBytes(configuredMemory)} is NOT enforced${note}. The launcher applies it (bun dev, ` + + 'run.sh / run.bat, bun run start) on Linux with systemd and on Windows; in Docker, set mem_limit', + ); + } else if (enforcedMemory !== null) { + parts.push(`memory capped at ${formatBytes(enforcedMemory)}`); + } + + return `Limits: ${parts.join('; ')}`; } diff --git a/apps/server/src/lib/windowsJob.ts b/apps/server/src/lib/windowsJob.ts new file mode 100644 index 0000000..90424e5 --- /dev/null +++ b/apps/server/src/lib/windowsJob.ts @@ -0,0 +1,114 @@ +/** + * Hard resource limits on Windows, via a job object — the Windows counterpart to a cgroup. The + * kernel enforces it: past JobMemoryLimit, commits fail; past the CPU rate, threads are descheduled. + * Every process started from inside the job (the server, and the ffmpeg / gallery-dl it spawns) + * lands in it automatically, which is why the launcher joins the job *itself* before spawning + * anything: there is no window in which a child exists outside it. + * + * Struct layouts are for 64-bit Windows (x64 and arm64 share them); Bun ships no 32-bit build. + */ +import { dlopen, FFIType } from 'bun:ffi'; + +const JobObjectExtendedLimitInformation = 9; +const JobObjectCpuRateControlInformation = 15; + +const JOB_OBJECT_LIMIT_JOB_MEMORY = 0x200; +// The launcher holds the only handle, so if it dies the server goes with it instead of lingering +// outside anyone's control. +const JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE = 0x2000; +const JOB_OBJECT_CPU_RATE_CONTROL_ENABLE = 0x1; +const JOB_OBJECT_CPU_RATE_CONTROL_HARD_CAP = 0x4; + +// JOBOBJECT_EXTENDED_LIMIT_INFORMATION: BasicLimitInformation (64 bytes, LimitFlags at 16), +// IoInfo (48), then ProcessMemoryLimit (112), JobMemoryLimit (120), and two peak counters. +const EXTENDED_LIMIT_SIZE = 144; +const LIMIT_FLAGS_OFFSET = 16; +const JOB_MEMORY_LIMIT_OFFSET = 120; + +/** JOBOBJECT_EXTENDED_LIMIT_INFORMATION capping the whole job's committed memory. */ +export function extendedLimitInfo(memoryBytes: number | null): Buffer { + const buf = Buffer.alloc(EXTENDED_LIMIT_SIZE); + let flags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE; + if (memoryBytes !== null) { + flags |= JOB_OBJECT_LIMIT_JOB_MEMORY; + buf.writeBigUInt64LE(BigInt(memoryBytes), JOB_MEMORY_LIMIT_OFFSET); + } + buf.writeUInt32LE(flags, LIMIT_FLAGS_OFFSET); + return buf; +} + +/** + * JOBOBJECT_CPU_RATE_CONTROL_INFORMATION with a hard cap, or null when the budget is the whole + * machine anyway. CpuRate is a share of *all* cores in hundredths of a percent. + */ +export function cpuRateInfo(cpus: number, logicalCores: number): Buffer | null { + if (cpus >= logicalCores) return null; + const rate = Math.min(10_000, Math.max(1, Math.round((cpus / logicalCores) * 10_000))); + const buf = Buffer.alloc(8); + buf.writeUInt32LE(JOB_OBJECT_CPU_RATE_CONTROL_ENABLE | JOB_OBJECT_CPU_RATE_CONTROL_HARD_CAP, 0); + buf.writeUInt32LE(rate, 4); + return buf; +} + +/** Read the job-wide memory limit back out of a JOBOBJECT_EXTENDED_LIMIT_INFORMATION. */ +export function jobMemoryLimitFrom(buf: Buffer): number | null { + if (!(buf.readUInt32LE(LIMIT_FLAGS_OFFSET) & JOB_OBJECT_LIMIT_JOB_MEMORY)) return null; + return Number(buf.readBigUInt64LE(JOB_MEMORY_LIMIT_OFFSET)); +} + +function kernel32() { + // Handles are pointer-sized; passed as 64-bit integers so the pseudo-handle -1 from + // GetCurrentProcess survives the round trip (as a JS number it would lose precision). + return dlopen('kernel32.dll', { + CreateJobObjectW: { args: [FFIType.ptr, FFIType.ptr], returns: FFIType.u64 }, + SetInformationJobObject: { args: [FFIType.u64, FFIType.i32, FFIType.ptr, FFIType.u32], returns: FFIType.i32 }, + QueryInformationJobObject: { + args: [FFIType.u64, FFIType.i32, FFIType.ptr, FFIType.u32, FFIType.ptr], + returns: FFIType.i32, + }, + AssignProcessToJobObject: { args: [FFIType.u64, FFIType.i64], returns: FFIType.i32 }, + GetCurrentProcess: { args: [], returns: FFIType.i64 }, + }).symbols; +} + +export interface JobResult { + memory: boolean; + cpu: boolean; + error?: string; +} + +/** + * Put the current process in a new job with the given limits. Memory and CPU are set separately so + * one being refused (CPU rate control can be, inside some nested jobs) doesn't cost the other. + */ +export function joinLimitedJob(memoryBytes: number | null, cpus: number | null, logicalCores: number): JobResult { + try { + const k = kernel32(); + const job = k.CreateJobObjectW(null, null); + if (!job) return { memory: false, cpu: false, error: 'CreateJobObject failed' }; + + const limit = extendedLimitInfo(memoryBytes); + const memory = k.SetInformationJobObject(job, JobObjectExtendedLimitInformation, limit, limit.length) !== 0; + const rate = cpus === null ? null : cpuRateInfo(cpus, logicalCores); + const cpu = + rate !== null && k.SetInformationJobObject(job, JobObjectCpuRateControlInformation, rate, rate.length) !== 0; + + if (!k.AssignProcessToJobObject(job, k.GetCurrentProcess())) { + return { memory: false, cpu: false, error: 'AssignProcessToJobObject failed' }; + } + return { memory: memory && memoryBytes !== null, cpu }; + } catch (err) { + return { memory: false, cpu: false, error: (err as Error).message }; + } +} + +/** The memory limit of the job this process is in, if any. A null handle means "my own job". */ +export function currentJobMemoryLimit(): number | null { + try { + const buf = Buffer.alloc(EXTENDED_LIMIT_SIZE); + const ok = kernel32().QueryInformationJobObject(0n, JobObjectExtendedLimitInformation, buf, buf.length, null); + return ok ? jobMemoryLimitFrom(buf) : null; + } catch { + return null; + } +} diff --git a/apps/server/src/libraryOrder.test.ts b/apps/server/src/libraryOrder.test.ts index cbdbff8..3cbefd5 100644 --- a/apps/server/src/libraryOrder.test.ts +++ b/apps/server/src/libraryOrder.test.ts @@ -1,11 +1,8 @@ import { expect, test } from 'bun:test'; -import fs from 'node:fs'; -import os from 'node:os'; -import path from 'node:path'; +import { useTestConfig } from './testConfig'; // Point the server at a throwaway data dir before anything opens the real database. -process.env.SAKUYA_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'sakuya-liborder-')); -process.env.PORT = '38773'; +useTestConfig('liborder', { port: 38773 }); const BASE = 'http://localhost:38773'; await import('./index'); // starts listening on PORT diff --git a/apps/server/src/limits.test.ts b/apps/server/src/limits.test.ts index 641f1f8..9f4ad09 100644 --- a/apps/server/src/limits.test.ts +++ b/apps/server/src/limits.test.ts @@ -1,13 +1,15 @@ import { describe, expect, test } from 'bun:test'; -import { describeLimits, parseMemory, resolveLimits } from './lib/limits'; +import { describeLimits, effectiveCpus, resolveLimits } from './lib/limits'; +import { cpuRateInfo, extendedLimitInfo, jobMemoryLimitFrom } from './lib/windowsJob'; /** - * lib/limits is pure and env-injected precisely so this file doesn't need a server, a data dir or - * a real .env — nothing here has the side effects that force `bun test --isolate` elsewhere. + * lib/limits is pure and takes its inputs as arguments precisely so this file doesn't need a + * server, a data dir or a config file — nothing here has the side effects that force + * `bun test --isolate` elsewhere. */ describe('resolveLimits', () => { - test('unset, reproduces the values the services used to hardcode', () => { - expect(resolveLimits({})).toEqual({ + test('with no budget, reproduces the values the services used to hardcode', () => { + expect(resolveLimits(null)).toEqual({ maxCpus: null, jobConcurrency: 2, ffmpegThreads: null, @@ -23,7 +25,7 @@ describe('resolveLimits', () => { * either the formula drifted or the old numbers were not what we thought. */ test('at 2 CPUs the derived values match the old hardcoded constants', () => { - const limits = resolveLimits({ SAKUYA_MAX_CPUS: '2' }); + const limits = resolveLimits(2); expect(limits.jobConcurrency).toBe(2); expect(limits.onnxIntraOpThreads).toBe(2); expect(limits.onnxInterOpThreads).toBe(1); @@ -33,7 +35,7 @@ describe('resolveLimits', () => { }); test('scales with the budget', () => { - const four = resolveLimits({ SAKUYA_MAX_CPUS: '4' }); + const four = resolveLimits(4); expect(four.jobConcurrency).toBe(4); expect(four.onnxIntraOpThreads).toBe(4); expect(four.sharpConcurrency).toBe(4); @@ -42,82 +44,100 @@ describe('resolveLimits', () => { }); test('a fractional budget still runs one of everything', () => { - const half = resolveLimits({ SAKUYA_MAX_CPUS: '0.5' }); + const half = resolveLimits(0.5); expect(half.jobConcurrency).toBe(1); expect(half.ffmpegThreads).toBe(1); expect(half.onnxIntraOpThreads).toBe(1); expect(half.sharpConcurrency).toBe(1); }); +}); - test('granular overrides beat the derived value', () => { - const limits = resolveLimits({ SAKUYA_MAX_CPUS: '4', SAKUYA_FFMPEG_THREADS: '3', SAKUYA_JOB_CONCURRENCY: '1' }); - expect(limits.jobConcurrency).toBe(1); - expect(limits.ffmpegThreads).toBe(3); - expect(limits.onnxIntraOpThreads).toBe(4); // untouched by the overrides +describe('effectiveCpus', () => { + test('nothing configured and no quota means no budget', () => { + expect(effectiveCpus(null, null)).toBeNull(); }); - test('granular overrides work with no CPU budget at all', () => { - const limits = resolveLimits({ SAKUYA_SHARP_CONCURRENCY: '1' }); - expect(limits.maxCpus).toBeNull(); - expect(limits.sharpConcurrency).toBe(1); - expect(limits.jobConcurrency).toBe(2); + /** Docker's `cpus` sizes the pools without being restated in sakuya.config.json. */ + test('a detected quota is used when the config has none', () => { + expect(effectiveCpus(null, 2)).toEqual({ cpus: 2, source: 'quota' }); }); - /** A typo in a limit should cost you the limit, not the server. */ - test('junk falls back instead of throwing', () => { - expect(resolveLimits({ SAKUYA_MAX_CPUS: 'two' })).toEqual(resolveLimits({})); - expect(resolveLimits({ SAKUYA_MAX_CPUS: '0' })).toEqual(resolveLimits({})); - expect(resolveLimits({ SAKUYA_MAX_CPUS: '-4' })).toEqual(resolveLimits({})); - expect(resolveLimits({ SAKUYA_JOB_CONCURRENCY: '2.5' }).jobConcurrency).toBe(2); - expect(resolveLimits({ SAKUYA_JOB_CONCURRENCY: '' }).jobConcurrency).toBe(2); + test('the tighter of the two wins', () => { + expect(effectiveCpus(1.5, 4)).toEqual({ cpus: 1.5, source: 'config' }); + expect(effectiveCpus(8, 2)).toEqual({ cpus: 2, source: 'quota' }); + expect(effectiveCpus(3, null)).toEqual({ cpus: 3, source: 'config' }); }); }); -describe('parseMemory', () => { - test('accepts the suffixes Docker accepts', () => { - expect(parseMemory('2g')).toBe(2 * 1024 ** 3); - expect(parseMemory('2G')).toBe(2 * 1024 ** 3); - expect(parseMemory('2gb')).toBe(2 * 1024 ** 3); - expect(parseMemory('512m')).toBe(512 * 1024 ** 2); - expect(parseMemory('1.5G')).toBe(Math.floor(1.5 * 1024 ** 3)); - expect(parseMemory('1024')).toBe(1024); +describe('describeLimits', () => { + const GiB = 1024 ** 3; + const report = (cpus: number | null, configuredMemory: number | null, enforcedMemory: number | null) => ({ + limits: resolveLimits(cpus), + cpuSource: cpus === null ? null : ('config' as const), + configuredMemory, + enforcedMemory, }); - test('rejects junk', () => { - expect(parseMemory(undefined)).toBeNull(); - expect(parseMemory('')).toBeNull(); - expect(parseMemory('lots')).toBeNull(); - expect(parseMemory('2x')).toBeNull(); - expect(parseMemory('-2g')).toBeNull(); + test('says nothing when nothing is configured or detected', () => { + expect(describeLimits(report(null, null, null))).toBeNull(); }); -}); -describe('describeLimits', () => { - test('says nothing when nothing is configured', () => { - expect(describeLimits({}, resolveLimits({}))).toBeNull(); + test('reports the derived CPU knobs', () => { + expect(describeLimits(report(2, null, null))).toBe( + 'Limits: 2 CPUs (config) -> 2 concurrent jobs, ffmpeg 1, onnx 2/1, libvips 2', + ); }); /** - * The failure mode this guards against is someone setting SAKUYA_MAX_MEMORY, seeing the server - * boot cleanly, and assuming they have a cap. Without the launcher they have not. + * The failure mode this guards against is someone setting limits.memory, seeing the server boot + * cleanly, and assuming they have a cap. Without the launcher applying it, they have not. */ - test('is explicit that a memory cap is unenforced without the launcher', () => { - const env = { SAKUYA_MAX_MEMORY: '2g' }; - const line = describeLimits(env, resolveLimits(env)); - expect(line).toContain('NOT enforced'); + test('is explicit that a configured memory limit is not binding', () => { + expect(describeLimits(report(null, 2 * GiB, null))).toContain('NOT enforced'); + // A looser ceiling elsewhere (say Docker's mem_limit) doesn't satisfy a tighter request either. + const looser = describeLimits(report(null, 2 * GiB, 3 * GiB)); + expect(looser).toContain('NOT enforced'); + expect(looser).toContain('3.0 GiB'); }); - test('confirms the cap once the launcher has applied it', () => { - const env = { SAKUYA_MAX_MEMORY: '2g', SAKUYA_HARD_LIMIT_APPLIED: 'true' }; - const line = describeLimits(env, resolveLimits(env)); - expect(line).toContain('Memory capped at 2g'); - expect(line).not.toContain('NOT enforced'); + test('confirms a memory limit the kernel enforces', () => { + const line = describeLimits(report(null, 2 * GiB, 2 * GiB)); + expect(line).toBe('Limits: memory capped at 2.0 GiB'); }); - test('reports the derived CPU knobs', () => { - const env = { SAKUYA_MAX_CPUS: '2' }; - expect(describeLimits(env, resolveLimits(env))).toBe( - 'Limits: 2 CPUs -> 2 concurrent jobs, ffmpeg 1, onnx 2/1, libvips 2', - ); + test('a byte count the kernel rounded down to a page still counts as enforced', () => { + expect(describeLimits(report(null, 1_000_000_000, 999_997_440))).not.toContain('NOT enforced'); + }); + + test("reports a container's memory ceiling even with nothing configured", () => { + expect(describeLimits(report(null, null, 3 * GiB))).toBe('Limits: memory capped at 3.0 GiB'); + }); +}); + +/** + * The job-object path only runs on Windows, so CI never calls into kernel32. What it can pin is + * the struct layout — the part that silently does the wrong thing if an offset is off. + */ +describe('windows job object structs', () => { + test('extended limit info carries the job memory limit at the documented offset', () => { + const buf = extendedLimitInfo(2 * 1024 ** 3); + expect(buf.length).toBe(144); + expect(buf.readUInt32LE(16)).toBe(0x2000 | 0x200); // KILL_ON_JOB_CLOSE | JOB_MEMORY + expect(buf.readBigUInt64LE(120)).toBe(2n * 1024n ** 3n); + expect(jobMemoryLimitFrom(buf)).toBe(2 * 1024 ** 3); + }); + + test('no memory limit leaves the flag off', () => { + const buf = extendedLimitInfo(null); + expect(buf.readUInt32LE(16)).toBe(0x2000); + expect(jobMemoryLimitFrom(buf)).toBeNull(); + }); + + test('cpu rate is a hard-capped share of all cores in hundredths of a percent', () => { + const buf = cpuRateInfo(2, 8)!; + expect(buf.readUInt32LE(0)).toBe(0x1 | 0x4); // ENABLE | HARD_CAP + expect(buf.readUInt32LE(4)).toBe(2500); + expect(cpuRateInfo(8, 8)).toBeNull(); + expect(cpuRateInfo(0.01, 64)!.readUInt32LE(4)).toBe(2); }); }); diff --git a/apps/server/src/mediaBatch.test.ts b/apps/server/src/mediaBatch.test.ts index 6253f83..765344a 100644 --- a/apps/server/src/mediaBatch.test.ts +++ b/apps/server/src/mediaBatch.test.ts @@ -3,10 +3,10 @@ import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; import { eq } from 'drizzle-orm'; +import { useTestConfig } from './testConfig'; // Point the server at a throwaway data dir before anything opens the real database. -process.env.SAKUYA_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'sakuya-batch-')); -process.env.PORT = '38774'; +useTestConfig('batch', { port: 38774 }); const BASE = 'http://localhost:38774'; const { db, sqlite, schema } = await import('./db'); diff --git a/apps/server/src/mediaList.test.ts b/apps/server/src/mediaList.test.ts index 9919abb..e35729d 100644 --- a/apps/server/src/mediaList.test.ts +++ b/apps/server/src/mediaList.test.ts @@ -1,11 +1,8 @@ import { expect, test } from 'bun:test'; -import fs from 'node:fs'; -import os from 'node:os'; -import path from 'node:path'; +import { useTestConfig } from './testConfig'; // Point the server at a throwaway data dir before anything opens the real database. -process.env.SAKUYA_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'sakuya-list-')); -process.env.PORT = '38775'; +useTestConfig('list', { port: 38775 }); const BASE = 'http://localhost:38775'; const { db, schema } = await import('./db'); diff --git a/apps/server/src/migrations.test.ts b/apps/server/src/migrations.test.ts index 8f04b5e..a681f3a 100644 --- a/apps/server/src/migrations.test.ts +++ b/apps/server/src/migrations.test.ts @@ -1,11 +1,9 @@ import { describe, expect, test } from 'bun:test'; import { Database } from 'bun:sqlite'; -import fs from 'node:fs'; -import os from 'node:os'; import path from 'node:path'; +import { useTestConfig } from './testConfig'; -const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), 'sakuya-migrations-')); -process.env.SAKUYA_DATA_DIR = dataDir; +const dataDir = useTestConfig('migrations'); // A DB last booted before the transcode release: it has every column up to v8, no transcoded_at, // and (like every DB from that era) user_version 0. Docker users who skipped that release hit this. diff --git a/apps/server/src/routes/settings.ts b/apps/server/src/routes/settings.ts index bef7044..ce8ce99 100644 --- a/apps/server/src/routes/settings.ts +++ b/apps/server/src/routes/settings.ts @@ -6,7 +6,7 @@ import { eq, isNotNull, isNull } from 'drizzle-orm'; import { sqlite, db, schema } from '../db'; import { wrap } from '../lib/http'; import { getAllSettings, getSetting, setSetting, gifsAsVideos } from '../lib/settings'; -import { THUMBS_DIR, DB_PATH, APP_VERSION, DATA_DIR, HOME_DATA_DIR, LOCAL_DATA_DIR } from '../lib/config'; +import { THUMBS_DIR, DB_PATH, APP_VERSION, DATA_DIR, DATA_DIR_PINNED, HOME_DATA_DIR, LOCAL_DATA_DIR } from '../lib/config'; import { migrateDataDir } from '../lib/storage'; import { enqueueBulkThumbnailRegenerate } from '../services/thumbnailer'; import { enqueueBulkTranscodeCheck } from '../services/transcoder'; @@ -92,7 +92,7 @@ settingsRouter.get( home: HOME_DATA_DIR, local: LOCAL_DATA_DIR, usingHome: DATA_DIR === HOME_DATA_DIR, - locked: Boolean(process.env.SAKUYA_DATA_DIR), + locked: DATA_DIR_PINNED, }; res.json(info); }), @@ -102,8 +102,8 @@ settingsRouter.post( '/api/system/storage/migrate', wrap(async (req, res) => { const { target } = z.object({ target: z.enum(['home', 'local']) }).parse(req.body); - if (process.env.SAKUYA_DATA_DIR) { - return res.status(400).json({ error: 'SAKUYA_DATA_DIR is set — change that env var instead (Docker sets it).' }); + if (DATA_DIR_PINNED) { + return res.status(400).json({ error: 'server.dataDir is set in sakuya.config.json — change it there instead.' }); } const to = target === 'home' ? HOME_DATA_DIR : LOCAL_DATA_DIR; if (to === DATA_DIR) return res.status(400).json({ error: 'Data is already stored there.' }); diff --git a/apps/server/src/search.test.ts b/apps/server/src/search.test.ts index 3731162..af6f021 100644 --- a/apps/server/src/search.test.ts +++ b/apps/server/src/search.test.ts @@ -3,9 +3,9 @@ import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; import { eq } from 'drizzle-orm'; +import { useTestConfig } from './testConfig'; -process.env.SAKUYA_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'sakuya-search-')); -process.env.PORT = '38777'; +useTestConfig('search', { port: 38777 }); const BASE = 'http://localhost:38777'; const { db, schema } = await import('./db'); diff --git a/apps/server/src/services/jobQueue.ts b/apps/server/src/services/jobQueue.ts index 176d45c..b86e819 100644 --- a/apps/server/src/services/jobQueue.ts +++ b/apps/server/src/services/jobQueue.ts @@ -1,7 +1,7 @@ import { EventEmitter } from 'node:events'; import { eq, desc } from 'drizzle-orm'; import { db, schema } from '../db'; -import { LIMITS } from '../lib/limits'; +import { LIMITS } from '../lib/config'; import type { Job, JobType } from '@sakuya/shared'; export const jobEvents = new EventEmitter(); diff --git a/apps/server/src/services/tagger.ts b/apps/server/src/services/tagger.ts index 8e1fb37..8379f6d 100644 --- a/apps/server/src/services/tagger.ts +++ b/apps/server/src/services/tagger.ts @@ -3,9 +3,8 @@ import fsp from 'node:fs/promises'; import sharp from 'sharp'; import { eq, and, isNull } from 'drizzle-orm'; import { db, sqlite, schema } from '../db'; -import { MODEL_PATH, MODEL_TAGS_PATH, modelRepoBase, DEFAULT_MODEL_ID, MODEL_REGISTRY } from '../lib/config'; +import { MODEL_PATH, MODEL_TAGS_PATH, modelRepoBase, DEFAULT_MODEL_ID, MODEL_REGISTRY, LIMITS } from '../lib/config'; import { confidenceThreshold, getSetting, setSetting } from '../lib/settings'; -import { LIMITS } from '../lib/limits'; import { thumbPathFor } from './thumbnailer'; import { enqueueJob, type JobHandle } from './jobQueue'; import { bumpTasteVersion } from '../lib/tasteVersion'; diff --git a/apps/server/src/services/transcoder.ts b/apps/server/src/services/transcoder.ts index cc58d9a..e7c545c 100644 --- a/apps/server/src/services/transcoder.ts +++ b/apps/server/src/services/transcoder.ts @@ -3,8 +3,7 @@ import fs from 'node:fs'; import { spawn } from 'node:child_process'; import ffmpegStatic from 'ffmpeg-static'; import { eq } from 'drizzle-orm'; -import { TRANSCODES_DIR } from '../lib/config'; -import { LIMITS } from '../lib/limits'; +import { TRANSCODES_DIR, LIMITS } from '../lib/config'; import { db, schema } from '../db'; import { probeVideo } from './scanner'; import { enqueueJob, type JobHandle } from './jobQueue'; diff --git a/apps/server/src/similar.test.ts b/apps/server/src/similar.test.ts index da24ef5..364bfaf 100644 --- a/apps/server/src/similar.test.ts +++ b/apps/server/src/similar.test.ts @@ -1,12 +1,9 @@ import { describe, expect, test } from 'bun:test'; import { Database } from 'bun:sqlite'; -import fs from 'node:fs'; -import os from 'node:os'; import path from 'node:path'; +import { useTestConfig } from './testConfig'; -const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), 'sakuya-similar-')); -process.env.SAKUYA_DATA_DIR = dataDir; -process.env.PORT = '38776'; +const dataDir = useTestConfig('similar', { port: 38776 }); const BASE = 'http://localhost:38776'; // Seed the database *before* importing ./db, so these rows look like ones hashed by a build that diff --git a/apps/server/src/testConfig.ts b/apps/server/src/testConfig.ts new file mode 100644 index 0000000..814d3c2 --- /dev/null +++ b/apps/server/src/testConfig.ts @@ -0,0 +1,21 @@ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; + +/** + * Point this test process at a throwaway sakuya.config.json whose data dir is a fresh temp folder, + * so a test never reads, creates or boots against the real one. Must run before anything imports + * lib/config — hence every server test dynamic-imports the app after calling this. + * + * Returns the data dir. + */ +export function useTestConfig( + name: string, + { port, auth }: { port?: number; auth?: { enabled: boolean; secret: string } } = {}, +): string { + const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), `sakuya-${name}-`)); + const file = path.join(dataDir, 'sakuya.config.json'); + fs.writeFileSync(file, JSON.stringify({ server: { dataDir, ...(port ? { port } : {}) }, ...(auth ? { auth } : {}) })); + process.env.SAKUYA_CONFIG = file; + return dataDir; +} diff --git a/apps/server/src/version.test.ts b/apps/server/src/version.test.ts index 5973298..ce7db99 100644 --- a/apps/server/src/version.test.ts +++ b/apps/server/src/version.test.ts @@ -1,11 +1,11 @@ import { describe, expect, test } from 'bun:test'; import fs from 'node:fs'; -import os from 'node:os'; import path from 'node:path'; +import { useTestConfig } from './testConfig'; // Importing lib/config has a side effect: it mkdirs the whole data tree. Point it at a temp dir // so running the suite doesn't create ~/.sakuya on a machine that has never run the app. -process.env.SAKUYA_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'sakuya-version-')); +useTestConfig('version'); /** * The app version lives in three places that used to drift apart: the root package.json, the diff --git a/apps/web/src/routes/settings/MiscTabs.tsx b/apps/web/src/routes/settings/MiscTabs.tsx index 1ef46c7..6f59752 100644 --- a/apps/web/src/routes/settings/MiscTabs.tsx +++ b/apps/web/src/routes/settings/MiscTabs.tsx @@ -139,7 +139,8 @@ export function SystemTab() { {storage?.locked && (
- Set by the SAKUYA_DATA_DIR environment variable (Docker) — change it there. + Pinned by server.dataDir in sakuya.config.json (Docker pins it to{' '} + /data) — change it there.
)} {movedTo && ( diff --git a/apps/web/vite.config.ts b/apps/web/vite.config.ts index 5f6264a..6dc7980 100644 --- a/apps/web/vite.config.ts +++ b/apps/web/vite.config.ts @@ -1,14 +1,29 @@ import { defineConfig } from 'vite'; import react from '@vitejs/plugin-react'; import tailwindcss from '@tailwindcss/vite'; +// Relative, not @sakuya/shared/config: Vite bundles relative imports of its config, while a bare +// workspace import would be handed to Node as raw TypeScript. +import { loadConfig } from '../../packages/shared/src/config'; + +// Same sakuya.config.json as the server, so changing server.port or server.host there can't leave +// the proxy pointing at the old address. Read-only: the server side creates the file. +const { config } = loadConfig(); + +// Connect to the server over loopback when it listens on every interface; otherwise to the one +// address it bound. IPv6 literals need brackets in a URL. +const wildcard = config.server.host === '0.0.0.0' || config.server.host === '::'; +const apiHost = wildcard ? '127.0.0.1' : config.server.host; export default defineConfig({ plugins: [react(), tailwindcss()], server: { - port: 5173, + // The web UI is useless on the network without the API behind it, and the API is exposed on + // purpose when server.host is — so the two follow one setting. + host: config.server.host, + port: config.web.port, proxy: { '/api': { - target: 'http://localhost:3777', + target: `http://${apiHost.includes(':') ? `[${apiHost}]` : apiHost}:${config.server.port}`, changeOrigin: true, }, }, diff --git a/docker-compose.yml b/docker-compose.yml index 01f3ed9..11b4080 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,25 +1,50 @@ +# Settings live in sakuya.config.json next to this file, same as outside Docker — create it first: +# `cp sakuya.config.example.json sakuya.config.json`, or run ./setup.sh. +# +# Docker is the one place settings can be overridden: any SAKUYA_* variable under `environment:` +# wins over the file (SAKUYA_AUTH_ENABLED, SAKUYA_AUTH_SECRET, SAKUYA_HTTPS, ...). The image already +# pins host, port and data dir, which are container plumbing. +# +# Machine-specific additions — your media folders, other ports, resource limits — go in +# docker-compose.override.yml, which compose merges in automatically and git ignores. For example: +# +# services: +# server: +# cpus: "2.0" +# mem_limit: "3g" +# volumes: +# - /path/to/your/Pictures:/path/to/your/Pictures +# web: +# ports: !override +# - "3000:80" +# +# Library folders must be mounted at the SAME absolute path the database references (visible via +# GET /api/libraries), or scanning and original file serving can't find them. services: server: build: context: . target: server restart: unless-stopped - environment: - PORT: "3777" - SAKUYA_DATA_DIR: /data + # The hard resource ceiling in Docker, instead of limits.* in sakuya.config.json: a container's + # limits can only be set from outside it. The app detects the CPU quota and sizes its job + # concurrency and thread pools to it, so it isn't restated anywhere else. Uncomment here, or set + # them in docker-compose.override.yml. + # cpus: "2.0" + # mem_limit: "3g" volumes: - # Reuses the existing apps/server/data folder, so the DB, downloaded tagger - # model, thumbnails, and uploads already on disk carry straight over. + # Long syntax so a missing file is an error. Short syntax would silently create a *directory* + # named sakuya.config.json on the host. + - type: bind + source: ./sakuya.config.json + target: /app/sakuya.config.json + read_only: true + bind: + create_host_path: false + # Data (DB, downloaded tagger model, thumbnails, uploads). - ./apps/server/data:/data - # Library folders must be mounted at the SAME absolute path the DB already - # references (visible via GET /api/libraries), otherwise scanning/original - # file serving can't find them. Edit these to match the absolute paths - # your libraries were added under: - - /path/to/your/Pictures:/path/to/your/Pictures - - /path/to/your/Videos:/path/to/your/Videos ports: - "3777:3777" - web: build: context: . @@ -27,9 +52,9 @@ services: restart: unless-stopped depends_on: - server - ports: - - "8080:80" volumes: - # Read-only so nginx serves thumbnails (/data/thumbnails) straight from - # disk, bypassing the proxy for grid loading. + # nginx serves thumbnails straight off disk (see docker/nginx.conf), + # so it needs the same data dir mounted as the server container. - ./apps/server/data:/data:ro + ports: + - "8080:80" diff --git a/docs/API.md b/docs/API.md index 6cc93f5..1a4ac09 100644 --- a/docs/API.md +++ b/docs/API.md @@ -325,12 +325,12 @@ Response (`SystemInfo`): `{ version, mediaCount, mediaBytes, dbBytes, thumbBytes ### `GET /api/system/storage` Response (`StorageInfo`): `{ current, home, local, usingHome, locked }` — where the data dir resolved -to this boot, the two candidates, and whether `SAKUYA_DATA_DIR` pins it. +to this boot, the two candidates, and whether `server.dataDir` in `sakuya.config.json` pins it. ### `POST /api/system/storage/migrate` Body: `{ target: 'home' | 'local' }`. Closes the DB, copies the whole data dir to the target, deletes the original, responds `{ movedTo }`, then exits the process — the new location is only picked up on -restart. 400 if `SAKUYA_DATA_DIR` is set, the target is already in use, or a database already exists +restart. 400 if `server.dataDir` is set, the target is already in use, or a database already exists there. ### `POST /api/system/clear-thumbnails` diff --git a/docs/SETUP.md b/docs/SETUP.md index c941df0..841249f 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -36,7 +36,7 @@ docker/ nginx config for the container build | Path | What it is | Why leave it alone | |---|---|---| -| `apps/server/data/` | Runtime data dir: SQLite DB (`tbge.db*`), `thumbnails/`, `transcodes/`, `uploads/`, `models/` (ONNX tagger), `downloader/` (gallery-dl bin + cookies) | Live user data and a live SQLite file (plus `-wal`/`-shm` sidecars). Editing/deleting while the server runs can corrupt the DB or drop real libraries. Gitignored — don't try to commit it. Override the location with `SAKUYA_DATA_DIR`, or move it to `~/.sakuya` from Settings > System — don't move it manually. | +| `apps/server/data/` | Runtime data dir: SQLite DB (`tbge.db*`), `thumbnails/`, `transcodes/`, `uploads/`, `models/` (ONNX tagger), `downloader/` (gallery-dl bin + cookies) | Live user data and a live SQLite file (plus `-wal`/`-shm` sidecars). Editing/deleting while the server runs can corrupt the DB or drop real libraries. Gitignored — don't try to commit it. Override the location with `server.dataDir` in `sakuya.config.json`, or move it to `~/.sakuya` from Settings > System — don't move it manually. | | `node_modules/` (root and per-workspace) | Installed deps | Managed by `bun install`. Never hand-edit; delete and reinstall if something's wrong. | | `apps/web/dist/` | Vite build output | Regenerated by `bun build`. Not source. | | `bun.lock` | Dependency lockfile | Only changes via `bun install`/`bun add`; hand-editing desyncs it from `node_modules`. | @@ -47,56 +47,107 @@ Anything under `apps/server/data/` is the one place that holds actual user state ## Configuration -No config is required to start. Env vars, if you need them: +Everything is configured in one file, **`sakuya.config.json`** at the repository root. It is created +with the defaults the first time Sakuya starts (or by `./setup.sh`), is gitignored, and is read by +the server, the web dev server and the Docker containers alike. Every key is optional. Restart after +editing it. + +The committed [`sakuya.config.example.json`](../sakuya.config.example.json) is the same file with +every setting at its default — copy it to `sakuya.config.json` to start by hand. Your own +`sakuya.config.json` never leaves your machine. + +```json +{ + "$schema": "./packages/shared/sakuya.config.schema.json", + "server": { "port": 3777, "host": "127.0.0.1", "dataDir": null, "https": false }, + "web": { "port": 5173 }, + "auth": { "enabled": false, "secret": "" }, + "limits": { "cpus": null, "memory": null } +} +``` + +The `$schema` line gives VS Code and JetBrains editors autocomplete and a description of each +setting on hover. -| Variable | Default | Purpose | +| Setting | Default | Purpose | |---|---|---| -| `PORT` | `3777` | Server port | -| `SAKUYA_DATA_DIR` | `~/.sakuya` if it exists, else `apps/server/data` | Where DB/thumbnails/uploads/models live. Unset, the location is autodetected on boot from whether `~/.sakuya` exists — no config file. Setting it wins over both and disables the in-app move. | -| `SAKUYA_HOST` | `127.0.0.1` | Interface to bind. Loopback by default, so a fresh install isn't exposed to the network: the server serves your whole library and can open a file manager on the host, and auth is off unless you turn it on. Set to `0.0.0.0` to reach it from other machines — preferably with `AUTH_ENABLED=true`. Docker images set this to `0.0.0.0` already, since a container must bind all interfaces for its published port to work. | -| `AUTH_ENABLED` | unset | Set to `true` to require login | -| `AUTH_SECRET` | unset | Required if `AUTH_ENABLED=true`. Changing it signs every logged-in session out. | -| `SAKUYA_HTTPS` | unset | Set to `true` when serving over HTTPS, to mark the auth cookie `Secure`. Off by default because a `Secure` cookie is silently dropped over plain HTTP, which is a supported LAN setup. | -| `SAKUYA_MAX_CPUS` | unset | CPU budget in cores, fractions allowed — the non-Docker counterpart to compose's `cpus`. Derives job concurrency and the ffmpeg / onnxruntime / libvips thread pools together. See [Resource limits](#resource-limits). | -| `SAKUYA_MAX_MEMORY` | unset | Memory ceiling (`2g`, `512m`, a byte count). **Not enforced by the app itself** — only by `./run.sh` on a systemd host. See [Resource limits](#resource-limits). | -| `SAKUYA_HARD_LIMIT` | unset | Set to `false` to keep the in-app limits but never wrap the process in a systemd scope. | -| `SAKUYA_JOB_CONCURRENCY` | `2` | Background jobs (scan/tag/thumbnail/transcode) allowed to run at once. Overrides what `SAKUYA_MAX_CPUS` derives. | -| `SAKUYA_FFMPEG_THREADS` | unset (ffmpeg picks) | `-threads` while transcoding. Thumbnail extraction is always 1 — a single frame gains nothing from more. | -| `SAKUYA_ONNX_THREADS` | `2` | onnxruntime threads within one operator, while AI tagging. The first knob to lower if tagging starves everything else. | -| `SAKUYA_SHARP_CONCURRENCY` | unset (sharp's own default, 1 on glibc Linux) | libvips worker pool for sharp (thumbnails, tagger preprocessing). Clamped to sharp's default — it can lower the pool, never raise it. | - -Start from [`.env.example`](../.env.example) — Bun loads `.env` on its own, so there is nothing to -install and no flag to pass. +| `server.port` | `3777` | API server port. The web dev server proxies `/api` to it, so changing it here is enough. | +| `server.host` | `127.0.0.1` | Interface the API **and** the web dev server bind to. Loopback by default, so a fresh install isn't exposed to the network: the server serves your whole library and can open a file manager on the host. Set `0.0.0.0` to reach Sakuya from other machines — turn `auth` on first. | +| `server.dataDir` | `null` | Where the DB, thumbnails, uploads and models live. `null` autodetects: `~/.sakuya` if it exists, else `apps/server/data`. A relative path is relative to the config file. Setting it disables the in-app move in Settings > System. | +| `server.https` | `false` | Marks the login cookie `Secure`. Only with HTTPS in front: a `Secure` cookie is silently dropped over plain HTTP, which is a supported LAN setup. | +| `web.port` | `5173` | Port of the web interface (the Vite dev server). | +| `auth.enabled` | `false` | Require a password. | +| `auth.secret` | `""` | The password; required when `auth.enabled` is on. Changing it signs every session out. | +| `limits.cpus` | `null` | CPU budget in cores, fractions allowed. See [Resource limits](#resource-limits). | +| `limits.memory` | `null` | Memory ceiling: `"2g"`, `"512m"`, or a byte count. See [Resource limits](#resource-limits). | + +A value of the wrong type stops Sakuya at startup with the file and key named, rather than quietly +falling back to a default. An unknown key (most likely a typo) is logged and ignored. + +**Environment variables are not read outside Docker.** One file is the whole story; a `PORT` or +`AUTH_ENABLED` left in your shell or an old `.env` is ignored, and the boot log says so. + +### Upgrading from `.env` + +The first start after upgrading creates `sakuya.config.json` from your old `.env`, `apps/server/.env` +and environment, so the port, data dir and login carry over, then logs what it did. Delete the +`.env` files afterwards. As a safety net, if an old `AUTH_ENABLED=true` is still around while the file +has `auth.enabled` off, Sakuya refuses to start rather than come up without a password. The old +per-subsystem knobs (`SAKUYA_JOB_CONCURRENCY`, `SAKUYA_FFMPEG_THREADS`, `SAKUYA_ONNX_THREADS`, +`SAKUYA_SHARP_CONCURRENCY`, `SAKUYA_HARD_LIMIT`) are gone; `limits.cpus` derives all of them. + +### Docker + +`docker-compose.yml` mounts the same `sakuya.config.json` into the server container, so create it +before `docker compose up` (`cp sakuya.config.example.json sakuya.config.json`, or `./setup.sh`). A +missing file is an error on purpose — Docker would otherwise create an empty *directory* in its place. + +Docker is the only place settings can be overridden: a `SAKUYA_*` variable under `environment:` in +compose wins over the file. + +| Variable | Overrides | +|---|---| +| `SAKUYA_PORT`, `SAKUYA_HOST`, `SAKUYA_DATA_DIR` | `server.port`, `server.host`, `server.dataDir` — already pinned by the image to `3777`, `0.0.0.0` and `/data`, which is container plumbing | +| `SAKUYA_HTTPS` | `server.https` | +| `SAKUYA_AUTH_ENABLED`, `SAKUYA_AUTH_SECRET` | `auth.enabled`, `auth.secret` | +| `SAKUYA_MAX_CPUS`, `SAKUYA_MAX_MEMORY` | `limits.cpus`, `limits.memory` | + +Ports the containers publish, the data folder and your media folders are compose's concern. Put +anything specific to your machine — media mounts especially — in a `docker-compose.override.yml` +next to it, which compose merges in automatically and git ignores. ## Resource limits -`docker-compose.yml` caps the server with `cpus: "2.0"` and `mem_limit: "3g"`. Running outside -Docker you can get most, but not all, of that from a `.env`. - -**CPU can be capped properly.** `SAKUYA_MAX_CPUS=2` bounds the things that actually consume cores: -how many background jobs run at once, and how many threads ffmpeg, onnxruntime and libvips each -spawn underneath them. Unset, libx264 sizes its own pool from the detected core count — which -inside a container is the *host's* core count, not the container's share, since `nproc` doesn't -know about the cgroup. (libvips is the exception: sharp already defaults it to 1 on glibc Linux. -The libvips knob is clamped so it can only ever lower that, never raise it.) - -**Memory cannot, from inside the process.** Docker's `mem_limit` is a kernel cgroup: the kernel -refuses the allocation and OOM-kills at the boundary. Nothing a process sets on itself is -equivalent, and `--max-old-space-size` would be actively misleading here, because most of this -app's memory is not JS heap — it is libvips buffers, onnxruntime arenas, ffmpeg child processes and -SQLite page cache. So `SAKUYA_MAX_MEMORY` is **not** read by the server. What it does instead: - -- `./run.sh` reads it and re-execs the process inside a transient systemd scope with a real - `MemoryMax`, which is a genuine cgroup ceiling — the same mechanism Docker uses. -- That needs Linux with systemd. It fails soft everywhere else (macOS, Windows, containers without - systemd, or when the cpu controller isn't delegated to user slices, which is common): you get a - note on stdout and the in-app limits still apply. -- The boot log states which of the two you got, so a `SAKUYA_MAX_MEMORY` that isn't binding says so - out loud instead of quietly implying a cap that isn't there. - -What the in-app limits *do* give you for memory is indirect but real: peak RSS is roughly -concurrency × per-task peak, so halving `SAKUYA_JOB_CONCURRENCY` roughly halves the worst case. - -Unset, every one of these keeps the value it had when the numbers were hardcoded across the -services, so adding a `.env` without them changes nothing. At `SAKUYA_MAX_CPUS=2` the derivations -reproduce those old constants exactly — `apps/server/src/limits.test.ts` pins that. +Two settings, `limits.cpus` and `limits.memory`, and Sakuya applies them in two layers: + +**In-app sizing (everywhere).** The CPU budget decides how many background jobs run at once and how +many threads ffmpeg, onnxruntime and libvips each spawn under them. That matters even under a hard +cap: those libraries size their pools from the core count they can *see*, which inside a quota is +still every core on the host, and they would then fight over the share they actually get. At +`cpus: 2` this reproduces the constants the deployment was originally tuned with +(`apps/server/src/limits.test.ts` pins it). With no budget, the old defaults apply. Peak memory is +roughly concurrency × per-task peak, so a lower CPU budget also lowers the worst case for memory. + +**A hard ceiling, enforced by the kernel.** A process can't impose one on itself, so the launcher +(`apps/server/src/launch.ts`, which `bun dev`, `bun run start`, `run.sh` and `run.bat` all use) puts it +in place before the server starts. It wraps only the server, not the web dev server: + +| Platform | Mechanism | Notes | +|---|---|---| +| Linux | transient systemd scope with `MemoryMax` / `CPUQuota` | Needs systemd, with the controller delegated to your user. Memory usually is; cpu sometimes isn't. Each limit is tried separately, so one being refused doesn't cost the other. | +| Windows | job object (memory limit + hard CPU-rate cap) | Everything the server spawns, ffmpeg included, inherits it. | +| macOS | none | In-app sizing only. | +| Docker | the container's `cpus` / `mem_limit`, set in `docker-compose.yml` or your `docker-compose.override.yml` | The app detects the CPU quota and sizes itself to it, so there's nothing to repeat in `sakuya.config.json`. | + +Past the memory ceiling, allocations fail (the process is OOM-killed on Linux), the same as Docker's +`mem_limit`. `--max-old-space-size` would not be a substitute: most of this app's memory is not JS +heap but libvips buffers, onnxruntime arenas, ffmpeg children and SQLite page cache. + +The launcher and the server's boot log both state what is actually enforced, not what was asked +for. A `limits.memory` that isn't binding says `NOT enforced` out loud, instead of a clean boot +implying a cap that isn't there: + +``` +[sakuya] Hard limits via systemd: memory 2.0 GiB, 2 CPUs +Limits: 2 CPUs (config) -> 2 concurrent jobs, ffmpeg 1, onnx 2/1, libvips 2; memory capped at 2.0 GiB +``` diff --git a/package.json b/package.json index edfc632..6ff209f 100644 --- a/package.json +++ b/package.json @@ -10,6 +10,7 @@ "dev": "bun run --filter '*' dev", "dev:server": "bun run --cwd apps/server dev", "dev:web": "bun run --cwd apps/web dev", + "start": "bun run --cwd apps/server start", "build": "bun run --cwd apps/web build", "lint": "eslint .", "format": "prettier --write .", diff --git a/packages/shared/package.json b/packages/shared/package.json index 1eb60f0..aa99195 100644 --- a/packages/shared/package.json +++ b/packages/shared/package.json @@ -6,6 +6,7 @@ "main": "src/index.ts", "types": "src/index.ts", "exports": { - ".": "./src/index.ts" + ".": "./src/index.ts", + "./config": "./src/config.ts" } } diff --git a/packages/shared/sakuya.config.schema.json b/packages/shared/sakuya.config.schema.json new file mode 100644 index 0000000..b32acd2 --- /dev/null +++ b/packages/shared/sakuya.config.schema.json @@ -0,0 +1,85 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Sakuya configuration", + "description": "sakuya.config.json, at the repository root. The only place Sakuya is configured; every key is optional. Restart after editing.", + "type": "object", + "additionalProperties": false, + "properties": { + "$schema": { "type": "string" }, + "server": { + "type": "object", + "additionalProperties": false, + "properties": { + "port": { + "description": "Port the API server listens on. The web dev server proxies /api here. Ignored in Docker, which pins 3777.", + "type": "integer", + "minimum": 1, + "maximum": 65535, + "default": 3777 + }, + "host": { + "description": "Interface to bind, for both the API and the web dev server. 127.0.0.1 = this machine only. 0.0.0.0 = reachable from your network; turn auth on first, since the server exposes your whole library and can open a file manager on this machine. Ignored in Docker, which must bind 0.0.0.0.", + "type": "string", + "default": "127.0.0.1" + }, + "dataDir": { + "description": "Where the database, thumbnails, uploads and models live. null = autodetect: ~/.sakuya if that folder exists, else apps/server/data. A relative path is relative to this file. Setting it disables the move button in Settings > System. Ignored in Docker, which mounts the data dir at /data.", + "type": ["string", "null"], + "default": null + }, + "https": { + "description": "Mark the login cookie Secure. Only turn on when serving over HTTPS: browsers silently drop Secure cookies over plain HTTP, which would lock you out.", + "type": "boolean", + "default": false + } + } + }, + "web": { + "type": "object", + "additionalProperties": false, + "properties": { + "port": { + "description": "Port of the web interface (the Vite dev server). In Docker the web interface is served by nginx; change its published port in docker-compose.yml.", + "type": "integer", + "minimum": 1, + "maximum": 65535, + "default": 5173 + } + } + }, + "auth": { + "type": "object", + "additionalProperties": false, + "properties": { + "enabled": { + "description": "Require a password to use Sakuya.", + "type": "boolean", + "default": false + }, + "secret": { + "description": "The password. Required when auth is enabled. Changing it signs every session out.", + "type": "string", + "default": "" + } + } + }, + "limits": { + "type": "object", + "additionalProperties": false, + "properties": { + "cpus": { + "description": "CPU budget in cores; fractions allowed (1.5). Sizes background job concurrency and the ffmpeg / AI tagger / image-processing thread pools, and is enforced as a hard cap on Linux (systemd) and Windows (job object). In Docker, set `cpus` in docker-compose.yml instead; the app detects it. null = no limit.", + "type": ["number", "null"], + "exclusiveMinimum": 0, + "default": null + }, + "memory": { + "description": "Memory ceiling: \"2g\", \"512m\", \"1.5g\", or a byte count. Enforced as a hard cap on Linux (systemd) and Windows (job object): past it, allocations fail rather than the machine swapping. Not enforceable on macOS. In Docker, set `mem_limit` in docker-compose.yml instead. null = no limit.", + "type": ["string", "integer", "null"], + "pattern": "^\\s*\\d+(\\.\\d+)?\\s*([kKmMgGtT][bB]?|[bB])?\\s*$", + "default": null + } + } + } + } +} diff --git a/packages/shared/src/config.ts b/packages/shared/src/config.ts new file mode 100644 index 0000000..9a53f05 --- /dev/null +++ b/packages/shared/src/config.ts @@ -0,0 +1,429 @@ +/** + * sakuya.config.json: the one place Sakuya is configured. + * + * Read by the server, by its launcher (for the hard resource limits) and by the Vite dev server + * (port, bind address, where to proxy /api), so a value changed there reaches everything that + * depends on it. It used to be spread across a root .env that the server never actually loaded + * (Bun reads .env from the working directory, and the server runs from apps/server), a second + * apps/server/.env, and constants hardcoded in vite.config.ts. + * + * Environment variables are deliberately NOT a second source outside Docker: two places to set the + * same thing is how the old setup ended up with a value that looked set and wasn't. Inside the + * container they are the override mechanism, because compose has to pin things the file can't know + * — the bind address, the port nginx proxies to, the data dir as mounted — see DOCKER_OVERRIDES. + * + * Runs under Node as well as Bun (Vite loads its config with Node), so: node: builtins only, and no + * dependencies — the web workspace doesn't have zod. + */ +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +export interface SakuyaConfig { + server: { + port: number; + /** Interface to bind. Loopback by default: the API serves the whole library and can open a file manager on the host. */ + host: string; + /** Absolute path, or null to autodetect (~/.sakuya if it exists, else apps/server/data). */ + dataDir: string | null; + /** Marks the auth cookie Secure. Only behind HTTPS — browsers drop Secure cookies over plain HTTP. */ + https: boolean; + }; + web: { + /** Vite dev server port. Docker serves the web app through nginx instead; publish that in compose. */ + port: number; + }; + auth: { + enabled: boolean; + /** The login password. Changing it signs every session out. */ + secret: string; + }; + limits: { + /** CPU budget in cores, fractions allowed. null = no limit. */ + cpus: number | null; + /** Memory ceiling in bytes (parsed from "2g" / "512m" / a byte count). null = no limit. */ + memory: number | null; + }; +} + +type Env = Record; + +export const CONFIG_FILE_NAME = 'sakuya.config.json'; + +/** packages/shared/src -> repo root. Also right inside the Docker image, which mirrors the layout under /app. */ +export const REPO_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..', '..'); + +export const DEFAULT_CONFIG: SakuyaConfig = { + server: { port: 3777, host: '127.0.0.1', dataDir: null, https: false }, + web: { port: 5173 }, + auth: { enabled: false, secret: '' }, + limits: { cpus: null, memory: null }, +}; + +export class ConfigError extends Error { + constructor(file: string, message: string) { + super(`${file}: ${message}`); + this.name = 'ConfigError'; + } +} + +const MEMORY_UNITS: Record = { + b: 1, + k: 1024, + kb: 1024, + m: 1024 ** 2, + mb: 1024 ** 2, + g: 1024 ** 3, + gb: 1024 ** 3, + t: 1024 ** 4, + tb: 1024 ** 4, +}; + +/** Accepts what Docker's mem_limit accepts — "2g", "512m", "1.5G" — plus a bare byte count. */ +export function parseMemory(raw: string | number | null | undefined): number | null { + if (typeof raw === 'number') return Number.isFinite(raw) && raw > 0 ? Math.floor(raw) : null; + if (!raw) return null; + const match = /^\s*(\d+(?:\.\d+)?)\s*([a-z]*)\s*$/i.exec(raw); + if (!match) return null; + const value = Number(match[1]); + const unit = match[2].toLowerCase(); + const multiplier = unit === '' ? 1 : MEMORY_UNITS[unit]; + if (!multiplier || !Number.isFinite(value) || value <= 0) return null; + return Math.floor(value * multiplier); +} + +export function formatBytes(bytes: number): string { + return bytes >= 1024 ** 3 ? `${(bytes / 1024 ** 3).toFixed(1)} GiB` : `${Math.round(bytes / 1024 ** 2)} MiB`; +} + +// --- Validation ------------------------------------------------------------------------------ +// +// A malformed value is an error, not a silent fallback: this file is the only knob, and a typo that +// quietly reverted to the default would look exactly like a setting that works. + +function isObject(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +const KNOWN_KEYS: Record = { + '': ['$schema', 'server', 'web', 'auth', 'limits'], + server: ['port', 'host', 'dataDir', 'https'], + web: ['port'], + auth: ['enabled', 'secret'], + limits: ['cpus', 'memory'], +}; + +/** + * Pure: validate parsed JSON into a full config, filling defaults for anything absent. Returns + * warnings for unknown keys rather than failing on them — most likely a typo, worth saying out loud. + */ +export function parseConfig(raw: unknown, file: string, baseDir: string): { config: SakuyaConfig; warnings: string[] } { + if (!isObject(raw)) throw new ConfigError(file, 'must contain a JSON object'); + const warnings: string[] = []; + const section = (name: keyof SakuyaConfig): Record => { + const value = raw[name]; + if (value === undefined) return {}; + if (!isObject(value)) throw new ConfigError(file, `"${name}" must be an object`); + return value; + }; + for (const [prefix, keys] of Object.entries(KNOWN_KEYS)) { + const obj = prefix ? raw[prefix] : raw; + if (!isObject(obj)) continue; + for (const key of Object.keys(obj)) { + if (!keys.includes(key)) + warnings.push(`unknown setting "${prefix ? `${prefix}.` : ''}${key}" in ${file} is ignored`); + } + } + + const port = (obj: Record, key: string, name: string, fallback: number): number => { + const value = obj[key]; + if (value === undefined) return fallback; + if (!Number.isInteger(value) || (value as number) < 1 || (value as number) > 65535) { + throw new ConfigError(file, `"${name}" must be a port number (1-65535)`); + } + return value as number; + }; + const bool = (obj: Record, key: string, name: string, fallback: boolean): boolean => { + const value = obj[key]; + if (value === undefined) return fallback; + if (typeof value !== 'boolean') throw new ConfigError(file, `"${name}" must be true or false`); + return value; + }; + const string = (obj: Record, key: string, name: string, fallback: string): string => { + const value = obj[key]; + if (value === undefined) return fallback; + if (typeof value !== 'string') throw new ConfigError(file, `"${name}" must be a string`); + return value; + }; + + const server = section('server'); + const web = section('web'); + const auth = section('auth'); + const limits = section('limits'); + + const host = string(server, 'host', 'server.host', DEFAULT_CONFIG.server.host).trim(); + if (!host) throw new ConfigError(file, '"server.host" must not be empty'); + + const rawDataDir = server.dataDir; + if (rawDataDir !== undefined && rawDataDir !== null && (typeof rawDataDir !== 'string' || !rawDataDir.trim())) { + throw new ConfigError(file, '"server.dataDir" must be a path, or null to autodetect'); + } + // Relative to the config file, not to wherever the process happened to start — that ambiguity + // is exactly what the old cwd-relative .env got wrong. + const dataDir = typeof rawDataDir === 'string' ? path.resolve(baseDir, rawDataDir.trim()) : null; + + const rawCpus = limits.cpus; + if ( + rawCpus !== undefined && + rawCpus !== null && + (typeof rawCpus !== 'number' || !Number.isFinite(rawCpus) || rawCpus <= 0) + ) { + throw new ConfigError(file, '"limits.cpus" must be a positive number of cores (e.g. 2 or 1.5), or null'); + } + + const rawMemory = limits.memory; + let memory: number | null = null; + if (rawMemory !== undefined && rawMemory !== null) { + memory = typeof rawMemory === 'string' || typeof rawMemory === 'number' ? parseMemory(rawMemory) : null; + if (memory === null) throw new ConfigError(file, '"limits.memory" must be a size like "2g" or "512m", or null'); + } + + return { + config: { + server: { + port: port(server, 'port', 'server.port', DEFAULT_CONFIG.server.port), + host, + dataDir, + https: bool(server, 'https', 'server.https', DEFAULT_CONFIG.server.https), + }, + web: { port: port(web, 'port', 'web.port', DEFAULT_CONFIG.web.port) }, + auth: { + enabled: bool(auth, 'enabled', 'auth.enabled', DEFAULT_CONFIG.auth.enabled), + secret: string(auth, 'secret', 'auth.secret', DEFAULT_CONFIG.auth.secret), + }, + limits: { cpus: (rawCpus as number | null | undefined) ?? null, memory }, + }, + warnings, + }; +} + +// --- Docker overrides ------------------------------------------------------------------------ + +/** + * The only way to override the file, and only inside the image (the Dockerfile sets SAKUYA_DOCKER). + * The image pins host, port and dataDir itself: the container must bind 0.0.0.0, nginx proxies to + * a fixed port, and the host's dataDir path means nothing inside the container. + */ +export const DOCKER_OVERRIDES = { + SAKUYA_PORT: 'server.port', + SAKUYA_HOST: 'server.host', + SAKUYA_DATA_DIR: 'server.dataDir', + SAKUYA_HTTPS: 'server.https', + SAKUYA_AUTH_ENABLED: 'auth.enabled', + SAKUYA_AUTH_SECRET: 'auth.secret', + SAKUYA_MAX_CPUS: 'limits.cpus', + SAKUYA_MAX_MEMORY: 'limits.memory', +} as const; + +/** Pre-config-file names, recognised only to migrate them into a new file and to warn when set. */ +const LEGACY_ENV: Record = { + PORT: 'server.port', + SAKUYA_HOST: 'server.host', + SAKUYA_DATA_DIR: 'server.dataDir', + SAKUYA_HTTPS: 'server.https', + AUTH_ENABLED: 'auth.enabled', + AUTH_SECRET: 'auth.secret', + SAKUYA_MAX_CPUS: 'limits.cpus', + SAKUYA_MAX_MEMORY: 'limits.memory', +}; + +/** Env vars from the old setup that no longer do anything; only mentioned so nobody keeps tuning them. */ +const RETIRED_ENV = [ + 'SAKUYA_HARD_LIMIT', + 'SAKUYA_JOB_CONCURRENCY', + 'SAKUYA_FFMPEG_THREADS', + 'SAKUYA_ONNX_THREADS', + 'SAKUYA_SHARP_CONCURRENCY', +]; + +/** Coerce an env string into the JSON value the file would hold, for the given setting path. */ +function envValue(setting: string, raw: string): unknown { + if (setting === 'server.port') return /^\d+$/.test(raw.trim()) ? Number(raw) : raw; + if (setting === 'server.https' || setting === 'auth.enabled') return raw.trim() === 'true'; + if (setting === 'limits.cpus') return raw.trim() && Number.isFinite(Number(raw)) ? Number(raw) : raw; + return raw; +} + +function setPath(target: Record>, setting: string, value: unknown): void { + const [section, key] = setting.split('.'); + (target[section] ??= {})[key] = value; +} + +// --- Legacy .env migration ------------------------------------------------------------------- + +/** KEY=value lines. Not `source`d or evaluated — data only. */ +function readDotenv(file: string): Env { + let text: string; + try { + text = fs.readFileSync(file, 'utf8'); + } catch { + return {}; + } + const out: Env = {}; + for (const line of text.split(/\r?\n/)) { + const match = /^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*?)\s*$/.exec(line); + if (!match) continue; + out[match[1]] = match[2].replace(/^(['"])(.*)\1$/, '$2'); + } + return out; +} + +/** + * Seed a first config file from whatever the old setup had, so upgrading keeps the port, the data + * dir and — most importantly — a login that was switched on. Later sources win: root .env, then + * apps/server/.env (the one Bun actually loaded), then the real environment. + */ +function legacySeed(dir: string, env: Env): { values: Record>; sources: string[] } { + const values: Record> = {}; + const sources: string[] = []; + const layers: [string, Env][] = [ + ['.env', readDotenv(path.join(dir, '.env'))], + [path.join('apps', 'server', '.env'), readDotenv(path.join(dir, 'apps', 'server', '.env'))], + ['environment', env], + ]; + for (const [label, layer] of layers) { + let used = false; + for (const [name, setting] of Object.entries(LEGACY_ENV)) { + const raw = layer[name]; + if (raw === undefined || raw === '') continue; + setPath(values, setting, envValue(setting, raw)); + used = true; + } + if (used) sources.push(label); + } + return { values, sources }; +} + +function initialFile(seed: Record>): string { + const content = { + $schema: './packages/shared/sakuya.config.schema.json', + server: { ...DEFAULT_CONFIG.server, ...seed.server }, + web: { ...DEFAULT_CONFIG.web }, + auth: { ...DEFAULT_CONFIG.auth, ...seed.auth }, + limits: { cpus: null, memory: null, ...seed.limits }, + }; + return `${JSON.stringify(content, null, 2)}\n`; +} + +// --- Loading --------------------------------------------------------------------------------- + +export interface LoadedConfig { + config: SakuyaConfig; + file: string; + /** Running inside the Docker image, where env overrides apply. */ + docker: boolean; + /** Set when this call wrote a fresh file — the caller should say so once. */ + created: string | null; + warnings: string[]; +} + +export interface LoadOptions { + env?: Env; + /** Write a default file (migrating any old .env) when none exists. The server side does; Vite only reads. */ + create?: boolean; +} + +export function configFilePath(env: Env = process.env): string { + // SAKUYA_CONFIG picks a different *file*; it overrides no values. The test suite points it at + // throwaway files so tests never read or create the real one. + return env.SAKUYA_CONFIG ? path.resolve(env.SAKUYA_CONFIG) : path.join(REPO_ROOT, CONFIG_FILE_NAME); +} + +export function loadConfig({ env = process.env, create = false }: LoadOptions = {}): LoadedConfig { + const file = configFilePath(env); + const dir = path.dirname(file); + const docker = env.SAKUYA_DOCKER === '1'; + let created: string | null = null; + + let text: string | null = null; + try { + text = fs.readFileSync(file, 'utf8'); + } catch (err) { + if ((err as NodeJS.ErrnoException).code !== 'ENOENT') throw err; + } + + if (text === null && create && !docker) { + const { values, sources } = legacySeed(dir, env); + const initial = initialFile(values); + try { + // wx: the server and Vite start in parallel; whoever loses the race just reads the winner's file. + fs.writeFileSync(file, initial, { flag: 'wx' }); + text = initial; + created = sources.length + ? `Created ${file} from your old settings (${sources.join(', ')}). Those are no longer read — ` + + 'edit this file instead, and delete any .env files.' + : `Created ${file} with the defaults. Edit it to configure Sakuya.`; + } catch (err) { + if ((err as NodeJS.ErrnoException).code !== 'EEXIST') throw err; + text = fs.readFileSync(file, 'utf8'); + } + } + + let raw: unknown = {}; + if (text !== null) { + try { + raw = JSON.parse(text); + } catch (err) { + throw new ConfigError(file, `is not valid JSON (${(err as Error).message})`); + } + } + + const parsed = parseConfig(raw, file, dir); + const warnings = parsed.warnings; + let config = parsed.config; + + if (docker) { + const overrides: Record> = {}; + for (const [name, setting] of Object.entries(DOCKER_OVERRIDES)) { + const value = env[name]; + if (value !== undefined && value !== '') setPath(overrides, setting, envValue(setting, value)); + } + // Re-validated as a whole so an override gets the same checks as the file. + const merged = { + server: { ...config.server, ...overrides.server }, + web: config.web, + auth: { ...config.auth, ...overrides.auth }, + limits: { ...config.limits, ...overrides.limits }, + }; + config = parseConfig(merged, `${file} + environment overrides`, dir).config; + } else { + const ignored = [...new Set([...Object.keys(LEGACY_ENV), ...Object.keys(DOCKER_OVERRIDES)])].filter( + (name) => env[name] !== undefined && env[name] !== '', + ); + if (ignored.length) { + warnings.push( + `ignoring ${ignored.join(', ')} from the environment — outside Docker, settings are only read from ${file}`, + ); + } + } + const retired = RETIRED_ENV.filter((name) => env[name] !== undefined && env[name] !== ''); + if (retired.length) { + warnings.push(`${retired.join(', ')} no longer exist; limits.cpus in ${file} derives all of them`); + } + + // The one ignored variable that must not just be a warning: someone who turned login on the old + // way would otherwise upgrade into a server that is silently open. + const wantsAuth = env.AUTH_ENABLED === 'true' || (!docker && env.SAKUYA_AUTH_ENABLED === 'true'); + if (wantsAuth && !config.auth.enabled) { + throw new ConfigError( + file, + 'login is enabled in your environment or .env (AUTH_ENABLED=true) but "auth.enabled" is false here. ' + + 'Environment variables are no longer read' + + (docker + ? ' under their old names — use SAKUYA_AUTH_ENABLED / SAKUYA_AUTH_SECRET.' + : '; set auth.enabled and auth.secret in this file.'), + ); + } + + return { config, file, docker, created, warnings }; +} diff --git a/run.bat b/run.bat index ebc4658..a0b06f2 100644 --- a/run.bat +++ b/run.bat @@ -23,9 +23,9 @@ if not exist "node_modules\" ( ) ) +rem Ports, bind address, login and resource limits all come from sakuya.config.json (created with +rem the defaults on first run). The server and web URLs are printed below once each is up. echo Starting development server... -echo Backend : http://localhost:3777 -echo Frontend : http://localhost:5173 echo. bun dev diff --git a/run.sh b/run.sh index 2d3ef31..005d48d 100755 --- a/run.sh +++ b/run.sh @@ -1,61 +1,6 @@ #!/usr/bin/env bash set -e -# --------------------------------------------------------------------------- -# Optional hard resource ceiling, the non-Docker counterpart to the `cpus` and -# `mem_limit` in docker-compose.yml. -# -# The SAKUYA_MAX_* knobs in .env cap what the app *asks for* (job concurrency, -# ffmpeg/onnx/libvips threads). They cannot stop a runaway the way Docker can, -# because Docker's limits are kernel cgroup limits. systemd can hand us the same -# cgroup outside Docker, so when it is available we re-exec this script inside a -# transient scope. Everything here fails soft: no systemd, an unwritable -# controller, or a bad value leaves the app running with in-app limits only, -# because a resource knob should never be the reason the server won't boot. -# -# Skipped entirely when neither variable is set. Set SAKUYA_HARD_LIMIT=false to -# keep the in-app limits but never wrap. -# --------------------------------------------------------------------------- -SCRIPT_PATH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")" - -# Read a single KEY=value out of .env. Deliberately not `source`: .env is data, -# and sourcing it would execute whatever happens to be in there. -env_value() { - [ -f .env ] || return 0 - sed -n "s/^[[:space:]]*$1[[:space:]]*=[[:space:]]*//p" .env | tail -n1 | tr -d "\"' \r" -} - -MAX_MEMORY="${SAKUYA_MAX_MEMORY:-$(env_value SAKUYA_MAX_MEMORY)}" -MAX_CPUS="${SAKUYA_MAX_CPUS:-$(env_value SAKUYA_MAX_CPUS)}" -HARD_LIMIT="${SAKUYA_HARD_LIMIT:-$(env_value SAKUYA_HARD_LIMIT)}" - -if [ -z "${SAKUYA_HARD_LIMIT_APPLIED:-}" ] && [ "$HARD_LIMIT" != "false" ] && - { [ -n "$MAX_MEMORY" ] || [ -n "$MAX_CPUS" ]; }; then - if command -v systemd-run &>/dev/null; then - props=() - [ -n "$MAX_MEMORY" ] && props+=(-p "MemoryMax=$(echo "$MAX_MEMORY" | tr '[:lower:]' '[:upper:]')") - # systemd counts a core as 100%, so 1.5 CPUs is CPUQuota=150%. - [ -n "$MAX_CPUS" ] && props+=(-p "CPUQuota=$(awk -v c="$MAX_CPUS" 'BEGIN{printf "%d", c*100}')%") - - # Dry-run the scope before committing to it. The cpu controller is often not delegated to - # user slices, and this is the only way to find out without either masking the real exit - # code or losing the chance to fall back. - if systemd-run --user --scope --collect --quiet "${props[@]}" true &>/dev/null; then - echo "Hard limits via systemd: ${props[*]}" - echo "" - export SAKUYA_HARD_LIMIT_APPLIED=true - exec systemd-run --user --scope --collect --quiet "${props[@]}" "$SCRIPT_PATH" "$@" - fi - echo "Note: systemd-run rejected ${props[*]} (the cpu controller is commonly not" - echo " delegated to user slices). Continuing with in-app limits only." - echo "" - else - echo "Note: systemd-run not found, so SAKUYA_MAX_MEMORY cannot be enforced as a" - echo " real ceiling. In-app concurrency/thread limits still apply." - echo "" - fi -fi - echo "========================================" echo " Sakuya - Dev Server" echo "========================================" @@ -71,9 +16,9 @@ if [ ! -d "node_modules" ]; then bun install fi +# Ports, bind address, login and resource limits all come from sakuya.config.json (created with the +# defaults on first run). The server and web URLs are printed below once each is up. echo "Starting development server..." -echo " Backend : http://localhost:3777" -echo " Frontend : http://localhost:5173" echo "" bun dev diff --git a/sakuya.config.example.json b/sakuya.config.example.json new file mode 100644 index 0000000..3eb6689 --- /dev/null +++ b/sakuya.config.example.json @@ -0,0 +1,20 @@ +{ + "$schema": "./packages/shared/sakuya.config.schema.json", + "server": { + "port": 3777, + "host": "127.0.0.1", + "dataDir": null, + "https": false + }, + "web": { + "port": 5173 + }, + "auth": { + "enabled": false, + "secret": "" + }, + "limits": { + "cpus": null, + "memory": null + } +} diff --git a/setup.bat b/setup.bat index c2d5df5..a9f1ab0 100644 --- a/setup.bat +++ b/setup.bat @@ -54,5 +54,8 @@ if %errorlevel% neq 0 ( ) echo. -echo Setup complete. Run run.bat to start the dev server. +bun apps\server\src\launch.ts --init + +echo. +echo Setup complete. Edit sakuya.config.json to configure Sakuya, then run run.bat. pause diff --git a/setup.sh b/setup.sh index 8b521f3..f51e06b 100755 --- a/setup.sh +++ b/setup.sh @@ -47,4 +47,7 @@ echo "Installing dependencies..." bun install echo "" -echo "Setup complete. Run ./run.sh to start the dev server." +bun apps/server/src/launch.ts --init + +echo "" +echo "Setup complete. Edit sakuya.config.json to configure Sakuya, then run ./run.sh."