Add clear script environment cache command (PEP 723 PR 13/16) (#1724)

> Part of #1602 (PEP 723 inline script env support). Design doc: #1601.
>
> Builds on the merged persistence work in #1697 and is rebased on
current `main`.

### Roadmap context

This is **PR 13 of 16** in the PEP 723 inline-script roadmap. It adds
the explicit, user-confirmed cache lifecycle operation that the later
TTL work will reuse.

| Phase 5: Lifecycle and polish | PR | Status |
|---|---|---|
| | PR 7: persistence (`get` / `set` + Memento) | merged (#1697) |
| | **PR 13: clear inline-script cache** | **this PR** |
| | PR 14: opportunistic 14-day TTL eviction | follow-up |
| | PR 15: lifecycle telemetry | separate |
| | PR 16: status-bar decision | resolved; no code PR |

### Why this PR

The extension can create extension-owned inline-script environments and
persist script associations, but it has no complete way to remove that
state. Clearing only the files would leave Memento associations and
`pythonProjects` entries pointing at deleted interpreters; clearing only
settings would leave disk usage behind.

This PR adds one coordinated lifecycle operation covering:

- extension-owned cache entries;
- persisted and in-memory script associations;
- active selection events; and
- generated inline-script project settings.

Because this is destructive and the cache is shared by extension-host
processes, the implementation is intentionally fail-closed around path
ownership and locks.

### What this PR does

**Adds an internal, confirmation-gated clear command**

- Registers `python-envs.clearScriptEnvCache` only while the hidden
inline-script feature flag is enabled.
- Does not contribute the command to `package.json` or the Command
Palette before rollout.
- Shows a modal warning covering cached environments, associations, and
project entries.
- Cancelling the prompt performs no filesystem, state, or settings
changes.
- Runs cache cleanup before settings removal, so failed/partial cache
cleanup does not silently rewrite project configuration.

**Keeps generic cache clearing behavior safe**

- The existing public `python-envs.clearCache` command continues to
clear existing non-inline managers.
- It skips the preview inline manager because the generic path has no
inline-specific confirmation or project-settings lifecycle.
- The dedicated command invokes the inline manager directly and performs
the complete cleanup transaction.

**Serializes in-process maintenance**

- Adds a manager-local maintenance queue and barrier.
- `create()`, `get()`, and `set()` cannot observe or mutate half-cleared
state.
- A clear request refuses to begin when creation already started.
- A creation request that arrives after clear begins waits for
maintenance to settle.
- Multiple maintenance requests are serialized without globally
serializing unrelated managers.

**Coordinates deletion across extension hosts**

- Acquires and holds each cache entry's cross-process lock through
deletion.
- Classifies locks as missing, held, retained, stale, orphaned,
malformed, or unavailable.
- Uses PID liveness to distinguish a live owner from a stale owner.
- Makes retained markers generation-specific by preserving the owner's
PID/nonce.
- Reclaims only the exact stale/retained generation marker that was
inspected.
- If another process replaces that generation before the atomic claim,
reclamation loses safely and touches nothing.
- Ambiguous legacy fixed `retained` markers remain recognizable but are
conservatively not reclaimed.

**Validates every destructive path**

Before deleting an entry, cleanup verifies that:

- global storage and `script-envs-v1` are normal directories rather than
symlinks/junctions;
- the versioned cache root is the expected direct child of global
storage;
- neither path is a filesystem root or dangerously shallow;
- physical `realpath` containment matches the expected ownership
boundary;
- the cache root has not changed since the cleanup snapshot; and
- the target entry is a normal direct-child directory inside that same
physical root.

The entry lock is acquired first, then root and entry ownership are
revalidated immediately before removal.

**Keeps state consistent through partial failures**

- Attempts cache entries independently and records successful removals.
- Aggregates and surfaces deletion/persistence failures rather than
returning success-shaped output.
- Invalidates only associations whose environment was removed or is
definitively missing.
- Preserves associations for cache entries that could not safely be
removed.
- Cancels pending rehydration for invalidated scripts.
- Clears warm environment and validation caches.
- Advances association revisions so stale async work cannot restore
removed selections.
- Emits `onDidChangeEnvironment` only for selections actually
invalidated.

**Removes generated inline project settings safely**

- Resolves `pythonProjects` entries independently from global,
workspace, and workspace-folder sources.
- Removes only entries whose manager is the inline-script manager.
- Preserves non-inline duplicate entries and higher-precedence
overrides.
- Handles same relative paths across multiple workspace roots.
- Aggregates global/workspace updates so each shared scope is written
once.
- Unloads only loaded projects that have no remaining configuration
source.

### Cleanup semantics

| Condition | Behavior |
|---|---|
| Entry is unlocked and physically owned | Lock, revalidate, delete |
| Lock belongs to a live process | Refuse that deletion |
| Exact stale/retained generation can be claimed | Reclaim, acquire a
fresh lock, delete |
| Lock is unavailable, malformed, orphaned, or legacy-ambiguous |
Preserve entry and surface failure |
| Root or entry is redirected/outside ownership boundary | Refuse
deletion |
| One entry fails after another was removed | Preserve valid survivors;
invalidate removed associations; report aggregate failure |
| Persistence update fails after disk cleanup | Keep in-memory state
consistent and surface the persistence error |
| Cache is already absent | Clear stale associations safely; remain
idempotent |

### Example

```text
Clear Script Environment Cache
→ modal confirmation
→ enter manager maintenance barrier
→ verify physical cache root
→ acquire exact per-entry lock
→ revalidate ownership immediately before deletion
→ delete safe entries
→ reconcile Memento + in-memory selections + events
→ remove generated inline pythonProjects settings
```

### Tests

Coverage includes:

- prompt cancellation and command ordering;
- generic clear behavior with the preview manager absent/present;
- in-process create/clear ordering;
- live, stale, retained, orphaned, malformed, unavailable, and legacy
lock states;
- exact-generation reclamation and delayed-reclaimer/new-creator races;
- holding entry locks through deletion;
- unsafe, shallow, redirected, symlinked, and root-swapped paths;
- successful, missing-cache, idempotent, and partial-failure cleanup;
- persistence failures and pending-rehydration races;
- global/workspace/workspace-folder setting precedence;
- multi-root projects and same-path entries; and
- default-off command registration.

Validation on the rebased branch:

- `npm run compile-tests`
- `npm run compile`
- `npm run lint`
- focused lock/cache-clear/settings/command suites: 38 passing

The full Windows unit run reaches 1638 passing and 5 pending; the
existing concurrent `writeMetaJson` rename test can still intermittently
fail with `EPERM` on Windows. That writer is unchanged by this PR and
the same failure is reproducible on `main`.

### Performance

- No activation scan, timer, or background maintenance is added.
- All work is initiated by the internal clear command.
- Per-entry locks avoid globally serializing independent environment
creation across extension hosts.
- The maintenance barrier exists only inside the enabled inline manager
and is active only during cleanup.

### User impact

**No default-path user impact.** The manager and command remain behind
the undeclared, default-off `python-envs.inlineScripts.enabled` flag,
and the command is not publicly contributed.

When the internal flag is manually enabled, the existing generic cache
command still behaves as before for non-inline managers. Inline cleanup
is available only through the dedicated confirmed lifecycle.

### Scope and follow-up

This PR intentionally does **not** implement:

- automatic routing or project registration;
- activation-time discovery;
- silent/opportunistic deletion;
- TTL expiration; or
- lifecycle telemetry.

PR 14 will reuse this safety and state-cleanup foundation to remove
entries whose `lastUsedAt` exceeds the planned 14-day TTL.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3cb82ae9-7424-40a4-9156-8c54ac6e0895
Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2

Author: StellaHuang95

src/common/lockfile.apis.ts

@@ -16,13 +16,33 @@ export interface AcquiredFileLock {
     readonly retain: () => Promise<void>;
 }
 
+export const FILE_LOCK_DIR_SUFFIX = '.lock';
+export const FILE_LOCK_OWNER_MARKER_PREFIX = 'owner-';
+export const FILE_LOCK_RETAINED_MARKER_PREFIX = 'retained-';
+/** Legacy retained marker. It remains recognizable but cannot be safely reclaimed. */
+export const FILE_LOCK_RETAINED_MARKER = 'retained';
+
+export type ProcessLiveness = 'live' | 'dead' | 'unavailable';
+export type FileLockState = 'missing' | 'held' | 'retained' | 'stale' | 'orphaned' | 'malformed' | 'unavailable';
+
+export interface InspectFileLockOptions {
+    readonly checkProcessLiveness?: (pid: number) => Promise<ProcessLiveness>;
+}
+
 type LockState = 'held' | 'released' | 'retained';
 
+export function getFileLockPath(filePath: string): string {
+    return `${path.resolve(filePath)}${FILE_LOCK_DIR_SUFFIX}`;
+}
+
 /** Acquire an atomic lock released only explicitly; interrupted operations remain locked. */
 export async function acquireFileLock(filePath: string, options: AcquireFileLockOptions): Promise<AcquiredFileLock> {
-    const lockPath = `${path.resolve(filePath)}.lock`;
-    const ownerMarker = path.join(lockPath, `owner-${process.pid}-${crypto.randomBytes(16).toString('hex')}`);
-    const retainedMarker = path.join(lockPath, 'retained');
+    const lockPath = getFileLockPath(filePath);
+    const ownerMarker = path.join(
+        lockPath,
+        `${FILE_LOCK_OWNER_MARKER_PREFIX}${process.pid}-${crypto.randomBytes(16).toString('hex')}`,
+    );
+    const retainedMarker = path.join(lockPath, getRetainedMarkerName(path.basename(ownerMarker)));
     const deadline = Date.now() + options.timeoutMs;
 
     while (true) {
@@ -51,22 +71,9 @@ export async function acquireFileLock(filePath: string, options: AcquireFileLock
                     }
                     state = 'retained';
                     try {
-                        await fsapi.writeFile(retainedMarker, '', { flag: 'wx' });
-                    } catch (error) {
-                        if (hasErrorCode(error, 'EEXIST')) {
-                            return;
-                        }
-                        try {
-                            await fsapi.rename(ownerMarker, retainedMarker);
-                        } catch (renameError) {
-                            if (!hasErrorCode(renameError, 'EEXIST')) {
-                                throw createLockError(
-                                    'Failed to mark the lock as retained',
-                                    'ERETAINFAILED',
-                                    lockPath,
-                                );
-                            }
-                        }
+                        await fsapi.rename(ownerMarker, retainedMarker);
+                    } catch (_error) {
+                        throw createLockError('Failed to mark the lock as retained', 'ERETAINFAILED', lockPath);
                     }
                 },
                 release: async () => {
@@ -100,10 +107,141 @@ export async function acquireFileLock(filePath: string, options: AcquireFileLock
     }
 }
 
-async function isRetainedLock(lockPath: string): Promise<boolean> {
+export async function inspectFileLock(filePath: string, options?: InspectFileLockOptions): Promise<FileLockState> {
+    return (await inspectFileLockSnapshot(filePath, options)).state;
+}
+
+interface FileLockSnapshot {
+    readonly state: FileLockState;
+    readonly marker?: string;
+    readonly markerKind?: 'owner' | 'retained';
+}
+
+async function inspectFileLockSnapshot(
+    filePath: string,
+    options?: InspectFileLockOptions,
+): Promise<FileLockSnapshot> {
+    const lockPath = getFileLockPath(filePath);
+
+    let stat;
     try {
-        await fsapi.lstat(path.join(lockPath, 'retained'));
+        stat = await fsapi.lstat(lockPath);
+    } catch (error) {
+        if (hasErrorCode(error, 'ENOENT')) {
+            return { state: 'missing' };
+        }
+        throw error;
+    }
+
+    if (!stat.isDirectory() || stat.isSymbolicLink()) {
+        return { state: 'malformed' };
+    }
+
+    const entries = await fsapi.readdir(lockPath);
+    const ownerEntries = entries.filter((entry) => entry.startsWith(FILE_LOCK_OWNER_MARKER_PREFIX));
+    const generationRetainedEntries = entries.filter((entry) => entry.startsWith(FILE_LOCK_RETAINED_MARKER_PREFIX));
+    const retainedEntries = entries.filter((entry) => entry === FILE_LOCK_RETAINED_MARKER);
+    const unknownEntries = entries.filter(
+        (entry) =>
+            !entry.startsWith(FILE_LOCK_OWNER_MARKER_PREFIX) &&
+            !entry.startsWith(FILE_LOCK_RETAINED_MARKER_PREFIX) &&
+            entry !== FILE_LOCK_RETAINED_MARKER,
+    );
+
+    if (
+        unknownEntries.length > 0 ||
+        ownerEntries.length > 1 ||
+        generationRetainedEntries.length > 1 ||
+        retainedEntries.length > 1 ||
+        generationRetainedEntries.length + retainedEntries.length > 1 ||
+        generationRetainedEntries.length + ownerEntries.length > 1
+    ) {
+        return { state: 'malformed' };
+    }
+    if (retainedEntries.length === 1) {
+        return { state: 'retained' };
+    }
+    if (generationRetainedEntries.length === 1) {
+        const retainedPid = parseMarkerPid(generationRetainedEntries[0], FILE_LOCK_RETAINED_MARKER_PREFIX);
+        if (retainedPid === undefined) {
+            return { state: 'malformed' };
+        }
+        return { state: 'retained', marker: generationRetainedEntries[0], markerKind: 'retained' };
+    }
+    if (ownerEntries.length === 1) {
+        const ownerPid = parseMarkerPid(ownerEntries[0], FILE_LOCK_OWNER_MARKER_PREFIX);
+        if (ownerPid === undefined) {
+            return { state: 'malformed' };
+        }
+        const liveness = await (options?.checkProcessLiveness ?? getProcessLiveness)(ownerPid);
+        if (liveness === 'dead') {
+            return { state: 'stale', marker: ownerEntries[0], markerKind: 'owner' };
+        }
+        return { state: liveness === 'live' ? 'held' : 'unavailable', marker: ownerEntries[0], markerKind: 'owner' };
+    }
+    return { state: 'orphaned' };
+}
+
+/**
+ * Claim and remove the exact observed stale or retained generation without releasing the lock directory.
+ */
+export async function reclaimFileLock(filePath: string, options?: InspectFileLockOptions): Promise<boolean> {
+    const lockPath = getFileLockPath(filePath);
+    const snapshot = await inspectFileLockSnapshot(filePath, options);
+    if (
+        (snapshot.state !== 'stale' && snapshot.state !== 'retained') ||
+        !snapshot.marker ||
+        !snapshot.markerKind
+    ) {
+        return false;
+    }
+
+    const claimedMarker = path.join(
+        lockPath,
+        `.reclaim-${process.pid}-${crypto.randomBytes(16).toString('hex')}-${snapshot.marker}`,
+    );
+    try {
+        await fsapi.rename(path.join(lockPath, snapshot.marker), claimedMarker);
+    } catch (error) {
+        if (hasErrorCode(error, 'ENOENT') || hasErrorCode(error, 'EEXIST')) {
+            return false;
+        }
+        throw error;
+    }
+
+    try {
+        await fsapi.unlink(claimedMarker);
+        await fsapi.rmdir(lockPath);
         return true;
+    } catch (error) {
+        if (hasErrorCode(error, 'ENOENT') || hasErrorCode(error, 'ENOTEMPTY')) {
+            return false;
+        }
+        throw error;
+    }
+}
+
+export async function getProcessLiveness(pid: number): Promise<ProcessLiveness> {
+    try {
+        process.kill(pid, 0);
+        return 'live';
+    } catch (error) {
+        if (hasErrorCode(error, 'ESRCH')) {
+            return 'dead';
+        }
+        if (hasErrorCode(error, 'EPERM') || hasErrorCode(error, 'EACCES')) {
+            return 'unavailable';
+        }
+        return 'unavailable';
+    }
+}
+
+async function isRetainedLock(lockPath: string): Promise<boolean> {
+    try {
+        const entries = await fsapi.readdir(lockPath);
+        return entries.some(
+            (entry) => entry === FILE_LOCK_RETAINED_MARKER || entry.startsWith(FILE_LOCK_RETAINED_MARKER_PREFIX),
+        );
     } catch (error) {
         if (hasErrorCode(error, 'ENOENT')) {
             return false;
@@ -118,6 +256,23 @@ function hasErrorCode(error: unknown, code: string): boolean {
     );
 }
 
+function getRetainedMarkerName(ownerMarker: string): string {
+    return `${FILE_LOCK_RETAINED_MARKER_PREFIX}${ownerMarker.slice(FILE_LOCK_OWNER_MARKER_PREFIX.length)}`;
+}
+
+function parseMarkerPid(entry: string, prefix: string): number | undefined {
+    const match = entry.match(new RegExp(`^${escapeRegExp(prefix)}(\\d+)-.+$`));
+    if (!match) {
+        return undefined;
+    }
+    const pid = Number(match[1]);
+    return Number.isSafeInteger(pid) && pid > 0 ? pid : undefined;
+}
+
+function escapeRegExp(value: string): string {
+    return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+}
+
 function createLockError(message: string, code: string, lockPath: string): NodeJS.ErrnoException {
     return Object.assign(new Error(message), { code, path: lockPath });
 }

src/extension.ts

@@ -45,6 +45,7 @@ import { ProjectCreatorsImpl } from './features/creators/projectCreators';
 import {
     addPythonProjectCommand,
     copyPathToClipboard,
+    clearScriptEnvironmentCacheCommand,
     createAnyEnvironmentCommand,
     createEnvironmentCommand,
     createTerminalCommand,
@@ -94,7 +95,12 @@ import { PythonStatusBarImpl } from './features/views/pythonStatusBar';
 import { updateViewsAndStatus } from './features/views/revealHandler';
 import { TemporaryStateManager } from './features/views/temporaryStateManager';
 import { PythonEnvTreeItem } from './features/views/treeViewItems';
-import { collectEnvironmentInfo, getEnvManagerAndPackageManagerConfigLevels, runPetInTerminalImpl } from './helpers';
+import {
+    collectEnvironmentInfo,
+    getEnvManagerAndPackageManagerConfigLevels,
+    isInlineScriptsFeatureEnabled,
+    runPetInTerminalImpl,
+} from './helpers';
 import { EnvironmentManagers, ProjectCreators, PythonProjectManager } from './internal.api';
 import { registerInlineScriptFeatures } from './managers/builtin/inlineScript/main';
 import { registerSystemPythonFeatures } from './managers/builtin/main';
@@ -386,6 +392,13 @@ export async function activate(context: ExtensionContext): Promise<PythonEnviron
             await envManagers.clearCache(undefined);
             await clearShellProfileCache(shellStartupProviders);
         }),
+        ...(isInlineScriptsFeatureEnabled()
+            ? [
+                  commands.registerCommand('python-envs.clearScriptEnvCache', async () => {
+                      await clearScriptEnvironmentCacheCommand(envManagers, projectManager);
+                  }),
+              ]
+            : []),
         commands.registerCommand('python-envs.runInTerminal', (item) => {
             return runInTerminalCommand(item, api, terminalManager);
         }),

src/features/envCommands.ts

@@ -26,7 +26,12 @@ import {
     ProjectCreators,
     PythonProjectManager,
 } from '../internal.api';
-import { removePythonProjectSetting, setEnvironmentManager, setPackageManager } from './settings/settingHelpers';
+import {
+    removeInlineScriptPythonProjectSettings,
+    removePythonProjectSetting,
+    setEnvironmentManager,
+    setPackageManager,
+} from './settings/settingHelpers';
 
 import { valid as pep440Valid } from '@renovatebot/pep440';
 import { executeCommand } from '../common/command.api';
@@ -50,8 +55,10 @@ import {
     showInputBox,
     showOpenDialog,
     showQuickPick,
+    showWarningMessage,
     withProgress,
 } from '../common/window.apis';
+import { INLINE_SCRIPT_MANAGER_ID } from '../common/constants';
 import { runAsTask } from './execution/runAsTask';
 import { runInTerminal } from './terminal/runInTerminal';
 import { TerminalManager } from './terminal/terminalManager';
@@ -662,6 +669,36 @@ export async function removePythonProject(
     wm.remove(item.project);
 }
 
+export async function clearScriptEnvironmentCacheCommand(
+    em: EnvironmentManagers,
+    wm: PythonProjectManager,
+): Promise<void> {
+    const manager = em.getEnvironmentManager(INLINE_SCRIPT_MANAGER_ID);
+    if (!manager || !manager.supportsClearCache()) {
+        throw new Error(
+            l10n.t('Inline-script environment cache is unavailable because the inline-script manager is not registered.'),
+        );
+    }
+
+    const clearLabel = l10n.t('Clear Cache');
+    const confirmation = await showWarningMessage(
+        l10n.t(
+            'This will delete all cached inline-script environments, forget their script associations, and remove inline-script project entries from settings.',
+        ),
+        { modal: true },
+        clearLabel,
+    );
+    if (confirmation !== clearLabel) {
+        return;
+    }
+
+    await manager.clearCache();
+    const loadedProjectsToRemove = await removeInlineScriptPythonProjectSettings(wm.getProjects());
+    if (loadedProjectsToRemove.length > 0) {
+        wm.remove(loadedProjectsToRemove);
+    }
+}
+
 export async function getPackageCommandOptions(
     e: unknown,
     em: EnvironmentManagers,

src/features/envManagers.ts

@@ -320,12 +320,16 @@ export class PythonEnvironmentManagers implements EnvironmentManagers {
 
     public async clearCache(scope: EnvironmentManagerScope): Promise<void> {
         if (scope === undefined) {
-            await Promise.all(this.managers.map((m) => m.clearCache()));
+            await Promise.all(
+                this.managers
+                    .filter((manager) => manager.id !== INLINE_SCRIPT_MANAGER_ID)
+                    .map((manager) => manager.clearCache()),
+            );
             return;
         }
 
         const manager = this.getEnvironmentManager(scope);
-        if (manager) {
+        if (manager && manager.id !== INLINE_SCRIPT_MANAGER_ID) {
             await manager.clearCache();
         }
     }

src/features/projectManager.ts

@@ -130,20 +130,17 @@ export class PythonProjectManagerImpl implements PythonProjectManager {
             // For each override, resolve its path and add as a project if not already present
             for (const o of overrides) {
                 let uriFromWorkspace: Uri | undefined = undefined;
-                // if override has a workspace property, resolve the path relative to that workspace
                 if (o.workspace) {
-                    //
                     const workspaceFolder = workspaces.find((ws) => ws.name === o.workspace);
                     if (workspaceFolder) {
                         if (workspaceFolder.uri.toString() !== w.uri.toString()) {
-                            continue; // skip if the workspace is not the same as the current workspace
+                            continue;
                         }
                         uriFromWorkspace = Uri.file(path.resolve(workspaceFolder.uri.fsPath, o.path));
                     }
                 }
                 const uri = uriFromWorkspace ? uriFromWorkspace : Uri.file(path.resolve(w.uri.fsPath, o.path));
 
-                // Check if the project already exists in the newProjects array
                 if (!newProjects.some((p) => p.uri.toString() === uri.toString())) {
                     newProjects.push(new PythonProjectsImpl(o.path, uri));
                 }