Skip to content

chore(api): Drop legacy app-centric tables and dead model classes#4676

Open
jp-agenta wants to merge 10 commits into
feat/oss-singleton-sweepfrom
chore/drop-legacy-tables
Open

chore(api): Drop legacy app-centric tables and dead model classes#4676
jp-agenta wants to merge 10 commits into
feat/oss-singleton-sweepfrom
chore/drop-legacy-tables

Conversation

@jp-agenta

@jp-agenta jp-agenta commented Jun 12, 2026

Copy link
Copy Markdown
Member

Context

The pre-project app ecosystem (apps/variants/bases, deployments, docker images, templates, old environments, old testsets, the old auto- and human-evaluation tables, and the Mongo-migration artifact ids_mapping) is dead code: its routers were deleted, its data was migrated to the new entities earlier in both migration chains, and the model classes were referenced only by an unimported re-export shim and unimported converters. EE also carries legacy nullable organization_id/workspace_id columns (plus deployments.cloud_map_service_id and auto_evaluator_configs.app_id) exclusively on these tables, so dropping them is the schema-parity fix for that drift. This is sequencing step 5 / parity step 2 of docs/designs/oss-ee-convergence/assessment-a-oss-multi-org.md.

Changes

Two migrations, one per chain (the chains are independent), each running DROP TABLE IF EXISTS … CASCADE over the same 20 tables, children before parents: human_evaluation_variants, human_evaluations_scenarios, human_evaluations, auto_evaluation_scenario_results, auto_evaluation_aggregated_results, auto_evaluation_evaluator_configs, auto_evaluation_scenarios, auto_evaluations, auto_evaluator_configs, environments_revisions, environments, app_variant_revisions, app_variants, bases, deployments, docker_images, templates, app_db, testsets, ids_mapping. Downgrade raises (irreversible; restore from backup). The migrations also drop the four enum types the table drops orphan (app_type_enum, templatetype, and nodetype/treetype, which the earlier nodes drop had already left behind); no live column or code uses them. The tracing chains get the same treatment: a new revision f3a4b5c6d7e8 in both editions drops nodetype/treetype there too (orphaned by the earlier nodes drop; spans uses spantype/tracetype/otelspankind/otelstatuscode, which stay).

Code cleanup in the same PR:

  • The 15 dead classes are deleted from oss/src/models/db_models.py (AppDB, DeploymentDB, VariantBaseDB, AppVariantDB, AppVariantRevisionsDB, AppEnvironmentDB, AppEnvironmentRevisionDB, TestsetDB, EvaluatorConfigDB, IDsMappingDB, and the five old Evaluation* classes), along with their now-unused imports.
  • oss/src/models/db/models.py (re-export shim) and oss/src/models/converters.py had zero importers; both files are deleted.
  • db_manager_ee.create_deployment had no callers; deleted with its DeploymentDB import.
  • deprecated_models.py / deprecated_transfer_models.py stay: historical data migrations import them, and deleting them would break replaying old chains.
  • Historical migrations that still imported the deleted classes directly (create_free_plans imported AppDB; the evaluators, environments, applications_workflow, and projects data migrations imported the others) now import frozen copies of the final pre-drop shapes, added to deprecated_models.py on a dedicated DroppedBase. Alembic loads every version script at startup, so without this the migration runner fails to boot on ImportError.

Not dropped: the new-architecture tables with similar names (evaluation_scenarios, testset_artifacts, environment_*) are untouched; the old/new names differ (auto_evaluations vs evaluation_runs, testsets vs testset_artifacts).

Lifecycle/FK standardization (per docs/designs/oss-ee-convergence/db-integrity-audit.md, migrations 4f5a6b7c8d9e OSS / 5a6b7c8d9e0f EE / a4b5c6d7e8f9 both tracing chains): every table gets the six lifecycle columns, fully nullable, created_at with a server default; lifecycle actor FKs are dropped (organizations x3, api_keys, webhook_subscriptions); owning-scope FKs added (webhook_deliveries to projects + subscriptions, secrets to projects/organizations, subscriptions to organizations, meters retargeted from subscriptions to organizations, all CASCADE); membership user FKs get the CASCADE the models declared; invitation user FK becomes SET NULL. The LegacyLifecycleDBA mixin is retired and models now agree with the database everywhere. New convention unit tests (oss/tests/pytest/unit/models/test_lifecycle_conventions.py and the EE counterpart) enforce the standard.

Tests / notes

  • ruff format and ruff check pass; all 175 alembic version scripts across both chains verified to import cleanly (this is what the migration runner does on boot); no remaining references to the dropped classes outside the frozen scaffolding (legacy oss/tests/legacy/old_tests are not collected per pytest.ini testpaths).
  • EE cloud must archive these 20 tables before this merges (pre-drop verification per the assessment); OSS deployments got their data migrated by earlier revisions in the chain.
  • Stacked on the convergence stack so each chain's migration extends the step-1 parity revisions linearly (a parallel branch would create two alembic heads).

🤖 Generated with Claude Code

@vercel

vercel Bot commented Jun 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
agenta-documentation Ready Ready Preview, Comment Jun 14, 2026 6:40pm

Request Review

@coderabbitai

coderabbitai Bot commented Jun 12, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 311064fb-b43d-4eb6-8ec7-5f37b6375fde

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch chore/drop-legacy-tables

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. refactoring A code change that neither fixes a bug nor adds a feature labels Jun 12, 2026
@github-actions

github-actions Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Railway Preview Environment

Preview URL https://gateway-production-0c74.up.railway.app/w
Image tag pr-4676-3ebefbc
Status Failed
Railway logs Open logs
Logs View workflow run
Updated at 2026-06-14T18:46:05.087Z

@junaway junaway force-pushed the feat/oss-singleton-sweep branch from ab96500 to 96c348b Compare June 12, 2026 13:50
@junaway junaway force-pushed the chore/drop-legacy-tables branch from 9f7bcee to 3d3750d Compare June 12, 2026 13:51
@junaway junaway force-pushed the chore/drop-legacy-tables branch from 3d3750d to fc900e5 Compare June 12, 2026 13:58
@junaway junaway force-pushed the feat/oss-singleton-sweep branch from b6d4a7d to dcb998f Compare June 12, 2026 15:40
@junaway junaway force-pushed the chore/drop-legacy-tables branch from df6a36a to c518250 Compare June 12, 2026 15:40
@junaway junaway force-pushed the chore/drop-legacy-tables branch from c518250 to 70650cd Compare June 12, 2026 15:46
@junaway junaway force-pushed the feat/oss-singleton-sweep branch from dcb998f to 91d979d Compare June 12, 2026 18:00
@junaway junaway force-pushed the chore/drop-legacy-tables branch from 18a1424 to 67ae791 Compare June 12, 2026 18:00
@junaway junaway force-pushed the feat/oss-singleton-sweep branch from 4356253 to 91d979d Compare June 14, 2026 13:03
@junaway junaway force-pushed the chore/drop-legacy-tables branch from a52e038 to 92530d5 Compare June 14, 2026 13:03
@junaway junaway force-pushed the feat/oss-singleton-sweep branch from 91d979d to 13535b9 Compare June 14, 2026 13:25
@junaway junaway force-pushed the chore/drop-legacy-tables branch from 92530d5 to 903f73e Compare June 14, 2026 13:25
jp-agenta and others added 10 commits June 14, 2026 20:37
Drops the 20 dead pre-project tables (apps/variants/bases, deployments,
docker_images, templates, old environments, old testsets, old auto/human
evaluations, ids_mapping) in both chains, deletes the 15 dead model classes,
the unimported models/db/models.py shim and models/converters.py, and the
dead create_deployment in db_manager_ee. The drop is the schema-parity fix
for the legacy organization_id/workspace_id columns EE carried on these
tables. deprecated_models scaffolding stays (historical migrations import it).
Alembic loads every version script at startup, and several historical
migrations still imported the model classes deleted with the legacy table
drop (create_free_plans imported AppDB, the evaluators/environments/
applications_workflow/projects data migrations imported the others). This
broke the migration runner on boot.

Freeze the final pre-drop shapes of the nine queried tables in
deprecated_models.py on a dedicated DroppedBase (a separate base so
extend_existing does not merge columns into the older partial shapes) and
point the migrations at them. Verified all 175 version scripts across both
chains now import cleanly.
The frozen classes keep their ForeignKey declarations, but projects/users/
folders are not in DroppedBase's metadata, so the ORM could not resolve FK
targets when a data migration flushes an insert (55bdd2e9a465 adds a testset
row). Stub tables with just the id column satisfy resolution; selects never
needed it, which is why the import sweep passed.
DROP TABLE leaves enum types behind: app_type_enum and templatetype are
orphaned by this revision's drops, nodetype and treetype were already
orphaned when the nodes table was dropped. No live column or code uses
any of the four; the schema dumps confirm zero references at head.
The nodes drop (cfa14a847972) left nodetype and treetype behind in both
editions' tracing databases; no live column uses them (spans uses spantype,
tracetype, otelspankind, otelstatuscode). New revision f3a4b5c6d7e8 on both
tracing chains drops them.
…rity audit

Every table gets the six lifecycle columns (created/updated/deleted x at and
by_id), fully nullable, created_at with a server default; the LegacyLifecycleDBA
mixin is retired (secrets moves to LifecycleDBA) and the per-table created_by_id
nullable overrides become unnecessary. Lifecycle actor columns lose their FKs
(organizations x3, api_keys, webhook_subscriptions) per the no-FKs-on-actors
rule. Owning-scope FKs are added where missing: webhook_deliveries -> projects +
webhook_subscriptions (event_id stays loose, events lives in the tracing
database), secrets -> projects/organizations, subscriptions -> organizations,
and meters retargets from subscriptions to organizations, all CASCADE.
Membership user FKs get the CASCADE the models already declared; invitation
user FK becomes SET NULL. Models and DB now agree everywhere; convention unit
tests enforce the standard from now on.

Migrations: 4f5a6b7c8d9e (OSS core), 5a6b7c8d9e0f (EE core), a4b5c6d7e8f9
(both tracing chains).
Drop pre-existing secrets/webhook_deliveries/subscriptions FKs by
introspection before recreating them, matching the meters idiom. On an
EE-origin database secrets_project_id_fkey already exists, so the bare
create_foreign_key raised DuplicateObjectError mid-migration. Applied to
both the EE (5a6b7c8d9e0f) and OSS (4f5a6b7c8d9e) lifecycle migrations.
… tests

The lifecycle/FK migration retargeted meters.organization_id from subscriptions
to organizations and dropped the old MeterDBE.subscription relationship, but the
meters DAO still joinedload()s it for the Stripe report path - crashing fetch/dump
and silently breaking usage reporting. Re-declare it as a viewonly nav on
organization_id (one subscription per org) so both joinedload sites and
_dbe_to_dto work unchanged.

Also add __init__.py to the oss/ee unit/models test dirs so the duplicate-basename
test_lifecycle_conventions.py modules import under distinct package paths.
…gration

Migration 1c2d3e4f5a6b ran select(WorkspaceDB)/select(ProjectDB) against the
live ORM models, which now declare deleted_at + audit columns added by the
later lifecycle_and_fk_cleanup migration. At 1c2d3e4f5a6b those columns do not
exist yet, so the SELECT failed with UndefinedColumnError: column
workspaces.deleted_at does not exist, aborting the EE core migration on boot.

Add frozen DeprecatedWorkspaceDB/DeprecatedProjectDB snapshots (schema as of
1c2d3e4f5a6b, no audit columns) and bind the data migration to them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t migration

The same migration's create_default_project_memberships did
select(WorkspaceMemberDB) against the live model, which now carries deleted_at
+ audit columns absent at revision 1c2d3e4f5a6b -> UndefinedColumnError:
column workspace_members.deleted_at does not exist. Extend the frozen-model
fix to DeprecatedWorkspaceMemberDB/DeprecatedProjectMemberDB so the whole
migration binds to the schema as of its own revision.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

refactoring A code change that neither fixes a bug nor adds a feature size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant