chore(api): Drop legacy app-centric tables and dead model classes#4676
chore(api): Drop legacy app-centric tables and dead model classes#4676jp-agenta wants to merge 10 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Railway Preview Environment
|
ab96500 to
96c348b
Compare
9f7bcee to
3d3750d
Compare
3d3750d to
fc900e5
Compare
b6d4a7d to
dcb998f
Compare
df6a36a to
c518250
Compare
c518250 to
70650cd
Compare
dcb998f to
91d979d
Compare
18a1424 to
67ae791
Compare
4356253 to
91d979d
Compare
a52e038 to
92530d5
Compare
91d979d to
13535b9
Compare
92530d5 to
903f73e
Compare
e66be2c to
01b03ff
Compare
8b4094e to
1a31273
Compare
53bc69b to
cbf5a5e
Compare
1a31273 to
7a49b6e
Compare
cbf5a5e to
b3b24ab
Compare
7a49b6e to
f39d1cc
Compare
b3b24ab to
4dd9f0f
Compare
Drops the 20 dead pre-project tables (apps/variants/bases, deployments, docker_images, templates, old environments, old testsets, old auto/human evaluations, ids_mapping) in both chains, deletes the 15 dead model classes, the unimported models/db/models.py shim and models/converters.py, and the dead create_deployment in db_manager_ee. The drop is the schema-parity fix for the legacy organization_id/workspace_id columns EE carried on these tables. deprecated_models scaffolding stays (historical migrations import it).
Alembic loads every version script at startup, and several historical migrations still imported the model classes deleted with the legacy table drop (create_free_plans imported AppDB, the evaluators/environments/ applications_workflow/projects data migrations imported the others). This broke the migration runner on boot. Freeze the final pre-drop shapes of the nine queried tables in deprecated_models.py on a dedicated DroppedBase (a separate base so extend_existing does not merge columns into the older partial shapes) and point the migrations at them. Verified all 175 version scripts across both chains now import cleanly.
The frozen classes keep their ForeignKey declarations, but projects/users/ folders are not in DroppedBase's metadata, so the ORM could not resolve FK targets when a data migration flushes an insert (55bdd2e9a465 adds a testset row). Stub tables with just the id column satisfy resolution; selects never needed it, which is why the import sweep passed.
DROP TABLE leaves enum types behind: app_type_enum and templatetype are orphaned by this revision's drops, nodetype and treetype were already orphaned when the nodes table was dropped. No live column or code uses any of the four; the schema dumps confirm zero references at head.
The nodes drop (cfa14a847972) left nodetype and treetype behind in both editions' tracing databases; no live column uses them (spans uses spantype, tracetype, otelspankind, otelstatuscode). New revision f3a4b5c6d7e8 on both tracing chains drops them.
…rity audit Every table gets the six lifecycle columns (created/updated/deleted x at and by_id), fully nullable, created_at with a server default; the LegacyLifecycleDBA mixin is retired (secrets moves to LifecycleDBA) and the per-table created_by_id nullable overrides become unnecessary. Lifecycle actor columns lose their FKs (organizations x3, api_keys, webhook_subscriptions) per the no-FKs-on-actors rule. Owning-scope FKs are added where missing: webhook_deliveries -> projects + webhook_subscriptions (event_id stays loose, events lives in the tracing database), secrets -> projects/organizations, subscriptions -> organizations, and meters retargets from subscriptions to organizations, all CASCADE. Membership user FKs get the CASCADE the models already declared; invitation user FK becomes SET NULL. Models and DB now agree everywhere; convention unit tests enforce the standard from now on. Migrations: 4f5a6b7c8d9e (OSS core), 5a6b7c8d9e0f (EE core), a4b5c6d7e8f9 (both tracing chains).
Drop pre-existing secrets/webhook_deliveries/subscriptions FKs by introspection before recreating them, matching the meters idiom. On an EE-origin database secrets_project_id_fkey already exists, so the bare create_foreign_key raised DuplicateObjectError mid-migration. Applied to both the EE (5a6b7c8d9e0f) and OSS (4f5a6b7c8d9e) lifecycle migrations.
… tests The lifecycle/FK migration retargeted meters.organization_id from subscriptions to organizations and dropped the old MeterDBE.subscription relationship, but the meters DAO still joinedload()s it for the Stripe report path - crashing fetch/dump and silently breaking usage reporting. Re-declare it as a viewonly nav on organization_id (one subscription per org) so both joinedload sites and _dbe_to_dto work unchanged. Also add __init__.py to the oss/ee unit/models test dirs so the duplicate-basename test_lifecycle_conventions.py modules import under distinct package paths.
…gration Migration 1c2d3e4f5a6b ran select(WorkspaceDB)/select(ProjectDB) against the live ORM models, which now declare deleted_at + audit columns added by the later lifecycle_and_fk_cleanup migration. At 1c2d3e4f5a6b those columns do not exist yet, so the SELECT failed with UndefinedColumnError: column workspaces.deleted_at does not exist, aborting the EE core migration on boot. Add frozen DeprecatedWorkspaceDB/DeprecatedProjectDB snapshots (schema as of 1c2d3e4f5a6b, no audit columns) and bind the data migration to them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t migration The same migration's create_default_project_memberships did select(WorkspaceMemberDB) against the live model, which now carries deleted_at + audit columns absent at revision 1c2d3e4f5a6b -> UndefinedColumnError: column workspace_members.deleted_at does not exist. Extend the frozen-model fix to DeprecatedWorkspaceMemberDB/DeprecatedProjectMemberDB so the whole migration binds to the schema as of its own revision. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
4dd9f0f to
5189159
Compare
Context
The pre-project app ecosystem (apps/variants/bases, deployments, docker images, templates, old environments, old testsets, the old auto- and human-evaluation tables, and the Mongo-migration artifact
ids_mapping) is dead code: its routers were deleted, its data was migrated to the new entities earlier in both migration chains, and the model classes were referenced only by an unimported re-export shim and unimported converters. EE also carries legacy nullableorganization_id/workspace_idcolumns (plusdeployments.cloud_map_service_idandauto_evaluator_configs.app_id) exclusively on these tables, so dropping them is the schema-parity fix for that drift. This is sequencing step 5 / parity step 2 ofdocs/designs/oss-ee-convergence/assessment-a-oss-multi-org.md.Changes
Two migrations, one per chain (the chains are independent), each running
DROP TABLE IF EXISTS … CASCADEover the same 20 tables, children before parents:human_evaluation_variants,human_evaluations_scenarios,human_evaluations,auto_evaluation_scenario_results,auto_evaluation_aggregated_results,auto_evaluation_evaluator_configs,auto_evaluation_scenarios,auto_evaluations,auto_evaluator_configs,environments_revisions,environments,app_variant_revisions,app_variants,bases,deployments,docker_images,templates,app_db,testsets,ids_mapping. Downgrade raises (irreversible; restore from backup). The migrations also drop the four enum types the table drops orphan (app_type_enum,templatetype, andnodetype/treetype, which the earlier nodes drop had already left behind); no live column or code uses them. The tracing chains get the same treatment: a new revisionf3a4b5c6d7e8in both editions dropsnodetype/treetypethere too (orphaned by the earlier nodes drop;spansusesspantype/tracetype/otelspankind/otelstatuscode, which stay).Code cleanup in the same PR:
oss/src/models/db_models.py(AppDB,DeploymentDB,VariantBaseDB,AppVariantDB,AppVariantRevisionsDB,AppEnvironmentDB,AppEnvironmentRevisionDB,TestsetDB,EvaluatorConfigDB,IDsMappingDB, and the five oldEvaluation*classes), along with their now-unused imports.oss/src/models/db/models.py(re-export shim) andoss/src/models/converters.pyhad zero importers; both files are deleted.db_manager_ee.create_deploymenthad no callers; deleted with itsDeploymentDBimport.deprecated_models.py/deprecated_transfer_models.pystay: historical data migrations import them, and deleting them would break replaying old chains.create_free_plansimportedAppDB; theevaluators,environments,applications_workflow, andprojectsdata migrations imported the others) now import frozen copies of the final pre-drop shapes, added todeprecated_models.pyon a dedicatedDroppedBase. Alembic loads every version script at startup, so without this the migration runner fails to boot onImportError.Not dropped: the new-architecture tables with similar names (
evaluation_scenarios,testset_artifacts,environment_*) are untouched; the old/new names differ (auto_evaluationsvsevaluation_runs,testsetsvstestset_artifacts).Lifecycle/FK standardization (per
docs/designs/oss-ee-convergence/db-integrity-audit.md, migrations4f5a6b7c8d9eOSS /5a6b7c8d9e0fEE /a4b5c6d7e8f9both tracing chains): every table gets the six lifecycle columns, fully nullable,created_atwith a server default; lifecycle actor FKs are dropped (organizations x3, api_keys, webhook_subscriptions); owning-scope FKs added (webhook_deliveries to projects + subscriptions, secrets to projects/organizations, subscriptions to organizations, meters retargeted from subscriptions to organizations, all CASCADE); membership user FKs get the CASCADE the models declared; invitation user FK becomes SET NULL. TheLegacyLifecycleDBAmixin is retired and models now agree with the database everywhere. New convention unit tests (oss/tests/pytest/unit/models/test_lifecycle_conventions.pyand the EE counterpart) enforce the standard.Tests / notes
ruff formatandruff checkpass; all 175 alembic version scripts across both chains verified to import cleanly (this is what the migration runner does on boot); no remaining references to the dropped classes outside the frozen scaffolding (legacyoss/tests/legacy/old_testsare not collected perpytest.initestpaths).🤖 Generated with Claude Code