Is there an existing issue for this?
Category of feature request
Analyzer
Problem statement
The Analyzer desktop app already tells users what they have (scripts, objects, complexity distribution) and how it connects — the Lineage tab lists every target table with its Upstream, Downstream, Written By and Read By, plus there's a "See lineage graph" view.
What it doesn't answer is the first question every migration team asks once the assessment is done: in what order do we migrate this, and what can we do in parallel?
Today that ordering is worked out by hand, by exporting the lineage data into a spreadsheet and tracing dependencies manually. On a 57-script project that's tedious. On a few-hundred-script estate it's error-prone enough that it tends to get skipped altogether — so teams migrate in effectively arbitrary order, discover a missing upstream table mid-sprint, and rework.
The dependency information required to answer the question is already computed and already on screen. What's missing is the layer that turns it into an execution plan.
Proposed Solution
Add a Migration Waves capability to the Analyzer, derived from the lineage graph it already builds. Two concepts:
Group — a set of scripts that transitively share at least one object. Groups have no dependencies on each other, so they are independently schedulable: different teams or sprints can take different groups in parallel.
Wave — the ordering within a group, by dependency depth. Wave 1 is scripts with no unmet dependency inside the batch (they read only objects that already exist, or nothing at all). Wave 2 is scripts consuming Wave 1's outputs, and so on. Waves within a group must be executed in order.
How this can be derived from the lineage graph that already exists:
- Add a script→script dependency edge wherever script A writes/creates an object that script B reads.
- Groups = connected components of the undirected form of that graph — scripts sharing any object land together.
- Waves = topological generations within each group. Ties broken deterministically (e.g. by name) so the same input always yields the same plan, which matters if the output is a customer deliverable.
- Group ordering — where a cross-group dependency does exist, order the groups relative to each other by topological sort at group level.
- Cycles — circular dependencies are common in legacy estates and shouldn't crash the plan or be silently mis-ordered. Isolate the cycle participants into their own explicitly flagged wave for manual resolution, and defer anything downstream of the cycle to a later wave. Being honest that a clean order doesn't exist is far more useful than guessing one.
- Pre-existing objects — objects that are read but never created by anything in the analyzed batch. These must already exist in the target before the consuming script can run. Surfacing them per-script gives a prerequisite checklist, and in practice this is one of the highest-value outputs: it pre-empts the most common class of migration failure, which is "we migrated it and it failed on a missing table."
Where it could surface, in rough priority order:
- A new Migration Waves tab, alongside Overview / Scripts / Objects & Functions / Lineage.
- Wave overlay on the existing lineage graph — color or swimlane the nodes by group and wave, so the ordering is visible in the view users already open via "See lineage graph".
- CSV / table export, flat, one row per script:
Group | Wave | Script | Pre-existing objects required | Notes, with Notes carrying the cycle flags. This is the artifact that gets pasted into a project plan or sprint board.
- CLI + JSON output, so the plan can feed external planning tooling instead of being locked inside the GUI.
Suggested scope: SQL first, as an MVP, since the lineage graph is already there for it. The approach is dialect-agnostic in principle — it operates on whatever lineage graph the Analyzer produces — so other sources could follow. Informatica would need one extra nesting level (Group → File → Wave of workflows) because workflows roll up under files.
Additional Context
Credit and origin of the idea. This is not my original idea, and the credit belongs elsewhere. The concept, and a working implementation of it, come from a Databricks-internal app built by Eri Adepoju, where migration waves are generated from Analyzer output and shipped to customers as part of the assessment deliverable. It has proven useful enough in that context that I think it belongs in Lakebridge itself, so that everyone using the Analyzer benefits rather than only those with access to an internal tool. I'm proposing the integration; the design credit is Eri's.
Why it matters in the field. The wave plan is consistently one of the most-referenced artifacts in an assessment handoff. It converts an inventory into something a delivery team can actually schedule, and the pre-existing-objects list prevents a predictable and expensive class of mid-migration failure.
Related existing issues. #29 (capture lineage between SQL files in a folder) and #220 (subcommand to generate lineage — closed, delivered lineage as a DOT export) are the foundation this would build on. #1157 uses a table dependency graph, but for job orchestration output rather than migration planning. As far as I can tell none of them propose sequencing or wave planning, so this shouldn't duplicate existing work — happy to be corrected if I've missed something.
Current UI, for reference. The SQL Analysis Report currently offers four tabs: Overview (report metadata, total scripts / lines / statements / object refs, complexity distribution, top statement categories and functions), Scripts, Objects & Functions, and Lineage. The Lineage tab lists each target table with Upstream, Downstream, Written By and Read By, filterable by "No upstream" and "Most downstreams", alongside a "See lineage graph" button. All the signal needed for ordering is present — nothing currently turns it into a sequence.
Environment observed. lakebridge@0.14.2, Analyzer desktop app, SQL analysis of a 57-script project with 28 tables reported in the Lineage tab.
Is there an existing issue for this?
Category of feature request
Analyzer
Problem statement
The Analyzer desktop app already tells users what they have (scripts, objects, complexity distribution) and how it connects — the Lineage tab lists every target table with its Upstream, Downstream, Written By and Read By, plus there's a "See lineage graph" view.
What it doesn't answer is the first question every migration team asks once the assessment is done: in what order do we migrate this, and what can we do in parallel?
Today that ordering is worked out by hand, by exporting the lineage data into a spreadsheet and tracing dependencies manually. On a 57-script project that's tedious. On a few-hundred-script estate it's error-prone enough that it tends to get skipped altogether — so teams migrate in effectively arbitrary order, discover a missing upstream table mid-sprint, and rework.
The dependency information required to answer the question is already computed and already on screen. What's missing is the layer that turns it into an execution plan.
Proposed Solution
Add a Migration Waves capability to the Analyzer, derived from the lineage graph it already builds. Two concepts:
Group — a set of scripts that transitively share at least one object. Groups have no dependencies on each other, so they are independently schedulable: different teams or sprints can take different groups in parallel.
Wave — the ordering within a group, by dependency depth. Wave 1 is scripts with no unmet dependency inside the batch (they read only objects that already exist, or nothing at all). Wave 2 is scripts consuming Wave 1's outputs, and so on. Waves within a group must be executed in order.
How this can be derived from the lineage graph that already exists:
Where it could surface, in rough priority order:
Group | Wave | Script | Pre-existing objects required | Notes, with Notes carrying the cycle flags. This is the artifact that gets pasted into a project plan or sprint board.Suggested scope: SQL first, as an MVP, since the lineage graph is already there for it. The approach is dialect-agnostic in principle — it operates on whatever lineage graph the Analyzer produces — so other sources could follow. Informatica would need one extra nesting level (Group → File → Wave of workflows) because workflows roll up under files.
Additional Context
Credit and origin of the idea. This is not my original idea, and the credit belongs elsewhere. The concept, and a working implementation of it, come from a Databricks-internal app built by Eri Adepoju, where migration waves are generated from Analyzer output and shipped to customers as part of the assessment deliverable. It has proven useful enough in that context that I think it belongs in Lakebridge itself, so that everyone using the Analyzer benefits rather than only those with access to an internal tool. I'm proposing the integration; the design credit is Eri's.
Why it matters in the field. The wave plan is consistently one of the most-referenced artifacts in an assessment handoff. It converts an inventory into something a delivery team can actually schedule, and the pre-existing-objects list prevents a predictable and expensive class of mid-migration failure.
Related existing issues. #29 (capture lineage between SQL files in a folder) and #220 (subcommand to generate lineage — closed, delivered lineage as a DOT export) are the foundation this would build on. #1157 uses a table dependency graph, but for job orchestration output rather than migration planning. As far as I can tell none of them propose sequencing or wave planning, so this shouldn't duplicate existing work — happy to be corrected if I've missed something.
Current UI, for reference. The SQL Analysis Report currently offers four tabs: Overview (report metadata, total scripts / lines / statements / object refs, complexity distribution, top statement categories and functions), Scripts, Objects & Functions, and Lineage. The Lineage tab lists each target table with Upstream, Downstream, Written By and Read By, filterable by "No upstream" and "Most downstreams", alongside a "See lineage graph" button. All the signal needed for ordering is present — nothing currently turns it into a sequence.
Environment observed.
lakebridge@0.14.2, Analyzer desktop app, SQL analysis of a 57-script project with 28 tables reported in the Lineage tab.