feat: implement Cloudreve to AsterDrive migration - #1
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthrough新增 Cloudreve 到 AsterDrive 的数据库迁移引擎、CLI 命令、分阶段迁移、检查点恢复、结果验证、SQLite 集成测试及使用文档，并调整实体映射与依赖声明。 Changes数据库迁移工具
Estimated code review effort: 4 (Complex) | ~75 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant MigrationEngine
participant CloudreveDB
participant AsterDriveDB
CLI->>MigrationEngine: 执行 check 或 migrate
MigrationEngine->>CloudreveDB: 加载源数据并检查兼容性
MigrationEngine->>AsterDriveDB: 校验 schema 与目标状态
MigrationEngine->>AsterDriveDB: 分阶段写入并保存 checkpoint
MigrationEngine-->>CLI: 输出 MigrationReport
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
src/migration/mod.rs (1)
254-263: 🚀 Performance & Scalability | 🔵 Trivial全量入内存 + 单事务写入，大库要留个心眼。
load对每张源表都.all(db)一次性拉进内存，随后所有阶段在同一个目标事务里写入。对小型实例没问题，但面对大规模 Cloudreve 部署，内存占用与目标库的长事务/锁持有时间都会成为瓶颈。若目标用户群里存在这种规模，建议按批分页读取与提交，或至少在文档里标注内存需求。🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/migration/mod.rs` around lines 254 - 263, Update SourceData::load and the migration flow that consumes it to avoid loading every source table with unbounded .all(db) calls and keeping all writes in one long transaction. Read and migrate records in bounded batches with periodic commits, preserving migration ordering and include_deleted behavior; alternatively, document the required memory and transaction-size limitations if batching cannot be introduced.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/cloudreve-entities/src/entities.rs`:
- Line 7: 为 entities 实体补充 PostgreSQL schema 绑定，更新 `cr::entities::Entity` 上的
SeaORM 表属性以显式指定 Cloudreve 使用的 schema，避免依赖连接默认 schema；同时保持 `src/migration/mod.rs`
中通过 `Database::connect` 查询该实体时能正确解析非 public schema，并补充对应的 PostgreSQL 集成测试。
In `@crates/cloudreve-entities/src/files.rs`:
- Around line 7-8: 保留 files::Model 中的 deleted_at 字段，并在文件迁移查询流程中对 files 应用
filter_deleted，与其他实体保持一致；确保 include_deleted 为 false 时排除已删除文件，避免其迁移到目标库。
---
Nitpick comments:
In `@src/migration/mod.rs`:
- Around line 254-263: Update SourceData::load and the migration flow that
consumes it to avoid loading every source table with unbounded .all(db) calls
and keeping all writes in one long transaction. Read and migrate records in
bounded batches with periodic commits, preserving migration ordering and
include_deleted behavior; alternatively, document the required memory and
transaction-size limitations if batching cannot be introduced.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 6e369702-3c4b-4b9f-a6fa-4469913354bc
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (23)
Cargo.tomlREADME.mdcrates/cloudreve-entities/src/dav_accounts.rscrates/cloudreve-entities/src/direct_links.rscrates/cloudreve-entities/src/entities.rscrates/cloudreve-entities/src/file_entities.rscrates/cloudreve-entities/src/files.rscrates/cloudreve-entities/src/fs_events.rscrates/cloudreve-entities/src/groups.rscrates/cloudreve-entities/src/metadata.rscrates/cloudreve-entities/src/nodes.rscrates/cloudreve-entities/src/oauth_clients.rscrates/cloudreve-entities/src/oauth_grants.rscrates/cloudreve-entities/src/passkeys.rscrates/cloudreve-entities/src/settings.rscrates/cloudreve-entities/src/shares.rscrates/cloudreve-entities/src/storage_policies.rscrates/cloudreve-entities/src/tasks.rscrates/cloudreve-entities/src/users.rssrc/lib.rssrc/main.rssrc/migration/mod.rssrc/migration/phases.rs
| #[sea_orm::model] | ||
| #[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)] | ||
| #[sea_orm(schema_name = "public", table_name = "entities")] | ||
| #[sea_orm(table_name = "entities")] |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n 'search_path|schema_name|CLOUDREVE_DATABASE_URL' src crates Cargo.tomlRepository: AsterCommunity/AsterDriveMigration
Length of output: 236
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== entities.rs ==\n'
cat -n crates/cloudreve-entities/src/entities.rs
printf '\n== migration/mod.rs ==\n'
cat -n src/migration/mod.rs
printf '\n== sea_orm / schema / search_path / database url references ==\n'
rg -n 'search_path|schema_name|Schema::|Database::connect|CLOUDREVE_DATABASE_URL|set_schema|with_schema|schema' src crates Cargo.tomlRepository: AsterCommunity/AsterDriveMigration
Length of output: 35564
🌐 Web query:
SeaORM #[sea_orm(schema_name = "...")] PostgreSQL default schema behavior table_name search_path
💡 Result:
In SeaORM, the #[sea_orm(schema_name = "...", table_name = "...")] attribute explicitly binds an entity to a specific PostgreSQL schema [1][2]. When schema_name is provided, SeaORM generates SQL queries that qualify the table name with the specified schema (e.g., "schema_name"."table_name") [3][4]. The interaction between schema_name and the PostgreSQL search_path is as follows: 1. Explicit Schema Binding: By using the schema_name attribute, you are telling SeaORM to always use that specific schema for the entity, regardless of the active search_path [3][4]. 2. PostgreSQL search_path: The search_path (configured via ConnectOptions::set_schema_search_path [5] or the database connection string [6]) is a PostgreSQL-level mechanism that tells the database which schemas to look into when an identifier is unqualified [6][7]. 3. Behavior: If you define an entity with a schema_name, SeaORM treats it as a fully qualified identifier. Consequently, the search_path has no effect on these specific entities because the query already includes the schema prefix [3]. If you do not specify a schema_name, SeaORM will not prefix the table, and PostgreSQL will rely on the search_path to resolve the table location [6]. Note that there are known edge cases, particularly with schema synchronization tools, where some generated SQL (such as for indexes or constraints) may not always consistently use the qualified schema name unless explicitly handled or configured [3]. For managing PostgreSQL schema configuration in SeaORM, you can use: - ConnectOptions: opt.set_schema_search_path("my_schema".into()) [5] - Connection String: postgres://user:pass@host/db?options[search_path]=my_schema [6]
Citations:
- 1: https://www.sea-ql.org/SeaORM/docs/generate-entity/entity-format/
- 2: https://www.sea-ql.org/SeaORM/docs/1.1.x/generate-entity/entity-structure/
- 3: schema-sync creates indexes against unqualified table names for entities in non-default PostgreSQL schemas SeaQL/sea-orm#3084
- 4: https://github.com/SeaQL/sea-orm/blob/63e6e491/sea-orm-macros/src/derives/entity_model.rs
- 5: https://github.com/SeaQL/sea-orm/blob/master/src/database/mod.rs
- 6: Use the
search_path=schema_namesuffix in PG connection string SeaQL/sea-orm#1272 - 7: https://www.sea-ql.org/sea-orm-cookbook/016-search-path.html
🌐 Web query:
SeaORM schema_name PostgreSQL default search_path public unqualified table name entity model
💡 Result:
In SeaORM, managing PostgreSQL schemas and unqualified table names is handled primarily through the schema_name attribute in the entity model and connection configuration [1][2][3]. To map an entity to a specific PostgreSQL schema, you use the schema_name attribute within the #[sea_orm(...)] derive macro [1][2]. This ensures the generated SQL appropriately qualifies the table name with the schema (e.g., "schema_name"."table_name") [4][2]. For connection-level handling, specifically when you want to avoid manual schema qualification or rely on the PostgreSQL search_path, you have two main options: 1. Connection Options: You can set the default PostgreSQL schema search path programmatically when establishing the database connection [5][6]. let mut opt = ConnectOptions::new("postgres://user:pass@localhost/db".to_owned); opt.set_schema_search_path("my_schema".into); let db = Database::connect(opt).await?; 2. Connection String: You can append the options[search_path] parameter to your connection string, which allows SQLx and SeaORM to honor the search path without requiring explicit schema qualification in your code [6]. postgres://user:pass@localhost/db?options[search_path]=my_schema Note that while schema_name explicitly defines the schema for an entity, relying on search_path can sometimes lead to issues with certain automated tools or generated queries (such as schema-sync creating indexes with unqualified table names if not handled correctly) [7][4]. If you explicitly define schema_name in your entity model, SeaORM typically generates schema-qualified identifiers [4][2]. If you require specific migration behaviors for non-public schemas, you can pass the -s <schema_name> flag to sea-orm-cli or set the DATABASE_SCHEMA environment variable [5][8].
Citations:
- 1: https://www.sea-ql.org/SeaORM/docs/generate-entity/entity-format/
- 2: https://github.com/SeaQL/sea-orm/blob/63e6e491/src/entity/base_entity.rs
- 3: https://www.sea-ql.org/SeaORM/docs/1.1.x/generate-entity/entity-format/
- 4: schema-sync creates indexes against unqualified table names for entities in non-default PostgreSQL schemas SeaQL/sea-orm#3084
- 5: Run migrations on PostgreSQL schema SeaQL/sea-orm#1056
- 6: Use the
search_path=schema_namesuffix in PG connection string SeaQL/sea-orm#1272 - 7: https://www.sea-ql.org/sea-orm-cookbook/016-search-path.html
- 8: [Migration] Create table with PostgreSQL schema name SeaQL/sea-orm#1029
补上 PostgreSQL schema 绑定
crates/cloudreve-entities/src/entities.rs:7 现在只有 table_name = "entities"；src/migration/mod.rs 里又是直接 Database::connect(url) 后跑 cr::entities::Entity::find().all(db)，没有任何 schema_name / search_path 配置。这样源表解析全靠连接默认 schema，非 public 部署会读错表或直接失败。要么显式固定 Cloudreve schema，要么把 search_path 配进连接串，再补 PostgreSQL 集成测试。
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/cloudreve-entities/src/entities.rs` at line 7, 为 entities 实体补充
PostgreSQL schema 绑定，更新 `cr::entities::Entity` 上的 SeaORM 表属性以显式指定 Cloudreve 使用的
schema，避免依赖连接默认 schema；同时保持 `src/migration/mod.rs` 中通过 `Database::connect`
查询该实体时能正确解析非 public schema，并补充对应的 PostgreSQL 集成测试。
| #[sea_orm(table_name = "files")] | ||
| pub struct Model { |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
不要移除 files.deleted_at，否则删除文件也会被迁移。
src/migration/mod.rs:235-285 对其他实体调用了 filter_deleted，但 files 直接使用查询结果；移除该字段后，include_deleted = false 无法过滤已删除文件，目标库会出现本应排除的记录。请保留字段，并对 files 同样应用 filter_deleted。
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/cloudreve-entities/src/files.rs` around lines 7 - 8, 保留 files::Model
中的 deleted_at 字段，并在文件迁移查询流程中对 files 应用 filter_deleted，与其他实体保持一致；确保
include_deleted 为 false 时排除已删除文件，避免其迁移到目标库。
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/migration/phases.rs`:
- Around line 455-491: 在迁移流程中更新标签关联逻辑：除现有的 tags 去重外，围绕
entity_properties::ActiveModel 插入增加按 (entity_type, entity_id, tag_id)
去重，确保同一实体复用同一 tag_id 时不会重复插入 system.tags 关联。仅首次出现该组合时插入并递增 migrated_properties 和
migrated_tag_assignments，重复关联应直接跳过。
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: d9f558bf-e792-4885-a64d-5de52a8434af
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (7)
Cargo.tomlREADME.mddocs/cloudreve-to-asterdrive-field-mapping.mddocs/migration-architecture.mdsrc/main.rssrc/migration/mod.rssrc/migration/phases.rs
🚧 Files skipped from review as they are similar to previous changes (2)
- src/main.rs
- Cargo.toml
| let tag_id = match tags.get(&(owner_user_id, normalized_name.clone())) { | ||
| Some(tag_id) => *tag_id, | ||
| None => { | ||
| let tag = ad::tags::ActiveModel { | ||
| scope_type: Set("personal".to_string()), | ||
| owner_user_id: Set(Some(owner_user_id)), | ||
| team_id: Set(None), | ||
| name: Set(name), | ||
| normalized_name: Set(normalized_name.clone()), | ||
| color: Set(target_tag_color(&metadata.value)), | ||
| sort_order: Set(0), | ||
| created_at: Set(metadata.created_at), | ||
| updated_at: Set(metadata.updated_at), | ||
| ..Default::default() | ||
| } | ||
| .insert(transaction) | ||
| .await | ||
| .wrap_err_with(|| format!("migrate tag metadata {}", metadata.id))?; | ||
| tags.insert((owner_user_id, normalized_name), tag.id); | ||
| report.migrated_tags += 1; | ||
| tag.id | ||
| } | ||
| }; | ||
| ad::entity_properties::ActiveModel { | ||
| entity_type: Set(entity_type.to_string()), | ||
| entity_id: Set(entity_id), | ||
| namespace: Set("system.tags".to_string()), | ||
| name: Set(tag_id.to_string()), | ||
| value: Set(None), | ||
| ..Default::default() | ||
| } | ||
| .insert(transaction) | ||
| .await | ||
| .wrap_err_with(|| format!("attach migrated tag for metadata {}", metadata.id))?; | ||
| report.migrated_properties += 1; | ||
| report.migrated_tag_assignments += 1; | ||
| continue; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# 确认 AD entity_properties 是否对 (entity_type, entity_id, namespace, name) 建唯一约束/索引
fd -t f . crates/asterdrive-entities/src | xargs rg -nP 'entity_properties|unique|index' -l
rg -nP -C3 'unique|index_name|primary_key' $(fd 'entity_propert' crates/asterdrive-entities/src)Repository: AsterCommunity/AsterDriveMigration
Length of output: 2670
标签关联也要去重，不然大小写变体会把迁移撞回去
tags 只去重了标签行，system.tags 这条 entity_properties 还是每条 metadata 都在插。entity_properties 对 (entity_type, entity_id, namespace, name) 有唯一键，同一文件上的 tag:Important 和 tag:important 会复用同一个 tag_id，第二次写入就直接撞唯一约束，整笔迁移回滚。这里得再按 (entity_type, entity_id, tag_id) 去重。
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/migration/phases.rs` around lines 455 - 491, 在迁移流程中更新标签关联逻辑：除现有的 tags
去重外，围绕 entity_properties::ActiveModel 插入增加按 (entity_type, entity_id, tag_id)
去重，确保同一实体复用同一 tag_id 时不会重复插入 system.tags 关联。仅首次出现该组合时插入并递增 migrated_properties 和
migrated_tag_assignments，重复关联应直接跳过。
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/main.rs (1)
24-26: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win给
source_url/target_url/default_password加上hide_env_values = true
src/main.rs:24-26,38-39这三个env参数会把已设置的值暴露到--help；direct_link_secret已经隐藏了，这里也该统一处理，避免泄露数据库连接串和默认密码。🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main.rs` around lines 24 - 26, Update the argument definitions for source_url, target_url, and default_password in the CLI arguments struct to include hide_env_values = true alongside their existing env settings. Match the existing direct_link_secret configuration and leave unrelated arguments unchanged.
🧹 Nitpick comments (2)
README.md (1)
59-59: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win补清
--allow-non-empty-target的边界说明。 README 现在只说它会跳过空库保护，但没说清楚它不是合并模式：正常迁移还是一堆insert，已有记录不会被覆盖，撞上主键/唯一约束就会失败。--resume也只是同一run_id的中断续跑，不是“对非空目标重复执行”的通用语义。🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` at line 59, 完善 README 中 --allow-non-empty-target 的说明，明确它仅跳过目标核心表必须为空的保护，不是合并或覆盖模式；迁移仍执行插入，已有记录不会被覆盖，遇到主键或唯一约束冲突可能失败。同时说明 --resume 仅用于续跑同一 run_id 的中断迁移，不支持对非空目标重复执行。src/main.rs (1)
86-94: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
emit_report建议先打印再写文件，并修正 bail 消息。当前顺序是先写文件再打印，若
write_json_report失败，用户在 stdout 上什么都看不到。建议调换顺序，让用户始终能看到报告内容。此外，行 92 的 bail 消息说 "inspect the JSON report"，但如果用户未提供--report-path，根本没有 JSON 文件可查，消息具有误导性。♻️ 建议修复：调换顺序并修正消息
fn emit_report(report_path: Option<&Path>, report: &MigrationReport) -> Result<()> { - if let Some(path) = report_path { - write_json_report(path, report)?; - } println!("{report}"); + if let Some(path) = report_path { + write_json_report(path, report)?; + } if report.validation.performed && !report.validation.passed { - bail!("migration committed but post-migration validation failed; inspect the JSON report"); + bail!("migration committed but post-migration validation failed; review the report above{}", + report_path.map(|_| "").unwrap_or(" (re-run with --report-path to save a JSON report)")); } Ok(()) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main.rs` around lines 86 - 94, Update emit_report to print the MigrationReport before calling write_json_report, ensuring report output appears even when file writing fails. Also revise the validation-failure bail message so it does not imply a JSON report exists when no report_path was provided, while retaining the existing failure behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/cloudreve-to-asterdrive-field-mapping.md`:
- Around line 520-538: 更新“JSON 报告字段”表，补充 MigrationReport 中缺失的
schema_version、generated_at、dry_run、warnings 和 skipped 字段说明；保留现有 skipped_by_type
与 skipped_objects 行，并确保表格完整覆盖报告结构体中的字段。
---
Outside diff comments:
In `@src/main.rs`:
- Around line 24-26: Update the argument definitions for source_url, target_url,
and default_password in the CLI arguments struct to include hide_env_values =
true alongside their existing env settings. Match the existing
direct_link_secret configuration and leave unrelated arguments unchanged.
---
Nitpick comments:
In `@README.md`:
- Line 59: 完善 README 中 --allow-non-empty-target
的说明，明确它仅跳过目标核心表必须为空的保护，不是合并或覆盖模式；迁移仍执行插入，已有记录不会被覆盖，遇到主键或唯一约束冲突可能失败。同时说明 --resume
仅用于续跑同一 run_id 的中断迁移，不支持对非空目标重复执行。
In `@src/main.rs`:
- Around line 86-94: Update emit_report to print the MigrationReport before
calling write_json_report, ensuring report output appears even when file writing
fails. Also revise the validation-failure bail message so it does not imply a
JSON report exists when no report_path was provided, while retaining the
existing failure behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 3dfb2848-af0d-46c8-860a-eb1d697fc4f2
📒 Files selected for processing (7)
README.mddocs/cloudreve-to-asterdrive-field-mapping.mddocs/migration-architecture.mdsrc/main.rssrc/migration/checkpoint.rssrc/migration/mod.rssrc/migration/phases.rs
🚧 Files skipped from review as they are similar to previous changes (3)
- docs/migration-architecture.md
- src/migration/mod.rs
- src/migration/phases.rs
| ## 16. JSON 报告字段 | ||
|
|
||
| 通过 `--report-path <path>` 可为 `check` 或 `migrate` 输出 `schema_version=1` 的 JSON 报告。 | ||
|
|
||
| | 报告字段 | 内容 | | ||
| |---|---| | ||
| | `source_*` | Cloudreve 各类源对象数量 | | ||
| | `migrated_*` | 本次写入 AD 的各类对象数量 | | ||
| | `skipped_by_type` | 按 `file`、`blob`、`share`、`direct_link` 等类型聚合的跳过数量 | | ||
| | `skipped_objects` | 每条跳过记录的对象类型、Cloudreve source ID 和明确原因 | | ||
| | `mappings` | policy、policy group、user、folder、blob、file、share、task 的排序 source ID -> target ID | | ||
| | `tag_assignments` | Cloudreve metadata ID、源 file/folder ID、AD entity ID、AD tag ID 和标签名 | | ||
| | `direct_links` | Cloudreve direct-link/file ID、AD file ID、新 URL、原名称、下载次数和限速 | | ||
| | `validation` | 是否执行/通过，以及每项检查的 expected、actual 和失败信息 | | ||
| | `run_id` | checkpoint run ID；未指定时由迁移工具生成 UUID | | ||
| | `resumed` | 本次执行是否从已有 checkpoint 恢复 | | ||
| | `completed_stages` | 已原子提交完成的迁移阶段列表 | | ||
|
|
||
| 当前提交后校验覆盖核心表增量数量、导入任务是否全为终态且无 lease、`system.tags` 关联是否存在、`cloudreve.direct_links` 属性中的 URL 是否与报告一致。报告不会保存数据库密码、存储密钥或 Cloudreve task private state，但会包含新 direct-link URL，因此必须限制报告文件访问权限。 |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
JSON 报告字段表漏了五个字段。
对照 MigrationReport 结构体，schema_version（正文提到但未列入表格）、generated_at、dry_run、warnings、skipped（总跳过数，表里只列了 skipped_by_type 和 skipped_objects）都没出现在表格中。用户照着文档解析报告会漏读这些字段。
📝 建议补充的表格行
| 报告字段 | 内容 |
|---|---|
+| `schema_version` | 报告 schema 版本，当前为 `1` |
+| `generated_at` | 报告生成时间（UTC） |
| `source_*` | Cloudreve 各类源对象数量 |
| `migrated_*` | 本次写入 AD 的各类对象数量 |
+| `skipped` | 跳过对象总数 |
+| `dry_run` | 是否为 dry-run 模式 |
+| `warnings` | 迁移过程中产生的警告信息列表 |
| `skipped_by_type` | 按 `file`、`blob`、`share`、`direct_link` 等类型聚合的跳过数量 |
| `skipped_objects` | 每条跳过记录的对象类型、Cloudreve source ID 和明确原因 |
| `mappings` | policy、policy group、user、folder、blob、file、share、task 的排序 source ID -> target ID |
| `tag_assignments` | Cloudreve metadata ID、源 file/folder ID、AD entity ID、AD tag ID 和标签名 |
| `direct_links` | Cloudreve direct-link/file ID、AD file ID、新 URL、原名称、下载次数和限速 |
| `validation` | 是否执行/通过，以及每项检查的 expected、actual 和失败信息 |
| `run_id` | checkpoint run ID；未指定时由迁移工具生成 UUID |
| `resumed` | 本次执行是否从已有 checkpoint 恢复 |
| `completed_stages` | 已原子提交完成的迁移阶段列表 |📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ## 16. JSON 报告字段 | |
| 通过 `--report-path <path>` 可为 `check` 或 `migrate` 输出 `schema_version=1` 的 JSON 报告。 | |
| | 报告字段 | 内容 | | |
| |---|---| | |
| | `source_*` | Cloudreve 各类源对象数量 | | |
| | `migrated_*` | 本次写入 AD 的各类对象数量 | | |
| | `skipped_by_type` | 按 `file`、`blob`、`share`、`direct_link` 等类型聚合的跳过数量 | | |
| | `skipped_objects` | 每条跳过记录的对象类型、Cloudreve source ID 和明确原因 | | |
| | `mappings` | policy、policy group、user、folder、blob、file、share、task 的排序 source ID -> target ID | | |
| | `tag_assignments` | Cloudreve metadata ID、源 file/folder ID、AD entity ID、AD tag ID 和标签名 | | |
| | `direct_links` | Cloudreve direct-link/file ID、AD file ID、新 URL、原名称、下载次数和限速 | | |
| | `validation` | 是否执行/通过，以及每项检查的 expected、actual 和失败信息 | | |
| | `run_id` | checkpoint run ID；未指定时由迁移工具生成 UUID | | |
| | `resumed` | 本次执行是否从已有 checkpoint 恢复 | | |
| | `completed_stages` | 已原子提交完成的迁移阶段列表 | | |
| 当前提交后校验覆盖核心表增量数量、导入任务是否全为终态且无 lease、`system.tags` 关联是否存在、`cloudreve.direct_links` 属性中的 URL 是否与报告一致。报告不会保存数据库密码、存储密钥或 Cloudreve task private state，但会包含新 direct-link URL，因此必须限制报告文件访问权限。 | |
| ## 16. JSON 报告字段 | |
| 通过 `--report-path <path>` 可为 `check` 或 `migrate` 输出 `schema_version=1` 的 JSON 报告。 | |
| | 报告字段 | 内容 | | |
| |---|---| | |
| | `schema_version` | 报告 schema 版本，当前为 `1` | | |
| | `generated_at` | 报告生成时间（UTC） | | |
| | `source_*` | Cloudreve 各类源对象数量 | | |
| | `migrated_*` | 本次写入 AD 的各类对象数量 | | |
| | `skipped` | 跳过对象总数 | | |
| | `dry_run` | 是否为 dry-run 模式 | | |
| | `warnings` | 迁移过程中产生的警告信息列表 | | |
| | `skipped_by_type` | 按 `file`、`blob`、`share`、`direct_link` 等类型聚合的跳过数量 | | |
| | `skipped_objects` | 每条跳过记录的对象类型、Cloudreve source ID 和明确原因 | | |
| | `mappings` | policy、policy group、user、folder、blob、file、share、task 的排序 source ID -> target ID | | |
| | `tag_assignments` | Cloudreve metadata ID、源 file/folder ID、AD entity ID、AD tag ID 和标签名 | | |
| | `direct_links` | Cloudreve direct-link/file ID、AD file ID、新 URL、原名称、下载次数和限速 | | |
| | `validation` | 是否执行/通过，以及每项检查的 expected、actual 和失败信息 | | |
| | `run_id` | checkpoint run ID；未指定时由迁移工具生成 UUID | | |
| | `resumed` | 本次执行是否从已有 checkpoint 恢复 | | |
| | `completed_stages` | 已原子提交完成的迁移阶段列表 | | |
| 当前提交后校验覆盖核心表增量数量、导入任务是否全为终态且无 lease、`system.tags` 关联是否存在、`cloudreve.direct_links` 属性中的 URL 是否与报告一致。报告不会保存数据库密码、存储密钥或 Cloudreve task private state，但会包含新 direct-link URL，因此必须限制报告文件访问权限。 |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/cloudreve-to-asterdrive-field-mapping.md` around lines 520 - 538,
更新“JSON 报告字段”表，补充 MigrationReport 中缺失的
schema_version、generated_at、dry_run、warnings 和 skipped 字段说明；保留现有 skipped_by_type
与 skipped_objects 行，并确保表格完整覆盖报告结构体中的字段。
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/migration/phases.rs (1)
300-314: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift文件迁移阶段全量加载 entities 和 file_entities 到内存
migrate_files一次性把源库的entities和file_entities全部拉进内存。blob 阶段已经做了分批处理，文件阶段却没有——大型 Cloudreve 实例（百万级文件 + 千万级 entities）跑下来直接 OOM 不是开玩笑的。建议参照
migrate_blobs_batched的分批策略，按 file ID 区间分批加载关联数据，避免单次查询撑爆内存。🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/migration/phases.rs` around lines 300 - 314, Update migrate_files to avoid loading all entities and file_entities at once: mirror migrate_blobs_batched’s batching strategy and query source records in file-ID ranges, processing and releasing each batch before fetching the next. Preserve the include_deleted filtering and existing migration/report behavior while ensuring memory usage remains bounded for large databases.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/migration/phases.rs`:
- Around line 352-355: 在计算 primary_entity_id 的回退逻辑中，保留 file.primary_entity 存在但不在
blob_mappings 时回退到 version_entities.last() 的行为，并在该回退发生时记录一条 warning，明确说明原
primary entity 无对应 blob 且当前版本已被替换。修改 primary_entity_id 附近的处理，避免无效或不存在的
primary_entity 触发无关警告。
---
Nitpick comments:
In `@src/migration/phases.rs`:
- Around line 300-314: Update migrate_files to avoid loading all entities and
file_entities at once: mirror migrate_blobs_batched’s batching strategy and
query source records in file-ID ranges, processing and releasing each batch
before fetching the next. Preserve the include_deleted filtering and existing
migration/report behavior while ensuring memory usage remains bounded for large
databases.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 92195ceb-a99d-4b32-88b6-ae1c23925b36
📒 Files selected for processing (7)
README.mddocs/cloudreve-to-asterdrive-field-mapping.mddocs/migration-architecture.mdsrc/main.rssrc/migration/checkpoint.rssrc/migration/mod.rssrc/migration/phases.rs
🚧 Files skipped from review as they are similar to previous changes (5)
- src/main.rs
- docs/cloudreve-to-asterdrive-field-mapping.md
- docs/migration-architecture.md
- README.md
- src/migration/mod.rs
| let primary_entity_id = file | ||
| .primary_entity | ||
| .filter(|id| blob_mappings.contains_key(id)) | ||
| .or_else(|| version_entities.last().map(|entity| entity.id)); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
primary entity 回退时静默吞掉了信息
file.primary_entity 存在但不在 blob_mappings 中时，代码直接回退到 version_entities.last()，没记录任何 warning。用户根本不知道这个文件的"当前版本"被换了——版本语义可能就此错乱。
🔧 建议在回退时补一条 warning
- let primary_entity_id = file
- .primary_entity
- .filter(|id| blob_mappings.contains_key(id))
- .or_else(|| version_entities.last().map(|entity| entity.id));
+ let primary_entity_id = match file.primary_entity {
+ Some(id) if blob_mappings.contains_key(&id) => Some(id),
+ Some(id) => {
+ report.warnings.push(format!(
+ "file {} primary entity {} was not migrated, falling back to latest version",
+ file.id, id
+ ));
+ version_entities.last().map(|entity| entity.id)
+ }
+ None => version_entities.last().map(|entity| entity.id),
+ };📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| let primary_entity_id = file | |
| .primary_entity | |
| .filter(|id| blob_mappings.contains_key(id)) | |
| .or_else(|| version_entities.last().map(|entity| entity.id)); | |
| let primary_entity_id = match file.primary_entity { | |
| Some(id) if blob_mappings.contains_key(&id) => Some(id), | |
| Some(id) => { | |
| report.warnings.push(format!( | |
| "file {} primary entity {} was not migrated, falling back to latest version", | |
| file.id, id | |
| )); | |
| version_entities.last().map(|entity| entity.id) | |
| } | |
| None => version_entities.last().map(|entity| entity.id), | |
| }; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/migration/phases.rs` around lines 352 - 355, 在计算 primary_entity_id
的回退逻辑中，保留 file.primary_entity 存在但不在 blob_mappings 时回退到 version_entities.last()
的行为，并在该回退发生时记录一条 warning，明确说明原 primary entity 无对应 blob 且当前版本已被替换。修改
primary_entity_id 附近的处理，避免无效或不存在的 primary_entity 触发无关警告。
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/migration/phases.rs (1)
196-207: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win你小子这个父目录存在性判断是拿
Vec硬扫的
while里套for，for里又对source.folders做.any()全表扫描——深层目录结构下就是妥妥的 O(n²)~O(n³)。一次性把 folder id 塞进HashSet<i64>再查，费不了你几行代码，大盘迁移时别让它在这儿卡到天荒地老。♻️ 预算一个 folder id 集合
let folder_ids: std::collections::HashSet<i64> = source.folders.iter().filter(|f| f.r#type == 1).map(|f| f.id).collect();if parent.is_some_and(|parent_id| { - source - .folders - .iter() - .any(|file| file.id == parent_id && file.r#type == 1) - && !context.folders.contains_key(&parent_id) + folder_ids.contains(&parent_id) && !context.folders.contains_key(&parent_id) }) {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/migration/phases.rs` around lines 196 - 207, 在迁移阶段围绕 pending 处理逻辑预先构建仅包含目录类型 folder id 的 HashSet，并将 `source.folders.iter().any(...)` 替换为该集合的 O(1) 成员查询；保留 `context.folders.contains_key` 判断及现有父目录筛选行为不变。
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/migration/phases.rs`:
- Around line 196-207: 在迁移阶段围绕 pending 处理逻辑预先构建仅包含目录类型 folder id 的 HashSet，并将
`source.folders.iter().any(...)` 替换为该集合的 O(1) 成员查询；保留
`context.folders.contains_key` 判断及现有父目录筛选行为不变。
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f5d9e1af-a8de-4d22-8310-742269be8551
📒 Files selected for processing (6)
README.mddocs/cloudreve-to-asterdrive-field-mapping.mddocs/migration-architecture.mdsrc/main.rssrc/migration/mod.rssrc/migration/phases.rs
🚧 Files skipped from review as they are similar to previous changes (4)
- src/main.rs
- README.md
- docs/cloudreve-to-asterdrive-field-mapping.md
- docs/migration-architecture.md
|
I wil review this pr soon |
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
…tching - add list, status, report, abort, cleanup, and resume CLI commands - export capability-free CSV source-to-target mapping reports - support aborting failed or running runs and cleaning completed metadata - add transactional blob and file batch cursors for resumable migrations - report committed batch elapsed time, throughput, and row-based ETA - add local storage reuse, verification, copy resume, and compensation coverage - expand migration flow, checkpoint, report, and CLI parsing tests - document post-migration checkpoint operations and progress metrics
AptS-1738
added
documentation
Switch from the in-tree `crates/asterdrive-entities` workspace member to the canonical `aster_drive_model` and `aster_drive_schema_migration` packages sourced directly from the AsterDrive git repository. Upgrade sea-orm to the stable2.0.0 release at the same time. - Remove `crates/asterdrive-entities` crate and all40+ generated entity files; source models from `aster_drive_model` instead - Add `aster_drive_schema_migration` (aliased to avoid name collision with this package's own migration library) for test target schema setup - Bump `sea-orm` from `2.0.0-rc.42` → `2.0.0` in root and `crates/cloudreve-entities` - Bump `serde` from `1.0.228` → `1.0.229` - Update `cloudreve-entities` `belongs_to` relation fields from `HasOne<>` to the correct `BelongsTo<>` type introduced in sea-orm 2.0stable, using `BelongsTo<Option<…>>` where the FK is nullable - Replace manual `create_table` calls in test fixtures with `aster_drive_schema_migration::Migrator::up` for target schema setup - Rename all entity references in tests from plural snake_case (`ad::users::Entity`) to singular (`ad::user::Entity`) to match upstream naming convention - Replace raw string comparisons for `status`, `kind`, `role`, and `driver_type` fields with typed enums from `aster_drive_model::types` - Update docs to reflect the new external dependency structure
… `cloudreve_schema` - Rename crate directory from `crates/cloudreve-entities` to `crates/cloudreve_schema` - Update package name in `Cargo.toml` from `cloudreve-entities` to `cloudreve_schema` - Update workspace member and dependency references in root `Cargo.toml` - Replace `use cloudreve_entities as cr` import alias with full `cloudreve_schema::` path throughout migration code - Replace `useaster_drive_model::entities as ad` alias with `use aster_drive_model as aster_drive_schema` and update all call sites - Update documentation reference in `docs/cloudreve-to-asterdrive-field-mapping.md` - Rename test function `parses_and_normalizes_cloudreve_tags_for_ad` to `parses_and_normalizes_cloudreve_tags_for_aster_drive`
…igration module - Remove all TUI interface code including screens, widgets, event handlers, and app state - Remove ratatui and crossterm dependencies from Cargo.toml - Remove CLI interface commands: config management, help, status, reset_password - Remove workspace feature flags (cli, tui, full) - Split monolithic migration/mod.rs into focused modules: engine, local_storage, model, report, validation - Move migration integration tests from src/migration/tests/ to top-level tests/ - Rename test fixtures module to tests/support/mod.rs - Remove resumes_local_copy_from_matching_temporary_file unit test (now covered by integration tests) - Update test imports to use aster_drive_migration public API
Extract schema-independent migration contracts into separate crates and remove physical object copying from the migration tool's responsibility: - Create `aster_drive_migration_core` crate with source-neutral domain types, conversion contracts, and ordered-stage state rules - Create `aster_drive_writer` crate encapsulating AsterDrive database writes for resolved migration values - Create `cloudreve_adapter` crate with Cloudreve-to-domain converters for policies, groups, users, and folders - Move object byte transfer outside migration scope: compatible storage is reused by reference; cross-root/bucket transfers handled by external infrastructure tools (rsync, rclone, snapshots) - Remove `--storage-mode copy-local` and related CLI flags; retain `--verify-local-storage` for final-path readability checks - Introduce `RunStatus` enum and `StagePlan` for explicit migration-run state transitions and stage ordering - Replace string-based stage/status checks with type-safe state rules that enforce terminal boundaries and resumption constraints - Add workspace-level lint configuration enforcing unsafe-op documentation and cast safety - Update documentation to clarify object-storage boundary: migration handles metadata and references, external tools handle byte movement - Preserve all integration tests; add new test verifying completed runs are terminal and cleanup removes metadata - Adjust profile settings to prioritize dev build speed: workspace code at O0, dependencies at O1
- Add `MigrationBlob`, `MigrationFile`, and `MigrationFileVersion` domain types to migration_core - Implement `write_blob` and `write_file` methods in AsterDriveWriter with atomic version insertion - Add converters for `CloudreveBlobRecord` and `CloudreveFileRecord` with validation and version ordering - Move `mime_guess` dependency from root to writer crate where MIME classification occurs - Delegate MIME type and file classification to AsterDrive's ActiveModelBehavior (Forge crate) - Skip Cloudreve thumbnails (type=1) since they lack AD's processor/version cache contract - Calculate blob reference counts from distinct file-blob pairs, excluding symbolic and non-file entities - Add comprehensive tests for writer atomicity, blob association counting, and version handling - Update field mapping documentation to clarify current-entity size derivation and thumbnail handling - Add clippy deny rules for unwrap/panic/expect in all migration crates (disabled in tests) - Remove obsolete `file_classification` and `opaque_blob_key` helper functions from model.rs - Ensure file and version insertion remains transactional when a referenced blob is missing
Add complete share migration pipeline from Cloudreve to AsterDrive with capability-based security and download limit tracking. Core changes: - Add `MigrationShare` and `MigrationShareTarget` domain types to migration_core with file/folder discrimination - Add `ResolvedShare` and `write_share` to aster_drive_writer with UUID v4 simple token generation (32-char hex capability) - Implement `CloudreveShareRecord` converter with file/folder target type detection, download limit computation from remaining+consumed counters, and validation for deleted/orphaned/negative-counter shares - Add share migration phase with converter integration, user/target resolution, Argon2 password hashing, and source-to-target ID mapping - Add preflight validation for negative counters and download limit overflow - Add duplicate active share target warning when multiple non-expired shares exist for same owner/resource pair - Rename `hash_password` to `hash_argon2_password` for clarity - Remove unused `target_optional_time` and `share_token` helpers Security properties: - Random UUID tokens replace predictable HashID-based Cloudreve URLs; old links cannot be migrated - Password hashing uses Argon2id with random salt per share - Writer enforces foreign key constraints on user_id and file_id/folder_id Test coverage: - File and folder share conversion with download semantics (unlimited and limited) - Boundary conditions: deleted shares, missing owner/target, unsupported target types, negative/overflow counters - Writer transaction rollback on invalid foreign keys - Unique token generation for duplicate targets - End-to-end migration flow with password hash verification - Duplicate active share target counting logic Compatibility notes: - Cloudreve allows multiple active shares per resource; AD preserves all during migration but UI enforces single active share going forward - Per-link speed limits and revocation semantics not preserved (documented limitation)
…igration BREAKING CHANGE: Tasks are no longer migrated as background_tasks history Added: - Migrate Cloudreve metadata as entity properties with namespaces - Convert `tag:*` metadata to native AD tags with personal scope - Generate AD v2 direct-link URLs with HMAC signing and archive mappings - New domain types: MigrationProperty, MigrationTagAssignment, MigrationDirectLink, MigrationMetadata, MigrationEntityRef - Writer methods: write_property, write_tag, write_tag_assignment, write_direct_link - Direct-link URL generation with base62 encoding and user-scoped HMAC - Tag conflict resolution: earliest created_at wins, conflicts reported - Comprehensive validation for tags, properties, and direct-link URLs Removed: - Task migration stage and all task-related domain/writer/converter logic - Tasks now only counted in source inventory, not written to target - `migrated_tasks` and task mappings removed from report schema - Dependency on base64, hmac, urlencoding moved from root to aster_drive_writer Changed: - Migration stage order: now ends at direct_links (was tasks) - Report schema version bumped to 1 - Checkpoint normalization: legacy "tasks" stage mapped to "direct_links" - Source fingerprint uses task count instead of migrated records - Preflight checks validate metadata/tag/direct-link relations - Post-migration validation verifies tag bindings and direct-link properties Tests: - End-to-end SQLite migration with metadata, tags, and direct-links - Tag conflict resolution across files and folders - Direct-link generation and property archival - Missing direct-link secret handling and deleted link filtering - Stage resume with legacy checkpoint normalization
…les into domain-specific submodules Split large monolithic writer and converter modules into focused submodules organized by entity type, improving maintainability and testability. **Writer module restructuring:** - Extract blob, file, folder, user, policy, share, metadata, and direct_link logic into dedicated submodules - Move writer structs and resolver types to writer/mod.rs - Separate integration tests into per-entity test modules (blob_file, direct_link, metadata, share) - Preserve all existing functionality and transaction semantics **Converter module restructuring:** - Extract blob, file, folder, user, policy, share, metadata, and direct_link conversion logic into dedicated submodules - Keep comprehensive test suite in converter/tests.rs - Maintain all validation, skip logic, and boundary checks - Preserve conversion trait implementations and error handling patterns
…files Add detection and skipping of Cloudreve-encrypted entities via `recycle_options.encrypt_metadata`. Implement: - `is_encrypted_entity()` helper to detect non-null `encrypt_metadata` - Skip encrypted blobs in converter with `cloudreve_encrypted_entity` reason - Skip files whose current entity is encrypted - Exclude encrypted entities from local/remote storage verification - Record skipped objects with type, source ID, and reason in migration report - Prevent sensitive keys/IVs from appearing in terminal or JSON output - Update field mapping documentation for `recycle_options` - Add integration test verifying encrypted entity detection, skipping, and safe reporting
…l path normalization Implement strict validation for Cloudreve storage policies and local storage paths to prevent migration of incompatible configurations and ensure safe object key resolution. **Storage Policy Validation:** - Add `storage_policy_skip_reason()` to reject unsupported drivers (oss, obs, custom COS endpoints) and incomplete configurations before migration - Validate Tencent COS policies require standard `bucket.cos.region.myqcloud.com` endpoints with matching bucket names - Check S3-compatible policies have required credentials, bucket, and endpoint fields - Reject policies with Cloudreve encryption enabled - Add `MigrationObjectStorageUploadStrategy` and `MigrationObjectStorageDownloadStrategy` enums to preserve relay vs presigned transfer modes - Map Cloudreve `relay` and `internal_proxy` settings to AD upload/download strategies **Local Storage Path Normalization:** - Add `normalize_local_storage_path()` to validate and convert Cloudreve entity paths - Accept relative paths after rejecting `..` parent-directory traversal - Validate absolute paths (POSIX, Windows drive, UNC) are within configured policy root - Strip absolute roots by complete path segment and convert to relative AD `storage_path` - Handle case-insensitive Windows and UNC paths, case-sensitive POSIX paths - Reject paths resolving to the storage root itself **Migration Engine:** - Split validation from migration in `migrate()` to run all preflight checks before any writes - Rename `unsupported_policy_types()` to `unsupported_policies()` to reflect configuration-level rejection - Pass local storage roots to blob converter for absolute path validation - Validate all local entity paths before target writes, with optional runtime file verification - Update remote validation to use adapter-level `supports_remote_validation()` check - Preserve S3 region settings and document AD #452 for runtime region support **Testing:** - Add unit tests for conversion boundaries, domain enums, and entity structures - Add property tests for base62 encoding, direct link URL generation, and folder cycle detection - Add integration tests for policy driver options, group mappings, and disabled user profiles - Add checkpoint round-trip tests for run metadata, stage cursors, and object mappings - Add end-to-end tests for unsupported policy rejection and local path validation - Add tests for CLI parsing, option validation, and batch size boundaries **Documentation:** - Update field mapping guide to reflect conditional migration status for oss/obs/cos - Clarify that absolute local paths must be inside configured roots and are converted to relative paths - Document S3 region contract gap pending AD #452 and ADM AsterCommunity#3 - Update README to describe standard-endpoint COS support and absolute path handling
- Add `OneDrive` variant to `MigrationStorageDriver` enum
- Add `MigrationOneDriveOptions` struct with cloud region, account mode,
drive/site/group IDs, and transfer topology fields
- Add supporting enums: `MigrationMicrosoftGraphCloud`,
`MigrationOneDriveAccountMode`, `MigrationProviderResumableUploadStrategy`,
`MigrationProviderDownloadStrategy`, `MigrationProviderDownloadFilenameMode`
- Parse Cloudreve `od_driver` path segments into account mode and IDs,
supporting `me/drive`, `drives/{id}`, `sites/{id}/drive`,
`sites/{id}/drives/{drive}`, `groups/{id}/drive`,
`groups/{id}/drives/{drive}`
- Derive `onedrive_cloud` enum from Graph endpoint host(`graph.microsoft.com` → global, `microsoftgraph.chinacloudapi.cn` → china)
- Map `settings.relay` / `settings.internal_proxy` to
`provider_resumable_upload_strategy` / `provider_download_strategy`
- Default OneDrive chunk size from0 to 50 MiB (Cloudreve runtime default)
- Strip legacy OAuth credentials (client ID, refresh token, client secret)from target policy fields and migration report
- Emit migration warning listing OneDrive policy IDs that require
re-authorization in AsterDrive before cutover
- Skip OneDrive policies with invalid endpoints or unsupported
`od_driver` paths via `unsupported_storage_configuration` reason
- Restrict `s3_path_style` option to S3/TencentCos drivers only
- Add integration test covering full OneDrive migration flow and
verifying no legacy credentials appear in DB or report
- Update field-mapping docs and README to reflect OneDrive support
…: migrate S3/KS3 signing region into AsterDrive policy options Add end-to-end support for preserving explicit S3/KS3 signing regions during Cloudreve → AsterDrive migration. - Add `s3_region: Option<String>` field to `MigrationStoragePolicy` domain struct - Add `storage_region_setting()` in `cloudreve_adapter` to parse and validate region from `settings.region` / `settings.s3_region`, with trimming and 1-128 printable ASCII (no whitespace or `/`) validation - Add `s3_policy_skip_reason()` to reject S3/KS3 policies with invalid region values at conversion time - Export `storage_region_setting` from `cloudreve_adapter::lib` for reuse - Refactor `storage_region` in `src/migration/remote.rs` to delegate to the shared `storage_region_setting` instead of duplicating inline logic - Write `options.s3_region` in `aster_drive_writer` when driver is S3 and region is present - Add unit tests for region validation boundary values, alias precedence (`region` over `s3_region`), skip-on-invalid-region, and writer output - Add integration test `migrates_s3_signing_region_into_asterdrive_policy_options` covering full migration flow with trimmed region - Update field-mapping docs and README to reflect completed implementation (replaces "pending AD #452/ ADM AsterCommunity#3" notes) - Bump `rust-version` to 1.95.0 and update `AsterDrive`/`AsterForge` dependency commits
…tion - Skip Cloudreve entities with upload_session_id set (incomplete uploads) during blob migration, recording them as skipped in the migration report - Skip entities not referenced by any migratable file (no primary_entity set), recording them as skipped - Filter files with null primary_entity in load_blob_association_info query to exclude unreferenced entities from reference counts - Move association info load before storage verification steps to enable pre-filtering of referenced entities - Add cos_s3_endpoint helper to strip bucket-qualified prefix from Tencent COS endpoints before passing to the AWS SDK - Handle null and missing file_type in Cloudreve policy settings as an empty allowed-types array instead of erroring - Add url dependency for COS endpoint URL parsing and normalization - Add tests for incomplete upload skipping, null/missing file_type, COS endpoint normalization, and integration flow coverage
Summary by CodeRabbit
check/migrate，可生成--report-pathJSON，并提供--run-id、--resume、--dry-run、--allow-non-empty-target、--skip-unsupported-policies与--blob-batch-size/--file-batch-size。direct_links（旧/f/...）不可继续有效。