chore: Simplify data setup and fix upload

[MohdRafi-Microsoft]

Purpose

This pull request streamlines and improves the deployment and data ingestion scripts, enhances multi-instance reliability in the backend, and updates the nginx configuration to support larger file uploads. The main themes are: deployment robustness, ingestion reliability, and backend correctness for multi-instance deployments.

Deployment and Script Improvements:

• The agent settings sync logic in connect-data.ps1 and setup-agent.ps1 now only sends non-empty, valid values to the API, preventing deployment failures due to empty or invalid settings (especially for USE_SQL). Skips sync if no values are resolved. [1] [2]
• Data upload logic in setup-data.ps1 is simplified: it no longer waits for full ingestion (indexing/chunking/embedding) but only confirms file registration with the backend, allowing the deployment to finish faster and minimizing the window of public network access. The script now provides clearer status messages and error handling for batch uploads. [1] [2] [3]

Backend Reliability for Multi-Instance Deployments:

• The ingestion timeout logic in router.py now re-checks the file status from the SQL source of truth before marking a file as failed, preventing false alarms in multi-instance setups.
• A new get_file_status method is added to sql_service.py to support accurate status checks directly from SQL, ensuring consistency across multiple API instances.

Infrastructure Configuration:

• The nginx configuration (nginx.conf) now sets client_max_body_size to 1024MB, matching backend upload limits and preventing silent failures on large file uploads.

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

• ...

Other Information

[Copilot]

Pull request overview

This PR streamlines the post-provision deployment scripts by shortening the data-ingestion “wait” behavior to only confirm backend registration, hardening azd env get-value handling to avoid propagating error text/invalid values, and adjusting nginx to accept large upload requests in private-networking mode.

Changes:

• Refactors setup-data.ps1 to stop polling for full ingestion completion and instead only confirm file registration via /api/ingestion/refresh.
• Improves environment-variable retrieval and agent settings sync logic to avoid sending empty/invalid app settings (notably USE_SQL).
• Increases nginx client_max_body_size to 1024MB to prevent 413s for large uploads.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
src/app/nginx.conf	Raises nginx request body limit to support large multipart uploads proxied to the backend.
infra/scripts/post-provision/setup-data.ps1	Simplifies upload/ingestion flow to confirm registration only; removes prior ingestion/index polling and upload retry helper.
infra/scripts/post-provision/setup-agent.ps1	Makes azd env get-value parsing more robust and omits invalid/empty agent settings when syncing to App Service.
infra/scripts/post-provision/connect-data.ps1	Aligns azd env get-value parsing and syncs App Service settings with filtered USE_SQL/DATA_SOURCE_TYPE.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

infra/scripts/post-provision/setup-data.ps1:336

• Confirm-UploadRegistered polls /api/ingestion/refresh, which calls ingestion_service.reload() and then forces a full SQL reload of documents/files/filter schema on every poll. For large datasets this can be expensive and can thrash the API cache (even if the caller only needs a durable count). Consider adding a lightweight endpoint (or a countOnly option) that returns the uploaded file registration count directly from SQL without clearing/reloading the in-memory cache, and have this script poll that instead.

    while ($elapsed -lt $TimeoutSec) {
        $cnt = -1
        try { $r = Invoke-RestMethod -Uri "$BackendUrl/api/ingestion/refresh" -Method POST -Headers $Headers; $cnt = [int]$r.files } catch {}
        if ($cnt -ge $expectedCount) { break }

infra/scripts/post-provision/setup-data.ps1:298

• Invoke-DataCleanup no longer retries transient gateway/unavailable failures (502/503/504). During deployments the backend can briefly return these while warming up; with the new logic the whole script exits immediately, making provisioning more brittle than before. A small bounded retry (even 3 attempts) keeps the simplification while preserving resilience.

    # Clear existing demo data (documents, insights cache) and any external data source
    # registrations so every scenario starts from a clean slate.
    Write-Host "Clearing existing data and external source connections for scenario isolation..." -ForegroundColor Yellow
    try {
        Invoke-RestMethod -Uri "$BackendUrl/api/ingestion/clear?include_external=true" -Method DELETE -Headers $Headers | Out-Null
        Write-Host "Previous data and external source registrations cleared." -ForegroundColor Green
    } catch {
        Write-Host "ERROR: Could not clear existing data before scenario load: $_" -ForegroundColor Red
        Write-Host "Aborting to prevent mixed data across use cases." -ForegroundColor Yellow
        exit 1
    }

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

src/api/storage/sql_service.py:482

• get_file_status closes the SQL connection only on the success path; if cursor.execute()/fetchone() raises, the connection can be left open, which can exhaust the connection pool over time. Wrap the query in a try/finally and always close the connection.

            conn = self._get_connection()
            cursor = conn.cursor()
            cursor.execute("SELECT status FROM uploaded_files WHERE id = ?", file_id)
            row = cursor.fetchone()
            conn.close()

infra/scripts/post-provision/setup-data.ps1:297

• This used to retry transient 502/503/504 errors while the backend was warming up, but now it fails fast on the first error. That can make deployments flaky (especially right after provisioning) and may contradict the script’s goal of robust setup. Consider reinstating a small retry loop for transient HTTP failures before aborting.

    Write-Host "Clearing existing data and external source connections for scenario isolation..." -ForegroundColor Yellow
    try {
        Invoke-RestMethod -Uri "$BackendUrl/api/ingestion/clear?include_external=true" -Method DELETE -Headers $Headers | Out-Null
        Write-Host "Previous data and external source registrations cleared." -ForegroundColor Green
    } catch {
        Write-Host "ERROR: Could not clear existing data before scenario load: $_" -ForegroundColor Red
        Write-Host "Aborting to prevent mixed data across use cases." -ForegroundColor Yellow
        exit 1

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

infra/scripts/post-provision/setup-agent.ps1:152

• The USE_SQL validation only allows "true"/"false", but the backend interprets USE_SQL using broader boolean semantics (e.g., "1"/"0", "yes"/"no", "on"/"off"). If a deployment/environment provides one of these valid encodings, this script will omit USE_SQL from the settings sync and the API may start with an unexpected value.

        $settingsArgs = @()
        if ($agentNameChat)  { $settingsArgs += "AGENT_NAME_CHAT=$agentNameChat" }
        if ($agentNameTitle) { $settingsArgs += "AGENT_NAME_TITLE=$agentNameTitle" }
        if ($useSql -match '^(?i:true|false)$') { $settingsArgs += "USE_SQL=$useSql" }
        if ($dataSourceType) { $settingsArgs += "DATA_SOURCE_TYPE=$dataSourceType" }

infra/scripts/post-provision/connect-data.ps1:119

• The USE_SQL validation only allows "true"/"false", but the backend accepts other standard boolean encodings (e.g., "1"/"0", "yes"/"no", "on"/"off"). With the current regex, a valid value like USE_SQL=1 from a .env/azd env would be silently omitted from the App Service settings sync, causing the API to run with an unexpected USE_SQL value.

    $settingsArgs = @("AGENT_NAME_CHAT=$agentNameChat", "AGENT_NAME_TITLE=$agentNameTitle")
    if ($useSql -match '^(?i:true|false)$') { $settingsArgs += "USE_SQL=$useSql" }
    if ($dataSourceType) { $settingsArgs += "DATA_SOURCE_TYPE=$dataSourceType" }