Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
215 changes: 215 additions & 0 deletions .agents/plans/manual-deployment-local-build.plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
# Plan: Manual Deployment — Local Docker Build → GHCR → ECS

## Summary

Deploy the latest code to Alibaba Cloud ECS by building the Docker image locally, pushing to GitHub Container Registry (GHCR), then pulling and restarting on ECS. Bypasses the broken CI/CD pipeline (self-hosted runner ruff check exit code 1) entirely. Fast path to production with manual verification steps.

## User Story

As a **developer**, I want to deploy the latest code to production with manual verification, so that the ECS instance runs the current source while we fix the CI pipeline.

## Type

OPERATIONS / BUG_FIX_WORKAROUND

## Complexity

LOW

---

## Prerequisites

- Docker installed locally with `ghcr.io` login
- `SSH_PRIVATE_KEY` / SSH access to `root@47.237.254.118`
- GitHub token with `write:packages` scope (for GHCR push)

---

## Patterns to Follow

### GHCR Authentication (Local)
```
// SOURCE: .github/workflows/deploy.yml:32-37
docker login ghcr.io -u <github-username> --password-stdin
```

### Docker Compose Production Config
```
// SOURCE: docker-compose.prod.yml:1-29
services:
app:
image: ghcr.io/wslag/workabroadai:latest
env_file: .env
restart: unless-stopped
networks: [app-network]
expose: ["8000"]
```

### SSH Deploy Procedure
```
// SOURCE: deploy.sh:47-57
ssh root@47.237.254.118 "cd /opt/workabroad-ai && \
docker compose pull && \
docker compose down --remove-orphans && \
docker compose up -d && \
for i in \$(seq 1 30); do \
curl -sf http://localhost/health && break; \
sleep 2; \
done"
```

### Health Check
```
// SOURCE: Dockerfile:44-45
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1
```

---

## Tasks

Execute in order. Each task is atomic and verifiable.

### Task 1: Verify Local Lint/Type/Tests

- **Location**: Local `repo/` directory
- **Action**: RUN
- **Implement**:
```bash
cd repo
ruff check . && mypy .
pytest -v --timeout=60
```
- **Validate**: All checks pass, 108 tests green
- **On failure**: Fix issues before proceeding

### Task 2: Build Docker Image Locally

- **Location**: Local `repo/` directory
- **Action**: RUN
- **Implement**:
```bash
docker build -t ghcr.io/wslag/workabroadai:latest .
```
- **Validate**: `docker images ghcr.io/wslag/workabroadai:latest` shows the image
- **Time**: ~20-40 minutes (torch + sentence-transformers download on first build)

### Task 3: Smoke Test Local Image

- **Location**: Local `repo/` directory
- **Action**: RUN
- **Implement**:
```bash
docker run --rm -d --name test-deploy -p 8000:8000 \
-e SUPABASE_URL=placeholder \
-e SUPABASE_SERVICE_KEY=placeholder \
-e SUPABASE_ANON_KEY=placeholder \
-e GROQ_API_KEY=placeholder \
-e CEREBRAS_API_KEY=placeholder \
-e API_KEY=placeholder \
-e LLM_PROVIDER=groq \
ghcr.io/wslag/workabroadai:latest
sleep 5
curl -f http://localhost:8000/health
docker kill test-deploy
```
- **Validate**: `{"status":"ok"}` response

### Task 4: Authenticate with GHCR Locally

- **Location**: Local machine
- **Action**: RUN
- **Implement**:
```bash
# Create a GitHub Personal Access Token with write:packages scope
# Then:
echo $GITHUB_TOKEN | docker login ghcr.io -u WSlag --password-stdin
```
- **Validate**: `docker login` succeeds (no error)

### Task 5: Push Image to GHCR

- **Location**: Local machine
- **Action**: RUN
- **Implement**:
```bash
docker push ghcr.io/wslag/workabroadai:latest
```
- **Validate**: Push completes without errors, image appears at `https://github.com/WSlag/workabroadai/pkgs/container/workabroadai`
- **Time**: ~2-5 minutes (image layer upload)

### Task 6: Deploy on ECS

- **Location**: Local machine
- **Action**: RUN
- **Implement**:
```bash
ssh root@47.237.254.118 "cd /opt/workabroad-ai && \
docker compose pull && \
docker compose down --remove-orphans && \
docker compose up -d && \
echo '==> Waiting for health check...' && \
for i in \$(seq 1 30); do \
curl -sf http://localhost/health && echo '' && break; \
sleep 2; \
done && \
if [ \$i -eq 30 ]; then \
echo 'Health check FAILED' && \
docker compose logs --tail=50 && \
exit 1; \
fi && \
echo 'Deployment successful!'"
```
- **Validate**: Script exits 0, `{"status":"ok"}` from `http://47.237.254.118/health`

### Task 7: Verify Production Endpoint

- **Location**: Local machine
- **Action**: RUN
- **Implement**:
```bash
curl -sf http://47.237.254.118/health
```
- **Validate**: `{"status":"ok"}`

---

## Risks

| Risk | Mitigation |
|------|------------|
| GHCR push fails (no write access) | Create PAT at `github.com/settings/tokens` with `write:packages` scope |
| Local Docker build incompatible with ECS CPU arch | Both are `linux/amd64` — verify with `docker inspect` before pushing |
| ECS pull fails (disk space) | Check with `df -h` before deploying — prune old images: `docker system prune -af` |
| Container starts but health check fails | `docker logs` for debugging; roll back to previous image via `sed -i` in compose |
| GHCR rate limiting (anonymous pulls) | Authenticated pulls on ECS: `docker login ghcr.io -u WSlag --password-stdin` (uses GITHUB_TOKEN secret) |

---

## Validation

```bash
# Task 1
ruff check . && mypy .
pytest -v --timeout=60

# Task 3 (smoke test local image)
docker run --rm -d --name test-deploy -p 8000:8000 ... && sleep 5 && curl -f http://localhost:8000/health

# Task 7 (verify production)
curl -sf http://47.237.254.118/health
```

---

## Acceptance Criteria

- [ ] Task 1: `ruff check . && mypy .` and `pytest -v` all pass
- [ ] Task 2: Docker image builds locally without errors
- [ ] Task 3: Local smoke test returns `{"status":"ok"}`
- [ ] Task 4: GHCR authentication succeeds
- [ ] Task 5: Image pushed to `ghcr.io/wslag/workabroadai:latest`
- [ ] Task 6: ECS pulls and restarts with new image, health check passes
- [ ] Task 7: `curl http://47.237.254.118/health` returns `{"status":"ok"}`
Loading
Loading