AI Document Processing Platform

A production-oriented backend system for asynchronous AI-powered document processing built with Node.js, Express, TypeScript, PostgreSQL (pgvector), Redis, BullMQ, Docker, GitHub Actions, and AWS EC2.

---

Overview

This project demonstrates how modern backend systems process large documents asynchronously using queues, workers, vector databases, and AI services.

Instead of processing uploaded documents inside an HTTP request, the API stores metadata, queues a background job, and returns immediately. Dedicated workers then extract text, generate AI embeddings using Google Gemini, and store vector representations inside PostgreSQL with pgvector.

The architecture follows production backend engineering principles including asynchronous processing, containerization, CI/CD, and cloud deployment.

---

Features

• JWT Authentication
• Document Upload API
• PDF Text Extraction
• AI Embedding Generation (Google Gemini)
• Vector Storage using pgvector
• Background Processing using BullMQ
• Redis Queue
• PostgreSQL Database
• Dockerized Services
• GitHub Actions CI/CD
• GitHub Container Registry (GHCR)
• AWS EC2 Deployment
• Integration Testing

---

System Architecture

                    Client
                       │
                       ▼
               Express REST API
                       │
             Store Metadata (Postgres)
                       │
                       ▼
                BullMQ Queue
                       │
                       ▼
              Redis Job Queue
                       │
                       ▼
               Background Worker
                       │
        ┌──────────────┼──────────────┐
        ▼              ▼              ▼
   Extract PDF     Chunk Text     Generate Embeddings
                                         │
                                         ▼
                             Google Gemini API
                                         │
                                         ▼
                             PostgreSQL + pgvector

Tech Stack

• Backend
• Node.js
• Express.js
• TypeScript

Database

• PostgreSQL
• pgvector
• Queue
• BullMQ
• Redis

AI

• Google Gemini Embedding API

DevOps

• CI/CD
• Docker
• Docker Compose
• GitHub Actions
• GitHub Container Registry
• AWS EC2

Testing

• Integration Tests

API Endpoints

Health

Method	Endpoint	Authentication	Description
GET	/health	No	Check API health status

---

Authentication

Method	Endpoint	Authentication	Description
POST	/api/auth/register	No	Register a new user
POST	/api/auth/login	No	Authenticate user and return access & refresh tokens
POST	/api/auth/refresh	No	Generate a new access token using a refresh token
POST	/api/auth/logout	Yes	Logout the current user and invalidate refresh token

---

Documents

Method	Endpoint	Authentication	Description
POST	/api/documents/upload	Yes	Upload a PDF for asynchronous AI processing
GET	/api/documents	Yes	Retrieve all uploaded documents
GET	/api/documents/:id	Yes	Retrieve a specific document by ID

---

Authentication Header

Protected endpoints require a JWT access token.

Authorization: Bearer <access_token>

---

Response Status Codes

Status Code	Description
200	Request completed successfully
201	Resource created successfully
202	Accepted for background processing
400	Bad request
401	Unauthorized
403	Forbidden
404	Resource not found
409	Conflict (resource already exists)
500	Internal server error

Project Structure

src
├── modules
│   ├── auth
│   └── document
│
├── infrastructure
│   ├── database
│   ├── queue
│   ├── workers
│   └── logger
│
├── shared
│
└── server.ts

Document Processing Workflow

Client Uploads PDF

        │

        ▼

Express API

        │

Save Document Metadata

        │

PostgreSQL

        │

Add Job

        │

BullMQ

        │

Redis Queue

        │

Worker

        │

Extract PDF

        │

Chunk Text

        │

Generate Embeddings

        │

Store Vectors

        │

PostgreSQL (pgvector)

CI/CP Pipeline

Git Push

      │

      ▼

GitHub Actions

      │

Run Tests

      │

Build Docker Images

      │

Push Images

      ▼

GitHub Container Registry

Production Deployment

The application is containerized using Docker Compose and deployed on AWS EC2.

Running services:

• API
• Worker
• PostgreSQL
• Redis

Deployment includes:

• Docker
• Docker Compose
• GitHub Container Registry
• AWS EC2

Local Development

Clone the repository

git clone https://github.com/<username>/document-processing-platform.git

Install dependencies

pnpm install

Run docker services

docker compose up -d

Run migrations

pnpm migrate

Start development server

pnpm dev

Running Tests

pnpm test

Future Improvements

• Semantic Search
• RAG Chat
• OpenTelemetry
• Prometheus & Grafana
• Kubernetes Deployment
• S3 Object Storage
• Multi-file Processing
• Horizontal Worker Scaling
• Rate Limiting
• API Gateway

Engineering Concepts Demonstrated

• REST API Design
• Authentication
• Repository Pattern
• Background Job Processing
• Queue-based Architecture
• Worker Pattern
• Vector Databases
• AI Integration
• Docker
• CI/CD
• Cloud Deployment
• Production Logging
• Integration Testing

Author

Kushal Chavan

Backend Engineer

GitHub: [https://github.com/Kushalchavan]