OptiSigns Knowledge Base Sync

Automatically scrape the OptiSigns Zendesk Help Center, convert articles to Markdown, detect changes, and synchronize only new or updated articles to an OpenAI Vector Store for Retrieval-Augmented Generation (RAG).

---

Features

• Scrape OptiSigns Zendesk Help Center
• Convert HTML articles to Markdown
• Store article metadata locally
• Detect new and updated articles using content hashing
• Upload only changed articles to OpenAI Vector Store
• Configurable chunking strategy
• Daily execution via Railway Cron
• Dockerized deployment
• Structured logging

---

Note

The Assistant Playground screenshot is not included because the OpenAI Playground requires a billing-enabled account.

Project Structure

app/
│
├── config/
├── openai/
├── scraper/
├── storage/
├── utils/
│
├── main.py
│
data/
│
├── html/
├── markdown/
├── metadata/
└── state/

Dockerfile
requirements.txt
README.md

---

Architecture

Zendesk API
      │
      ▼
 Fetch Articles
      │
      ▼
 Detect Changes
(Hash Comparison)
      │
      ▼
 Save HTML
      │
      ▼
 Convert Markdown
      │
      ▼
 Upload Delta Files
      │
      ▼
 OpenAI Vector Store

---

Installation

Clone the repository

git clone https://github.com/VoHuan/ChatBot.git
cd ChatBot

Create virtual environment

python -m venv .venv

Activate

Windows

.venv\Scripts\activate

Linux / macOS

source .venv/bin/activate

Install dependencies

pip install -r requirements.txt

---

Environment Variables

Create a .env file

OPENAI_API_KEY=your_api_key

---

Run Locally

python -m app.main

---

Docker

Build

docker build -t optisigns-sync .

Run

docker run --rm \
-e OPENAI_API_KEY=YOUR_KEY \
optisigns-sync

---

Incremental Synchronization

Each execution performs:

1. Fetch latest Zendesk articles
2. Calculate content hash
3. Compare with previous hashes
4. Detect

• Added
• Updated
• Skipped

Only added and updated articles are uploaded to OpenAI.

---

Chunking Strategy

OpenAI Vector Store performs automatic embedding.

Static chunking configuration:

chunking_strategy={
    "type": "static",
    "static": {
        "max_chunk_size_tokens": 1200,
        "chunk_overlap_tokens": 400
    }
}

Logging

Each execution generates a summary similar to:

============================================================
OptiSigns Daily Sync Summary
============================================================
Fetched        : 404
Added          : 2
Updated        : 1
Skipped        : 401

Uploaded Files : 3

Vector Store   : vs_xxxxxxxxx
============================================================

Deployment

The application is deployed as a Railway Cron Job.

Execution schedule:

0 1 * * *

(Every day at 01:00 UTC)

---

Screenshots

Railway Cron

Sample Job Log

---

State Management

The project maintains several local state files.

data/state/

Includes

• hashes.json
• vector_store.json
• vector_files.json

These files allow incremental synchronization between executions.

---

Technologies

• Python 3.13
• OpenAI SDK 2.x
• OpenAI Vector Store
• BeautifulSoup
• Markdownify
• Docker
• Railway
• Zendesk Help Center API

Author

Vo Van Huan