Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 4 additions & 7 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,8 @@
API_URL=https://api.parktrack.live/api/v1
API_TOKEN=your_token_here

# Model storage path (relative to project root)
MODEL_PATH=models/forecast_model.pkl
# Path where model artifacts are stored (mounted volume in Docker)
MODEL_PATH=./models

# Comma-separated forecast horizons in minutes
FORECAST_HORIZONS=15,30,60

# How many days of historical data to use for training
TRAIN_DAYS_BACK=90
# Days of historical occupancy data to use for training
TRAIN_DAYS_BACK=200
73 changes: 45 additions & 28 deletions .github/workflows/build-and-push.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,11 @@ on:
branches: ["main", "development"]
pull_request:
branches: ["main"]
workflow_dispatch:

env:
REGISTRY: ghcr.io
IMAGE_BASE: ${{ github.repository }}
IS_DEFAULT: ${{ github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}

jobs:
build-and-push:
Expand All @@ -29,34 +30,50 @@ jobs:
- name: Checkout
uses: actions/checkout@v4

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4

- name: Log in to GHCR
uses: docker/login-action@v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata
id: meta
uses: docker/metadata-action@v6
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_BASE }}-${{ matrix.image_suffix }}
tags: |
type=raw,value=development,enable=${{ github.ref != format('refs/heads/{0}', github.event.repository.default_branch) }}
type=raw,value=latest,enable={{is_default_branch}}
type=ref,event=branch
type=sha,format=short
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ${{ env.REGISTRY }} -u ${{ github.actor }} --password-stdin

- name: Build and push
uses: docker/build-push-action@v7
if: github.event_name != 'pull_request'
run: |
IMAGE=${{ env.REGISTRY }}/$(echo "${{ github.repository }}" | tr '[:upper:]' '[:lower:]')-${{ matrix.image_suffix }}
BRANCH=${{ github.ref_name }}
SHA=$(echo ${{ github.sha }} | cut -c1-7)

docker build -f ${{ matrix.dockerfile }} -t $IMAGE:$BRANCH -t $IMAGE:$SHA .

if [ "$BRANCH" = "main" ]; then
docker tag $IMAGE:$BRANCH $IMAGE:latest
docker push $IMAGE:latest
fi

docker push $IMAGE:$BRANCH
docker push $IMAGE:$SHA
deploy:
needs:
- build-and-push

runs-on: ubuntu-latest

permissions:
contents: read
packages: write

if: github.event_name == 'push'

steps:
- name: Deploy via SSH
uses: appleboy/ssh-action@v1.0.3
with:
context: .
file: ${{ matrix.dockerfile }}
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha,scope=${{ matrix.image_suffix }}
cache-to: type=gha,mode=max,scope=${{ matrix.image_suffix }}
host: ${{ secrets.SERVER_IP }}
username: ${{ secrets.USERNAME }}
key: ${{ secrets.SSH_PRIVATE_KEY }}
script: |
if [ "${{ env.IS_DEFAULT }}" = "true" ]; then
cd ${{ secrets.COMPOSE_DIRECTORY_PATH }}
docker compose up -d
else
cd ${{ secrets.DEVELOPMENT_COMPOSE_DIRECTORY_PATH }}
docker compose -p parktrack-dev up -d
fi

3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@ venv/
dist/
build/

# Trained models (large binary files — store in object storage or registry)
# Trained models and artifacts
models/
artifacts/
*.pkl
*.lgb
*.json.model
5 changes: 3 additions & 2 deletions Dockerfile.predict
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
FROM python:3.11-slim

RUN apt-get update && apt-get install -y --no-install-recommends libgomp1 && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY requirements.txt .
Expand All @@ -9,5 +11,4 @@ COPY parktrack_ml/ ./parktrack_ml/

ENV PYTHONPATH="/app"

# Mount model volume at /app/models when running
CMD ["python", "-m", "parktrack_ml.predict"]
CMD ["python", "-m", "parktrack_ml.service"]
2 changes: 2 additions & 0 deletions Dockerfile.train
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
FROM python:3.11-slim

RUN apt-get update && apt-get install -y --no-install-recommends libgomp1 && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY requirements.txt .
Expand Down
174 changes: 101 additions & 73 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,132 +1,160 @@
# parktrack-ml

ML microservice for parking occupancy forecasting. Consists of two independent scripts:
ML-микросервис для прогнозирования загруженности парковок. Работает как автономный процесс: сам обучается, сам генерирует прогнозы и публикует их в ParkTrack API.

- **train** — fetches historical occupancy data from ParkTrack API, trains a LightGBM model, saves it to disk.
- **predict** — loads the saved model, generates forecasts for configurable future horizons, posts them to `POST /forecasts/new`.
Всё взаимодействие — только через ParkTrack REST API. Прямого доступа к БД нет.

Both scripts are packaged as separate Docker images and can be scheduled as cron jobs.
---

## Как это работает

Сервис запускается и делает три вещи по расписанию:

- **каждые 30 минут** — генерирует прогнозы на ближайшие 24 часа для всех активных зон и постит их в `POST /forecasts/new`
- **каждые 5 минут** — подтягивает актуальную погоду
- **ежедневно в 02:00 UTC** — переобучает модель на свежих данных

При первом запуске сервис сам обучает модель, если артефакта нет.

---

## Project structure
## Структура проекта

```
parktrack_ml/
├── api_client.py # ParkTrack API wrapper (Bearer token auth)
├── features.py # Feature engineering
├── train.py # Training script entry point
└── predict.py # Prediction script entry point
Dockerfile.train
Dockerfile.predict
├── api_client.py — HTTP-клиент к ParkTrack API (Bearer auth)
├── config.py — все настройки и константы
├── data_loader.py — загрузка данных через API
├── features.py — построение признаков
├── model.py — LightGBM-обёртка + LR-fallback
├── train.py — скрипт обучения
├── predict.py — логика предсказания
├── forecaster.py — генерация и публикация прогнозов
├── weather.py — работа с погодными данными
├── interfaces.py — публичный API пакета
└── service.py — точка входа (планировщик)
Dockerfile.train — образ для разового обучения
Dockerfile.predict — образ основного сервиса
docker-compose.yml
.env.example
requirements.txt
```

---

## Quick start
## Быстрый старт

### 1. Configure environment
### 1. Настроить окружение

```bash
cp .env.example .env
# Edit .env and set API_URL and API_TOKEN
# Заполнить API_URL и API_TOKEN
```

### 2. Run locally (without Docker)
### 2. Локально (без Docker)

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Train
# Разовое обучение
python -m parktrack_ml.train

# Predict (requires a trained model)
python -m parktrack_ml.predict
# Запуск сервиса (обучение + прогнозы по расписанию)
python -m parktrack_ml.service
```

### 3. Run with Docker Compose
### 3. Docker Compose

```bash
# Train
# Разовое обучение
docker compose --profile train up --build

# Predict
docker compose --profile predict up --build
# Основной сервис (работает постоянно)
docker compose --profile predict up -d --build
```

---

## Environment variables
### 4. Проверить что сервис работает

| Variable | Required | Default | Description |
|---------------------|----------|------------------------------|--------------------------------------------------|
| `API_URL` | yes | — | ParkTrack API base URL (e.g. `https://api.parktrack.live/api/v1`) |
| `API_TOKEN` | yes | — | Bearer token with `forecasts.write` permission |
| `MODEL_PATH` | no | `models/forecast_model.pkl` | Path to save/load the trained model |
| `FORECAST_HORIZONS` | no | `15,30,60` | Comma-separated list of forecast horizons (minutes) |
| `TRAIN_DAYS_BACK` | no | `90` | Days of historical data to use for training |
```bash
# Логи
docker logs <container_name> -f

# Разовый тест предсказания прямо в контейнере
docker exec <container_name> python -c "
from parktrack_ml.interfaces import predict
from datetime import datetime, timezone
r = predict(zone_id=3, predicted_for=datetime.now(timezone.utc))
print(r)
"
```

---

## Docker images

Images are published to GHCR automatically via GitHub Actions on every push to `main` or `development`:
## Переменные окружения

| Image | Tag |
|-------|-----|
| `ghcr.io/parktrack-project/parktrack-ml-train` | `latest` / `development` / `sha-xxxxxxx` |
| `ghcr.io/parktrack-project/parktrack-ml-predict` | `latest` / `development` / `sha-xxxxxxx` |
| Переменная | Обязательна | По умолчанию | Описание |
|-------------------|-------------|--------------|----------|
| `API_URL` | да | — | Базовый URL ParkTrack API |
| `API_TOKEN` | да | — | Bearer-токен |
| `MODEL_PATH` | нет | `./models` | Путь к папке с артефактами модели |
| `TRAIN_DAYS_BACK` | нет | `200` | Глубина истории для обучения (дни) |

---

## Model details
## Интеграция в deploy

- **Algorithm**: LightGBM (`LGBMRegressor`)
- **Features**: `zone_id`, `hour`, `minute`, `day_of_week`, `is_weekend`, `month`, `horizon_minutes`
- **Target**: `occupied` N minutes into the future
- **Training split**: 85% train / 15% validation (time-ordered, no shuffle)
- **Metric**: Mean Absolute Error (MAE) on validation set
Добавить в основной `docker-compose.yml`:

The model artifact (`.pkl`) includes the trained model, feature column names, configured horizons, and a per-zone capacity map derived from training data.
```yaml
services:
ml:
image: ghcr.io/parktrack-project/parktrack-ml-predict:development
env_file: .env # API_URL, API_TOKEN
environment:
MODEL_PATH: /app/models
volumes:
- ml_models:/app/models
restart: unless-stopped

volumes:
ml_models:
```

Сервис не требует открытых портов — он только ходит в ParkTrack API.

---

## Cron scheduling
## Использование пакета напрямую (Python)

Train weekly, predict every 15 minutes (example crontab):
Если нужно вызвать предсказание из другого Python-сервиса:

```cron
# Retrain every Sunday at 02:00 UTC
0 2 * * 0 docker compose -f /opt/parktrack-ml/docker-compose.yml --profile train up --build
```python
from parktrack_ml.interfaces import predict
from datetime import datetime, timezone

# Predict every 15 minutes
*/15 * * * * docker compose -f /opt/parktrack-ml/docker-compose.yml --profile predict up
result = predict(
zone_id=3,
predicted_for=datetime(2026, 5, 26, 14, 0, tzinfo=timezone.utc),
)

print(result.occupancy_class) # "Low" | "Medium" | "High"
print(result.predicted_occupied) # 4 (машин из capacity)
print(result.capacity) # 10
print(result.confidence) # 0.89
print(result.probability_free_space) # 0.95
print(result.prob_low) # 0.67
print(result.prob_medium) # 0.31
print(result.prob_high) # 0.02
```

---

## Integration with ParkTrack deploy

Add to the `deploy/docker-compose.yml`:

```yaml
ml-predict:
image: ghcr.io/parktrack-project/parktrack-ml-predict:latest
environment:
- API_URL=http://api-server:8000/api/v1
- API_TOKEN=${ML_API_TOKEN}
- MODEL_PATH=/models/forecast_model.pkl
- FORECAST_HORIZONS=15,30,60
volumes:
- ml_models:/models
depends_on:
- api-server
restart: unless-stopped
## Модель

volumes:
ml_models:
```
- **Алгоритм**: LightGBM (multiclass)
- **Классы**: Low (< 33%), Medium (33–67%), High (> 67%)
- **Признаков**: 28 — история загруженности (lag/MA), время (с циклическим кодированием), погода, зона, праздники
- **Данных для обучения**: ~18 000 часовых наблюдений (150 дней по 15 зонам)
- **Точность на валидации**: 90.9% (temporal split 80/20)
- **Артефакты**: `models/model.lgb`, `models/zone_meta.json`
Loading
Loading