AI News Aggregator

A modular system that collects AI-related content (YouTube, RSS), converts it into readable digests using LLMs, ranks items by user relevance, and delivers personalized daily email summaries.

Summary

This repository implements a configurable pipeline for aggregating and transforming AI news from multiple sources into concise, personalized digests. It is designed for easy extension with new scrapers, processors, and delivery channels.

Core capabilities:

• Aggregate content from YouTube and RSS feeds
• Convert articles and transcripts into markdown
• Generate summarized digests using LLMs
• Rank and curate content per user preferences
• Deliver HTML email digests and prevent duplicates

Architecture

graph LR
  Sources[Sources\nYouTube\nRSS] --> Scrapers[Scrapers\nRegistry Pattern]
  Scrapers --> DB[(PostgreSQL)]
  DB --> Processors[Processors\nMarkdown, Transcripts, Digests]
  Processors --> DB
  DB --> Curator[Curator\nLLM Ranking]
  Curator --> Email[Email\nDigest Generation]
  Email --> Delivery[Delivery\nSMTP]

Loading

Major components live under app/ and follow a clear separation of concerns: scraping, processing, curation, and delivery.

How the Pipeline Works

1. Scrape: registered scrapers fetch new items and store raw content in the database (app/runner.py).
2. Process: source-specific processors convert HTML/transcripts to markdown and produce LLM summaries (app/services/).
3. Curate: the curator scores and ranks summaries against user profiles (app/services/process_curator.py).
4. Email: the email generator builds a personalized digest and marks items as sent (app/services/process_email.py).
5. Deliver: HTML emails are sent via SMTP (app/services/email.py).

The run_daily_pipeline() function coordinates these steps for a scheduled run.

Quick Project Layout

See the app/ folder for the main modules:

app/
├── agent/              # LLM agents for processing
│   ├── base.py        # Base agent class
│   ├── curator_agent.py   # Article ranking
│   ├── digest_agent.py    # Summary generation
│   └── email_agent.py     # Email content generation
├── config.py          # Configuration (YouTube channels)
├── database/          # Database layer
│   ├── models.py      # SQLAlchemy models
│   ├── repository.py # Data access layer
│   └── connection.py  # DB connection & environment
├── profiles/          # User profile configuration
│   └── user_profile.py
├── scrapers/          # Content scrapers
│   ├── base.py        # Base scraper for RSS feeds
│   ├── anthropic.py   # Anthropic RSS scraper
│   ├── openai.py      # OpenAI RSS scraper
│   └── youtube.py     # YouTube channel scraper
├── services/          # Processing services
│   ├── base.py        # Base process service
│   ├── process_anthropic.py
│   ├── process_youtube.py
│   ├── process_digest.py
│   ├── process_curator.py
│   ├── process_email.py
│   └── email.py       # Email sending
├── daily_runner.py    # Main pipeline orchestrator
└── runner.py          # Scraper registry & execution

Adding a Scraper

RSS scrapers can extend the provided BaseScraper. Example pattern:

from typing import List
from .base import BaseScraper, Article

class MyScraper(BaseScraper):
   @property
   def rss_urls(self) -> List[str]:
      return ["https://example.com/feed.xml"]

   def get_articles(self, hours: int = 24) -> List[Article]:
      return [Article(**a.model_dump()) for a in super().get_articles(hours)]

Register your scraper in app/runner.py and add a save-handler that stores the parsed items.

For non-RSS sources, implement a custom scraper class exposing get_articles().

Setup

Prerequisites:

• Python 3.12+
• PostgreSQL
• OpenAI API key (or configured LLM credentials)
• SMTP credentials for email delivery

Install and prepare:

1. Install dependencies (project uses uv for virtual environment management):

uv sync

2. Copy example env and set required variables:

cp app/example.env .env
# Then edit .env with your values (OPENAI_API_KEY, DATABASE_URL, MY_EMAIL, APP_PASSWORD, etc.)

3. Initialize the database:

uv run python -m app.database.create_tables

4. Configure sources and user profiles in app/config.py and app/profiles/user_profile.py.

Running

Run the full scheduled pipeline:

uv run main.py

Run individual steps for debugging or development:

# Scrape
uv run python -m app.runner

# Processing
uv run python -m app.services.process_anthropic
uv run python -m app.services.process_youtube
uv run python -m app.services.process_digest

# Curation
uv run python -m app.services.process_curator

# Email
uv run python -m app.services.process_email

Features

• Modular design for easy extension
• LLM-driven summarization and curation
• Personalized digests based on user profiles
• Duplicate prevention for sent digests

Tech Stack

• Python 3.12+
• PostgreSQL + SQLAlchemy
• Pydantic for validation
• OpenAI (or compatible LLM) integration
• feedparser and youtube-transcript-api for scraping

License

This project is available under the MIT License.