Lead Pipeline

Overview

This repository implements a full-featured, AI-powered B2B lead pipeline designed for agencies and SaaS platforms that require scalable, automated outbound lead generation. The pipeline covers the entire journey from raw contact data to enriched, personalized outreach, and is built to be modular, extensible, and production-ready.

Key Features:

• Contact Data Normalization: Cleans and standardizes raw lead data.
• Automated Web Scraping: Gathers company context (website, LinkedIn, etc.) for each contact.
• AI Market Research: Summarizes company background and pain points using LLMs.
• Personalized Outreach: Generates custom first lines/emails based on research.
• Seamless Integration: Pushes enriched leads into outbound platforms (e.g., SalesBlaster AI Voice).
• Batch & Incremental Processing: Supports large-scale, bi-weekly lead drops and incremental enrichment.
• Multi-Tenant Ready: Designed for agencies serving multiple clients with customizable workflows.

---

Use Case & Context

This pipeline was built to power the SalesBlaster DFY High Ticket Offer—a 3-month program for agencies and SMBs to automate and scale outbound campaigns using AI. The system is designed to be generic and configurable, allowing agencies to onboard new clients by simply swapping out a few variables and context files.

Typical Workflow:

1. Input: Upload a CSV of raw leads.
2. Normalization: Standardize and validate contact data.
3. Scraping: Collect company context (website, LinkedIn, etc.).
4. Market Research: Use LLMs to generate a research report for each company.
5. Personalization: Generate a custom first-line/email for each contact.
6. Export: Push enriched leads to outbound platforms for automated campaigns.

---

Project Structure

lead-pipeline/
│
├── .input/                  # All user-supplied data and configuration
│   ├── config.json          # Offer-specific config (pain points, value props, etc.)
│   ├── contacts.csv         # Input CSV(s) of raw leads
│   ├── collection/          # Example configs for different agencies/clients
│   └── ...                  # Prompt variables, templates, etc.
│
├── libs/
│   ├── lead-pipeline/       # Main pipeline logic
│   │   ├── src/
│   │   │   ├── main.ts      # Entry point for running the pipeline
│   │   │   ├── lib/
│   │   │   │   ├── csv/
│   │   │   │   │   └── pipe.ts      # Core CSV processing and enrichment logic
│   │   │   │   ├── llm/
│   │   │   │   │   └── prompts/     # LLM prompt templates and logic
│   │   │   │   └── ...              # Other helpers/utilities
│   │   └── ...
│   └── lead-pipeline-scraper/       # Apify/Crawlee-based web scraper
│       └── ...                      # Scraper code for collecting website markdown
│
├── storage/
│   └── datasets/            # Scraped website data, organized by dataset name
│
├── cli/                     # (Deprecated) Old CLI interface (see notes below)
│
├── README.md
└── package.json

---

How to Use

1. Install Dependencies

cd lead-pipeline
npm install

2. Prepare Input Data

• Place your input CSV(s) in the .input/ directory.
• Edit .input/config.json to reflect your offer, value props, and pain points.
• (Optional) Add or modify prompt templates in .input/ or src/lib/llm/prompts/.

3. Scrape Company Context (Optional but Recommended)

• Use the Apify/Crawlee scraper in libs/lead-pipeline-scraper to collect website markdown for each company.
• Scraped data will be stored in storage/datasets/<dataset_name>/.
• The pipeline will automatically match dataset names to the correct context.

4. Run the Pipeline

# Basic usage
tsx libs/lead-pipeline/src/main.ts <dataset_name>

# With input file
tsx libs/lead-pipeline/src/main.ts <dataset_name> contacts.csv

# With research and AI options
tsx libs/lead-pipeline/src/main.ts <dataset_name> contacts.csv -r 10 -a 10

# Flags:
# -r / --research: Number of new market research entries to generate
# -a / --ai: Number of new personalized lines to generate

• The pipeline will only process rows that do not already have research or personalized lines, allowing for incremental/batch processing.

5. Output

• Enriched, personalized leads are output to the specified location or pushed directly to your outbound platform (e.g., SalesBlaster AI Voice).

---

Customization & Extensibility

• Multi-Tenant Support: To onboard a new agency/client, simply add a new config in .input/collection/ and point the pipeline to the correct dataset.
• Prompt Engineering: All LLM prompts and variables are easily editable in .input/ and src/lib/llm/prompts/.
• Scraper: The web scraper is modular and can be extended to collect additional context (e.g., social profiles, news, etc.).
• Integration: The final step can be customized to push leads to any outbound platform or CRM.

---

Notes

• The /cli directory contains a deprecated CLI interface. The current approach is to run the pipeline directly via tsx for simplicity and reproducibility.
• Make sure to run all commands from the directory containing your .env file for environment variable support.

---

Migration to Workflow Engines

This codebase was built as a proof-of-concept for a fully automated, AI-driven lead pipeline. The next step is to migrate this logic to a workflow engine (ideally with a no-code UI) to:

• Reduce key-person risk
• Enable non-technical users to manage pipelines via a UI
• Support multi-tenant, client-facing workflows
• Leverage pre-built templates and integrations

If you are evaluating this repository for migration or partnership, focus on:

• src/main.ts and src/lib/csv/pipe.ts for the core logic
• .input/ for configuration and prompt management
• libs/lead-pipeline-scraper for scraping context

---

About SalesBlaster DFY High Ticket Offer

The AI Growth Infrastructure program is designed to help agencies and SMBs scale outbound campaigns using AI-driven automation. This repository powers the backend for our DFY (Done-For-You) offer, enabling high-volume, high-quality lead generation and outreach at scale.

---

Happy automating!