**A clean, production-ready system for analyzing data center projects in U.S. interconnection queues.**A production-ready, AI-powered web scraper for extracting data center project intelligence from news articles.
---## Overview
📖 OverviewDataCenter COMPASS automatically discovers and extracts structured data about data center projects from news articles across the web. It uses advanced AI and robust web scraping techniques to handle real-world challenges like SSL certificates, bot detection, and diverse article formats.
DataCenter COMPASS identifies and enriches data center projects from ISO/RTO interconnection queues using:## Key Features
-
Raw data collection from all major U.S. ISOs/RTOs
-
Heuristic-based detection using capacity, naming, and location signals✅ AI-Powered Extraction: OpenAI GPT-4 intelligently extracts multiple projects per article
-
LLM-powered enrichment for deep research on candidates✅ SSL Certificate Bypass: Handles problematic SSL certificates automatically
-
Transparent, layered architecture for reproducible results✅ Multi-Project Detection: Finds multiple data center projects in single articles
✅ Production Ready: Robust error handling and fallback systems
✅ Bot Evasion: Advanced techniques to bypass website blocking
1. Install Dependencies✅ Diverse Source Support: Works with construction news, tech blogs, local news, industry publications
🧪 (Experimental) Grid Queue Intelligence: Optional complementary script to pull ISO interconnection queue data (starting with PJM) using the open-source gridstatus library to analyze power availability timelines.
# Create and activate virtual environment## Quick Start
python3 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate### 1. Install Dependencies
# Install requirements```bash
pip install -r requirements.txtpip install -r requirements.txt
```python -m spacy download en_core_web_sm
Create a .env file in the root directory:
```bashexport OPENAI_API_KEY="your-openai-api-key"
# Required for PJM API access```
PJM_API_KEY=your_pjm_api_key_here
*Note: Without OpenAI API key, the system will use NLP fallback (lower accuracy)*
# Required for LLM enrichment (optional)
AZURE_OPENAI_ENDPOINT=your_azure_endpoint### 3. Extract Data from Articles
AZURE_OPENAI_API_KEY=your_azure_key
AZURE_OPENAI_DEPLOYMENT=your_deployment_name**Single Article:**
AZURE_OPENAI_API_VERSION=2024-02-15-preview```bash
```python datacenter_scraper.py "https://www.constructiondive.com/news/amazon-data-centers-mississippi/706228/"
Get a PJM API key: https://dataminer2.pjm.com/
Multiple Articles:
python datacenter_scraper.py \
# Fetch raw data from all ISOs "https://sanantonioreport.org/san-antonio-vantage-data-centers-microsoft-west-side/" \
python -m queue_pipeline_v2.run fetch --isos ALL "https://www.constructiondive.com/news/google-plans-2-new-data-centers-ohio-columbus-lancaster/649470/"
python -m queue_pipeline_v2.run normalize## Output
python -m queue_pipeline_v2.run signals
python -m queue_pipeline_v2.run curate --profile balanced --target 30{
"total_projects": 5,
### 4. Generate LLM Batches (Optional) "projects": [
{
```python "company_name": "Amazon Web Services",
from queue_pipeline_v2.v2_batch_llm import batch_llm "project_name": null,
from pathlib import Path "location_city": "Madison County",
"location_state": "MS",
batch_llm( "investment_amount": "$10 billion",
curate_dir=Path('queue_pipeline_v2/curate'), "capacity_mw": null,
out_dir=Path('queue_pipeline_v2/batches'), "size_sqft": null,
batch_size=10, "timeline": "partially open by 2027",
limit=30 "project_status": "announced",
) "article_url": "https://www.constructiondive.com/news/amazon-data-centers-mississippi/706228/",
``` "article_title": "",
"extraction_confidence": "high"
--- }
]
## 📁 Project Structure}
DataCenterCOMPASS/### 📊 `datacenter_projects.csv`
│Clean tabular format ready for analysis in Excel, Google Sheets, or data analysis tools.
├── queue_pipeline_v2/ # ⭐ Main pipeline (USE THIS)
│ ├── v2_raw.py # Fetch raw ISO queues## Data Schema
│ ├── v2_normalize.py # Normalize to canonical schema
│ ├── v2_signals.py # Compute DC detection signalsEach extracted project includes:
│ ├── v2_curate.py # Curate high-quality candidates
│ ├── v2_batch_llm.py # Generate LLM enrichment batches| Field | Description | Example |
│ ├── run.py # Unified CLI interface|-------|-------------|---------|
│ ├── README.md # Detailed documentation| `company_name` | Company developing the data center | "Amazon Web Services" |
│ │| `project_name` | Specific project name (if mentioned) | "Prime Data Center Campus" |
│ ├── raw/ # Raw ISO data (generated)| `location_city` | City location | "Madison County" |
│ ├── normalize/ # Normalized data (generated)| `location_state` | State (abbreviated) | "MS" |
│ ├── signals/ # Signal-enriched data (generated)| `investment_amount` | Investment amount | "$10 billion" |
│ ├── curate/ # Curated candidates (generated)| `capacity_mw` | Power capacity in megawatts | "15" |
│ └── batches/ # LLM batches (generated)| `size_sqft` | Size in square feet | "85000" |
│| `timeline` | Construction timeline | "partially open by 2027" |
├── data/ # Data storage| `project_status` | Project status | "announced", "under construction", "operational" |
│ ├── queue_json/ # Archival ISO queue snapshots| `extraction_confidence` | AI confidence level | "high", "medium", "low" |
│ │ └── raw_isos_YYYY-MM-DD/ # Date-stamped raw data| `article_url` | Source article URL | Original URL |
│ └── reference/ # Reference data (cities, states, brands)
│## Performance
├── config/ # Configuration files
│ ├── brands.json # Known DC operator brands**Tested Performance (Production Validation):**
│ ├── cities.json # DC-heavy cities- ✅ **100% Success Rate** across 12 diverse news sources
│ └── states.json # State mappings- ✅ **38 Projects Extracted** from 12 articles (4.2 projects/article average)
│- ✅ **8.1 seconds** average processing time per URL
├── prompts/ # LLM prompt templates- ✅ **100% Data Completeness** for essential fields (company, location, status)
│ ├── deep_research_prompt.txt
│ └── deep_research_enhanced_prompt.txt**Supported Sources:**
│- Construction Industry News (ConstructionDive, etc.)
├── archive/ # Deprecated/old code- Data Center Industry Publications (DataCenterDynamics, etc.)
│ ├── queue_pipeline_v1_2025-10-07/ # Old pipeline (archived)- Local News Outlets
│ ├── root_scripts_2025-10-07/ # Old root scripts (archived)- Tech Company Engineering Blogs
│ ├── deprecated/ # Legacy experimental code- Investment and Infrastructure News
│ └── legacy_queue_refactor_2025-09-26/ # Historical refactors
│## System Requirements
├── .env # Environment variables (not in git)
├── .env.example # Environment template- Python 3.8+
├── .gitignore # Git ignore rules- Internet connection
├── requirements.txt # Python dependencies- OpenAI API key (recommended)
└── README.md # This file- ~100MB disk space
openai>=1.0.0- AI extraction
newspaper3k>=0.2.8- Article extraction
requests>=2.31.0- HTTP requests
Collect raw interconnection queue data from all ISOs and save for reference:- fake-useragent>=1.4.0 - Bot evasion
gridstatus>=0.30.1,<0.32- (Experimental) ISO interconnection queue data (PJM, expandable)
# Fetch to date-stamped folder### PJM Interconnection Queue (Now Included by Default)
python -m queue_pipeline_v2.run fetch \
--isos ALL \The ingestion pipeline under `queue_pipeline/ingestion/fetch_queue.py` now ALWAYS attempts to include **PJM** interconnection queue data using the public fetch implemented in the `gridstatus` library. An API key is **not required** for the PJM queue itself (older code previously skipped PJM when `PJM_API_KEY` was absent). If you provide `PJM_API_KEY`, it will simply be passed through (useful only for other authenticated PJM endpoints not yet used here).
--out-dir data/queue_json/raw_isos_$(date +%Y-%m-%d) \
--insecureBasic one‑liner to fetch & score all ISOs including PJM (year ≥ 2020):
```bash
# Result: ~23,000+ projects from 7 ISOspython -m queue_pipeline.ingestion.fetch_queue --iso ALL --queue-year-min 2020 --out queue_pipeline/qp_outputs/raw_candidates.jsonl
Output: data/queue_json/raw_isos_YYYY-MM-DD/#### Focusing on PJM Load / Data Center Candidates
-
PJM.raw.jsonl(9,251 projects) -
MISO.raw.jsonl(3,549 projects)Add the new--pjm-load-onlyflag to restrict PJM rows to those heuristically classified as load (resource_typeLOADorLARGE_LOAD, or rows with a structured load indicator): -
SPP.raw.jsonl(2,798 projects)```bash -
CAISO.raw.jsonl(2,278 projects)python -m queue_pipeline.ingestion.fetch_queue --iso PJM --queue-year-min 2020 --pjm-load-only --recall-bias --out queue_pipeline/qp_outputs/pjm_load_candidates.jsonl -
NYISO.raw.jsonl(2,120 projects)``` -
ERCOT.raw.jsonl(1,847 projects) -
ISONE.raw.jsonl(1,751 projects)Combine with other ISOs while filtering only PJM rows:
### Workflow 2: Identify Data Center Candidatespython -m queue_pipeline.ingestion.fetch_queue --iso ALL --queue-year-min 2020 --pjm-load-only --recall-bias --limit 300
Run the full pipeline to find and curate data center candidates:
# Step 1: Fetch (to default pipeline location)
python -m queue_pipeline_v2.run fetch --isos ALL#### Output Signals (Relevant for PJM Load Filtering)
The filter relies on:
# Step 2: Normalize- `resource_type` in {`LOAD`, `LARGE_LOAD`}
python -m queue_pipeline_v2.run normalize- `structured_load_flag` (derived from structured PJM columns containing load / data center indicators)
# Step 3: Compute signalsRows outside those criteria are excluded only for PJM when the flag is set; other ISOs are unaffected.
python -m queue_pipeline_v2.run signals
> Tip: Pair with `--prune-nonload` (if/when ported to the orchestrator) for a fully load-focused multi-ISO dataset.
# Step 4: Curate (adjust --target for more/fewer candidates)
python -m queue_pipeline_v2.run curate --profile balanced --target 30
```## Error Handling
**Output:** `queue_pipeline_v2/curate/`The system includes comprehensive error handling:
- `full_enrichment.jsonl` - Complete candidate data- **SSL Certificate Issues**: Automatic bypass
- `minimal_review.jsonl` - Minimal fields for quick review- **Bot Detection**: Multiple user agent strategies
- **Network Timeouts**: Configurable timeouts and retries
### Workflow 3: Focus on Single ISO (PJM)- **Content Extraction Failures**: Multiple parsing strategies
- **AI API Failures**: Automatic NLP fallback
```bash
# Fetch only PJM## Production Deployment
python -m queue_pipeline_v2.run fetch --isos PJM
This system is **production-ready** and includes:
# Run remaining steps as normal- Comprehensive error handling and logging
python -m queue_pipeline_v2.run normalize- SSL certificate bypass for problematic sites
python -m queue_pipeline_v2.run signals- Multiple fallback strategies
python -m queue_pipeline_v2.run curate --profile balanced --target 30- Clean, structured output
```- Automatic timestamped backups
### Workflow 4: Prepare for LLM Enrichment## License
```pythonMIT License - See LICENSE file for details.
# Generate batches for manual or automated LLM enrichment
python << 'EOF'## Support
from queue_pipeline_v2.v2_batch_llm import batch_llm
from pathlib import PathFor issues or questions, please check the existing documentation or create an issue in the repository.
batch_llm(---
curate_dir=Path('queue_pipeline_v2/curate'),## 🧪 (Archived) Unified Data Center Interconnection Queue Pipeline (`dc_queue`)
out_dir=Path('queue_pipeline_v2/batches'),
batch_size=10, # Projects per batchThe original experimental `dc_queue/` package has been fully archived under `archive/legacy_queue_refactor_2025-09-26/` and removed from the active codebase. A fresh, simplified, and modular implementation now lives under `queue_pipeline/` and should be used for all future ingestion, preparation, curation, batching, validation, and summarization tasks.
limit=30, # Total projects to batch
use_full=False # Use minimal_review.jsonlIf you are looking for the old scripts referenced below (e.g. `dc_queue.prepare_llm_batches`, `make_llm_batches`, `profile_prepared`, `quick_grid.py`), they now exist only in the archive for historical comparison. Commands in this README referencing `dc_queue` are deprecated; migrate to the orchestrator:
)
EOF```
```python -m queue_pipeline.qp ingest --iso ALL --year-min 2020 --limit 150 --recall-bias
Output: queue_pipeline_v2/batches/
-
llm_batch_0001.jsonl(10 projects)Additional stages (prepare, curate, batch, validate, summarize) will be added toqpas they are ported. -
llm_batch_0002.jsonl(10 projects) -
llm_batch_0003.jsonl(10 projects)### Why a New Pipeline? -
BATCH_SUMMARY.md(documentation)Heterogeneous ISO schemas, evolving service codes, and the need to combine structural + textual + contextual signals (geo clusters, developer roles) required a layered, extensible architecture instead of ad‑hoc heuristics.
---### Architecture Overview
## 🔧 Configurationdc_queue/
adapters/ # ISO-specific raw -> canonical mapping functions
### Curation Profiles core/ # Normalization primitives
enrichment/ # Cluster & developer role feature augmentation
Control the balance between recall and precision: config/ # JSON config (geo clusters, developer roles, service mapping)
run.py # Orchestrated pipeline CLI (preferred entrypoint)
| Profile | Description | Use Case |```
|---------|-------------|----------|
| `recall` | Maximizes candidates found | Initial exploration |### 🔍 LLM / RAG Integration Guide (Context & Retrieval)
| `balanced` | **Default** - Good mix | Most use cases |
| `precision` | Minimizes false positives | High-confidence only |The pipeline now supports rich LLM-oriented context generation with two optional flags:
| `exploratory` | Experimental signals | Research & development |
| Flag | Purpose | Output Columns Added |
```bash|------|---------|----------------------|
# Example: High recall (more candidates)| `--llm-context` | Adds verbose, labeled context & search query fields | `llm_context`, `llm_search_query` |
python -m queue_pipeline_v2.run curate --profile recall --target 50| `--llm-compact` | Compresses labels & replaces long phrases (token efficient) | Shortened `llm_context` |
When --llm-context is used without explicit JSON options, the pipeline auto-enables JSON Lines export with a curated column subset suitable for ingestion into a vector store or downstream RAG index.
Example (verbose):
| Variable | Required | Description |```
|----------|----------|-------------|python -m dc_queue.run --iso ALL --recall-bias --year-min 2020 --llm-context --limit 500
| PJM_API_KEY | Yes | PJM API access key |```
| AZURE_OPENAI_* | No | For LLM enrichment |
Compact mode:
---```
python -m dc_queue.run --iso ALL --recall-bias --llm-context --llm-compact --limit 500
Pipe-delimited segments capturing high-signal attributes:
| Field | Type | Description |```
|-------|------|-------------|ISO=SPP | ID=GEN-2025-SR9 | State=OK | County='Fort Gibson' | ReqType='Energy Resource' | CapMW=410.0 | Submitted=2025-05-31 00:00:00 | Scores S=0.55 T=0.00 E=0.00 Total=0.55 | Tier=LOW
| iso | string | ISO/RTO identifier (PJM, MISO, etc.) |```
| iso_native_id | string | Original queue ID from ISO |Segments (if available):
| project_name | string | Project name |ISO, ID, Project, Entity, State, County, ReqType, CapMW, LoadMW, Submitted, ProposedISD, Scores (S/T/E/Total), Tier, Status, Stage
| capacity_mw | float | Capacity in megawatts |
| state | string | U.S. state code |Suppressed automatically:
| county | string | County name |- Empty / null-like fields (e.g., project_name == None)
| submitted_dt | datetime | Queue submission date |- ProposedISD if datetime missing / NaT
| proposed_in_service_dt | datetime | Proposed online date |
| status | string | Project status |#### Compact Mode Differences (--llm-compact)
- Removes pipe separators (
|) and joins tokens with spaces
- Maps long request types back to short codes (Energy Resource→ER, Network Resource→NR)
| Field | Description |- Keeps score component abbreviations (S, T, E) and Tier
|-------|-------------|
| dc_signal_strength | weak/medium/strong |Lifecycle tokens (Status=<class> and Stage=<funnel>) are appended when available; they help downstream retrieval distinguish active vs withdrawn vs operational rows.
| candidate_confidence | 0.0 - 1.0 score |
| signals.capacity_flag | True if capacity matches DC profile |Example compact:
| signals.name_flag | True if name suggests DC |```
| signals.brand_flag | True if known DC operator |ISO=SPP ID=GEN-2025-SR9 State=OK County='Fort Gibson' ReqType='ER' CapMW=410.0 Sub=2025-05-31 00:00:00 S=0.55 T=0.00 E=0.00 Total=0.55 Tier=LOW
---
#### `llm_search_query` Field
## 🧪 Testing & ValidationConstructed to seed external news / document search or retrieval expansion.
Token sources (conditionally added):
### Quick Validation- Entity (developer/operator)
- Project name (if present)
```bash- County + State or State alone
# Check raw data- ISO
wc -l queue_pipeline_v2/raw/*.jsonl- Year token (from `submitted_dt`)
- Capacity bucket ("50 MW" if ≥50, "100 MW" if ≥100)
# Check normalized- Select lifecycle token (only if `withdrawn`, `canceled`, or `in_service`) to bias search toward status-confirming evidence
wc -l queue_pipeline_v2/normalize/normalized.jsonl
Example:
# View top candidates```
head -n 5 queue_pipeline_v2/curate/full_enrichment.jsonl | python -m json.toolFort Gibson OK SPP 2025 100 MW
Example: Count by ISOYou can feed this directly into web/news/API search, then attempt to align returned article metadata against the structured queue row.
import jsonFor RAG, prefer storing BOTH:
from collections import Counter1. `llm_context` (as the retrievable text) — high signal density
2. A structured JSON object (original fields) — for programmatic filters (e.g., capacity_mw ≥ 100)
with open('queue_pipeline_v2/signals/signals.jsonl') as f:
isos = [json.loads(line)['iso'] for line in f]Recommended JSONL ingestion schema:
```jsonl
print(Counter(isos)){"id": "SPP:GEN-2025-SR9", "text": "ISO=SPP ... Tier=LOW", "iso": "SPP", "iso_native_id": "GEN-2025-SR9", "state": "OK", "county": "Fort Gibson", "capacity_mw": 410.0, "dc_score": 0.55, "dc_tier": "LOW"}
---#### Retrieval Prompting Pattern
After retrieving top-k rows given a user/article snippet, instruct the model to cross-check:
You are linking news articles to interconnection queue entries. Given:
-
Pipeline Details:
queue_pipeline_v2/README.mdArticle snippet: "A 400 MW data center load proposed in eastern Oklahoma..." -
ISO Data Reference:
data/queue_json/QUICK_REFERENCE.mdCandidate rows: -
Batch Summary:
queue_pipeline_v2/batches/BATCH_SUMMARY.md1. ISO=SPP ID=GEN-2025-SR9 State=OK County='Fort Gibson' CapMW=410.0 ... -
Old System:
archive/queue_pipeline_v1_2025-10-07/ARCHIVED_README.md2. ISO=SPP ID=GEN-2025-SR6 State=OK County='Minco' CapMW=200.0 ... -
Scraper Examples:
archive/root_scripts_2025-10-07/datacenter_scraper.pyReturn the most likely match with reasoning (or NONE if mismatch).
---
### 🚀 MVP: From Raw Queue Candidates to Enriched DC Insights
## 🗄️ Archive Policy
This lightweight flow turns `pipeline_candidates.jsonl` into an analytics-ready enriched dataset with model-confirmed data center classifications, temporal metrics, and summary stats.
Old/deprecated code is preserved in `archive/` with documentation:
- **Don't delete** - Historical reference value#### 1. Prepare LLM Candidates
- **Don't modify** - Frozen snapshotsDEPRECATED (old prepare step shown for archival reference only):
- **Don't use** - Active development uses `queue_pipeline_v2/````bash
# OLD (archived) -- do not use
---python -m dc_queue.prepare_llm_batches --year 2025
When adding new features:python -m queue_pipeline.qp prepare --year 2025
-
Use queue_pipeline_v2/ - Don't create new top-level scripts
-
Follow layered architecture - raw → normalize → signals → curate##### Heuristic Enhancements (v2)
-
Add tests - Validate with real data samplesThe prepare step now adds richer pre-LLM discriminators:
-
Document - Update READMEs and add examples
-
Preserve provenance - Keep original field names in raw layer| Field | Meaning |
|-------|---------|
---| dc_heuristic_score | 0..3 additive score: regex DC terms + brand + structural (≥300MW in key states) minus 0.5 if generation only |
| dc_likelihood_tier | HIGH (≥2), MED (≥1), LOW (>0), NONE (≤0) |
| generation_flag | Contains generation tokens (solar, wind, battery, etc.) |
Internal research tool - not for public distribution.| structural_dc_flag | Large-capacity structural inference |
| negative_reason | GENERATION_ONLY or NO_DC_SIGNAL for NONE tier rows |
---| priority_bucket | New ordering: withdrawn_dc_high, dc_high, dc_mid, withdrawn, high_value, mid_value, other |
Deprecated old profiling command (archived):
Add --insecure flag to fetch command:python -m dc_queue.profile_prepared --prepared dc_queue_pipeline_outputs/prepared_llm_candidates.jsonl
bash
python -m queue_pipeline_v2.run fetch --isos ALL --insecureNew command (after tool port) will be:
python -m queue_pipeline.qp profile
### PJM API Key Not Working```
Outputs tier mix, score histogram, brand × generation table, withdrawn rate by tier.
1. Check `.env` file has `PJM_API_KEY=your_key`
2. Verify key at https://dataminer2.pjm.com/#### 2. Create Batches (choose round_robin for early diversity)
3. Test with: `python -c "import os; from dotenv import load_dotenv; load_dotenv(); print(os.getenv('PJM_API_KEY'))"`Deprecated batching (archived):
```bash
### No Candidates Found# OLD (archived)
python -m dc_queue.make_llm_batches --batch-size 40 --mix-mode round_robin
- Try `--profile recall` for higher sensitivity```
- Check `signals/signals.jsonl` for signal distributionsFuture replacement:
- Verify raw data was fetched: `ls queue_pipeline_v2/raw/````bash
python -m queue_pipeline.qp batch --batch-size 40 --mix-mode round_robin --ensure-likely-dc 12
### Empty Batch Files```
Send each `*_prompt_strict.txt` to the model, capture only lines between BEGIN_JSONL / END_JSONL into `llm_results/batchXXXX.jsonl`.
- Ensure curate step produced results: `wc -l queue_pipeline_v2/curate/*.jsonl`
- Check `--limit` parameter in batch_llm()#### 3. Concatenate Model Output
- Verify `minimal_review.jsonl` exists```bash
cat llm_results/batch*.jsonl > llm_results/full_all.jsonl
---```
## 🎯 Roadmap#### 4. Validate (heuristic precision/recall sanity check)
```bash
- [ ] Add automated LLM enrichment workflowpython -m dc_queue.validate_llm_results \
- [ ] Create visualization dashboard for candidates --prepared dc_queue_pipeline_outputs/prepared_llm_candidates.jsonl \
- [ ] Add historical trend analysis --llm-output llm_results/full_all.jsonl \
- [ ] Expand to international ISOs/TSOs --dc-threshold 0.75
- [ ] Add project clustering by location/operator```
---#### 5. Export Classification-Only (clean minimal schema)
```bash
**Last Updated:** October 7, 2025 python -m dc_queue.export_classification_only \
**Version:** 2.0 (Clean Architecture) --llm-output llm_results/full_all.jsonl \
--out llm_results/full_all_classification_only.jsonl
```
#### 6. Merge Enrichment (adds queue_days, lifecycle, evidence counts)
```bash
python -m dc_queue.merge_llm_enrichment \
--prepared dc_queue_pipeline_outputs/prepared_llm_candidates.jsonl \
--llm-output llm_results/full_all.jsonl \
--out dc_queue_pipeline_outputs/enriched_candidates.jsonl \
--dc-threshold 0.75
```
#### 7. Summarize & Generate Charts
```bash
python -m dc_queue.summarize_enriched \
--enriched dc_queue_pipeline_outputs/enriched_candidates.jsonl \
--summary-out dc_queue_pipeline_outputs/enriched_summary.json \
--charts-dir figures/enriched_mvp --html
```
Optional: `pip install plotly` for HTML chart generation.
#### 8. Visual / Dashboard Inputs
Key derived fields after merge:
| Field | Meaning |
|-------|---------|
| `dc_confirmed_flag` | Model-high-confidence DC load classification |
| `queue_days` | Days since submission (approx latency so far) |
| `lead_time_days` | (If proposed date exists) planned construction window |
| `lifecycle_stage` | `active_dc_candidate`, `withdrawn_candidate`, `generic_generation_candidate`, `ambiguous_candidate` |
| `evidence_snippet_count` | Quick proxy for support robustness |
#### 9. MVP Storytelling Angles
Use `enriched_summary.json` + charts to answer:
* How many MW of confirmed data center load are active vs withdrawn?
* Average days_in_queue for confirmed DC projects vs generic generation.
* Which ISOs or states show longer queue latency (extend summarizer grouping if needed).
* Withdrawn share and (once root causes appear) leading withdrawal reasons.
#### 10. Extending to Dashboards
Feed `enriched_candidates.jsonl` into:
* A lightweight Streamlit or Dash prototype
* A BI tool (convert to Parquet/CSV) filtering `dc_confirmed_flag==True`
---
Future enhancements (see CLEAN_WORKFLOW.md) include timeline phase segmentation, geographic heatmaps, and confidence calibration plots.
### Clean Restart (Mini 30-Row Diversified Test)
Archive/remove prior artifacts and generate a fresh curated mini set with new heuristics:
```bash
python -m dc_queue.reset_pipeline_artifacts --archive
python -m dc_queue.fetch_queue_candidates --iso ALL --year-min 2020 --recall-bias --out dc_queue_pipeline_outputs/pipeline_candidates.jsonl
python -m dc_queue.prepare_llm_batches --candidates-json dc_queue_pipeline_outputs/pipeline_candidates.jsonl --output dc_queue_pipeline_outputs/prepared_llm_candidates.jsonl --year 2025 --config-dir config
python -m dc_queue.profile_prepared --prepared dc_queue_pipeline_outputs/prepared_llm_candidates.jsonl
python -m dc_queue.curate_mini_dataset --prepared dc_queue_pipeline_outputs/prepared_llm_candidates.jsonl --total 30 --min-high 4 --min-med 4
python -m dc_queue.make_llm_batches --prepared dc_queue_pipeline_outputs/mini_curated_30.jsonl --out-dir dc_queue_pipeline_outputs/llm_batches_mini --batch-size 15 --mix-mode round_robin --ensure-likely-dc 8 --strict-prompts --download-friendly
```
Then send the two prompt files for classification, validate, merge, and summarize as before.
#### Post-Retrieval Heuristic Filters (Optional)
Apply lightweight rules before model invocation to reduce token usage:
- Drop rows where `abs(capacity_mw - mentioned_capacity) / mentioned_capacity > 0.6`
- Prefer same state; penalize county mismatch unless only one candidate remains
- Boost recency (lower `queue_age_days` if available)
#### Potential Enhancements (Future Flags)
- `--llm-verify` (planned): run an LLM sanity pass to produce structured justification & false-positive reduction
- `--llm-match <article_jsonl>`: batch entity/timeline linking between articles and queue rows
- Optional capacity bucket customization (e.g. 25/75/150 MW)
#### Custom Capacity Buckets
To customize buckets now, you can fork and adjust the logic in `_mk_search` (capacity threshold section) inside `dc_queue/run.py`.
#### Including Additional Fields
If future adapters retain richer raw fields (e.g., substation name, phase status), extend `_mk_context` to append them; stable label prefixes keep embeddings robust.
#### Best Practices Summary
| Goal | Recommendation |
|------|----------------|
| Minimize tokens | Use `--llm-compact`, drop low-signal numeric columns from JSONL |
| Max recall | Enable `--recall-bias`, ingest all candidates (Tier LOW + MEDIUM + HIGH) |
| Precision triage | Filter to `dc_tier != 'REJECT'` and capacity ≥ threshold |
| Temporal focus | Use `--year-min 2020` (or relevant recent cutoff) |
| External linking | Use `llm_search_query` to drive article/API search |
---
Canonical Fields (partial):
`iso, iso_native_id, project_name, entity, request_type, request_category, capacity_mw, state, submitted_dt, proposed_in_service_dt, queue_age_days, in_service_lead_days, queue_status_raw, queue_status_class, status_stage, withdrawn_flag, canceled_flag, in_service_flag, dc_score, dc_flag, dc_flags_explain, dc_tier`
### Core Stages
1. Fetch: multi-ISO pull via `gridstatus` (PJM requires `PJM_API_KEY`).
2. Adapter Normalize: ISO-specific column extraction → canonical schema.
3. Enrich: add `cluster_weight`, `developer_category`, role bonuses/penalties.
4. Temporal: derive `queue_age_days`, `in_service_lead_days` (timezone-safe UTC).
5. Classify: advanced scoring (structural + textual + enrichment) with optional recall bias.
6. Tier: map score to `HIGH | MEDIUM | LOW | REJECT` for triage.
7. Status Lifecycle: normalize raw queue status to `queue_status_class`, flags (`withdrawn_flag`, `canceled_flag`, etc.), and funnel stage (`status_stage`).
8. Output: full CSV, candidate CSV, summary JSON.
### Lifecycle / Status Normalization
Added module `status.py` derives canonical lifecycle taxonomy:
| Column | Meaning |
|--------|---------|
| `queue_status_raw` | Raw text from ISO sheet (verbatim) |
| `queue_status_class` | Canonical bucket {active, withdrawn, canceled, suspended, in_service, unknown} |
| `status_stage` | Funnel stage {early, study, permitting, construction, operational, dead, unknown} |
| `withdrawn_flag` | True if withdrawn / removed |
| `canceled_flag` | Explicitly canceled / terminated |
| `suspended_flag` | Temporarily on hold |
| `in_service_flag` | Commercial / operational |
| `active_flag` | Still progressing (non-withdrawn/canceled/suspended) |
| `opposition_flag` | Placeholder (future: external signals / news NLP) |
| `status_notes` | Short derived explanation |
These fields enable attrition rate analytics (e.g., withdrawn vs total by county) and guide enrichment queries.
### LLM-Ready Preset (`--llm-ready`)
One-command bundling for downstream enrichment:
```
python -m dc_queue.run --iso ALL --year-min 2020 --llm-ready --recall-bias
```
Enables: compact LLM context, generation noise suppression, capacity floor (≥40 MW), lifecycle columns, curated JSONL export, and search-friendly query strings.
### Run (Recommended)
Recall-biased (max capture for later LLM pruning):
```bash
python -m dc_queue.run --iso ALL --limit 500 --recall-bias
```
Higher precision mode:
```bash
python -m dc_queue.run --iso ALL --limit 500
```
Outputs (default `dc_queue_pipeline_outputs/` unless overridden):
| File | Description |
|------|-------------|
| `pipeline_full.csv` | All processed rows with enrichment + scoring + tiers |
| `pipeline_candidates.csv` | Subset where `dc_flag == True` |
| `pipeline_summary.json` | Aggregated statistics & threshold metadata |
Summary JSON Keys:
`isos_processed, rows_total, candidates, candidate_pct, recall_bias, threshold, tiers_distribution, capacity_stats`
### Research-Oriented Signals & Slices
These new fields enable fast, programmatic branching of analysis (model training vs strategic scanning vs root-cause research):
| Field | Type | Meaning |
|-------|------|---------|
| `generation_noise_flag` | bool | Heuristic: row text/name looks like pure generation (solar / wind / storage) |
| `generation_override_flag` | bool | True if generation-looking row also contains strong DC override terms (kept as candidate) |
| `opposition_flag` | bool | Raw status / notes contain opposition / denial / protest keywords |
| `opposition_terms` | str | Comma-joined matched opposition keywords (debugging / transparency) |
| `capacity_band` | str | Capacity bucket used for attrition aggregation (0-50, 50-100, 100-200, 200-500, 500+) |
| `attrition_rate` | float | Laplace-smoothed withdrawn/canceled rate for (state, capacity_band) |
| `attrition_risk_tier` | str | {very_low, low, moderate, high, very_high, unknown} derived from `attrition_rate` thresholds |
| `status_reason` | str? | Heuristic reason extraction (economic, transmission, permitting, environmental, withdrawn_unspecified) |
#### How to Use the New Signals (Programmatic Filters)
Common research slices can be expressed directly (same logic the optional slice exporter uses):
1. Filter out likely false positives while keeping them for modeling (negative examples):
```
generation_noise_flag == True and generation_override_flag == False
```
2. Prioritize stable projects (lower local attrition risk + still active):
```
attrition_risk_tier in ['very_low','low'] and queue_status_class == 'active'
```
3. Investigate potential local pushback:
```
opposition_flag == True
```
4. Funnel targeted withdrawn root-cause follow-up (permits / transmission constraints):
```
withdrawn_flag == True and status_reason in ('permitting','transmission')
```
#### Generation Noise Handling (Suppress vs Tag)
Two mutually exclusive modes:
* `--suppress-generation-noise`: Drop rows classified as pure generation (fast precision boost, but loses negatives for model training).
* `--tag-generation-noise`: Keep rows and set `generation_noise_flag`; downstream filtering can exclude them while preserving data for future classifier refinement. If both flags supplied, tagging wins (no drops).
Override Behavior: If a row matches generation heuristics BUT also contains strong DC override terms (brand + data center token), `generation_override_flag` becomes True (kept as candidate regardless of suppression) and the row is not dropped.
#### Opposition & Attrition
`opposition_flag` / `opposition_terms` are simple keyword scans (e.g., "opposition", "protest", "denial", "lawsuit", "concern"). They are intentionally recall-biased. Future versions may externalize the term list into config and add NLP sentiment classification.
`attrition_rate` is computed with Laplace smoothing per (state, capacity_band): `(withdrawn+canceled + 1) / (total + 2)`. Tiers:
* `very_high` ≥ 0.60
* `high` ≥ 0.45
* `moderate` ≥ 0.30
* `low` ≥ 0.15
* `very_low` < 0.15
* `unknown` insufficient data / missing bucket
Use low/very_low + active for stable pipeline emphasis; investigate high/very_high clusters for systemic risk or permitting friction.
#### Status Reason Heuristic
`--infer-status-reasons` enables a lightweight pattern pass over `queue_status_raw` (lowercased) mapping to provisional buckets:
* economic → contains economics / cost / financial
* transmission → contains transmission / upgrade
* permitting → contains permit / zoning
* environmental → contains environment / wetland
* withdrawn_unspecified → contains withdraw (no clearer anchor)
These are placeholders for a future LLM-backed enrichment pass; absence of a match leaves `status_reason` null.
#### Automated Slice Export
Enable `--export-research-slices` to auto-write canonical subsets in `dc_queue_pipeline_outputs/slices/` (CSV + JSONL):
| Slice Filename Stem | Filter Logic |
|---------------------|--------------|
| `slice_generation_noise_negative` | generation_noise_flag == True and generation_override_flag == False |
| `slice_stable_active_low_attrition` | attrition_risk_tier in (very_low, low) and queue_status_class == 'active' |
| `slice_opposition` | opposition_flag == True |
| `slice_withdrawn_root_cause` | withdrawn_flag == True and status_reason in ('permitting','transmission') |
Each slice export respects `--json-cols` (if set) and also writes `slices_summary.json` with counts. Warning printed if `--limit` was applied (slices may be truncated).
Example combined run (recall-biased, LLM context, slices):
```bash
python -m dc_queue.run --iso ALL --year-min 2020 --recall-bias \
--llm-context --llm-compact --tag-generation-noise \
--infer-status-reasons --export-research-slices --limit 500
```
Resulting directories:
```
dc_queue_pipeline_outputs/
pipeline_full.csv
pipeline_candidates.csv
pipeline_summary.json
pipeline_full.jsonl (if JSON export enabled)
slices/
slice_generation_noise_negative.csv
slice_stable_active_low_attrition.csv
slice_opposition.csv
slice_withdrawn_root_cause.csv
slices_summary.json
```
Use these slices directly to seed downstream investigations (e.g., permitting bottleneck study, local opposition heatmap, negative training set curation).
### Scoring Highlights
Signals (illustrative — see `dc_load_classifier.py` for specifics):
- Structural: request/service type patterns, absence of generation fuel, capacity thresholds.
- Textual: strong vs weak DC terms, company/brand names, geographic DC cluster hits.
- Enrichment: `cluster_weight` scaling, `developer_category` bonus (hyperscale > colo > generic), penalties for generation-only hints.
Recall Bias Mode lowers the candidate threshold (e.g., 0.55 → 0.45) to minimize false negatives at the cost of more manual/LLM review.
### Migration from `dc_queue_filter.py`
`dc_queue_filter.py` is now a thin wrapper delegating to `dc_queue.run` and will be removed in a future cleanup.
Update automation to call:
```bash
python -m dc_queue.run --iso ALL --recall-bias --limit 500
```
### Configuration Files
| File | Purpose |
|------|---------|
| `config/cluster_locations.json` | Geo clusters with weights (e.g., Loudoun, Prince William, Santa Clara) |
| `config/developer_roles.json` | Categorizes developers (hyperscale, colocation, infrastructure) |
| `config/service_type_mapping.json` | (Planned) Maps ISO-specific service codes → canonical `request_category` |
Tuning: edit JSON, re-run pipeline; changes applied without code modification.
### Temporal Safety
All datetime computations forced to UTC-aware to avoid tz-naive subtraction errors (`queue_age_days` stability fix implemented 2025-09-26).
### Roadmap (Next Iterations)
1. Integrate `service_type_mapping.json` into adapters for deterministic `request_category` classification.
2. Add structured confidence breakdown (separate structural vs textual vs enrichment score components).
3. Introduce optional LLM validation layer (entity verification & generation disambiguation) behind `--llm-verify`.
4. Add caching & incremental diff mode (only new/changed queue rows).
5. Calibration notebook: precision/recall tuning with manually labeled sample.
### Quick Diff vs Legacy Tools
| Aspect | Old (`dc_queue_filter.py`) | New (`dc_queue.run`) |
|--------|---------------------------|-----------------------|
| Enrichment | Minimal | Geo clusters + developer roles + temporal |
| Tiers | No | Yes (`dc_tier`) |
| Temporal Features | None | `queue_age_days`, `in_service_lead_days` |
| Config JSON | Limited | Centralized + extensible |
| Wrapper Support | N/A | Legacy script delegates |
| Future LLM Hook | Manual | Planned flag + modular insertion |
### Troubleshooting
- Missing PJM data: ensure `PJM_API_KEY` exported (`echo $PJM_API_KEY`).
- SSL warnings: pass `--secure` to enforce verification, or omit for convenience (insecure mode prints a warning).
- Extremely high candidate rate (>85%): adjust threshold in classifier or remove recall bias.
- Empty outputs: network/API errors or schema shift; re-run with `--debug` and inspect intermediate CSV.
### Contributing
Open issues with: ISO name, sample row (sanitized), desired canonical mapping, and rationale for new signals.
---
## 🧪 Experimental: Interconnection Queue Data (Enhanced Tooling)
Two complementary utilities now support early-stage analysis of ISO interconnection queues for potential large data center (load) signals.
### 1. `quick_grid.py` (Fast CSV Slice)
Focused on rapid filtering & optional heuristic / config-driven classification.
Key Flags:
- `--classify` – adds dc_likelihood / dc_candidate columns (uses `dc_classifier` if present)
- `--candidates-only` – retain only rows flagged dc_candidate (implies `--classify`)
- `--export-json-meta` – writes per-ISO lightweight schema + 10-row sample to `data/queue_json/`
Example:
```bash
python quick_grid.py --iso ALL --limit 100 --classify --export-json-meta
```
Outputs:
- `quick_queue_filtered.csv` – full filtered set
- `quick_queue_filtered_slim.csv` – subset of key columns (when `--classify`)
- `data/queue_json/<iso>_queue_meta.json` – schema + sample snapshot
### 2. `export_iso_queue_json.py` (Schema & Raw JSON Export)
Purpose-built for deeper schema discovery and downstream normalization planning.
Flags:
- `--iso ALL` or comma list
- `--limit N` – limit normalized CSV rows (does not change raw schema)
- `--sample N` – number of sample rows in meta JSON (default 15)
- `--no-raw` – skip large raw JSON file (only meta + normalized)
- `--secure` – enable SSL verification (default disabled for convenience)
Example:
```bash
python export_iso_queue_json.py --iso PJM,SPP,NYISO --limit 250 --sample 20
```
Artifacts (written to `data/queue_json_full/`):
- `<iso>_meta.json` – field list, dtype map, sample rows (datetime-safe)
- `<iso>_raw.json` – full raw queue dump (omit with `--no-raw`)
- `<iso>_normalized.csv` – light canonical slice (`capacity_mw`, `state`, `submitted_dt`, etc.)
### Config‑Driven Classifier (`dc_classifier.py`)
Moves beyond embedded keyword lists. Loads brand & geo lists from `config/brands.json` and `config/cities.json` (with fallbacks). Scoring is additive + transparent:
| Component | Weight |
|-----------|--------|
| Strong DC term (e.g., "data center", "hyperscale") | 0.55 |
| Weak infra term (e.g., "campus", "facility") | 0.30 |
| Brand hit (AWS, Microsoft, Equinix, etc.) | 0.25 |
| Geo cluster hit (Ashburn, Loudoun, etc.) | 0.18 |
| Generation penalty (no strong DC term) | -0.50 |
Candidate condition: score ≥ 0.50 and not generation-only.
### Known Limitations (Active Work Area)
- Raw queue schemas differ significantly by ISO (naming, presence of load vs generation flags).
- Load vs generation determination still heuristic; needs structural signals (e.g., service type codes) integrated.
- No current disambiguation of utility-owned generation vs. dedicated campus load proxies.
### Planned Next Steps (Roadmap)
1. Field Frequency Analyzer – aggregate column presence across all ISOs (meta JSON driven).
2. Canonical Normalization Module (`queue_normalizer.py`) – ISO adapters mapping raw → canonical keys.
3. Enhanced Classifier – incorporate structural signals (request type, absence of fuel code) before textual.
4. Labeled Sample Set – manual annotation of ~100 rows for precision/recall tuning.
5. Enrichment – join county-level DC cluster index & operator category list.
### NEW: Advanced Data Center Queue Filter
`dc_queue_filter.py` performs a multi-stage pipeline:
1. Fetch raw queues (multi-ISO) via `gridstatus`.
2. Normalize into canonical fields (`queue_normalizer.py`).
3. Apply advanced recall-biased classifier (`dc_load_classifier.py`).
4. Emit full classified CSV, slim candidate CSV, and JSON summary.
Run (precision mode):
```bash
python dc_queue_filter.py --iso ALL --limit 500
```
Run (recall-biased mode – recommended for downstream LLM pruning):
```bash
python dc_queue_filter.py --iso ALL --limit 500 --recall-bias
```
Outputs (default `dc_queue_outputs/`):
- `dc_queue_classified_full.csv` – All normalized rows with `dc_score` & components.
- `dc_queue_candidates.csv` – Subset where `dc_flag` is True.
- `dc_queue_summary.json` – High-level stats (candidate percentage, capacity distribution, thresholds).
Classifier Signal Layers:
- Structural: service/request type contains load/network/service, absence of fuel with capacity, large MW thresholds (≥50 / ≥100).
- Textual: strong/weak DC phrases, brand hits, geo cluster hits.
- Penalty: explicit generation fuel without a load service anchor.
Bias Strategy:
- `--recall-bias` lowers threshold (0.55 → 0.45) to intentionally over-capture for later enrichment filtering (LLM + external sources).
Future Enhancements (post-normalization):
- Service type code dictionaries per ISO for deterministic load tagging.
- County-level DC cluster scoring & fiber/power corridor overlays.
- Historical conversion modeling (submitted → in-service lag) to weight recency.
- LLM verification pipeline: verify entity categories & weed out generation-only false positives.
### Contributing Feedback
If you have internal mappings (e.g., service type → load), open an issue or drop a structured snippet so we can incorporate into the normalization layer.
---
## 🧪 Experimental: Grid / Interconnection Queue Explorer
This repository now includes an optional complementary module (`grid_status_explorer.py`) that pulls ISO interconnection queue data (initial focus: **PJM**) using the open-source [`gridstatus`](https://github.com/kmax12/gridstatus) library. This data can help contextualize:
- Regional power capacity pipeline vs. hyperscale expansion
- Queue backlog / project aging as a proxy for grid constraints
- Technology mix (renewables, storage) potentially supporting future data center loads
### Why Separate?
The grid queue explorer is intentionally decoupled from the article-based project intelligence until alignment is established on how to correlate the datasets (e.g., geographic joins, power availability scoring, timeline merging). No assumptions are currently made about linkage.
### Install (if not already):
```bash
pip install gridstatus
```
### Run (Fetch PJM Queue):
```bash
python grid_status_explorer.py --iso PJM --limit 500
```
Options:
- `--iso` (default: PJM) – placeholder for future expansion
- `--limit` – truncate rows for a fast exploratory run
- `--output` – directory for artifacts (default: `grid_outputs`)
### Outputs
The script writes timestamped files to `grid_outputs/` (created if missing):
| File Type | Pattern | Description |
|-----------|---------|-------------|
| Raw CSV | `interconnection_queue_raw_<timestamp>.csv` | Full queue dataset from the ISO (unmodified) |
| Enriched CSV | `interconnection_queue_enriched_<timestamp>.csv` | Light temporal + capacity normalization subset |
| Summary JSON | `interconnection_queue_summary_<timestamp>.json` | Aggregated stats (year distribution, MW stats) |
### Example Summary Snippet
```json
{
"generated_at": "2025-09-25T14:12:03Z",
"iso": "PJM",
"records": 8921,
"year_distribution": {"2023": 812, "2024": 765},
"capacity_stats": {"count": 8700, "sum_mw": 524000.5, "median_mw": 110.0, "p90_mw": 450.0, "max_mw": 1380.0}
}
```
(*Values illustrative; actual numbers depend on current queue data.)
### Roadmap Ideas (Not Yet Implemented)
- Geographic alignment to project locations (state/county normalization)
- Power availability score blending with article-derived timelines
- Filtering for high-MW renewable + storage near key hyperscale regions
- Temporal forecasting: interconnection to in-service conversion lag
If you want to prioritize any of these, open an issue with context.
---
### 🔐 PJM API Key (Required for PJM Queue)
Recent versions of `gridstatus` require a PJM API key for accessing certain PJM endpoints (including Queue Point data). You must have Account Manager API access provisioned for your user or a system account.
Steps:
1. Ensure your PJM Account Manager profile is active (you have a username like `bpulluta`).
2. Request API access (Account Manager API + Queue Point read scope). Your company account managers must approve.
3. Once the key is issued, set it locally:
```bash
export PJM_API_KEY=YOUR_KEY_VALUE
# or create .env (copy pjm.env.example) and add PJM_API_KEY=YOUR_KEY_VALUE
```
4. Run the explorer:
```bash
python grid_status_explorer.py --filter-states VA,TX --min-mw 100 --parquet --limit 1000
```
Optional: supply inline without exporting
```bash
python grid_status_explorer.py --pjm-api-key YOUR_KEY_VALUE --min-mw 150 --filter-states VA,TX
```
Security Notes:
- Do NOT commit your real key (.env is git-ignored if you add it to .gitignore)
- Rotate keys per PJM security policies
- For multi-user environments, use a system account with least privilege
If the script exits with: `PJM API key missing or invalid` – confirm environment variable is exported in the *same* shell session (`echo $PJM_API_KEY`).
---
## ⚡ Fast Start: Minimal PJM Queue Pull
The former lightweight helper script `quick_grid.py` has been retired and archived. Use the ingestion stage of the new pipeline instead:
```bash
python -m queue_pipeline.qp ingest --iso ALL --year-min 2020 --limit 150 --recall-bias
```
### 1. Provide PJM API Key
Either export it for the session:
```bash
export PJM_API_KEY=YOUR_KEY_VALUE
```
Or create a `.env` file in the repo root:
```
PJM_API_KEY=YOUR_KEY_VALUE
```
(`quick_grid.py` will auto-load it if `python-dotenv` is installed; uncomment `python-dotenv` in `requirements.txt` if needed.)
### 2. Run a Simple Filter
```bash
python -m queue_pipeline.qp ingest --iso ALL --year-min 2020 --limit 500
```
### 3. Inspect Output
`quick_queue_filtered.csv` (overwritten each run) will contain only rows matching your filters (if those columns exist in the source feed).
### Script Behavior
- Auto-detects likely capacity (MW) column and queue submission date
- Adds normalized columns (`capacity_mw_norm`, `state_norm`, `year` when discoverable)
- Prints a concise summary (row counts, MW totals, years)
- Produces a single CSV: `quick_queue_filtered.csv`
Use the richer `grid_status_explorer.py` if you need caching, enrichment artifacts, summaries, or Parquet output.
### Handling Corporate / Self-Signed Certificates
If you hit SSL errors (certificate verify failed), you can retry once with:
```bash
python -m queue_pipeline.qp ingest --iso ALL --year-min 2020 --limit 500 --insecure
```
Warning: This disables SSL verification for the run. Use only if you trust the network environment.
---
## ⚡ ERCOT Interconnection MVP (Queue Load Focus)
An MVP script demonstrates ERCOT-specific large flexible load (LFL) style metrics using existing queue ingestion heuristics.
### Script
`queue_pipeline/tools/ercot_mvp.py`
### Metrics Produced
1. Total requested MW for ERCOT (sum of `capacity_mw` for ERCOT rows)
2. Heuristic LFL subset ("LFL Approx"): rows where
- `iso == ERCOT`
- `resource_type` in {`DATA_CENTER_CANDIDATE`, `LARGE_LOAD`}
- Not `generation_flag` and not `transmission_flag`
- Capacity ≥ `--min-lfl-capacity` (default 50 MW)
3. Average proposed in-service year across LFL Approx subset (if dates present)
### Run (use existing raw candidates first)
```bash
python queue_pipeline/tools/ercot_mvp.py --min-lfl-capacity 50
```
### Force Fresh ERCOT Fetch
```bash
python queue_pipeline/tools/ercot_mvp.py --fetch --queue-year-min 2020 --min-lfl-capacity 50
```
Example output:
```
=== ERCOT MVP Summary ===
Projects (all ERCOT): 14
Total Requested MW (approx): 1260.3
LFL Approx Projects: 0
LFL Approx Total MW: 0
Avg Proposed In-Service Year (LFL subset): N/A
Summary JSON -> queue_pipeline/qp_outputs/ercot_mvp_summary.json
```
If LFL counts are unexpectedly zero, lower threshold:
```bash
python queue_pipeline/tools/ercot_mvp.py --min-lfl-capacity 20
```
Or broaden heuristics during fetch (when using --fetch):
```bash
python queue_pipeline/tools/ercot_mvp.py --fetch --structural-broad --recall-experimental --min-lfl-capacity 40
```
### Output JSON
`queue_pipeline/qp_outputs/ercot_mvp_summary.json`:
```json
{
"ercot_total_projects": 14,
"ercot_total_requested_mw": 1260.3,
"ercot_lfl_projects": 0,
"ercot_lfl_total_mw": 0.0,
"ercot_lfl_avg_in_service_year": null,
"params": {"queue_year_min": 2020, "min_lfl_capacity": 50.0}
}
```
### Limitations
- LFL semantics approximated; no official ERCOT LLI classification parsed yet.
- Proposed in-service dates often sparse; average may be noisy.
- Does not yet segment by corridor (Houston / DFW / West Texas) or filter hydrogen/crypto explicitly.
### Roadmap Enhancements
1. Parse ERCOT Long-Term Load Interconnection (LLI) PDF/XLS directly for authoritative LFL labels.
2. Add classifier differentiating AI / hyperscale vs crypto vs hydrogen.
3. Add geographic clustering + transmission corridor MW aggregation.
4. Track delta vs previous month (time-series trend of load-side queue accumulation).
5. Integrate with batching/curation stage for targeted article retrieval about largest pending loads.
This MVP validates that the pipeline can isolate potential data center / large flexible load signals and produce investor/developer‑relevant early analytics for ERCOT.