CodeSentinel - GitHub Risk Scraper

🔍 CodeSentinel is an open-source risk scraper that analyzes trending GitHub repositories to calculate trustworthiness scores. It evaluates repositories across multiple dimensions to provide comprehensive risk assessment for organizations evaluating OSS dependencies.

Features

• Multi-dimensional Risk Analysis: Calculates risk scores based on:

  • License Score (20% weight): MIT/Apache = high, GPL/AGPL = medium, None = low
  • Activity Score (30% weight): Based on days since last commit and commit frequency
  • Maintainer Score (25% weight): Based on contributor count
  • Stability Score (25% weight): Based on project age, stars, and fork ratio

• Beautiful Output:

  • Rich console tables with color-coded risk scores and interactive progress indicators
  • Professional HTML reports with detailed breakdowns and summary statistics
  • CSV export for data analysis and spreadsheet integration
  • JSON export for API integration and programmatic access

• Advanced Filtering & Sorting:

  • Filter by risk score range (--min-risk, --max-risk)
  • Filter by programming language (--language)
  • Show top N repositories (--top)
  • Sort by multiple criteria (--sort-by: risk_score, stars, activity_score, etc.)

• Comprehensive Analysis:

  • Analyzes trending repos across all languages
  • Searches by stars + recent activity (last 30 days)
  • Shows sub-component score breakdowns
  • Risk grade (A-F) for easy interpretation
  • Repository URLs for quick access
  • Summary statistics in console and HTML reports

• Reliability:

  • Automatic rate limit handling with exponential backoff retry
  • Graceful error handling for API failures

Requirements

• Python 3.7+
• Dependencies listed in requirements.txt:
  • requests - GitHub API calls
  • pandas - Vectorized data operations
  • rich - Beautiful console formatting

Installation

pip install -r requirements.txt

Usage

Basic Usage

python main.py

This will analyze 50 trending repositories by default and generate risk_report.html.

Advanced Usage

# Analyze specific number of repos
python main.py --limit 100

# Custom output filename
python main.py --output my_report.html

# Combine options
python main.py --limit 75 --output analysis.html

# Export to CSV for data analysis
python main.py --csv risk_report.csv

# Export to JSON
python main.py --json risk_report.json

# Filter by risk score
python main.py --min-risk 70 --max-risk 90

# Filter by language
python main.py --language Python

# Show top 10 repositories sorted by stars
python main.py --top 10 --sort-by stars

# Combine all options
python main.py --limit 100 --output report.html --csv data.csv --json data.json --min-risk 60 --top 20 --sort-by activity_score

Output

Console Output

The console displays a beautiful Rich table with:

• Repository name and language
• Overall risk score (color-coded: green/yellow/red)
• Sub-component scores (License, Activity, Maintainer, Stability)
• Star count

HTML Report

An HTML report is generated with:

• Sortable table of all repositories
• Color-coded risk indicators
• Detailed score breakdowns
• Explanation of scoring methodology

CSV Export

A CSV file can be exported (optional, using --csv flag) with:

• All repository data in spreadsheet-friendly format
• Human-readable column names
• Complete risk scores, risk grades (A-F), and sub-component breakdowns
• Repository metrics (stars, forks, contributors, commits)
• Repository URLs for quick access
• Perfect for Excel, Google Sheets, or data analysis tools

JSON Export

A JSON file can be exported (optional, using --json flag) with:

• Complete repository data in JSON format
• Perfect for API integration and programmatic access
• All risk scores, grades, and metrics included
• Easy to parse and process programmatically

Summary Statistics

Summary statistics are displayed in:

• Console: After the main table, showing total repos, average risk score, distribution, and most common language
• HTML Report: At the top of the report with key metrics

Interactive Progress Indicators

Real-time progress indicators show:

• Spinners for fetching trending repos
• Progress bar for detailed metadata fetching (with repo names)
• Spinners for risk calculation
• Status updates for report generation

Risk Score Interpretation

• 90-100 (Grade A, Green): Excellent - Very low risk, safe to use
• 80-89 (Grade B, Green): Good - Low risk, generally safe
• 70-79 (Grade C, Green): Acceptable - Low to medium risk
• 60-69 (Grade D, Yellow): Moderate - Medium risk, use with caution
• 50-59 (Grade E, Yellow): Risky - Medium to high risk
• 0-49 (Grade F, Red): High risk - Requires careful evaluation

Score Components

1. License Score: Evaluates license compatibility and risk - MIT/Apache-2.0: 100 points - BSD: 90 points - GPL/AGPL: 40 points - None: 0 points

2. Activity Score: Measures project maintenance - Fresh commits (<30 days): High score - Moderate activity (30-90 days): Medium score - Stale (>90 days): Low score - Boost for high commit frequency

3. Maintainer Score: Assesses community involvement - Based on contributor count - Bonus for repos with 10+ contributors

4. Stability Score: Evaluates project maturity - Project age (older = more stable) - Star count (popularity indicator) - Fork ratio (community engagement)

Architecture

• Single Loop Constraint: The entire program uses only ONE loop (in main.py)
• Vectorized Operations: Uses pandas for efficient data processing
• Functional Programming: Leverages recursion, apply(), and other functional patterns
• Modular Design: Clean separation of concerns across modules

Project Structure

CodeSentinel/
├── main.py              # Entry point and orchestration (ONE LOOP)
├── github_api.py        # GitHub API interactions (recursion, rate limit handling)
├── risk_calculator.py   # Risk scoring with grades (vectorized pandas, no loops)
├── output_formatter.py  # Rich tables + HTML + CSV + JSON + stats (apply(), no loops)
└── requirements.txt     # Dependencies

Example Output

CodeSentinel - GitHub Repository Risk Analysis
┌──────────────────────────────┬────────────┬─────────────┬──────────┬──────────┬────────────┬──────────┬────────┐
│ Repository                   │ Language   │ Risk Score  │ License  │ Activity │ Maintainer │ Stability│ Stars  │
├──────────────────────────────┼────────────┼─────────────┼──────────┼──────────┼────────────┼──────────┼────────┤
│ vue                          │ JavaScript │ [green]85.2[/] │ 100.0    │ 95.0     │ 90.0       │ 70.0     │ 210k   │
│ react                        │ JavaScript │ [green]82.5[/] │ 100.0    │ 90.0     │ 88.0       │ 65.0     │ 215k   │
...

Technical Highlights

• One-Loop Warrior: Maximum 1 loop in entire program (in main.py)
• Professional Builder: Under 500 lines total (271 lines - highly optimized!)
• Pure Fundamentals: No frameworks, just core Python
• Efficient API Usage: Batch processing, recursion for pagination, automatic rate limit handling
• Vectorized Calculations: Pandas operations for fast risk computation (no loops)
• Advanced Filtering: All filters use vectorized pandas operations
• Interactive UI: Real-time progress bars and spinners using Rich library
• Multiple Export Formats: HTML, CSV, and JSON exports available

License

This project is open source and available for use in the Code Olympics hackathon.

Command-Line Options

• --limit N: Number of repos to analyze (default: 50)
• --output FILE: HTML report filename (default: risk_report.html)
• --csv FILE: Export to CSV format
• --json FILE: Export to JSON format
• --min-risk SCORE: Filter by minimum risk score (0-100)
• --max-risk SCORE: Filter by maximum risk score (0-100)
• --language LANG: Filter by programming language
• --top N: Show only top N repositories
• --sort-by COLUMN: Sort by column (risk_score, stars, activity_score, maintainer_score, stability_score)

Notes

• Uses GitHub API (no authentication required for public repos)
• Automatic rate limit handling with exponential backoff retry
• Results are sorted by risk score by default (highest = lowest risk)
• All filtering and sorting uses vectorized pandas operations (no additional loops)