Log Analysis 🖥️🔍

A real-time system & network log analysis desktop application built with PySide6. It combines live network packet monitoring, machine‑learning‑based log anomaly detection, and cross‑platform (Windows/Linux) system log collection in a single dashboard — with a rule engine and ML model to flag suspicious activity and raise alerts.

📄 Research paper publication: "Log Analysis: Understanding and Enhancing System Monitoring" — IJARCCE

---

Table of Contents

• Features
• Architecture
• Tech Stack
• Project Structure
• Getting Started (Run Locally)
• Building a Standalone Executable
• Screenshots
• Certificate
• Known Limitations
• License

---

Features

The app is organized into three main dashboard tabs, each backed by its own Python module:

🏠 Home — Live Network Monitoring & Threat Detection

• Live packet capture using Scapy (sniff) with start/stop controls.
• Rule-based detection engine for:
  • Port scans
  • SYN floods
  • DNS tunneling (entropy-based)
  • ARP spoofing
  • Data exfiltration patterns
  • C2 (command-and-control) communication patterns
  • Known malicious IP matching
  • Suspicious/broadcast traffic
  • Time-based anomalies & brute-force login attempts
• Alerting system with severity levels and optional sound alerts.
• Live traffic summary, alert feed, and log counters exposed to the UI.
• All events persisted to a local SQLite database.

📄 Log — Log Upload, Search & ML-Based Anomaly Detection

• Upload/save log files, browse folders, and search within collected logs.
• Rule/pattern-based anomaly scanning plus a trained ML classifier (scikit-learn RandomForestClassifier + CountVectorizer pipeline, persisted with joblib in models/log_analysis_model.joblib).
• "Deep analysis" mode with risk classification of log content.
• Export analysis results to CSV.

🖥️ System — Cross-Platform System Log Collection

• One-click collection of system diagnostics, running in a background thread with live progress reporting to the UI.
• Windows: Event Logs, system info, network info, running processes, services.
• Linux: system info, network info, process info, and security-related logs.
• Collected logs are written under collected_logs/upload_log/{windows|linux}/.

🎨 UI

• Frameless, custom-styled window (drag, minimize/maximize/close handled manually).
• Left navigation menu + collapsible center info/help menu.
• Each tab embeds an HTML/JS frontend (via QWebEngineView) that talks to its Python backend over a QWebChannel bridge — see Architecture.

---

Architecture

The app is a Qt (PySide6) desktop shell. Each of the three tabs loads a local HTML/CSS/JS page inside an embedded Chromium view (QWebEngineView), and that page talks to a dedicated Python "backend" object over Qt WebChannel (JS ⇄ Python bridge). This keeps the UI logic in HTML/JS while all the heavy lifting (packet capture, ML inference, system calls) runs in Python.

flowchart TD
    A["main.py<br/>MainWindow (PySide6, frameless QMainWindow)"] --> B[Home Tab]
    A --> C[Log Tab]
    A --> D[System Tab]

    B -->|QWebEngineView| B1["assets/html/home.html + JS"]
    C -->|QWebEngineView| C1["assets/html/log.html + JS"]
    D -->|QWebEngineView| D1["assets/html/system_analysis.html + JS"]

    B1 <-->|QWebChannel bridge| B2["BackendClass_hom<br/>assets/py/home.py"]
    C1 <-->|QWebChannel bridge| C2["BackendClass_log<br/>assets/py/log.py"]
    D1 <-->|QWebChannel bridge| D2["BackendClass_sys<br/>assets/py/system_analysis.py"]

    B2 --> E1["Scapy packet sniffing"]
    B2 --> E2[("SQLite<br/>log_analysis.db")]
    B2 --> E3["Rule engine: port scan, SYN flood,<br/>DNS tunneling, ARP spoof, C2, etc."]

    C2 --> F1["RandomForestClassifier + CountVectorizer<br/>(scikit-learn, joblib)"]
    C2 --> F2["models/log_analysis_model.joblib"]

    D2 --> G1["SystemLogCollector"]
    G1 --> G2["Windows: Event Log, psutil, subprocess"]
    G1 --> G3["Linux: /proc, journalctl-style, psutil"]
    G1 --> G4["collected_logs/upload_log/"]

Loading

Key design points:

• main.py builds the MainWindow from the Qt Designer file Dashboard_main.ui (compiled to ui_Dashboard_main.py), then wires up navigation, window controls, and three QWebEngineViews (Home / Log / System).
• Each backend (BackendClass_hom, BackendClass_log, BackendClass_sys) is a QObject exposing @Slot methods and Signals, registered on a QWebChannel so the embedded JS can call Python and receive live updates (e.g. new alerts, scan progress).
• resource_path() helpers in each module resolve file paths correctly both when running from source and when frozen into an executable via PyInstaller.

---

Tech Stack

Layer	Technology
Desktop UI shell	PySide6 (Qt for Python), Qt Designer .ui file
Embedded frontend	HTML, CSS, JavaScript (via QWebEngineView + QWebChannel)
Packet capture	Scapy
Machine learning	scikit-learn (RandomForestClassifier, CountVectorizer pipeline), joblib, pandas, numpy
System info	psutil, subprocess, Windows Event Log APIs
Storage	SQLite (log_analysis.db), local file system (collected_logs/)
Packaging	PyInstaller (Log Analysis.spec)

---

Project Structure

Log-Analysis/
├── main.py                     # App entry point, MainWindow, navigation & window logic
├── debug_launcher.py           # Diagnostic launcher (verbose import/startup logging)
├── Dashboard_main.ui            # Qt Designer UI definition
├── ui_Dashboard_main.py         # Compiled UI (generated from the .ui file)
├── resource.qrc / resource_rc.py# Qt resource file & compiled resources
├── style.json                  # UI style/theme config
├── log_analysis.db             # SQLite database used by the Home tab
├── requirements.txt             # Python dependencies
├── Log Analysis.spec           # PyInstaller build spec
├── Qss.zip / icons.zip          # Packaged stylesheets / icons
├── assets/
│   ├── py/
│   │   ├── home.py             # BackendClass_hom — network monitoring & alerts
│   │   ├── log.py              # BackendClass_log — log search & ML anomaly detection
│   │   └── system_analysis.py  # BackendClass_sys / SystemLogCollector — OS log collection
│   ├── html/                   # home.html, log.html, system_analysis.html (frontend pages)
│   ├── css/                    # Stylesheets for the embedded frontend
│   └── js/                     # Frontend JavaScript
├── models/
│   └── log_analysis_model.joblib  # Trained RandomForest log-classification model
├── collected_logs/
│   └── upload_log/             # Output folder for collected/uploaded logs
└── images/                     # README screenshots

---

Getting Started (Run Locally)

Prerequisites

• Python 3.9–3.11 (PySide6 6.9 and the ML stack are most stable in this range)
• Git
• Windows or Linux (some features — Windows Event Log collection, winsound alert tones — are Windows-only; on Linux those specific calls will need adjusting/guard-clauses)
• Recommended: run packet-capture features with sufficient privileges (Administrator on Windows, sudo/CAP_NET_RAW on Linux), since Scapy needs raw socket access to sniff packets.

1. Clone the repository

git clone https://github.com/rupeshrb/Log-Analysis.git
cd Log-Analysis

2. Create and activate a virtual environment

# Windows
python -m venv venv
venv\Scripts\activate

# macOS/Linux
python3 -m venv venv
source venv/bin/activate

3. Install dependencies

pip install -r requirements.txt

⚠️ Note: requirements.txt currently contains a large, generic list of packages (it looks like it was generated from a full environment pip freeze rather than only this project's direct dependencies). At minimum, the app needs: PySide6, PySide6-Addons, PySide6-Essentials, scapy, scikit-learn, pandas, numpy, joblib, psutil, and QT-PyQt-PySide-Custom-Widgets (imported as Custom_Widgets). If pip install -r requirements.txt fails or is too slow, you can install just these core packages instead and add others only as needed.

4. Run the application

python main.py

If the app fails to start (missing modules, blank window, etc.), run the diagnostic launcher instead — it prints detailed environment/import info and writes a traceback to error_log.txt:

python debug_launcher.py

5. Using the app

• Home tab — start/stop live packet capture and watch real-time alerts, traffic summaries, and detected threats.
• Log tab — upload or browse log files, search them, and run anomaly/deep analysis (rule-based + ML). Export results to CSV.
• System tab — click to collect system logs/info for the current OS; progress is shown live, and results are saved under collected_logs/upload_log/.

---

Building a Standalone Executable

The repo includes a PyInstaller spec file for producing a Windows .exe:

pip install pyinstaller
pyinstaller "Log Analysis.spec"

The built executable will be created in the generated dist/ folder. A pre-built Windows release is also available under the repository's Releases page (LogAnalysis_WINDOWS_SETUP_V_1.0).

---

Screenshots

Certificate

---

Known Limitations

• Some features (winsound alert tones, Windows Event Log collection) are Windows-specific and will need platform guards or alternatives to run fully on macOS/Linux.
• requirements.txt is broader than the project's actual dependencies — consider regenerating it with pip freeze from a clean virtual environment built solely for this app, or trimming it to the core packages listed above.
• The repo currently commits some generated/binary artifacts (log_analysis.db, models/log_analysis_model.joblib, Qss.zip, icons.zip, __pycache__/). Consider a .gitignore for __pycache__/ and regenerated files going forward.
• Packet capture requires elevated privileges; without them, the Home tab's live monitoring will fail silently or raise permission errors.

License

No license file is currently included in this repository. If you intend this project to be reused or contributed to by others, consider adding a LICENSE file (e.g., MIT, Apache-2.0) to clarify usage terms.