NSE/BSE Data Downloader v1.2.0 (PySide6)

Upgrading from v1.0.1? Read UPGRADE.md first.

v1.1.0 replaced PyQt6 with PySide6 and requires Python 3.10 or newer. Copying these files over an existing v1.0.1 install without running pip install -r requirements.txt will stop the app from starting.

Prebuilt applications that need no setup are on the releases page.

Using a v1.1.0 prebuilt application? Replace it. Those builds shipped without a certificate store and could not download anything; they were withdrawn, and v1.1.1 and later fix it. Source installs were never affected.

Coming from v1.1.1? Nothing to migrate or install — the top of UPGRADE.md covers the two new columns and the optional database.

A desktop downloader that turns legacy and current NSE/BSE reports into one stable daily-file format.

Features

• NSE Equity, Futures, SME and Index downloads.
• BSE Equity and Index downloads.
• Date-aware URL selection for old and current exchange report eras.
• NSE/BSE delivery quantity and percentage merged into cash-market files.
• NSE futures open interest and change in open interest.
• Symbol-wise text histories such as NSE/SYMBOLS/reliance.txt.
• Split, consolidation and equity-bonus adjustments on pre-ex-date symbol prices and share counts, so price and turnover stay continuous.
• Pending delivery retry, atomic file replacement and corporate-action audit state.
• A read-only --audit command that verifies digests, coverage, row counts and symbol histories without changing a single byte of the data folder.
• Turnover and previous close from every supported report era, normalised to rupees, with a per-segment schema marker beside the data.
• Only dates each NSE segment was actually published are offered, from NSE equity's first trading day on 1994-11-03.
• A rotating diagnostic log outside the data folder, opened from Help → Open Log Folder.
• A SQLite mirror of every download, with commands that prove it regenerates your files byte for byte. On by default; the two settings that read it are not.
• Staged per-date publication with deterministic SME/Index combination.
• Calendar-based historical/custom date ranges with automatic mode retained.
• Individually collapsible Exchange, Date, Options, Progress and Status panels.
• IST-aware trading dates and a 24-hour cached official NSE holiday calendar.
• Cooperative Stop/close behavior with success, partial, pending, warning, repair-required, cancelled and failed outcomes.
• Responsive 720-pixel default window with complete date controls and Donate action kept beside the main controls.
• Automatic/manual version checks, remembered skipped versions and a reset action.

Daily bhavcopy files remain official unadjusted market records. Corporate-action adjustments are applied only to symbol-wise histories.

Every file carries TURNOVER and PREV_CLOSE as its last two columns, in rupees — NSE publishes F&O turnover in lakhs before 2024-07-08 and index turnover in crores, and both are converted. Where an exchange publishes neither (BSE index turnover, NSE F&O previous close before 2024-07-08, NSE index previous close) the field is left empty rather than zero. Daily files stay headerless; each segment folder carries a SCHEMA.json naming the columns of every width this application has published, so a file can be read by counting its columns and looking the width up.

Installation

Requirements: Python 3.10 or newer and an internet connection.

git clone https://github.com/pparesh25/NSE_BSE_Downloader.git
cd NSE_BSE_Downloader
pip install -r requirements.txt
python main.py

Core dependencies include PySide6, aiohttp, pandas, NumPy and PyYAML. Source-mode launches also set the process title to NSE BSE Data Downloader; packaged releases use the same native product and bundle identity.

Upgrading from v1.0.1

Every version since 1.1 keeps the existing ~/NSE_BSE_Data/ data root and ~/.nse_bse_downloader/ preference directory. Existing seven-column daily files remain readable and are not rewritten merely by launching the application. A date downloaded again is published under the current eleven-column EQ/SME/FO contract. Back up important user data before a major upgrade and use the official GitHub Release assets, or the immutable v1.2.0 tag, rather than an archive from a mutable branch.

Nuitka packaging and release trust

Nuitka packaging and project-owned cross-platform icons are configured. GitHub Actions builds unsigned macOS ARM64, Windows x64, and Linux x64 test candidates; a separate manual workflow is reserved for credential-gated macOS signing/notarization and Windows Authenticode signing. No workflow publishes a release or enables the updater automatically.

The default local commands only validate resources/dependencies and print the build command:

python build_nuitka_cross_platform.py --target-platform=darwin
./build_nuitka_macos.sh

Neither command installs Nuitka, deletes output nor starts compilation without an explicit --build. Runtime config/QR paths work independently of the launch directory; user data and preferences remain outside the read-only app bundle. See PACKAGING.md and RELEASE_TRUST_ASSETS.md for build evidence, credential setup, clean-machine verification, and publication gates. Every internal engineering record lives under docs/engineering/.

Using the app

1. Select the exchange segments.
2. Leave Date Range in automatic mode, or select a custom start/end date.
3. Choose delivery, FO open-interest and symbol-history options.
4. Collapse panels you do not need, or use View → Expand/Collapse All.
5. Click Start Download.

Stop Download requests cooperative cancellation. The app waits for the current atomic operation instead of forcibly terminating its worker thread, so already-published files remain valid. Closing the window while download/update work is active follows the same safe shutdown path.

Files are stored under ~/NSE_BSE_Data/ by default.

After the downloads finish, symbol histories are brought up to date. That stage has its own Symbol histories row in the progress panel, with a bar and a count, so a large update never looks like a frozen window.

If something goes wrong, Help → Open Log Folder opens ~/.nse_bse_downloader/logs/. Each run records the application version, platform and certificate status at the top, so attaching the log to an issue is usually enough to identify the build. The files are capped and rotated, and nothing is written into your data folder.

Output contracts

NSE/BSE Equity and SME:

SYMBOL,DATE,OPEN,HIGH,LOW,CLOSE,VOLUME,DELIVERY_QTY,DELIVERY_PERCENT,TURNOVER,PREV_CLOSE

NSE Futures:

SYMBOL,DATE,OPEN,HIGH,LOW,CLOSE,VOLUME,OPEN_INTEREST,CHANGE_IN_OI,TURNOVER,PREV_CLOSE

Index files:

SYMBOL,DATE,OPEN,HIGH,LOW,CLOSE,VOLUME,TURNOVER,PREV_CLOSE

Equity, SME and FO files always retain their stable eleven-column shape, and index files their nine. If delivery or FO open interest is disabled or unavailable, the corresponding optional fields are blank rather than removed. Files written by earlier versions keep the width they were written with; the SCHEMA.json in each segment folder names the columns of every width, so a reader tells the generations apart by counting. NSE delivery is matched by SYMBOL + SERIES; BSE delivery is matched by security code.

When the append options are enabled, the app always assembles combined cash files in a fixed order, independent of concurrent download completion order:

• NSE: EQ → SME → INDEX
• BSE: EQ → INDEX

Each segment is first saved as a validated named-column component. The public EQ file is replaced atomically only after all enabled dependencies succeed. A failed or interrupted dependency leaves the previous public EQ file untouched and the manifest schedules that date for repair on the next run. Index rows are aligned by column name, so the two delivery fields remain blank.

Data organization

~/NSE_BSE_Data/
├── .state/                 # Internal pending/action/raw rebuild state
│   ├── components/         # Validated inputs for combined-file restart/rebuild
│   └── eod.sqlite3         # Optional database mirror, only if you turn it on
├── NSE/
│   ├── EQ/
│   ├── FO/
│   ├── SME/
│   ├── INDEX/
│   └── SYMBOLS/
│       └── reliance.txt
└── BSE/
    ├── EQ/
    ├── INDEX/
    └── SYMBOLS/

Each symbol file has one header and one row per date:

DATE,OPEN,HIGH,LOW,CLOSE,VOLUME,SERIES,TOTAL_TRADES,QTY_PER_TRADE,DELIVERY_QTY,DELIVERY_PERCENT,ISIN,TURNOVER,PREV_CLOSE

ISIN identifies the security; TURNOVER and PREV_CLOSE come last, so a tool that reads the earlier columns by position keeps working.

Symbol files are sorted, idempotently updated and renamed/merged using ISIN or exchange security code when available. Split, consolidation and equity-bonus actions adjust pre-ex-date rows only: OPEN, HIGH, LOW and CLOSE are divided by the factor, and VOLUME, DELIVERY_QTY and QTY_PER_TRADE are multiplied by it, so price x volume is unchanged across the ex-date. TOTAL_TRADES counts transactions rather than shares and DELIVERY_PERCENT is a ratio of two columns that both scale, so neither is touched. FO open interest is not adjusted.

Histories written by version 1.1.0 or earlier used a different rule: prices were snapped to a 0.05 grid and share counts were left unadjusted. An audited action is never applied twice, so those bars keep the old arithmetic until the exchange is rebuilt. The application reports which exchanges are affected on startup; --rebuild-exchange replays the checksummed .state/raw snapshots through the current rule.

State files and symbol histories are validated before any update. Damaged bytes are preserved under .state/quarantine and the operation stops with a repair-required error instead of treating corrupt state as empty. Corporate actions use a prepared/committed journal, so an interruption between history replacement and ledger commit is recovered without applying the factor twice.

Symbol files are written directly, without a second copy. .state/raw is the recoverable source: it is checksummed, keyed by date, and is what the rebuild commands below read. The diagnostic trees beside it — .state/quarantine and .state/raw_revisions — are pruned after each run under state_retention in config.yaml, so a multi-year backfill cannot fill the disk with copies. .state/raw itself is never pruned.

Raw snapshots include checksum metadata; replaced revisions are retained under .state/raw_revisions. They can be used for explicit repairs:

python main.py --rebuild-symbol NSE RELIANCE
python main.py --rebuild-exchange BSE
python main.py --rebuild-registry
python main.py --rebuild-all
python main.py --rebuild-combined NSE 2026-07-31

Symbol rebuild commands validate snapshot checksums and replay committed corporate actions before publishing repaired histories. --rebuild-combined uses the current append preferences and persisted EQ/SME/Index components. Existing files remain in place when validation or transaction recovery cannot be completed safely.

Checking the database without changing it

The rebuild commands above repair; --audit only looks. It answers "is this database complete and self-consistent?" and writes nothing at all, so it is safe to run at any time, including while a download is in progress.

python main.py --audit                # the whole data folder
python main.py --audit NSE_EQ BSE_EQ  # only these segments

It reports:

• every sha256 the pipeline recorded, checked against the file on disk — published files, reconciliation components, combined outputs and the .state/raw snapshots;
• coverage from base_start_date to the most recent session the exchanges have published, so a truncated head and a stale tail are both visible, with trading holidays taken from the offline calendar rather than guessed;
• row counts against what was recorded and against neighbouring sessions, which is the only plausibility check available for files written before the pipeline database existed;
• every SYMBOLS/*.txt history cross-checked against the raw snapshots, in both directions;
• orphans: components and snapshots for dates that were never published, metadata without its snapshot, temporary files from writes that did not finish, and registry entries naming files that do not exist.

Exit codes are 0 for a clean database, 1 when there are findings, and 2 when the audit itself could not run. Notices — data older than the pipeline database, a tail you have not downloaded yet, the older symbol-file column set — are shown but do not fail the command.

The optional EOD database

This version also keeps every download in a SQLite database, .state/eod.sqlite3, as the first steps towards making it the archive's system of record. The mirror is on by default from v1.2.1, because the settings that read the database can only read what it has already written. It costs about 450 MB a year for all six segments, and dual_write_eod_database: false stops it without touching anything already written. Nothing reads it yet: the two settings that do are off. Each step was proven byte for byte against real downloads, and each has its own setting under download_options:

Setting	What it does
dual_write_eod_database	Mirrors every download into the database. On by default. About 450 MB a year for all six segments. The other two only read what this writes.
publish_histories_from_database	Brings symbol histories up to date by extending each file rather than rewriting it.
read_snapshots_from_database	Lets the history journal, the rebuild commands, the rebuild prompt and --audit read snapshots from the database. .state/raw is still written.

Three commands work with it:

python main.py --verify-eod-parity                     # regenerate every file from the database and diff it
python main.py --republish-histories                   # write symbol histories out of the database
python main.py --snapshot-revisions NSE EQ 2026-08-18  # what the exchange changed when it republished a day

--verify-eod-parity and --snapshot-revisions only read. --republish-histories writes symbol files and takes the same lock a download does. The measurements behind each step are in PHASE_5_1_DUAL_WRITE_REPORT.md.

Date-aware report support

The application automatically selects the proper official source:

• NSE Equity/FO: legacy archives before 8 July 2024; UDiFF reports from that date.
• BSE Equity: ISIN legacy reports before 17 August 2022, second-generation reports through 7 July 2024, and UDiFF reports from 8 July 2024.
• NSE SME: two-digit-year filenames through 10 October 2025 and four-digit-year filenames from 13 October 2025.

Users do not need separate old/new downloader scripts.

Configuration

Application defaults are in config.yaml. Per-user choices are saved to:

~/.nse_bse_downloader/user_preferences.json

Effective settings use one precedence rule: built-in schema defaults, then config.yaml, then validated saved user choices. Unknown keys and invalid types are discarded or bounded before use. Automatic update checks can be disabled from Settings; a skipped version can be reset there, while Help → Check for Updates always performs a manual check.

Market-date decisions use Asia/Kolkata time. Trading holidays are read from NSE's official capital-market calendar by year, cached for 24 hours, and refreshed automatically. If refresh fails, the last valid cache is retained, and below that a calendar for 2013–2026 is bundled with the application, so a blocked or unreachable holiday API cannot make a holiday look like a trading day. NSE serves no calendar before 2013; for those years a report the exchange never published is recognised from its own response and is not retried, unless no other date in that year downloaded — in which case the source itself is suspect and the dates stay queued.

New v1.1 options:

download_options:
  include_delivery_data: true
  include_fo_open_interest: true
  generate_symbol_files: true
  apply_corporate_actions: true

If a delivery report is late or temporarily unavailable, the price bhavcopy is still saved. The date is recorded under .state and retried on the next run — but not forever: a report that has not appeared after 30 days is treated as absent rather than late, and the date stops being re-queued.

NSE published no separate delivery report before 2019-09-30 and BSE none before 2006-01-02, while both price archives go back much further. For dates below those, the delivery stage is marked disabled rather than requested, so a historical backfill completes instead of waiting for a report that has never existed.

Custom date mode intentionally allows existing historical dates to be downloaded again. Daily and symbol files are updated atomically rather than duplicated. The selected dates and each panel's expanded/collapsed state are remembered for the next launch.

Testing

Run the suite on both supported Python versions:

python -m pytest -q

The maintainer validates each release on Python 3.10 and 3.13 before publishing; GitHub Actions runs the same suite on both.

The tests cover URL cutovers, legacy/current schemas, delivery keys, FO OI, pending state, symbol reruns/renames, corporate-action idempotency and GUI date range/collapse behavior. They also cover every NSE/BSE component arrival order, restart rebuilds, disabled/failed dependencies, old component in-memory upgrade and staged publication. The state-integrity matrix injects interrupted ledger commits, failed history publication, corrupt JSON/CSV state, unexpected history revisions and rebuilds from checksummed raw snapshots. Lifecycle coverage also checks cooperative cancellation/window close, settings precedence, skipped updates, IST boundaries, official-calendar parsing, TTL refresh and stale fallback.

The --audit tests cover every finding it can report and, separately, the promise that it reports them without writing: one test fingerprints every path, mtime, size and content digest under a data root around a full run and requires them to be identical afterwards.

The EOD database tests regenerate daily files, symbol histories and raw snapshots through the real writers and compare them byte for byte, and fingerprint the data root around every command that promises only to read.

GitHub Actions runs the suite on Python 3.10 and 3.13 and fails below 70% coverage. Ruff, mypy and a no-build Nuitka command validation are separate release gates. The equivalent local commands are:

python -m ruff check .
python -m mypy src main.py app_metadata.py runtime_paths.py runtime_identity.py build_nuitka_cross_platform.py package_release_artifact.py
python -m pytest --cov=src --cov=main --cov=runtime_paths --cov=runtime_identity --cov=app_metadata --cov=version --cov-fail-under=70
python build_nuitka_cross_platform.py --target linux

They also verify bundle-root config/QR lookup, compiled-module version detection, platform-specific Nuitka command generation and the default no-build guard. The strict project mypy configuration currently passes all 63 source files.

Version history

v1.2.1 (2026-09-16)

• Turned the SQLite mirror on by default, so a new data folder starts collecting .state/eod.sqlite3 with its first download. The two settings that read the database stay off.
• Gave symbol histories a section of their own, and made their progress run once across the whole stage, naming the batch it is on instead of restarting at zero for each.
• A delivery report the exchange has not published yet is shown as pending rather than as an error, so a day that did download no longer reads "Completed" in red.

v1.2.0 (2026-09-15)

• Added a read-only --audit command that checks every recorded checksum, coverage, row counts and symbol histories without writing anything.
• Added turnover and previous close to every daily file and symbol history, from every report era, with a SCHEMA.json in each folder naming the columns of every width.
• Limited the date picker to dates each NSE segment was actually published.
• Added a rotating diagnostic log outside the data folder and Help → Open Log Folder.
• Gave the symbol-history stage its own progress row after downloads finish.
• Measured delivery joins, so a report that matched nothing no longer publishes silently.
• Added an optional SQLite mirror with --verify-eod-parity, --republish-histories and --snapshot-revisions; off by default.

v1.1.1 (2026-08-18)

• Gave the packaged application its own certificate authorities. v1.1.0 builds shipped without a trust store, so every download failed certificate verification and the host circuit breaker opened. Source runs were unaffected.
• Routed the corporate-action endpoints through the same trust store as every other request.
• Made packaging prove it: a compiled build must complete one verified HTTPS request before it can become a release, and a build with no certificate bundle now fails the packaging dry run.

v1.1.0 (2026-07-31)

• Unified date-aware old/current download sources.
• Added NSE/BSE delivery fields and NSE FO OI fields.
• Added symbol-wise histories with audited corporate-action adjustment.
• Added pending delivery retry, atomic writes and stable nine-column EQ/SME/FO output contracts.
• Added fail-closed state quarantine, crash-recoverable corporate-action transactions and raw-snapshot repair commands.
• Replaced arrival-order append logic with deterministic, restart-safe combined bhavcopy assembly and an explicit combined-file rebuild command.
• Added remembered calendar date ranges and collapsible GUI panels.
• Added IST-aware official holiday refresh, validated settings precedence, cooperative cancellation and typed GUI completion outcomes.
• Added remembered update-check and skipped-version controls.
• Prepared deterministic Nuitka app metadata, bundle-root resources and an explicit dry-run/build split; no packaged artifact is included in this release.
• Fixed the NSE SME filename-era change and HTML-as-data responses.

v1.0.1 (2025-08-07)

• Improved download logging and market-hours behavior.
• Migrated the GUI runtime to PySide6.
• Added cross-platform Nuitka build tooling.

v1.0.0 (2025-07-31)

• Initial multi-exchange production release.

License and support

Licensed under GPL-3.0; see LICENSE. Please use GitHub Issues for bugs and support.

© 2026 Paresh Patel. All rights reserved.