SmartNFTparser

Public Release v1, update coming soon...

SmartNFTparser turns Telegram gift browsing into a practical research and collection tool. Instead of manually opening gift links one by one, users can look up exact gifts, sample collections, filter by traits, find rare gifts, compare collections, inspect market metadata, export results, watch collections over time, and review historical trends from inside Telegram. Admins also get parser health, runtime controls, proxy management, broadcasts, and database reset tools.

Main menu, discovery, and tools:

Full crawl flow:

Gift lookup:

Functionality

The bot currently exposes multiple gift-focused modes from the main menu. Start the process, open the bot in Telegram, send /start, choose a section, and use the inline buttons to navigate results, exports, refreshes, and saved lists.

Mode	What It Does and How to Use It
Random view	Selects a random collection, samples a gift-number range, and returns public NFT gift links and metadata.
Gift lookup	Accepts https://t.me/nft/Collection-Number, Collection-Number, or Collection Number, then returns metadata for that exact public gift.
Trait explorer	Shows known models, backdrops, and patterns for a selected collection from the static collection constants.
Favorites	Lets users save gift lookup results and reopen saved public gift links later.
Filtered view	Lets users select collections, models, and backdrops, then returns matching sampled gifts with pagination and refresh controls.
Filter presets	Saves a configured filtered-view setup under a name so it can be loaded again later.
Collection statistics	Samples one collection and reports aggregate counts, sampled number range, average rarity score, and top traits.
Collection comparison	Compares two selected collections side by side using sampled stats and rarity/trait summaries.
Rare gifts	Samples one collection, ranks gifts by combined rarity score, and returns the rarest sampled public gift links.
Rarity search	Finds sampled gifts under a selected combined rarity threshold.
Market snapshot	Samples a collection and summarises gifts with market metadata, numeric market values when available, and detected market sources.
Deep scan	Scans a selected number of pages without random range sampling and caches results for pagination, sorting, and export.
Full crawl	Lets users choose how many gifts to scan, splits the crawl into gift-number chunks, stores durable progress, archives checkpoints, and supports cancel/resume.
Crawl jobs	Shows the latest full-crawl job state from the database, including page cap, scanned pages, gifts found, cancel status, and resume controls.
Archive database	Stores parsed public gift metadata from lookup, random, filter, stats, rare, market, deep-scan, and full-crawl modes for later analytics.
Archive search	Searches archived gifts by collection, gift number, model, pattern, backdrop, or title and returns clickable gift links.
Leaderboards	Ranks archived collections by archive size, average rarity, newest observed gift number, and market coverage.
Collection watchlist	Saves collections for manual refreshes, stored snapshots, new-since review, and scheduled monitoring.
Watchlist alerts	Notifies users about meaningful sampled-stat changes, linked newly observed gift numbers, upgrade availability, floor-like drops, and sale events when source data is available.
Scheduled collection crawls	Queues scheduled full crawls for watchlisted collections from a separate scheduler loop so archived data can refresh without manual button presses.
Collection trends	Shows stored collection snapshots from stats, compare, and watchlist checks.
CSV/JSON export	Exports random and filtered result sets without owner identifiers.
Result sorting	Sorts random, filtered, deep-scan, and crawl result sets by rarity or gift number where cached result data is available.
Language	Switches the bot UI between English and Russian. Telegram language is used only as the initial default.
Parser health dashboard	Admin-only view for parser attempts, success rate, failures, JSON errors, latency, and HTTP status counts.
Runtime parser settings	Admin-only controls for parser range size, max pages, and request delay until restart.
GPT-filtered view	Optional demographic-inference mode hidden by default. When deliberately enabled, it uses the OpenAI Responses API, defaults to gpt-5.6-luna, disables response storage, and requires a server key or encrypted user-key flow.

Admin tools are separate from user modes. Admins configured in ADMIN_IDS can open /admin for bot statistics, parser health, runtime parser settings, proxy management, broadcasts, and database reset actions.

Clone this bot wisely

1. Create a Bot Token

1. Open Telegram and message @BotFather.
2. Create a bot with /newbot.
3. Copy the token into your local .env file as BOT_TOKEN.

2. Create Environment File

Copy the example file:

copy .env.example .env

• Keep WEBHOOK_HOST empty for local long-polling.
• In webhook mode, set a separate WEBHOOK_SECRET_TOKEN; the bot token is never placed in the public route.
• Use sqlite:///smartnftparser.db for local development unless you specifically want PostgreSQL.
• Leave REDIS_URL empty if you do not run Redis locally. The bot will use in-memory cache automatically.
• Keep GPT_FILTER_ENABLED=0 for the privacy-safe default. Set GPT_KEY_ENCRYPTION_SECRET before allowing users to save personal GPT keys.
• Do not commit .env or real tokens, proxy credentials, database passwords, or API keys.

Generate a GPT-key encryption secret with:

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Generate a webhook secret with:

python -c "from secrets import token_urlsafe; print(token_urlsafe(32))"

3. Install Dependencies

python -m venv .venv
python -m pip install -r requirements.txt
python -m compileall .

4. Start the Bot Locally

& ".venv\Scripts\python.exe" -X utf8 -u "main.py"

Expected startup flow:

TELEGRAM NFT PARSER BOT
Initializing database...
Database initialized: sqlite
Initializing Redis...
Initializing proxy manager...
Started workers
Starting in polling mode...
Run polling for bot @your_bot

If Redis is not running and REDIS_URL is empty or unreachable, startup can still succeed. The bot logs a warning and falls back to in-memory cache.

5. Stop the Bot

Stop the terminal where the bot is running - the shutdown handler stops background workers, closes cache/database connections, and closes the bot session.

If a process is stuck on Windows:

Stop-Process -Id <PROCESS_ID>

Admin Usage

1. Add your numeric Telegram ID to ADMIN_IDS.
2. Restart the bot.
3. Send /admin to the bot.
4. Use statistics, parser health, runtime parser settings, broadcasts, proxy management, and database reset actions.

Running in Production

Polling is a simplest run style and works without a public HTTPS URL:

WEBHOOK_HOST=

Webhook mode is useful on hosted platforms that provide a public HTTPS URL.

WEBHOOK_HOST=https://your-domain.example
WEBHOOK_PATH=/telegram/webhook
WEBHOOK_SECRET_TOKEN=<random-secret>
PORT=8080

When WEBHOOK_HOST is set, the bot registers:

https://your-domain.example/telegram/webhook

Telegram sends the secret in its webhook header and aiogram verifies it before dispatching an update. The app listens on 0.0.0.0:${PORT} and exposes GET /healthz for hosting-platform liveness checks; the response contains no tokens, URLs, user data, or parser error details.

Local Database

The project has a local database when DATABASE_URL=sqlite:///smartnftparser.db.

SQLite files are created automatically in the project root:

smartnftparser.db
smartnftparser.db-wal
smartnftparser.db-shm

What is stored:

Table	Purpose
users	Telegram users who started or interacted with the bot, including Telegram ID, username, first name, request count, and blocked status.
requests	Parser request history: user ID, mode, collection label, result count, and timestamp.
user_filters	Saved filtered-view selections per user.
proxies	Admin-managed parser proxies and health status.
favorite_gifts	Saved public gift links per user.
filter_presets	Named saved filter setups per user.
user_settings	Per-user settings, including encrypted GPT API key data when personal key storage is enabled.
collection_watchlists	Saved collections and last sampled stat snapshot per user.
collection_snapshots	Historical sampled stats for trends and watchlist comparisons.
archived_gifts	Durable archive of parsed public gift metadata used by archive summaries and leaderboards.
crawl_jobs	Durable progress/cancel state for full-crawl jobs.
market_events	Sale, floor-like, and upgrade events inferred from parser-provided market metadata.

GPT Key Handling

GPT-filtered view is registered but hidden and blocked unless GPT_FILTER_ENABLED=1. When enabled, a user can send an OpenAI API key only when GPT_KEY_ENCRYPTION_SECRET is configured with a valid Fernet key.

• The bot attempts to delete the Telegram message containing the key after saving it.
• The key is encrypted before it is stored in the configured bot database.
• If GPT_KEY_ENCRYPTION_SECRET is empty, invalid, or unavailable, the bot refuses to save the key and the personal-key flow stays disabled.
• Legacy plaintext GPT keys from older local databases are not used; users must re-add their key after encryption is configured.
• Admin database reset deletes stored GPT keys.
• OpenAI calls use the Responses API with store=False, a hashed per-user safety identifier, validated Telegram usernames, and gpt-5.6-luna by default. The model and reasoning effort remain configurable.

For a stronger production setup, keep GPT-filtered mode hidden or disabled unless you have a clear privacy policy and a managed encrypted secrets store.

Project Structure

%%{init: {"flowchart": {"curve": "linear", "nodeSpacing": 35, "rankSpacing": 55}} }%%
flowchart TB
	ENTRY["Application Runtime<br/>main.py / webhook.py"]
	CONFIG["Configuration & Localisation<br/>config.py / .env.example"]
	RUNTIME["Runtime Support<br/>parser_runtime.py / background_tasks.py / security.py"]

	UI["User Bot Interface<br/>handlers.py / filter_handlers.py"]
	ADMIN["Admin Interface<br/>admin.py"]
	QUEUE["Task Queue<br/>async_queue.py"]

	SERVICES["Feature Services<br/>parser_wrapper.py / watchlist_runner.py"]
	PARSER["Parser Client<br/>async_parser.py / user_agents.py"]
	INTEGRATIONS["External Integrations<br/>market providers / proxies / optional GPT<br/><br/>market_api.py / proxy_manager.py / gpt_filter.py"]

	DB["Persistence<br/>database.py / postgres_schema.sql"]
	CACHE["Cache<br/>redis_cache.py"]
	DATA["Static Gift Data<br/>gifts_constants.py"]
	RELEASE["Release Assets<br/>media / Procfile / requirements.txt"]

	ENTRY --> CONFIG
	ENTRY --> RUNTIME
	ENTRY --> UI
	ENTRY --> ADMIN
	UI --> QUEUE
	ADMIN --> QUEUE
	QUEUE --> SERVICES
	UI --> SERVICES
	ADMIN --> DB
	UI --> DATA
	SERVICES --> PARSER
	SERVICES --> DB
	SERVICES --> CACHE
	SERVICES --> INTEGRATIONS
	PARSER --> INTEGRATIONS
	DB --> RUNTIME
	CONFIG --> SERVICES
	RELEASE --> ENTRY

	classDef entry fill:#1f2937,stroke:#94a3b8,stroke-width:1.5px,color:#f8fafc;
	classDef config fill:#312e81,stroke:#a5b4fc,stroke-width:1.5px,color:#eef2ff;
	classDef ui fill:#0f172a,stroke:#60a5fa,stroke-width:1.5px,color:#eff6ff;
	classDef admin fill:#7c2d12,stroke:#fdba74,stroke-width:1.5px,color:#fff7ed;
	classDef queue fill:#064e3b,stroke:#6ee7b7,stroke-width:1.5px,color:#ecfdf5;
	classDef parser fill:#164e63,stroke:#67e8f9,stroke-width:1.5px,color:#ecfeff;
	classDef data fill:#3f3f46,stroke:#d4d4d8,stroke-width:1.5px,color:#fafafa;
	classDef store fill:#581c87,stroke:#d8b4fe,stroke-width:1.5px,color:#faf5ff;
	classDef release fill:#713f12,stroke:#fde68a,stroke-width:1.5px,color:#fffbeb;

	class ENTRY entry;
	class CONFIG config;
	class UI ui;
	class ADMIN admin;
	class QUEUE queue;
	class SERVICES queue;
	class PARSER parser;
	class DATA data;
	class DB store;
	class CACHE store;
	class INTEGRATIONS parser;
	class RUNTIME config;
	class RELEASE release;

Loading

File	Purpose
main.py	Application entry point. Initialises logging, database, cache, proxy manager, task queue, bot, routers, and polling/webhook lifecycle.
config.py	Reads environment variables, parser settings, rate limits, button labels, and bot messages.
handlers.py	Main Telegram handlers: /start, help, main menu, random view, collection statistics, rare gifts, GPT-filtered view, task status, pagination, and refresh.
filter_handlers.py	Filtered-view workflow: collection/model/backdrop selection, stored user filters, result pagination, and refresh.
parser_wrapper.py	High-level parser functions used by queue workers. Converts parser results into bot-ready dictionaries and computes stats/rarity rankings.
async_parser.py	Low-level asynchronous parser for the public gift API. Handles headers, compression, proxy rotation, ranges, metadata parsing, and privacy-safe export.
database.py	SQLite/PostgreSQL persistence for users, settings, filters, favorites, watchlists, snapshots, archives, crawl jobs, market events, and proxies.
redis_cache.py	Result cache. Uses Redis when available, otherwise falls back to in-memory TTL storage.
async_queue.py	In-process async task queue and worker pool for parser jobs. Enforces one active full crawl at a time.
background_tasks.py	Retains fire-and-forget bot tasks until completion and logs otherwise-unobserved exceptions.
proxy_manager.py	Proxy storage, validation, health checks, active-proxy rotation, and masking for admin display.
admin.py	Admin panel with stats, broadcasts, and proxy management.
webhook.py	Polling and authenticated webhook startup, clean runner shutdown, and a secret-free /healthz endpoint.
gifts_constants.py	Static collection, model, backdrop, and pattern lists used by keyboards and filters.
gpt_filter.py	Optional GPT-5.6 Responses API username-classification helper with stateless requests and strict output validation.
user_agents.py	Randomised request headers for parser requests.
watchlist_runner.py	Background watchlist checker plus separate scheduled-crawl scheduler loop.
market_api.py	Provider-neutral external market API helper.
parser_runtime.py	In-memory parser runtime settings and health counters.
security.py	Encryption enforcement helpers for user-supplied GPT keys.
postgres_schema.sql	Manual PostgreSQL schema matching the bot's auto-created PostgreSQL tables.
media/	Optional README screenshots and portfolio images.
requirements.txt	Python dependencies.
Procfile	Deployment start command for platforms that support Procfile apps.