Skip to content

Add anonymous public data spider - #57

Open
xluos wants to merge 1 commit into
cv-cat:masterfrom
xluos:feature/anonymous-public-spider
Open

Add anonymous public data spider#57
xluos wants to merge 1 commit into
cv-cat:masterfrom
xluos:feature/anonymous-public-spider

Conversation

@xluos

@xluos xluos commented Jun 12, 2026

Copy link
Copy Markdown

Summary

  • Add public_spider.py as a CLI entrypoint for anonymous-cookie public data collection.
  • Collect public user profile data, user videos, comments, and comment image URLs into JSON/JSONL outputs.
  • Extend cookie generation so callers can pass a target URL, headless mode, timeout, and receive a reusable cookie string.
  • Document the new anonymous public spider workflow and ignore generated outputs/ data.

Validation

  • git diff --check origin/master..HEAD
  • .venv/bin/python -m compileall public_spider.py utils/cookie_util.py
  • .venv/bin/python public_spider.py --help

@xluos
xluos marked this pull request as ready for review June 12, 2026 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant