Capture robots.txt + llms.txt as crawl-time compliance witnesses - #10

Capture robots.txt + llms.txt as crawl-time compliance witnesses#10

MirjamOdile wants to merge 1 commit into

maindiscourselab/scrapai-cli:mainfrom

pr/04-compliance-witnessesdiscourselab/scrapai-cli:pr/04-compliance-witnessesCopy head branch name to clipboard