Missing active sales

[theitchtime]

Noticed a whole bunch not in the active sales list

https://itch.io/s/86526/bundle-sale
https://itch.io/s/88459/year-of-the-dragon
https://itch.io/s/102043/free-to-get
https://itch.io/s/108977/buttworm-ate-the-price-d
https://itch.io/s/110941/free-gift
https://itch.io/s/115108/everything-is-free
https://itch.io/s/116899/sale-symbio-games
https://itch.io/s/117435/support-me-
https://itch.io/s/122261/aquapop-sale
https://itch.io/s/122579/spring-summer-sale
https://itch.io/s/124458/a-very-very-very-goooooooooooooood-sale
https://itch.io/s/125390/summer-sale
https://itch.io/s/127091/summer-sale
https://itch.io/s/127385/claudia-for-free-summer-2024
https://itch.io/s/128902/vacation-sale-100
https://itch.io/s/129069/king-boo-sale
https://itch.io/s/129092/for-free

Note that 129092 seems to be a short ?1-day? sale. Also wish there was easier way to scrape all these unknown finds (and what else I missed out).

But I love your program! Incredibly helpful.

EDIT: I've been randomly clicking a bunch of sales and scanning for 100% tags, which is how I found most of these new ones.

EDIT2:
https://itch.io/s/59746/free
https://itch.io/s/107895/old-game-sale
https://itch.io/s/115229/2024-free
https://itch.io/s/115331/play-now
https://itch.io/s/116859/curtain-play-it
https://itch.io/s/122432/oceanox-development-sale
https://itch.io/s/122615/gone-out-of-business-sale
https://itch.io/s/127260/bedrotting
https://itch.io/s/127938/amareica-going-for-free-until-end-of-year

EDIT3:
Another thing I noticed is that sales sometimes show up on the feeds.
https://itch.io/games/newest/on-sale.xml?page=1

https://itch.io/feed/sales.xml
New sales was blank for 2024-07-19 but 59028 appeared for 2024-07-20.

But I've yet to figure out a master list for all those missing entries listed above.

[Smart123s]

Wow, that's a huge list, thanks! I've manually refreshed those sales in the list. Sorry for the late response, I didn't have much time on hand. I'll look into why those weren't cached as soon as I can. Also thanks for the hint on how you found them. I check sale pages such as https://itch.io/games/on-sale and even https://itch.io/tools/on-sale. I'll check if those XML have information in addition to those pages. Any other methods you have found sales? Everything would be useful.

[theitchtime]

I haven't figured out a reliable method yet.

Just been manually scrubbing the sales, which I think your scanner is pretty reliable.

https://itch.io/games/newest/on-sale?page=1
https://itch.io/tools/newest/on-sale?page=1
https://itch.io/game-assets/newest/on-sale?page=1
https://itch.io/comics/newest/on-sale?page=1
https://itch.io/books/newest/on-sale?page=1
https://itch.io/physical-games/newest/on-sale?page=1
https://itch.io/soundtracks/newest/on-sale?page=1
https://itch.io/game-mods/newest/on-sale?page=1
https://itch.io/misc/newest/on-sale?page=1

The rest I've been randomly entering sale id #s and then looking for 100% off tags. Surprisingly I picked up a lot of "hidden" things this way, but had to manually do this over maybe 100-ish pages.

I noticed that BRUD sale just started but it's not showing on the sales.xml feed. So meh.
http://itchclaim.tmbpeter.com/data/2594408.json

Also want to report that some items not having a "claim" page are still claimable. There's usually no download links but good for future-proofing.
https://flagimtoshi.itch.io/claudia-plus
https://kool-games-dev-jarmele.itch.io/phineas-and-ferb-original
(enough others)

Thank you for adding all those links btw! This tool is exceptionally amazing. :)

[Smart123s]

The script actually scrapes the exact same URLs, just in JSON format:

ItchClaim/ItchClaim/DiskManager.py

Line 164 in be8880f

r = requests.get(f"https://itch.io/{category}/newest/on-sale?page={page}&format=json",

ItchClaim/ItchClaim/__main__.py

Lines 87 to 90 in be8880f

	for category in ['games', 'tools', 'game-assets', 'comics', 'books', 'physical-games',
	'soundtracks', 'game-mods', 'misc']:
	print(f'Collecting sales from {category} list')
	DiskManager.get_all_sale_pages(category=category)

And if you have noticed, the sale URLs have an incremental ID. So the script also goes through those. It starts where it last left off, and goes until it gets 404 errors.

The https://itch.io/games/newest/on-sale.xml?page=1 URL seems to be the same as the one without the .xml extension.
And https://itch.io/feed/sales.xml doesn't seem to be too useful. It has only one item in to right now: https://web.archive.org/web/20240723205609/https://itch.io/feed/sales.xml

[Smart123s]

If you notice any more missing sales, please let me know here. And if you could, I would greatly appreciate a link to where you have found it. (Maybe with an archive.org link too, in case the list's content changes?)

[theitchtime]

See area below for Wendigo 100%
http://web.archive.org/web/20240724204926/https://itch.io/s/87970/find-yourself
==>
https://itch.io/s/129248/wendigo-sale

And this one by chance.
https://itch.io/s/121220/

[Smart123s]

Thanks! I think I have found the root cause of the problem.

Seems like at job https://github.com/Smart123s/ItchClaim/actions/runs/9258985901/job/25470049749 (archived logs), itch.io was down for 5+ hours, so the script continued to scrape sales. Then itch.io came back online, the script got a 404 error, which is a valid stop condition, so it stopped, and noted 129906 as the last checked sale.
I have started a recheck of all sales after 123964. That's where the buggy run started.
I've pushed an update (24ba668), so that if a network error happens again, it'll abort the running refresh immediately.

[Smart123s]

https://itch.io/s/121220/ didn't get cached, because it doesn't have a price. Will be curious to see what happens to this game on Friday, when the sale ends.

[myzinetime]

Missing:
https://itch.io/s/135203/halloween-sale-2024
https://itch.io/s/135396/halloween-sale-2024
https://itch.io/s/135522/halloween-special
https://itch.io/s/135582/halloween-sale-2024
https://itch.io/s/135621/halloween-sale-2024
https://itch.io/s/135713/soulstice-2024

Edit: An idea is to check the highest scraped sale # and if it's newer (135713) than your last checked one (135096), you're missing out on a ton

[Smart123s]

Missing: https://itch.io/s/135203/halloween-sale-2024 https://itch.io/s/135396/halloween-sale-2024 https://itch.io/s/135522/halloween-special https://itch.io/s/135582/halloween-sale-2024 https://itch.io/s/135621/halloween-sale-2024 https://itch.io/s/135713/soulstice-2024

Edit: An idea is to check the highest scraped sale # and if it's newer (135713) than your last checked one (135096), you're missing out on a ton

There were a lot of "404 Not found" sale pages after 135096 (the first valid one after that is 135171), and the script assumes, that there are no more sales, if it finds 10+ "404 Not found" sale pages. I've manually skipped these in https://github.com/Smart123s/ItchClaim/actions/runs/11540667072/job/32121759254.