Getting tons of false negatives when scanning common usernames. Is there a way to reduce rate limiting?

[BittuChan]

Hey everyone,

I've been using Sherlock for some OSINT work and running into an issue where I'm getting way more false negatives than expected, especially when scanning relatively common usernames like "alex_2023" or "john_dev". The scan finishes quickly but only returns maybe 5-10 results when I know for a fact the username exists on 20+ platforms.

Here's my current setup:
sherlock alex_2023 --timeout 120 --csv

I'm running this from a residential IP (no VPN/proxy) and the timeout seems generous enough. A few observations:

1. Some sites consistently fail such as Reddit, Instagram, TikTok almost never show up in results even when the username definitely exists there
2. The scan runs FAST like suspiciously fast, completing 400+ sites in under 90 seconds
3. Verbose mode shows connection errors, there are lots of "Error: 429" and "Error: Timeout" messages

I feel that I'm getting rate-limited by a bunch of sites, but I'm not sure how to work around this without making the scan take forever. I tried adding --timeout 180 but that didn't really help.

[BittuChan]

Is there a recommended delay/throttle setting to avoid triggering rate limits? I didn't see this in the --help output but maybe I'm missing something.
Also would using --tor actually help here, or would that make things worse since Tor exit nodes are often flagged/rate-limited even more aggressively?

[Shiven0504]

hey @BittuChan, thanks for sharing your problem, here are a few solutions that should ideally work in your case:

1. Target Specific Sites
   Instead of scanning everything at once, use the --site flag to focus on specific platforms:
   sherlock alex_2023 --site Instagram --site Twitter --site GitHub --timeout 60

2. Don't Use Tor (Seriously)
   Tor actually makes this worse for most sites. Tor exit nodes are heavily rate-limited and often blocked entirely by major platforms. I've found Tor useful only when scanning obscure sites that don't have sophisticated anti-bot systems.

3. Residential Proxy Rotation
   This is the real solution for high-volume scans. You need rotating residential proxies (not datacenter IPs). Services like Bright Data, Smartproxy, or Oxylabs work well.
   Basic proxy usage:
   sherlock username --proxy socks5://your-proxy:1080
   For rotation, you'd need to write a simple wrapper script:

#!/bin/bash
PROXIES=("proxy1:1080" "proxy2:1080" "proxy3:1080")
for proxy in "${PROXIES[@]}"; do
    sherlock username --site Instagram --proxy "socks5://$proxy"
done

4. Increase Timeout + Add Delays
   Sherlock doesn't have a built-in rate limit/delay flag (which honestly it should). But you can run sequential scans with delays:

sherlock user1 --timeout 120
sleep 30
sherlock user2 --timeout 120  
sleep 30
sherlock user3 --timeout 120

Not elegant, but it works for small batches.

[Shiven0504]

One More Thing -
If you're getting consistent false negatives on specific sites (Instagram, Reddit), it might also be because those sites recently changed their detection logic and Sherlock's site definitions (data.json) haven't been updated yet.

Check the GitHub issues to see if anyone else is reporting problems with those specific sites.

Hope this helps!

[BittuChan]

Hii @Shiven0504, This is super helpful, thank you!
I switched to the targeted --site approach and things are working way better now. Breaking the scan into batches with 30-second delays between each batch completely solved the rate limiting issue.

Also checked the GitHub issues like you suggested and found out that Instagram detection had recently broken in the latest version. Pinned to an older version (v0.14.2) temporarily and that fixed the Instagram false negatives.

Marking this as resolved. Thanks again for the detailed breakdown!

[ppfeister]

@BittuChan Since you're running 0.14.2, are you using --local?

I'm surprised that an older version started working again. Might have to take a look at that when I have some time, see what can be brought back for ig