Repository navigation
Add NetAPI threat intelligence provider - #956
Open
Slava Olesik (slavaolesik) wants to merge 3 commits into
Open
Slava Olesik (slavaolesik) wants to merge 3 commits into
Slava Olesik (slavaolesik) wants to merge 3 commits into
Conversation
Author
|
@microsoft-github-policy-service agree company="CodeRockers LTD" |
Author
|
NetAPI is discontinuing its compromised IP feed, so I have narrowed this PR to domains, hostnames and URLs. The ipv4 and ipv6 queries are gone, and the provider is now tested in its own module instead of the shared harness, because that harness looks up IP addresses in bulk. I have updated the description to match. |
Author
|
Update: NetAPI keeps its compromised IP feed after all, so I put the IP part of this PR back as it was first submitted. Sorry for the back and forth. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A
NetAPIthreat intelligence provider. NetAPI publishes a free feed of compromised domains and IP addresses, the hosts that public threat feeds (URLhaus, CERT Polska, blackbook and others) report for malware or phishing, and the NetAPI Top 1M, a daily popularity ranking. A lookup answers "is this host in the feed now, was it in the past, and since when".Supported observables:
dns,hostname,url,ipv4,ipv6. No file hashes.The API needs no key (30 requests a minute and 1,000 a day per IP), so the config entry has no
Argsblock, as for ScanMalware:Behaviour
compromised-check. A current listing (seen within the last 24 hours) is a hit with severityhigh. A past listing only is a hit with severityinformation: most of those are incidents that are over, often a legitimate site that was compromised and cleaned up. A value that was never listed is a negative result.Detailscarries the listing state, the matched entry, first and last seen, the reporting feed and the registrar.Matched Valueshows which entry matched, so a listing ofexample.comis not presented as a listing ofwww.example.comitself.ranksubtype (dns-rank,hostname-rank) callsdomain-rankand returns the position of the registered domain in the NetAPI Top 1M, the same kind of context as the OpenPageRank provider._QUERIESentries, one request each, no override of the lookup flow. No auth header is sent. The provider sets its ownUser-Agent,netapi-msticpy/1.0.Verification
pytest tests/context/test_tiproviders.py: 41 passed on unmodifiedmain, 42 passed with this change.pytest tests/context/test_netapi.py: 11 passed: current, past and never-listed results, the registered-domain match, URLs, both rank results, a 429, an empty answer, no auth header and unsupported hashes. Coverage of the new module is 100% (measured withcoverage runandCOVERAGE_CORE=sysmon, sincepytest-covhits the samenumpy: cannot load module more than once per processerror mentioned in Add ScanMalware threat intelligence provider #938).ruff checkandruff format --checkclean,pylint10.00/10,pydocstyle --convention=numpyandmypyclean on the new module.TILookupagainst the live API:Disclosure
I run NetAPI. Happy to adjust anything here, including dropping the provider from the default config.