[Dennys Kaluzny Pereira (protonnegativo)]

IoCExtract URL-decodes each URL it finds and adds the decoded URL and its host to the results (_check_decode_url). That only works for the first URL in the input.

Reproducing

>>> from msticpy.transform.iocextract import IoCExtract
>>> ie = IoCExtract()
>>> ie.extract("http://%65vil.com/x http://good.org/y")["dns"]
{'65vil.com', 'evil.com', 'good.org'}
>>> ie.extract("http://good.org/y http://%65vil.com/x")["dns"]
{'65vil.com', 'good.org'}

http://evil.com/x is also missing from the url results in the second case.

Cause

_scan_for_iocs passed match_pos, the end offset of the previous match in the source string, to _check_decode_url, which used it as the start position for finditer on the decoded URL. For the first URL the offset is 0. For later ones it points into, or past the end of, a different and shorter string, so nothing is found.

The change

_check_decode_url scans the decoded URL from the start. With that, match_pos was no longer read anywhere (finditer(src, match_pos) only ever saw 0), so I removed it.

Testing

• Added test_url_encoded_host in tests/transform/test_ioc_extractor.py, with the encoded URL first and second. The second case fails on main and passes with the change.
• pytest tests/transform/test_ioc_extractor.py tests/context/test_lookup.py tests/context/test_vtlookup.py: same results as main apart from the new test (the remaining errors are the optional passivetotal import).
• ruff check and ruff format --check pass on iocextract.py.

[Dennys Kaluzny Pereira (protonnegativo)]

@microsoft-github-policy-service agree