Skip to content

Fix ParseEmailFilesV2 root attachment parsing - #46012

Open
otsimed-contribute wants to merge 5 commits into
demisto:contrib/otsimed-contribute_fix/parse-email-envelope-regressionfrom
otsimed-contribute:fix/parse-email-envelope-regression
Open

otsimed-contribute wants to merge 5 commits into
demisto:contrib/otsimed-contribute_fix/parse-email-envelope-regressionfrom
otsimed-contribute:fix/parse-email-envelope-regression

Conversation

@otsimed-contribute

@otsimed-contribute otsimed-contribute commented Sep 25, 2026 •

Copy link
Copy Markdown

Summary

  • Decode attached .eml payloads when MIME parsing returns serialized message objects, then parse nested emails while respecting max depth and nesting selection.
  • Remove empty unnamed MIME artifacts so they are not emitted as attachments.
  • Add synthetic regression fixtures, test coverage, README notes, and a CommonScripts release note.

Validation

  • ParseEmailFilesV2_test.py: 33 passed.
  • Ruff format/check: passed.
  • demisto-sdk format and demisto-sdk validate: passed (Docker unavailable for Docker-dependent checks).

relates: https://jira-dc.paloaltonetworks.com/browse/CIAC-18446

@CLAassistant

CLAassistant commented Sep 25, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@content-bot content-bot added Contribution Thank you! Contributions are always welcome! External PR Xsoar Support Level Indicates that the contribution is for XSOAR supported pack labels Sep 25, 2026
@content-bot
content-bot changed the base branch from master to contrib/otsimed-contribute_fix/parse-email-envelope-regression September 25, 2026 14:37
@content-bot

Copy link
Copy Markdown
Contributor

Thank you for your contribution. Your generosity and caring are unrivaled! Make sure to register your contribution by filling the Contribution Registration form, so our content wizard @kamalq97 will know the proposed changes are ready to be reviewed.
For your convenience, here is a link to the contributions SLAs document.

@content-bot

Copy link
Copy Markdown
Contributor

Hi @otsimed-contribute, thanks for contributing to the XSOAR marketplace. To receive credit for your generous contribution please follow this link.

@content-bot

Copy link
Copy Markdown
Contributor

🤖 AI-Powered Code Review Available

Hi @kamalq97, you can leverage AI-powered code review to assist with this PR!

Available Commands:

  • @marketplace-ai-reviewer start review - Initiate a full AI code review
  • @marketplace-ai-reviewer re-review - Incremental review for new commits

@kamalq97 kamalq97 closed this Sep 27, 2026
@kamalq97 kamalq97 reopened this Sep 27, 2026
@kamalq97 kamalq97 added Contribution Form Filled Whether contribution form filled or not. Community labels Sep 27, 2026
@kamalq97 kamalq97 assigned TheL0L and unassigned kamalq97 Sep 28, 2026
@kamalq97
kamalq97 requested review from TheL0L and removed request for kamalq97 September 28, 2026 07:13
@TheL0L TheL0L added the ready-for-ai-review The PR is ready for reviewing the PR with the AI Reviewer. label Oct 1, 2026
@TheL0L

TheL0L commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Hello @otsimed-contribute, thank you for your contribution.

I have started the review, and would like to clarify the justification for remove_empty_unnamed_attachments().

Soon I will post the full review.

@TheL0L

TheL0L commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Additionally, please note that the terms listed in the CLA have been updated.
Please review the changes and re-sign the CLA.

CLA assistant check Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 2 committers have signed the CLA.❌ Copilot❌ otsimed-contributeYou have signed the CLA already but the status is still pending? Let us recheck it.

@marketplace-ai-reviewer marketplace-ai-reviewer removed the ready-for-ai-review The PR is ready for reviewing the PR with the AI Reviewer. label Oct 1, 2026
@marketplace-ai-reviewer

Copy link
Copy Markdown
Contributor

🤖 Analysis started. Please wait for results...

@marketplace-ai-reviewer

Copy link
Copy Markdown
Contributor
🤖 AI Review Disclaimer

This review was generated by an AI-powered tool and may contain inaccuracies. Please be advised, and we extend our sincere apologies for any inconvenience this may cause.

@marketplace-ai-reviewer marketplace-ai-reviewer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi! Thanks for your contribution to the ParseEmailFilesV2 script. I've left a few notes to help polish the code and documentation before we can merge.

Mainly, please ensure you use .get() for safe dictionary access, fix a variable shadowing issue, and add the missing type hints. You'll also need to adjust the release notes formatting and remove internal test documentation from the README.

Thanks again for your hard work on this!

Additionally, please address the following file-level notes:

  • Packs/CommonScripts/ReleaseNotes/1_22_58.md: The release notes filename 1_22_58.md does not match the currentVersion in pack_metadata.json.

@TheL0L please review and approve the results generated by the AI Reviewer by responding 👍 on this comment.

Comment thread Packs/CommonScripts/Scripts/ParseEmailFilesV2/ParseEmailFilesV2.py Outdated
Comment thread Packs/CommonScripts/Scripts/ParseEmailFilesV2/ParseEmailFilesV2.py Outdated
Comment thread Packs/CommonScripts/Scripts/ParseEmailFilesV2/README.md Outdated
@content-bot

Copy link
Copy Markdown
Contributor

For the Reviewer: Successfully created a pipeline in GitLab with url: https://gitlab.xdr.pan.local/xdr/cortex-content/content/-/pipelines/13751687

@content-bot content-bot removed the ready-for-instance-test In contribution PRs, this label will cause a trigger of a build with a modified pack from the PR. label Oct 13, 2026

@TheL0L TheL0L left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @otsimed-contribute for addressing my previous comments.
I have reviewed the contribution once more and left 2 comments.

Once these are addressed we can proceed with merging the PR.

Comment thread Packs/CommonScripts/Scripts/ParseEmailFilesV2/ParseEmailFilesV2.py Outdated
Comment thread Packs/CommonScripts/Scripts/ParseEmailFilesV2/ParseEmailFilesV2.py Outdated
@TheL0L TheL0L added the pending-contributor The PR is pending the response of its creator label Oct 14, 2026
@otsimed-contribute
otsimed-contribute force-pushed the fix/parse-email-envelope-regression branch from 406f52e to ba51f17 Compare October 14, 2026 14:38
@TheL0L TheL0L added docs-approved ready-for-instance-test In contribution PRs, this label will cause a trigger of a build with a modified pack from the PR. and removed pending-contributor The PR is pending the response of its creator labels Oct 15, 2026
@content-bot

Copy link
Copy Markdown
Contributor

For the Reviewer: Trigger build request has been accepted for this contribution PR.

@content-bot

Copy link
Copy Markdown
Contributor

For the Reviewer: Successfully created a pipeline in GitLab with url: https://gitlab.xdr.pan.local/xdr/cortex-content/content/-/pipelines/13814142

@content-bot content-bot removed the ready-for-instance-test In contribution PRs, this label will cause a trigger of a build with a modified pack from the PR. label Oct 15, 2026
@TheL0L TheL0L added the ready-for-pipeline-running Whether the pr is ready for running the whole pipeline, including testing on SAAS machines label Oct 15, 2026
@TheL0L

TheL0L commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

All contributors must sign the CLA for their contributions to be accepted.
Please read and sign the CLA.

CLA assistant check Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 2 committers have signed the CLA.❌ Copilot❌ otsimed-contributeYou have signed the CLA already but the status is still pending? Let us recheck it.

@TheL0L TheL0L left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @otsimed-contribute for the continued cooperation!
I have reviewed the changes, and we can move forward.

While automatic build workflows validate the changes, please prepare a demo.

We can schedule a meeting or you can share a video recording.

Feel free to reach out to me here or via our Slack DFIR Community.

@TheL0L TheL0L added pending-demo Demo pending pending-contributor The PR is pending the response of its creator labels Oct 15, 2026
@otsimed-contribute

Copy link
Copy Markdown
Author
PA.recording-20261020_153209UTC-Meeting.Recording-compressed.mp4

@TheL0L

TheL0L commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Thank you @otsimed-contribute for the detailed demo.
Please confirm whether you have signed the CLA, and I will proceed with merging the PR.

@TheL0L TheL0L added post-demo and removed pending-demo Demo pending labels Oct 21, 2026
@otsimed-contribute

otsimed-contribute commented Oct 21, 2026 •

Copy link
Copy Markdown
Author

Yep, I have!

@TheL0L TheL0L removed the pending-contributor The PR is pending the response of its creator label Oct 22, 2026
@TheL0L

TheL0L commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Sorry for the inconvenience @otsimed-contribute, but it looks like the CLA was not submitted.

Please note that it must be submitted via this specific link as it is tied to this PR.

Also note that the copilot agent is listed in the required signatures.

Please try once more, and if it does not work, you can open a new contribution PR with a single clean commit authored by you.

Once again, sorry for the inconvenience.

@TheL0L TheL0L added the pending-contributor The PR is pending the response of its creator label Oct 22, 2026
…marked as an attachment

The email parser handles an email whose top-level part has 'Content-Disposition: attachment' as a single attachment, so the emails attached to it are not parsed, and their files contain a placeholder instead of the email. If such an email contains an attached email (message/rfc822), the script parses a copy of the file without that header, so the parser handles it as a regular email. Other emails are parsed as before.
…rked as an attachment

Adds a synthetic email whose top-level part is marked as an attachment and has an attached email (test_data/reported_message_root_attachment.eml, which attaches test_data/original_message.eml), and tests for the parsed emails, the saved attachments, parse_only_headers, and the copy of the file.
…ctet-stream

The parser also parses an attached application/octet-stream file named .eml as an email, so such an email is handled as well, as in a regular email. text/plain files are not parsed as emails by the parser, so they are not handled.
…el part

Use get_content_disposition(), which returns only the disposition type, in lowercase, as suggested in the review: the type is case-insensitive (RFC 2183), and a parameter which contains "attachment" (for example, the file name) does not mark the part as an attachment.
Bumps the pack version to 1.22.65 and adds the release notes for the ParseEmailFilesV2 fix.
@otsimed-contribute
otsimed-contribute force-pushed the fix/parse-email-envelope-regression branch from ba51f17 to d629675 Compare October 23, 2026 14:16

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Community Contribution Form Filled Whether contribution form filled or not. Contribution Thank you! Contributions are always welcome! docs-approved External PR pending-contributor The PR is pending the response of its creator post-demo ready-for-pipeline-running Whether the pr is ready for running the whole pipeline, including testing on SAAS machines Xsoar Support Level Indicates that the contribution is for XSOAR supported pack

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants