Add optional owningProjectId to GlueDQResultToDataZonePublisher to avoid posting to the wrong project - #182
Open
ashwinrh wants to merge 1 commit into
Conversation
search_asset_id resolves the target asset via a domain-wide SearchListings by table name and posts DQ results to every match. When the same schema/table is published under more than one project, this posts to assets owned by projects the caller may not be a member of, causing AccessDeniedException on PostTimeSeriesDataPoints (and aborting the job). Add an optional owningProjectId parameter; when set, only assets owned by that project are selected. Defaults to None, preserving existing behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GlueDQResultToDataZonePublisherresolves the target DataZone asset insearch_asset_idvia a domain-wideSearchListingson the table name, then posts the data quality results to every matching asset.SearchListingsspans all owning projects, so when the same schema/table is published under more than one project, the transform posts to assets owned by projects the caller may not be a member of.PostTimeSeriesDataPointsauthorizes the write against the asset's owning project (there is noprojectIdinput on the API). If the caller is not a member of that project, the call fails with:Because the post loop has no per-item error handling, one denied asset aborts the whole Glue job.
Fix
Add an optional
owningProjectIdparameter topost_dq_results_to_datazone/search_asset_id. When set, only assets owned by that project are selected (eachSearchListingsresult already carriesowningProjectId); same-named assets in other projects are skipped and logged.owningProjectIddefaults toNone, so the change is fully backward compatible — existing jobs that omit it behave exactly as before.Changes
post_dq_results_to_datazone.py: new optionalowningProjectIdarg on the entry function andsearch_asset_id; filter on the asset's owning project when provided.post_dq_results_to_datazone.json: new optionalowningProjectIdparameter so it surfaces in the AWS Glue Studio custom transform UI.Usage
Pass the id of the project that owns the asset you intend to publish DQ results to, e.g.
owningProjectId="<yourProjectId>". Omit it to keep the previous name-only matching behavior.