`sesv2 list-suppressed-destinations` is not paginated (only first page is returned)

[adam-nielsen]

Describe the bug

Since there is no way to search the list of SES suppressed e-mails for a given address, I launched CloudShell and used this command:

aws sesv2 list-suppressed-destinations --query "SuppressedDestinationSummaries[*].[EmailAddress]" --output text | grep example.com

However I thought I had better test that this is actually returning the full list of addresses, so I counted the number of addresses returned:

aws sesv2 list-suppressed-destinations --query "SuppressedDestinationSummaries[*].[EmailAddress]" --output text | wc -l
1000

It seemed suspicious exactly 1000 addresses were returned, so I tried lowering the page size:

aws sesv2 list-suppressed-destinations --page-size 500 --query "SuppressedDestinationSummaries[*].[EmailAddress]" --output text | wc -l
500

Here, instead of making multiple requests for 500 records each, it only stopped after the first request. I tried to disable pagination to confirm:

aws sesv2 list-suppressed-destinations --page-size 500 --no-paginate --query "SuppressedDestinationSummaries[*].[EmailAddress]" --output text | wc -l
500

Again, I get 500 records. So whether pagination is enabled or disabled, I only get one page of results.

I believe this is a bug, and aws list-suppressed-destinations is not automatically retrieving additional pages when present, as the documentation says it should and as it apparently does for other services.

Expected Behavior

I expected to get the full list of records returned, not limited to just 1000 or 500.

Current Behavior

Only the first page of results were returned.

Reproduction Steps

Add 10 or 15 addresses to the suppression list, then query them via AWS CloudShell:

aws sesv2 list-suppressed-destinations --page-size 5 --query "SuppressedDestinationSummaries[*].[EmailAddress]" --output text | wc -l

You should get a number that matches however many addresses you added.

Possible Solution

Reuse the same pagination code already in use by the other commands.

Additional Information/Context

I have been given an e-mail address that is experiencing issues, and I am trying to find out whether it is included on any of our SES suppression lists.

CLI version used

AWS CloudShell / ap-southeast-2

Environment details (OS name and version, etc.)

AWS CloudShell

[aBurmeseDev]

Hi @adam-nielsen - thanks for reaching out. Could you help clarify your use case by answering a few questions below?

• what's the total number of records you're expecting and what's currently being returned?
• If not all the records were returned on single page, NextToken should be returned. Could you confirm that?
• which version is CLI are you using? To check, run aws --version.
• could you try running the command without --query and/or --page-size like so aws sesv2 list-suppressed-destinations?
• Debug logs would also give us more insight if you could share, without any sensitive info. You can get the logs by adding --debug to the command.

Also for reference on PageSize,

The number of results to show in a single call to ListSuppressedDestinations. If the number of results is larger than the number you specified in this parameter, then the response includes a NextToken element, which you can use to obtain additional results.

[adam-nielsen]

Hi @aBurmeseDev,

what's the total number of records you're expecting and what's currently being returned?

According to the AWS console, 3481:

If not all the records were returned on single page, NextToken should be returned. Could you confirm that?

Looks like it:

$ aws sesv2 list-suppressed-destinations | tail -n 5
            "LastUpdateTime": "2022-02-09T13:36:05.844000+00:00"
        }
    ],
    "NextToken": "AYADeOiYtjuUo4/Wv...
}

which version is CLI are you using? To check, run aws --version.

aws-cli/2.11.15 Python/3.11.3 Linux/4.14.255-291-231.527.amzn2.x86_64 exec-env/CloudShell exe/x86_64.amzn.2 prompt/off

could you try running the command without --query and/or --page-size like so aws sesv2 list-suppressed-destinations?

$ aws sesv2 list-suppressed-destinations | grep EmailAddress | wc -l
1000

Debug logs would also give us more insight if you could share, without any sensitive info. You can get the logs by adding --debug to the command.

They contain some sensitive info, so here are what look to be the relevant bits:

[...]
2023-05-03 04:36:57,130 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListSuppressedDestinations) with params: {'url_path': '/v2/email/suppression/addresses', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.11.15 Python/3.11.3 Linux/4.14.255-291-231.527.amzn2.x86_64 exec-env/CloudShell exe/x86_64.amzn.2 prompt/off command/sesv2.list-suppressed-destinations'}, 'body': b'', 'url': 'https://email.ap-southeast-2.amazonaws.com/v2/email/suppression/addresses', 'context': {'client_region': 'ap-southeast-2', 'client_config': <botocore.config.Config object at 0x7f8d2650a390>, 'has_streaming_input': False, 'auth_type': None}}
[...]
2023-05-03 04:36:57,294 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"NextToken":"AYADeH... [list of suppressed e-mail addresses removed]
2023-05-03 04:38:16,474 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sesv2.ListSuppressedDestinations: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7f8d26514490>>
2023-05-03 04:38:16,474 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2023-05-03 04:38:16,474 - MainThread - botocore.hooks - DEBUG - Event after-call.sesv2.ListSuppressedDestinations: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7f8d2650b010>>
2023-05-03 04:38:16,474 - MainThread - awscli.formatter - DEBUG - RequestId: 0598f5bb-8d95-4406-a277-44593c46043e
[end of --debug output]

So it looks like it picks up NextToken and knows it has to issue another request, but then doesn't for some reason?

[Ifedi]

I have the exact same issue, I think. Or at least, a very related one.

In my own case I'm trying to automate retrieval of all pages on the suppresion list using:

aws sesv2 list-suppressed-destinations --page-size 1000 --query 'sort_by(SuppressedDestinationSummaries,&LastUpdateTime)'

When I run the command without "--query", the output is like this:

{
    "SuppressedDestinationSummaries": [
        {
            "EmailAddress": "aaabbb@qq.com",
            "Reason": "BOUNCE",
            "LastUpdateTime": "2022-01-15T13:36:04.834000+01:00"
        },
        {
            "EmailAddress": "cccddd@yahoo.com",
            "Reason": "BOUNCE",
            "LastUpdateTime": "2022-01-20T19:12:25.620000+01:00"
        }
    ],
    "NextToken": "<LongString>"
}

On the other hand, running the command WITH --query returns only:

[
	{
		"EmailAddress": "aaabbb@qq.com",
		"Reason": "BOUNCE",
		"LastUpdateTime": "2022-01-15T13:36:04.834000+01:00"
	},
	{
		"EmailAddress": "cccddd@yahoo.com",
		"Reason": "BOUNCE",
		"LastUpdateTime": "2022-01-20T19:12:25.620000+01:00"
	}
]

Therefore, it seems that running the command with --query (for the purpose of sorting the output) returns no "NextToken" and, in fact, pruduces a different JSON structure in the returned payload. The consequence of this is to break the looping script that was expecting the "NextToken" key
.

[balonik]

I have the same issue with AWS CLI $ aws --version aws-cli/2.13.7 Python/3.11.4 Linux/6.4.4-100.fc37.x86_64 exe/x86_64.fedora.37 prompt/off where aws sesv2 list-suppressed-destinations --no-paginate returns only first 1000 entries with NextToken.

Additionally, in other commands (like ec2 describe-instances) it is not possible to use --no-paginate and --page-size together. It results in an error Cannot specify --no-paginate along with pagination arguments: --page-size. However, in sesv2 list-suppressed-destinations such error is not produced. This would indicate that the --no-paginate is not parsed/processed/taken at all.