aws: fix NextToken invalidation in paginated queries#18830
Merged
Conversation
Remove the upper time bound from Security Hub, GuardDuty, and Inspector httpjson templates. The httpjson input re-evaluates request.transforms on every pagination page, so (now) in the filter body changes between pages. When the value crosses a time boundary, AWS rejects the NextToken with "Input Query has changed". The lower bound from the cursor remains stable during pagination and provides sufficient windowing. A CEL rewrite can reintroduce the upper bound safely since it can pin the value once per collection interval. Without an upper bound, findings created during pagination could extend the result set. Since results are sorted ascending by update time, new findings appear at the tail. The creation rate would need to exceed the pagination rate (~60 findings/second at 100 per page) to cause unbounded pagination. Security Hub, GuardDuty, and Inspector finding rates are orders of magnitude below this.
💚 Build Succeeded
cc @efd6 |
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
zmoog
approved these changes
May 6, 2026
kcreddy
approved these changes
May 7, 2026
|
Package aws - 6.14.2 containing this change is available at https://epr.elastic.co/package/aws/6.14.2/ |
5 tasks
herrBez
pushed a commit
to herrBez/integrations
that referenced
this pull request
Jun 1, 2026
Remove the upper time bound from Security Hub, GuardDuty, and Inspector httpjson templates. The httpjson input re-evaluates request.transforms on every pagination page, so (now) in the filter body changes between pages. When the value crosses a time boundary, AWS rejects the NextToken with "Input Query has changed". The lower bound from the cursor remains stable during pagination and provides sufficient windowing. A CEL rewrite can reintroduce the upper bound safely since it can pin the value once per collection interval. Without an upper bound, findings created during pagination could extend the result set. Since results are sorted ascending by update time, new findings appear at the tail. The creation rate would need to exceed the pagination rate (~60 findings/second at 100 per page) to cause unbounded pagination. Security Hub, GuardDuty, and Inspector finding rates are orders of magnitude below this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots