Skip to content

aws: fix NextToken invalidation in paginated queries#18830

Merged
efd6 merged 1 commit into
elastic:mainfrom
efd6:s7155-aws
May 7, 2026
Merged

aws: fix NextToken invalidation in paginated queries#18830
efd6 merged 1 commit into
elastic:mainfrom
efd6:s7155-aws

Conversation

@efd6

@efd6 efd6 commented May 5, 2026

Copy link
Copy Markdown
Contributor

Proposed commit message

aws: fix NextToken invalidation in paginated queries

Remove the upper time bound from Security Hub, GuardDuty, and
Inspector httpjson templates. The httpjson input re-evaluates
request.transforms on every pagination page, so (now) in the
filter body changes between pages. When the value crosses a time
boundary, AWS rejects the NextToken with "Input Query has changed".

The lower bound from the cursor remains stable during pagination
and provides sufficient windowing. A CEL rewrite can reintroduce
the upper bound safely since it can pin the value once per
collection interval.

Without an upper bound, findings created during pagination could
extend the result set. Since results are sorted ascending by update
time, new findings appear at the tail. The creation rate would need
to exceed the pagination rate (~60 findings/second at 100 per page)
to cause unbounded pagination. Security Hub, GuardDuty, and
Inspector finding rates are orders of magnitude below this.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@efd6 efd6 self-assigned this May 5, 2026
@efd6 efd6 added Integration:aws AWS bugfix Pull request that fixes a bug issue Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels May 5, 2026
Remove the upper time bound from Security Hub, GuardDuty, and
Inspector httpjson templates. The httpjson input re-evaluates
request.transforms on every pagination page, so (now) in the
filter body changes between pages. When the value crosses a time
boundary, AWS rejects the NextToken with "Input Query has changed".

The lower bound from the cursor remains stable during pagination
and provides sufficient windowing. A CEL rewrite can reintroduce
the upper bound safely since it can pin the value once per
collection interval.

Without an upper bound, findings created during pagination could
extend the result set. Since results are sorted ascending by update
time, new findings appear at the tail. The creation rate would need
to exceed the pagination rate (~60 findings/second at 100 per page)
to cause unbounded pagination. Security Hub, GuardDuty, and
Inspector finding rates are orders of magnitude below this.
Comment thread packages/aws/changelog.yml
@elasticmachine

Copy link
Copy Markdown

💚 Build Succeeded

cc @efd6

@efd6 efd6 marked this pull request as ready for review May 6, 2026 01:17
@efd6 efd6 requested review from a team as code owners May 6, 2026 01:17
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@muthu-mps muthu-mps left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code owner approval!

@efd6 efd6 merged commit cd8d67e into elastic:main May 7, 2026
10 checks passed
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown

Package aws - 6.14.2 containing this change is available at https://epr.elastic.co/package/aws/6.14.2/

herrBez pushed a commit to herrBez/integrations that referenced this pull request Jun 1, 2026
Remove the upper time bound from Security Hub, GuardDuty, and
Inspector httpjson templates. The httpjson input re-evaluates
request.transforms on every pagination page, so (now) in the
filter body changes between pages. When the value crosses a time
boundary, AWS rejects the NextToken with "Input Query has changed".

The lower bound from the cursor remains stable during pagination
and provides sufficient windowing. A CEL rewrite can reintroduce
the upper bound safely since it can pin the value once per
collection interval.

Without an upper bound, findings created during pagination could
extend the result set. Since results are sorted ascending by update
time, new findings appear at the tail. The creation rate would need
to exceed the pagination rate (~60 findings/second at 100 per page)
to cause unbounded pagination. Security Hub, GuardDuty, and
Inspector finding rates are orders of magnitude below this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes a bug issue Integration:aws AWS Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants