Skip to content
This repository was archived by the owner on Mar 23, 2026. It is now read-only.

Logs: harden dict view iterations#13272

Merged
nik-localstack merged 1 commit into
mainfrom
logs/improve-dictionary-handling
Oct 15, 2025
Merged

Logs: harden dict view iterations#13272
nik-localstack merged 1 commit into
mainfrom
logs/improve-dictionary-handling

Conversation

@nik-localstack

@nik-localstack nik-localstack commented Oct 15, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Motivation

The following issue came up when looking in telemetry for issues with the new Batch service. Batch provider calls the DescribeLogGroups API when creating a new task.

exception while calling logs.DescribeLogGroups: dictionary changed size during iterationTraceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/rolo/gateway/chain.py", line 166, in handle
    handler(self, self.context, response)
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/handlers/service.py", line 116, in __call__
    handler(chain, context, response)
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/handlers/service.py", line 86, in __call__
    skeleton_response = self.skeleton.invoke(context)
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/skeleton.py", line 155, in invoke
    return self.dispatch_request(serializer, context, instance)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/skeleton.py", line 169, in dispatch_request
    result = handler(context, instance) or {}
             ~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/forwarder.py", line 137, in _call
    return handler(context, req)
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/skeleton.py", line 117, in __call__
    return self.fn(*args, **kwargs)
           ~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/api/core.py", line 181, in operation_marker
    return fn(*args, **kwargs)
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/services/logs/provider.py", line 108, in describe_log_groups
    copy_groups = copy.deepcopy(region_backend.groups)
  File "/usr/local/lib/python3.13/copy.py", line 137, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.13/copy.py", line 222, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/copy.py", line 163, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.13/copy.py", line 260, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.13/copy.py", line 137, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.13/copy.py", line 222, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/copy.py", line 137, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.13/copy.py", line 221, in _deepcopy_dict
    for key, value in x.items():
                      ~~~~~~~^^
localstack.aws.api.core.CommonServiceException: exception while calling logs.DescribeLogGroups: dictionary changed size during iteration

The error occurs because the dictionary region_backend.groups is being modified by another thread while copy.deepcopy() is trying to iterate over it. This is a classic race condition where concurrent modifications to the dictionary cause the "dictionary changed size during iteration" error.

Changes

  • Fixes the race condition, by creating a shallow copy first to avoid the race condition, then do the deep copy:

@nik-localstack nik-localstack self-assigned this Oct 15, 2025
@nik-localstack nik-localstack added semver: patch Non-breaking changes which can be included in patch releases docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes labels Oct 15, 2025
@github-actions

github-actions Bot commented Oct 15, 2025

Copy link
Copy Markdown

Test Results - Preflight, Unit

22 337 tests  ±0   20 587 ✅ ±0   15m 20s ⏱️ -31s
     1 suites ±0    1 750 💤 ±0 
     1 files   ±0        0 ❌ ±0 

Results for commit f49043e. ± Comparison against base commit 395f363.

@github-actions

github-actions Bot commented Oct 15, 2025

Copy link
Copy Markdown

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 20s ⏱️ -4s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit f49043e. ± Comparison against base commit 395f363.

@github-actions

github-actions Bot commented Oct 15, 2025

Copy link
Copy Markdown

LocalStack Community integration with Pro

    2 files  ±    0      2 suites  ±0   53m 43s ⏱️ - 1h 7m 10s
2 016 tests  - 2 801  1 923 ✅  - 2 557  93 💤  - 244  0 ❌ ±0 
2 018 runs   - 2 801  1 923 ✅  - 2 557  95 💤  - 244  0 ❌ ±0 

Results for commit f49043e. ± Comparison against base commit 395f363.

This pull request removes 2801 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

@github-actions

github-actions Bot commented Oct 15, 2025

Copy link
Copy Markdown

Test Results (amd64) - Integration, Bootstrap

    5 files  ±    0      5 suites  ±0   1h 17m 4s ⏱️ - 1h 24m 15s
2 040 tests  - 3 151  1 949 ✅  - 2 745  91 💤  - 406  0 ❌ ±0 
2 046 runs   - 3 151  1 949 ✅  - 2 745  97 💤  - 406  0 ❌ ±0 

Results for commit f49043e. ± Comparison against base commit 395f363.

This pull request removes 3151 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

@nik-localstack nik-localstack marked this pull request as ready for review October 15, 2025 11:14
pinzon
pinzon approved these changes Oct 15, 2025
View reviewed changes

@pinzon pinzon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nik-localstack nik-localstack merged commit 814a597 into main Oct 15, 2025
56 checks passed
@nik-localstack nik-localstack deleted the logs/improve-dictionary-handling branch October 15, 2025 13:50
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@steffyP steffyP Awaiting requested review from steffyP

1 more reviewer

@pinzon pinzon pinzon approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@nik-localstack nik-localstack

Labels

docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes semver: patch Non-breaking changes which can be included in patch releases

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nik-localstack @pinzon