Skip to content

[Bug]: _background_ingestion_task stuck in crashing loop #990

@scottly06

Description

@scottly06

Describe the bug

CC @o-love @jgong

_background_ingestion_task stuck in this crash loop backtrace:

2026-01-28 00:28:50,146 [ERROR] memmachine.semantic_memory.semantic_memory - background task crashed, restarting
  + Exception Group Traceback (most recent call last):
  |   File "/app/.venv/lib/python3.12/site-packages/memmachine/semantic_memory/semantic_memory.py", line 282, in _background_ingestion_task
  |     await ingestion_service.process_set_ids(dirty_sets)
  |   File "/app/.venv/lib/python3.12/site-packages/memmachine/semantic_memory/semantic_ingestion.py", line 68, in process_set_ids
  |     raise ExceptionGroup("Failed to process set ids", errors)
  | ExceptionGroup: Failed to process set ids (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/app/.venv/lib/python3.12/site-packages/memmachine/semantic_memory/semantic_ingestion.py", line 98, in _process_single_set
    |     raise ValueError("Failed to retrieve messages. Invalid history_ids")
    | ValueError: Failed to retrieve messages. Invalid history_ids
    +------------------------------------
2026-01-28 00:28:54,868 [INFO] mmp_api_server.main - client=10.0.47.75 method=GET path="/v2/health" status=200 duration_ms=0.41 size=43
INFO:     10.0.47.75:43218 - "GET /v2/health HTTP/1.1" 200 OK
2026-01-28 00:29:04,869 [INFO] mmp_api_server.main - client=10.0.47.75 method=GET path="/v2/health" status=200 duration_ms=0.61 size=43
INFO:     10.0.47.75:60016 - "GET /v2/health HTTP/1.1" 200 OK
2026-01-28 00:29:04,890 [INFO] mmp_api_server.main - client=10.0.35.238 method=GET path="/v2/metrics" status=200 duration_ms=1.08 size=11443
INFO:     10.0.35.238:53940 - "GET /v2/metrics HTTP/1.1" 200 OK
2026-01-28 00:29:10,092 [INFO] memmachine.semantic_memory.semantic_ingestion - Starting ingestion processing for set ids: ['mem_session_00fc6cd7-cb59-4fbc-b810-9866ecf5b570/00fc6cd7-cb59-4fbc-b810-9866ecf5b570']
2026-01-28 00:29:10,103 [ERROR] memmachine.semantic_memory.semantic_memory - background task crashed, restarting
  + Exception Group Traceback (most recent call last):
  |   File "/app/.venv/lib/python3.12/site-packages/memmachine/semantic_memory/semantic_memory.py", line 282, in _background_ingestion_task
  |     await ingestion_service.process_set_ids(dirty_sets)
  |   File "/app/.venv/lib/python3.12/site-packages/memmachine/semantic_memory/semantic_ingestion.py", line 68, in process_set_ids
  |     raise ExceptionGroup("Failed to process set ids", errors)
  | ExceptionGroup: Failed to process set ids (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/app/.venv/lib/python3.12/site-packages/memmachine/semantic_memory/semantic_ingestion.py", line 98, in _process_single_set
    |     raise ValueError("Failed to retrieve messages. Invalid history_ids")
    | ValueError: Failed to retrieve messages. Invalid history_ids
    +------------------------------------
2026-01-28 00:29:14,868 [INFO] mmp_api_server.main - client=10.0.47.75 method=GET path="/v2/health" status=200 duration_ms=0.96 size=43
INFO:     10.0.47.75:37384 - "GET /v2/health HTTP/1.1" 200 OK
2026-01-28 00:29:24,868 [INFO] mmp_api_server.main - client=10.0.47.75 method=GET path="/v2/health" status=200 duration_ms=0.97 size=43
INFO:     10.0.47.75:47528 - "GET /v2/health HTTP/1.1" 200 OK
2026-01-28 00:29:29,693 [INFO] memmachine.semantic_memory.semantic_ingestion - Starting ingestion processing for set ids: ['mem_session_00fc6cd7-cb59-4fbc-b810-9866ecf5b570/00fc6cd7-cb59-4fbc-b810-9866ecf5b570']
2026-01-28 00:29:29,702 [ERROR] memmachine.semantic_memory.semantic_memory - background task crashed, restarting
  + Exception Group Traceback (most recent call last):
  |   File "/app/.venv/lib/python3.12/site-packages/memmachine/semantic_memory/semantic_memory.py", line 282, in _background_ingestion_task
  |     await ingestion_service.process_set_ids(dirty_sets)
  |   File "/app/.venv/lib/python3.12/site-packages/memmachine/semantic_memory/semantic_ingestion.py", line 68, in process_set_ids
  |     raise ExceptionGroup("Failed to process set ids", errors)
  | ExceptionGroup: Failed to process set ids (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/app/.venv/lib/python3.12/site-packages/memmachine/semantic_memory/semantic_ingestion.py", line 98, in _process_single_set
    |     raise ValueError("Failed to retrieve messages. Invalid history_ids")
    | ValueError: Failed to retrieve messages. Invalid history_ids
    +------------------------------------
2026-01-28 00:29:34,868 [INFO] mmp_api_server.main - client=10.0.47.75 method=GET path="/v2/health" status=200 duration_ms=0.43 size=43
INFO:     10.0.47.75:59088 - "GET /v2/health HTTP/1.1" 200 OK
2026-01-28 00:29:34,890 [INFO] mmp_api_server.main - client=10.0.35.238 method=GET path="/v2/metrics" status=200 duration_ms=1.10 size=11430
INFO:     10.0.35.238:60948 - "GET /v2/metrics HTTP/1.1" 200 OK
2026-01-28 00:29:44,868 [INFO] mmp_api_server.main - client=10.0.47.75 method=GET path="/v2/health" status=200 duration_ms=0.39 size=43
INFO:     10.0.47.75:32822 - "GET /v2/health HTTP/1.1" 200 OK
2026-01-28 00:29:50,893 [INFO] memmachine.semantic_memory.semantic_ingestion - Starting ingestion processing for set ids: ['mem_session_00fc6cd7-cb59-4fbc-b810-9866ecf5b570/00fc6cd7-cb59-4fbc-b810-9866ecf5b570']
2026-01-28 00:29:50,902 [ERROR] memmachine.semantic_memory.semantic_memory - background task crashed, restarting
  + Exception Group Traceback (most recent call last):
  |   File "/app/.venv/lib/python3.12/site-packages/memmachine/semantic_memory/semantic_memory.py", line 282, in _background_ingestion_task
  |     await ingestion_service.process_set_ids(dirty_sets)
  |   File "/app/.venv/lib/python3.12/site-packages/memmachine/semantic_memory/semantic_ingestion.py", line 68, in process_set_ids
  |     raise ExceptionGroup("Failed to process set ids", errors)
  | ExceptionGroup: Failed to process set ids (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/app/.venv/lib/python3.12/site-packages/memmachine/semantic_memory/semantic_ingestion.py", line 98, in _process_single_set
    |     raise ValueError("Failed to retrieve messages. Invalid history_ids")
    | ValueError: Failed to retrieve messages. Invalid history_ids
    +------------------------------------
    +------------------------------------

Steps to reproduce

Run Platform Integration Regression Pipeline.

Expected behavior

The semantic memory added successfully.

Environment

AWS

Additional context

No response

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions