Skip to content

fix: skip eval invocations without user content to prevent ValidationError#6134

Open
tcconnally wants to merge 3 commits into
google:mainfrom
Perseus-Computing-LLC:fix/eval-user-content-validation-error
Open

fix: skip eval invocations without user content to prevent ValidationError#6134
tcconnally wants to merge 3 commits into
google:mainfrom
Perseus-Computing-LLC:fix/eval-user-content-validation-error

Conversation

@tcconnally

Copy link
Copy Markdown

Problem

When a session contains invocations without user-authored events (author != "user"), convert_events_to_eval_invocations() creates an Invocation with an empty Content(parts=[]). This causes a Pydantic ValidationError because Invocation.user_content is typed as genai_types.Content and empty Content triggers validation failure.

Error when calling POST /apps/{app_name}/eval-sets/{eval_set_id}/add-session:

ValidationError on Invocation.user_content

Fix

Skip invocations entirely when no user event is found. Evaluations without user input are not meaningful, so filtering them out is both a correctness fix and an efficiency improvement.

Before

for invocation_id, events in events_by_invocation_id.items():
    user_content = Content(parts=[])
    ...
    invocations.append(Invocation(user_content=user_content, ...))

After

for invocation_id, events in events_by_invocation_id.items():
    user_content = Content(parts=[])
    ...
    if not user_content.parts:  # No user event found
        continue
    invocations.append(Invocation(user_content=user_content, ...))

Closes #3760

@rohityan rohityan self-assigned this Jun 15, 2026
@haranrk haranrk assigned haranrk and unassigned rohityan Jun 16, 2026
@rohityan rohityan added eval [Component] This issue is related to evaluation request clarification [Status] The maintainer need clarification or more information from the author labels Jun 17, 2026
@rohityan

Copy link
Copy Markdown
Collaborator

Hi @tcconnally , Thank you for your contribution! We appreciate you taking the time to submit this pull request. Can you please fix the failing mypy-diff tests before we can proceed with the review.

…Error

When a session contains invocations without user-authored events,
convert_events_to_eval_invocations() created an Invocation with an
empty Content(parts=[]). This caused a Pydantic ValidationError because
Invocation.user_content is typed as genai_types.Content (non-optional)
and an empty Content with no parts triggered validation failure.

Fix: skip invocations entirely when no user event is found in the
event list, since evaluations without user input are not meaningful.

Closes google#3760
@tcconnally

Copy link
Copy Markdown
Author

Fixed the mypy issue — added user_content None guard alongside the empty parts check. Rebased on main.

@tcconnally tcconnally force-pushed the fix/eval-user-content-validation-error branch from 41a49b0 to 9352f62 Compare June 17, 2026 18:40
@rohityan rohityan added needs review [Status] The PR/issue is awaiting review from the maintainer and removed request clarification [Status] The maintainer need clarification or more information from the author labels Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval [Component] This issue is related to evaluation needs review [Status] The PR/issue is awaiting review from the maintainer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluation Invocation.user_content ValidationError when session has invocations without user events

3 participants