Skip to content

ValueError: Couldn't find cache for livecodebench/code_generation_lite for config 'default-version_tag=release_latest' Available configs in the cache: ['release_latest-version_tag=release_v5'] #131

Description

@fkggggg

I followed the README to set up the environment using uv and successfully downloaded the dataset via Hugging Face:

lcb_codegen = load_dataset("livecodebench/code_generation_lite", version_tag="release_v5")

However, when running the evaluation command:

python -m lcb_runner.runner.main --local_model_path {model_name} --scenario codegeneration --evaluate --release_version release_v5

I encountered the following error:

ValueError: Couldn't find cache for livecodebench/code_generation_lite for config 'default-version_tag=release_latest'
Available configs in the cache: ['release_latest-version_tag=release_v5']

I noticed this is caused by the load_dataset call in code_generation.py:
https://github.com/LiveCodeBench/LiveCodeBench/blob/main/lcb_runner/benchmarks/code_generation.py

As suggested in #108, I already removed trust_remote_code=True and updated the call to:

load_dataset("livecodebench/code_generation_lite", release_version, split="test")

or equivalently:

load_dataset("livecodebench/code_generation_lite", split="test", version_tag=release_version)

Despite this change, the ValueError persists. It seems the code is still trying to look for a config named default-version_tag=release_latest, which doesn't match the cached config release_latest-version_tag=release_v5.

Any guidance would be appreciated. Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions