Summary

When a GCS bucket directory contains a very large number of files, Cloud Storage
FUSE currently has no mechanism to warn operators or degrade gracefully. In
production we observe:

• ~50,000 files in a single directory → significant read/list performance
  degradation
• 100,000+ files in a single directory → very poor performance that can
  lead to filesystem state corruption, causing extended downtime and potential
  file loss

There is also no way to prevent new files from being written into an already
over-populated directory, so the problem compounds silently until it causes an
outage.

Current Behavior

gcsfuse performs a full ListObjects (paginated) on every readdir() call
(when kernel-list-cache-ttl-secs=0) or on first access within a TTL window.
With 100k objects under one prefix this means:

• 10+ sequential paginated GCS API calls per listing
• Unbounded memory growth in the metadata/stat cache
• No warning in logs that a directory is approaching a problematic size
• No way to prevent further writes into an already over-populated directory
• No documented hard limit — operators discover the problem only after
  degradation or corruption occurs

Requested Features

1. Write guard — block new file creation above a threshold

When a directory's entry count (known from the stat/list cache, or from a
lightweight ListObjects prefix count) meets or exceeds a configurable limit,
reject O_CREAT operations on that directory with ENOSPC and emit a
structured WARNING log.

file-system:
  max-dir-entries: 50000  # block new creates and warn when parent dir hits this

Returning ENOSPC gives applications and scripts a clear, standard error
signal (No space left on device) rather than silently allowing the directory
to grow into the corruption zone.

2. Warning threshold log (pre-limit signal)

Emit a structured WARNING log when a ListObjects response for a single
directory prefix exceeds a lower configurable threshold, giving operators
time to act before the hard limit is reached.

file-system:
  large-dir-warning-threshold: 10000  # log WARNING when readdir exceeds this

3. Streaming / incremental readdir

Instead of buffering the entire ListObjects result before returning to the
kernel, stream entries back incrementally. This caps peak memory usage and
allows the kernel to begin iterating entries before the full GCS list
completes — directly addressing latency and OOM risk at extreme scale.

4. Documentation of limits

Publish a supported maximum directory size (entries per prefix) in the
official documentation with explicit guidance on mitigation strategies
(subdirectory sharding, HNS buckets, etc.).

Why the write guard matters

Read-path mitigations (kernel list cache TTL, larger stat cache) reduce the
frequency of expensive listing but do not stop directories from growing further.
Without a write guard, every new file written into an already over-populated
directory makes the next readdir more expensive and moves the mount closer to
the corruption threshold. A write guard closes this feedback loop at the source.

Impact

This affects any multi-tenant or user-generated-content workload where
individual users can accumulate files in a flat directory structure without
architectural guardrails. The lack of early warning means operators only
discover the limit under production load, at which point recovery requires
unmounting and remounting — causing service interruption.

Workaround

Currently we rely on a combination of:

• kernel-list-cache-ttl-secs: 30 to reduce frequency of full listings
• stat-cache-max-size-mb: 128 to size the cache for 100k entries

These mitigate the performance impact but do not prevent corruption at
extreme file counts, and require operators to know the limits in advance.

Environment

• Cloud Storage FUSE version: 2.x (config-file mode)
• Mount mode: --config-file with implicit-dirs: true
• GCS bucket type: standard (non-HNS)