How to optimize the speed of processing large JSON files in Python without using too

[sophia2005]

Hello everyone, I am processing very large JSON files (GB) in Python, but my RAM is limited, so I often run out of memory. Can everyone suggest a way to read/write and process large JSON in the most RAM-saving way? Should I use any library other than the default JSON? Thanks, everyone, in advance! Monkey Mart
[link removed]

[Pedroski55]

A json file is just a text file.

What are you doing to have GB size text files??

Have a look at this link for some tips.

[paul18fr]

never tried this for a while, but maybe can you use pandas (to be check):

df = pd.read_json(jsonFile)

[snippsat]

Pandas as mention is ok,it has chunksize build in.

import pandas as pd

reader = pd.read_json("big.json", lines=True, chunksize=100_000)
for chunk in reader:
    filtered = chunk[chunk['value'] > 100]
    # further processing

Using Polars would be better.

import polars as pl

# Lazy scan of newline-delimited JSON
# builds(stream) a query plan without immediately reading all rows
df = pl.scan_json("big.json")  

# Example: filter and select
result = (
    df.filter(pl.col("value") > 100)
      .select(["id", "value", "timestamp"])
      .with_column(pl.col("value") * 2)
)

# Collect into memory only the reduced result
df_small = result.collect()