Skip to content
The Chat Archive
chatgpt-4o-latest-20250326· OpenAI LMArena Human Preference 140K

How to read only some parts of large json with python

  1. User

    How to read only some parts of large json with python

  2. Assistant

    Reading only parts of a large JSON file in Python is often necessary to conserve memory and improve efficiency. Here are a few strategies depending on the structure of your JSON file:


    🧩 Case 1: JSON is an Array of Objects

    This is common for large JSON files exported from databases or logs.

    [
      {"id": 1, "name": "Alice"},
      {"id": 2, "name": "Bob"},
      ...
    ]
    

    ✅ Solution: Use ijson — an iterative JSON parser

    pip install ijson
    
    import ijson
    
    with open('large.json', 'rb') as f:
        # Parse items inside the top-level array
        objects = ijson.items(f, 'item')  # 'item' refers to each element in top-level array
        for obj in objects:
            print(obj['id'])  # Access only fields you care about
            if obj['id'] == 10:
                break
    

    🧩 Case 2: JSON is an Object with Nested Structures

    Example:

    {
      "users": [...],
      "metadata": {...},
      "settings": {...}
    }
    

    ✅ Solution: Extract only part using ijson

    import ijson
    
    with open('large.json', 'rb') as f:
        users = ijson.items(f, 'users.item')
        for user in users:
            print(user['name'])
            # Stop early if needed
            if user['name'] == 'Alice':
                break
    

    🧩 Case 3: Your data is newline-delimited JSON (JSONL / NDJSON)

    This format is like:

    {"id": 1, "name": "Alice"}
    {"id": 2, "name": "Bob"}
    ...
    

    ✅ Solution: Read line by line

    import json
    
    with open('large.json', 'r') as f:
        for line in f:
            obj = json.loads(line)
            print(obj['name'])
            # Optional break condition
            if obj['id'] > 100:
                break
    

    🧠 Bonus Tips:

    • Use generators to lazily process data.
    • Avoid loading entire JSON with json.load() if it's huge.
    • Use Pandas for line-delimited JSON (pd.read_json(..., lines=True)).

    🛠️ When to Use ijson vs Built-in json?

    ScenarioUse
    Full in-memory loadingjson.load()
    Partial reading (big file)ijson
    Newline-delimited JSONjson.loads() line-by-line

    Let me know the structure of your large JSON if you want a tailored example.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.