Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro-preview-03-25· GoogleHugging Face LMArena Human Preference 140K

How to read only some parts of large json with python

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with chatgpt-4o-latest-20250326’s answer.
  1. User

    How to read only some parts of large json with python

  2. Gemini Assistant

    Okay, reading only parts of a large JSON file efficiently in Python is a common problem. The standard json.load() or json.loads() reads the entire file into memory first, which is impractical for large files (gigabytes or more).

    Here are the best approaches, depending on the JSON structure and what you need to extract:

    1. If the JSON is formatted as JSON Lines / ndjson:

    • Structure: The file contains multiple independent JSON objects, one per line.

      {"id": 1, "data": "value1"}
      {"id": 2, "data": "value2"}
      {"id": 3, "data": "value3"}
      
    • Approach: This is the most memory-efficient scenario. You can read the file line by line and parse each line individually.

    • Code:

      import json
      import sys # For handling potential errors on specific lines
      
      filename = 'large_data.jsonl' # Or .ndjson
      target_ids = {10, 500, 10005} # Example: Only process records with these IDs
      results = []
      
      try:
          with open(filename, 'r', encoding='utf-8') as f:
              for line_num, line in enumerate(f):
                  try:
                      # Remove leading/trailing whitespace (like newline char)
                      line = line.strip()
                      if not line: # Skip empty lines
                          continue
                      
                      record = json.loads(line) # Parse just one line
      
                      # --- Your logic to select/process the record ---
                      if 'id' in record and record['id'] in target_ids:
                          # Process or store the relevant record
                          print(f"Found relevant record on line {line_num + 1}: {record}")
                          results.append(record['data']) # Example: extract 'data' field
                      # ---------------------------------------------
      
                  except json.JSONDecodeError as e:
                      print(f"Error decoding JSON on line {line_num + 1}: {e}", file=sys.stderr)
                      # Decide how to handle errors: skip line, stop processing, etc.
                      continue # Skip this line
                  except Exception as e:
                      print(f"Error processing line {line_num + 1}: {e}", file=sys.stderr)
                      continue # Skip this line
      
      except FileNotFoundError:
          print(f"Error: File '{filename}' not found.", file=sys.stderr)
      except Exception as e:
          print(f"An unexpected error occurred: {e}", file=sys.stderr)
      
      print("\nFinished processing.")
      print(f"Extracted data for target IDs: {results}")
      
    • Pros: Extremely memory efficient, simple Python code.

    • Cons: Only works if the file is in JSON Lines format.

    2. Using an Iterative/Streaming JSON Parser (like ijson)

    • Structure: The file contains a single, potentially very large and nested, JSON object or array.

      {
        "metadata": { ... },
        "items": [
          {"id": 1, "value": "..."}, // Potentially millions of these
          {"id": 2, "value": "..."},
          ...
        ],
        "summary": { ... }
      }
      
    • Approach: Use a library like ijson that parses the JSON incrementally, yielding data as it encounters it, without loading the whole structure. You specify a prefix path to the data you're interested in.

    • Installation: pip install ijson

    • Code:

      import ijson
      import sys
      
      filename = 'large_single_object.json'
      # Example: We want to extract items from the 'items' array inside the root object
      # The prefix 'items.item' tells ijson to yield each object inside the 'items' array.
      # 'item' is a special keyword representing array elements.
      prefix_to_extract = 'items.item' 
      
      # Example: If you wanted objects where key='user' inside 'items' array items:
      # prefix_to_extract = 'items.item.user' 
      
      # Example: If you wanted all values associated with a key 'target_value' anywhere:
      # You might need more complex event handling (see ijson docs for lower-level parsing)
      # or iterate through known prefixes like:
      # prefix_to_extract = 'results.item.data.target_value' # More specific path
      
      results = []
      count = 0
      limit = 10 # Example: Stop after finding 10 matching items
      
      try:
          with open(filename, 'rb') as f: # Open in binary mode for ijson
              # Use ijson.items to yield Python objects matching the prefix
              parser = ijson.items(f, prefix_to_extract)
              
              print(f"Streaming items with prefix '{prefix_to_extract}'...")
              for item in parser:
                  # --- Your logic to select/process the item ---
                  # Example: Select items with an even ID
                  if isinstance(item, dict) and 'id' in item and item['id'] % 2 == 0:
                      print(f"Processing item: {item}")
                      results.append(item)
                      count += 1
                      if count >= limit:
                          print(f"Reached limit of {limit}, stopping.")
                          break 
                  # ---------------------------------------------
      
      except FileNotFoundError:
          print(f"Error: File '{filename}' not found.", file=sys.stderr)
      except ijson.JSONError as e:
           print(f"Error parsing JSON: {e}", file=sys.stderr)
      except Exception as e:
          print(f"An unexpected error occurred: {e}", file=sys.stderr)
          
      print("\nFinished streaming.")
      print(f"Collected {len(results)} results.")
      # print(results) # Might still be large!
      
    • Pros: Memory efficient (doesn't load the whole file), handles arbitrarily large single JSON objects/arrays, flexible path specification.

    • Cons: Can be slower than json.load if you end up needing most of the file anyway, requires understanding ijson's prefix syntax, might still need to read/scan through large parts of the file on disk to find the desired data. Requires an external library.

    3. Using External Tools (like jq) via subprocess

    • Approach: Use a powerful command-line JSON processor like jq to filter/extract the desired data before it even reaches Python. Pipe the output of jq into your Python script.

    • Prerequisites: jq must be installed on your system.

    • Code:

      import subprocess
      import json
      import sys
      
      filename = 'large_single_object.json'
      # jq query to extract only objects from the 'items' array where the 'id' is greater than 100
      # '.' refers to the root, '.items[]' iterates the array, 'select(.id > 100)' filters
      jq_query = '.items[] | select(.id > 100)' 
      
      # Or to extract just the 'value' field from those filtered items:
      # jq_query = '.items[] | select(.id > 100) | .value' 
      
      results = []
      
      try:
          # Start jq process
          process = subprocess.Popen(
              ['jq', jq_query, filename],
              stdout=subprocess.PIPE,  # Capture standard output
              stderr=subprocess.PIPE,  # Capture standard error
              text=True,               # Work with text streams (auto-decodes stdout/stderr)
              encoding='utf-8'
          )
      
          # Read jq's output line by line (jq often outputs JSON Lines)
          for line in process.stdout:
              try:
                  item = json.loads(line)
                  # Process the item extracted by jq
                  print(f"Received from jq: {item}")
                  results.append(item)
              except json.JSONDecodeError:
                  # If jq outputs non-JSON strings (e.g., extracting primitive values)
                  # Handle `line` directly as a string
                  print(f"Received non-JSON from jq: {line.strip()}")
                  results.append(line.strip()) # Store the raw string output
      
          # Wait for jq to finish and check for errors
          stdout, stderr = process.communicate() # stdout will be empty now as we read it line-by-line
          
          if process.returncode != 0:
              print(f"jq command failed with error code {process.returncode}:", file=sys.stderr)
              print(stderr, file=sys.stderr)
          else:
               print("\nFinished processing jq output.")
               print(f"Collected {len(results)} results.")
      
      except FileNotFoundError:
          # This could be because 'jq' isn't installed or the JSON file is missing
          print("Error: 'jq' command not found or input file missing.", file=sys.stderr)
          print("Make sure 'jq' is installed and in your PATH.", file=sys.stderr)
      except Exception as e:
          print(f"An unexpected error occurred: {e}", file=sys.stderr)
      
      # print(results)
      
    • Pros: Leverages the power and speed of jq for complex filtering/transformation, can be very fast and memory efficient (jq handles the heavy lifting), keeps Python code potentially simpler if the extraction logic is complex.

    • Cons: Requires jq to be installed, adds an external dependency, overhead of starting a separate process, debugging jq queries can be separate from Python debugging.

    Which method to choose?

    1. Is your file JSON Lines (.jsonl, .ndjson)? Use Method 1 (Line-by-line reading). It's the simplest and most efficient for this format.
    2. Is your file one large JSON object/array? Use Method 2 (ijson). It's designed for this exact scenario and keeps everything within Python.
    3. Do you need complex filtering/transformation better suited for jq, or are you already using command-line tools in a pipeline? Consider Method 3 (jq + subprocess).

    Avoid trying to read the entire file with json.load or manually parsing with regular expressions (which is brittle and error-prone for complex JSON).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.