How to read only some parts of large json with python
- User
How to read only some parts of large json with python
Assistant
Okay, reading only parts of a large JSON file efficiently in Python is a common problem. The standard
json.load()orjson.loads()reads the entire file into memory first, which is impractical for large files (gigabytes or more).Here are the best approaches, depending on the JSON structure and what you need to extract:
1. If the JSON is formatted as JSON Lines / ndjson:
-
Structure: The file contains multiple independent JSON objects, one per line.
{"id": 1, "data": "value1"} {"id": 2, "data": "value2"} {"id": 3, "data": "value3"} -
Approach: This is the most memory-efficient scenario. You can read the file line by line and parse each line individually.
-
Code:
import json import sys # For handling potential errors on specific lines filename = 'large_data.jsonl' # Or .ndjson target_ids = {10, 500, 10005} # Example: Only process records with these IDs results = [] try: with open(filename, 'r', encoding='utf-8') as f: for line_num, line in enumerate(f): try: # Remove leading/trailing whitespace (like newline char) line = line.strip() if not line: # Skip empty lines continue record = json.loads(line) # Parse just one line # --- Your logic to select/process the record --- if 'id' in record and record['id'] in target_ids: # Process or store the relevant record print(f"Found relevant record on line {line_num + 1}: {record}") results.append(record['data']) # Example: extract 'data' field # --------------------------------------------- except json.JSONDecodeError as e: print(f"Error decoding JSON on line {line_num + 1}: {e}", file=sys.stderr) # Decide how to handle errors: skip line, stop processing, etc. continue # Skip this line except Exception as e: print(f"Error processing line {line_num + 1}: {e}", file=sys.stderr) continue # Skip this line except FileNotFoundError: print(f"Error: File '{filename}' not found.", file=sys.stderr) except Exception as e: print(f"An unexpected error occurred: {e}", file=sys.stderr) print("\nFinished processing.") print(f"Extracted data for target IDs: {results}") -
Pros: Extremely memory efficient, simple Python code.
-
Cons: Only works if the file is in JSON Lines format.
2. Using an Iterative/Streaming JSON Parser (like
ijson)-
Structure: The file contains a single, potentially very large and nested, JSON object or array.
{ "metadata": { ... }, "items": [ {"id": 1, "value": "..."}, // Potentially millions of these {"id": 2, "value": "..."}, ... ], "summary": { ... } } -
Approach: Use a library like
ijsonthat parses the JSON incrementally, yielding data as it encounters it, without loading the whole structure. You specify a prefix path to the data you're interested in. -
Installation:
pip install ijson -
Code:
import ijson import sys filename = 'large_single_object.json' # Example: We want to extract items from the 'items' array inside the root object # The prefix 'items.item' tells ijson to yield each object inside the 'items' array. # 'item' is a special keyword representing array elements. prefix_to_extract = 'items.item' # Example: If you wanted objects where key='user' inside 'items' array items: # prefix_to_extract = 'items.item.user' # Example: If you wanted all values associated with a key 'target_value' anywhere: # You might need more complex event handling (see ijson docs for lower-level parsing) # or iterate through known prefixes like: # prefix_to_extract = 'results.item.data.target_value' # More specific path results = [] count = 0 limit = 10 # Example: Stop after finding 10 matching items try: with open(filename, 'rb') as f: # Open in binary mode for ijson # Use ijson.items to yield Python objects matching the prefix parser = ijson.items(f, prefix_to_extract) print(f"Streaming items with prefix '{prefix_to_extract}'...") for item in parser: # --- Your logic to select/process the item --- # Example: Select items with an even ID if isinstance(item, dict) and 'id' in item and item['id'] % 2 == 0: print(f"Processing item: {item}") results.append(item) count += 1 if count >= limit: print(f"Reached limit of {limit}, stopping.") break # --------------------------------------------- except FileNotFoundError: print(f"Error: File '{filename}' not found.", file=sys.stderr) except ijson.JSONError as e: print(f"Error parsing JSON: {e}", file=sys.stderr) except Exception as e: print(f"An unexpected error occurred: {e}", file=sys.stderr) print("\nFinished streaming.") print(f"Collected {len(results)} results.") # print(results) # Might still be large! -
Pros: Memory efficient (doesn't load the whole file), handles arbitrarily large single JSON objects/arrays, flexible path specification.
-
Cons: Can be slower than
json.loadif you end up needing most of the file anyway, requires understandingijson's prefix syntax, might still need to read/scan through large parts of the file on disk to find the desired data. Requires an external library.
3. Using External Tools (like
jq) viasubprocess-
Approach: Use a powerful command-line JSON processor like
jqto filter/extract the desired data before it even reaches Python. Pipe the output ofjqinto your Python script. -
Prerequisites:
jqmust be installed on your system. -
Code:
import subprocess import json import sys filename = 'large_single_object.json' # jq query to extract only objects from the 'items' array where the 'id' is greater than 100 # '.' refers to the root, '.items[]' iterates the array, 'select(.id > 100)' filters jq_query = '.items[] | select(.id > 100)' # Or to extract just the 'value' field from those filtered items: # jq_query = '.items[] | select(.id > 100) | .value' results = [] try: # Start jq process process = subprocess.Popen( ['jq', jq_query, filename], stdout=subprocess.PIPE, # Capture standard output stderr=subprocess.PIPE, # Capture standard error text=True, # Work with text streams (auto-decodes stdout/stderr) encoding='utf-8' ) # Read jq's output line by line (jq often outputs JSON Lines) for line in process.stdout: try: item = json.loads(line) # Process the item extracted by jq print(f"Received from jq: {item}") results.append(item) except json.JSONDecodeError: # If jq outputs non-JSON strings (e.g., extracting primitive values) # Handle `line` directly as a string print(f"Received non-JSON from jq: {line.strip()}") results.append(line.strip()) # Store the raw string output # Wait for jq to finish and check for errors stdout, stderr = process.communicate() # stdout will be empty now as we read it line-by-line if process.returncode != 0: print(f"jq command failed with error code {process.returncode}:", file=sys.stderr) print(stderr, file=sys.stderr) else: print("\nFinished processing jq output.") print(f"Collected {len(results)} results.") except FileNotFoundError: # This could be because 'jq' isn't installed or the JSON file is missing print("Error: 'jq' command not found or input file missing.", file=sys.stderr) print("Make sure 'jq' is installed and in your PATH.", file=sys.stderr) except Exception as e: print(f"An unexpected error occurred: {e}", file=sys.stderr) # print(results) -
Pros: Leverages the power and speed of
jqfor complex filtering/transformation, can be very fast and memory efficient (jq handles the heavy lifting), keeps Python code potentially simpler if the extraction logic is complex. -
Cons: Requires
jqto be installed, adds an external dependency, overhead of starting a separate process, debuggingjqqueries can be separate from Python debugging.
Which method to choose?
- Is your file JSON Lines (
.jsonl,.ndjson)? Use Method 1 (Line-by-line reading). It's the simplest and most efficient for this format. - Is your file one large JSON object/array? Use Method 2 (
ijson). It's designed for this exact scenario and keeps everything within Python. - Do you need complex filtering/transformation better suited for
jq, or are you already using command-line tools in a pipeline? Consider Method 3 (jq+subprocess).
Avoid trying to read the entire file with
json.loador manually parsing with regular expressions (which is brittle and error-prone for complex JSON).-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.