chatgpt-4o-latest-20250326· OpenAI LMArena Human Preference 140K
How to read only some parts of large json with python
- User
How to read only some parts of large json with python
- Assistant
Reading only parts of a large JSON file in Python is often necessary to conserve memory and improve efficiency. Here are a few strategies depending on the structure of your JSON file:
🧩 Case 1: JSON is an Array of Objects
This is common for large JSON files exported from databases or logs.
[ {"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}, ... ]✅ Solution: Use
ijson— an iterative JSON parserpip install ijsonimport ijson with open('large.json', 'rb') as f: # Parse items inside the top-level array objects = ijson.items(f, 'item') # 'item' refers to each element in top-level array for obj in objects: print(obj['id']) # Access only fields you care about if obj['id'] == 10: break
🧩 Case 2: JSON is an Object with Nested Structures
Example:
{ "users": [...], "metadata": {...}, "settings": {...} }✅ Solution: Extract only part using
ijsonimport ijson with open('large.json', 'rb') as f: users = ijson.items(f, 'users.item') for user in users: print(user['name']) # Stop early if needed if user['name'] == 'Alice': break
🧩 Case 3: Your data is newline-delimited JSON (JSONL / NDJSON)
This format is like:
{"id": 1, "name": "Alice"} {"id": 2, "name": "Bob"} ...✅ Solution: Read line by line
import json with open('large.json', 'r') as f: for line in f: obj = json.loads(line) print(obj['name']) # Optional break condition if obj['id'] > 100: break
🧠 Bonus Tips:
- Use generators to lazily process data.
- Avoid loading entire JSON with
json.load()if it's huge. - Use Pandas for line-delimited JSON (
pd.read_json(..., lines=True)).
🛠️ When to Use
ijsonvs Built-injson?Scenario Use Full in-memory loading json.load()Partial reading (big file) ijsonNewline-delimited JSON json.loads()line-by-line
Let me know the structure of your large JSON if you want a tailored example.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.