Create a Python tool to validate and clean JSON data.
- User
Create a Python tool to validate and clean JSON data.
Assistant
Planning the Python Tool
I've got a grasp of the initial goals: building a Python tool to validate and clean JSON. My focus now is zeroing in on the must-have features. Validation is paramount, naturally, but the cleaning functionalities are going to be key, and I'm keen to make sure I don't overlook any common JSON issues.
Structuring the Tool's Core
I'm starting to sketch out the class structure, as I described. The class-based approach seems sound, and I'm leaning towards a
JSONProcessorclass to handle both validation and cleaning, rather than two separate classes. Thejsonmodule will be core to the parsing, and I'm definitely going to incorporatejsonschema. Regular expressions will be vital for those quick fixes.Refining the Approach
I'm now contemplating how to combine the raw string manipulation with the parsing. A hybrid strategy seems best: try
json.loadsfirst, and if it fails, engage regex to remove trailing commas, fix single quotes (with caution), and remove comments before trying to parse again. I'm investigating permissive parsers, too, though, to try to balance simplicity and accuracy.Evaluating the Hybrid Strategy
I'm now really weighing the pros and cons of the hybrid approach. The core idea remains: use regex for common issues before attempting
json.loads. I've firmed up on usingjsonschemafor structural validation. The key is in ordering the cleaning steps and being very careful with the regex replacements. I'm exploring adding optional parameters to theclean_stringmethod to make some changes configurable.Considering Additional Cleaning Steps
I'm now contemplating additional cleaning methods, such as handling
NaNandInfinity, even though they aren't standard JSON. Also, I'm thinking about includingast.literal_evalto convert strings that look like Python dictionaries, if applicable. A configuration parameter for "strictness" seems like a practical way to balance flexibility.Formulating the Class Design
I'm now deep in the implementation. The
JSONToolclass is taking shape, and the methods for cleaning and validating are being implemented. I'm focusing on the order of operations: first, cleaning with regex, then parsing withjson.loads. After that, I will validate the schema, or return an error. I'm also thinking of adding an option to includeast.literal_evalto convert Python-like dictionaries.Here is a comprehensive Python tool used to validate, clean, and format JSON data.
This tool goes beyond basic syntax checking. It handles:
- Syntax Validation: Checks if the JSON is parsable.
- Schema Validation: Checks if the valid JSON matches a specific structure (using
jsonschema). - Cleaning: Automatically fixes common errors like trailing commas, single quotes, and comments (C-style).
- Formatting: Pretty-printing the output.
Prerequisites
You will need the
jsonschemalibrary for structural validation.pip install jsonschemaThe Python Tool (
json_tool.py)import json import re import ast from typing import Dict, Any, Union, Tuple from jsonschema import validate, ValidationError class JSONTool: def __init__(self, schema: Dict = None): """ Initialize the tool with an optional JSON Schema. """ self.schema = schema def _remove_comments(self, json_str: str) -> str: """Removes C-style comments (//... and /*...*/) from the string.""" pattern = r'//.*?$|/\*.*?\*/' # Flags: MULTILINE for line comments, DOTALL for block comments regex = re.compile(pattern, re.MULTILINE | re.DOTALL) return regex.sub("", json_str) def _fix_trailing_commas(self, json_str: str) -> str: """Removes trailing commas in objects and arrays.""" # Look for a comma followed immediately by whitespace and a closing bracket/brace pattern = r',(?=\s*[}\]])' return re.sub(pattern, "", json_str) def _convert_single_quotes(self, json_str: str) -> str: """ Attempts to convert single quotes to double quotes. Note: This is a heuristic and might fail on complex strings containing apostrophes. """ # Check if it looks like a Python dict string representation if "'" in json_str and '"' not in json_str: try: # Use ast.literal_eval to safely parse python-like dict strings # then dump it back to strict JSON py_obj = ast.literal_eval(json_str) return json.dumps(py_obj) except (ValueError, SyntaxError): pass return json_str def clean(self, raw_data: str) -> str: """ Attempts to clean 'dirty' JSON strings. 1. Removes comments. 2. Fixes single quotes (if strictly single-quoted). 3. Removes trailing commas. """ if not isinstance(raw_data, str): return raw_data cleaned = self._remove_comments(raw_data) cleaned = self._convert_single_quotes(cleaned) cleaned = self._fix_trailing_commas(cleaned) return cleaned.strip() def process(self, raw_data: Union[str, Dict]) -> Tuple[bool, Any, str]: """ Main entry point. Validates syntax, cleans, interprets, checks schema. Returns tuple: (is_valid, parsed_data_or_none, message) """ data_obj = None # 1. Handle non-string input if isinstance(raw_data, (dict, list)): data_obj = raw_data else: # 2. Try parsing raw string first try: data_obj = json.loads(raw_data) except json.JSONDecodeError: # 3. If parsing fails, try cleaning the string cleaned_str = self.clean(raw_data) try: data_obj = json.loads(cleaned_str) except json.JSONDecodeError as e: return False, None, f"Syntax Error: {str(e)}" except Exception as e: return False, None, f"Parsing Error: {str(e)}" # 4. Schema Validation (if schema provided) if self.schema: try: validate(instance=data_obj, schema=self.schema) except ValidationError as e: return False, data_obj, f"Schema Error: {e.message}" return True, data_obj, "Valid JSON" def format(self, data: Any, indent: int = 4) -> str: """Returns a pretty-printed string representation.""" return json.dumps(data, indent=indent) # ========================================== # Usage Examples # ========================================== if __name__ == "__main__": # Example 1: A "Dirty" JSON string with comments, single quotes, trailing comma dirty_json = """ { 'id': 1, 'name': 'Test Tool', // This is a comment 'active': true, } """ # Optional: Define a schema to enforce structure user_schema = { "type": "object", "properties": { "id": {"type": "number"}, "name": {"type": "string"}, "active": {"type": "boolean"} }, "required": ["id", "name"] } tool = JSONTool(schema=user_schema) print("--- Processing Dirty JSON ---") is_valid, data, msg = tool.process(dirty_json) if is_valid: print(f"Status: Success ({msg})") print("Cleaned Data:", tool.format(data)) else: print(f"Status: Failed ({msg})") print("\n--- Processing Invalid Schema ---") # Valid syntax, but 'id' is a string instead of number bad_schema_json = '{"id": "one", "name": "Test", "active": true}' is_valid, data, msg = tool.process(bad_schema_json) if not is_valid: print(f"Status: Failed ({msg})")How it works
_remove_comments: Uses Regex to strip out//and/* */style comments, which are illegal in standard JSON but common in config files._fix_trailing_commas: Uses Regex loop-ahead to find commas at the end of lists or objects ([1, 2,]becomes[1, 2])._convert_single_quotes: If the string uses Python-style single quotes ({'key': 'val'}), standardjson.loadswill fail. The tool attempts to fix this using Python's AST evaluator purely for parsing the structure, then converts it to standard JSON.processworkflow:- It tries to load the data normally.
- If that fails, it runs the cleaning functions and tries loading again.
- If it parses successfully, it validates against the
jsonschema(if provided).
Integration
You can use this in your own project simply by importing the class:
from json_tool import JSONTool # Simple syntax check cleaner = JSONTool() valid, data, error = cleaner.process("{'foo': 'bar',}") if valid: print(data['foo']) # outputs: bar
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.