Skip to content
TheChatArchiveThe Chat Archive
GeminiGemini 3 Pro Preview· googleHugging Face Frontier Reasoning Traces

Create a Python tool to validate and clean JSON data.

  1. User

    Create a Python tool to validate and clean JSON data.

  2. Gemini Assistant

    Planning the Python Tool

    I've got a grasp of the initial goals: building a Python tool to validate and clean JSON. My focus now is zeroing in on the must-have features. Validation is paramount, naturally, but the cleaning functionalities are going to be key, and I'm keen to make sure I don't overlook any common JSON issues.

    Structuring the Tool's Core

    I'm starting to sketch out the class structure, as I described. The class-based approach seems sound, and I'm leaning towards a JSONProcessor class to handle both validation and cleaning, rather than two separate classes. The json module will be core to the parsing, and I'm definitely going to incorporate jsonschema. Regular expressions will be vital for those quick fixes.

    Refining the Approach

    I'm now contemplating how to combine the raw string manipulation with the parsing. A hybrid strategy seems best: try json.loads first, and if it fails, engage regex to remove trailing commas, fix single quotes (with caution), and remove comments before trying to parse again. I'm investigating permissive parsers, too, though, to try to balance simplicity and accuracy.

    Evaluating the Hybrid Strategy

    I'm now really weighing the pros and cons of the hybrid approach. The core idea remains: use regex for common issues before attempting json.loads. I've firmed up on using jsonschema for structural validation. The key is in ordering the cleaning steps and being very careful with the regex replacements. I'm exploring adding optional parameters to the clean_string method to make some changes configurable.

    Considering Additional Cleaning Steps

    I'm now contemplating additional cleaning methods, such as handling NaN and Infinity, even though they aren't standard JSON. Also, I'm thinking about including ast.literal_eval to convert strings that look like Python dictionaries, if applicable. A configuration parameter for "strictness" seems like a practical way to balance flexibility.

    Formulating the Class Design

    I'm now deep in the implementation. The JSONTool class is taking shape, and the methods for cleaning and validating are being implemented. I'm focusing on the order of operations: first, cleaning with regex, then parsing with json.loads. After that, I will validate the schema, or return an error. I'm also thinking of adding an option to include ast.literal_eval to convert Python-like dictionaries.

    Here is a comprehensive Python tool used to validate, clean, and format JSON data.

    This tool goes beyond basic syntax checking. It handles:

    1. Syntax Validation: Checks if the JSON is parsable.
    2. Schema Validation: Checks if the valid JSON matches a specific structure (using jsonschema).
    3. Cleaning: Automatically fixes common errors like trailing commas, single quotes, and comments (C-style).
    4. Formatting: Pretty-printing the output.

    Prerequisites

    You will need the jsonschema library for structural validation.

    pip install jsonschema
    

    The Python Tool (json_tool.py)

    import json
    import re
    import ast
    from typing import Dict, Any, Union, Tuple
    from jsonschema import validate, ValidationError
    
    class JSONTool:
        def __init__(self, schema: Dict = None):
            """
            Initialize the tool with an optional JSON Schema.
            """
            self.schema = schema
    
        def _remove_comments(self, json_str: str) -> str:
            """Removes C-style comments (//... and /*...*/) from the string."""
            pattern = r'//.*?$|/\*.*?\*/'
            # Flags: MULTILINE for line comments, DOTALL for block comments
            regex = re.compile(pattern, re.MULTILINE | re.DOTALL)
            return regex.sub("", json_str)
    
        def _fix_trailing_commas(self, json_str: str) -> str:
            """Removes trailing commas in objects and arrays."""
            # Look for a comma followed immediately by whitespace and a closing bracket/brace
            pattern = r',(?=\s*[}\]])'
            return re.sub(pattern, "", json_str)
    
        def _convert_single_quotes(self, json_str: str) -> str:
            """
            Attempts to convert single quotes to double quotes.
            Note: This is a heuristic and might fail on complex strings containing apostrophes.
            """
            # Check if it looks like a Python dict string representation
            if "'" in json_str and '"' not in json_str:
                try:
                    # Use ast.literal_eval to safely parse python-like dict strings
                    # then dump it back to strict JSON
                    py_obj = ast.literal_eval(json_str)
                    return json.dumps(py_obj)
                except (ValueError, SyntaxError):
                    pass
            return json_str
    
        def clean(self, raw_data: str) -> str:
            """
            Attempts to clean 'dirty' JSON strings.
            1. Removes comments.
            2. Fixes single quotes (if strictly single-quoted).
            3. Removes trailing commas.
            """
            if not isinstance(raw_data, str):
                return raw_data
    
            cleaned = self._remove_comments(raw_data)
            cleaned = self._convert_single_quotes(cleaned)
            cleaned = self._fix_trailing_commas(cleaned)
            return cleaned.strip()
    
        def process(self, raw_data: Union[str, Dict]) -> Tuple[bool, Any, str]:
            """
            Main entry point. Validates syntax, cleans, interprets, checks schema.
            
            Returns tuple: (is_valid, parsed_data_or_none, message)
            """
            data_obj = None
    
            # 1. Handle non-string input
            if isinstance(raw_data, (dict, list)):
                data_obj = raw_data
            else:
                # 2. Try parsing raw string first
                try:
                    data_obj = json.loads(raw_data)
                except json.JSONDecodeError:
                    # 3. If parsing fails, try cleaning the string
                    cleaned_str = self.clean(raw_data)
                    try:
                        data_obj = json.loads(cleaned_str)
                    except json.JSONDecodeError as e:
                        return False, None, f"Syntax Error: {str(e)}"
                    except Exception as e:
                        return False, None, f"Parsing Error: {str(e)}"
    
            # 4. Schema Validation (if schema provided)
            if self.schema:
                try:
                    validate(instance=data_obj, schema=self.schema)
                except ValidationError as e:
                    return False, data_obj, f"Schema Error: {e.message}"
    
            return True, data_obj, "Valid JSON"
    
        def format(self, data: Any, indent: int = 4) -> str:
            """Returns a pretty-printed string representation."""
            return json.dumps(data, indent=indent)
    
    # ==========================================
    # Usage Examples
    # ==========================================
    
    if __name__ == "__main__":
        # Example 1: A "Dirty" JSON string with comments, single quotes, trailing comma
        dirty_json = """
        {
            'id': 1,
            'name': 'Test Tool', // This is a comment
            'active': true,
        }
        """
    
        # Optional: Define a schema to enforce structure
        user_schema = {
            "type": "object",
            "properties": {
                "id": {"type": "number"},
                "name": {"type": "string"},
                "active": {"type": "boolean"}
            },
            "required": ["id", "name"]
        }
    
        tool = JSONTool(schema=user_schema)
    
        print("--- Processing Dirty JSON ---")
        is_valid, data, msg = tool.process(dirty_json)
        
        if is_valid:
            print(f"Status: Success ({msg})")
            print("Cleaned Data:", tool.format(data))
        else:
            print(f"Status: Failed ({msg})")
    
        print("\n--- Processing Invalid Schema ---")
        # Valid syntax, but 'id' is a string instead of number
        bad_schema_json = '{"id": "one", "name": "Test", "active": true}'
        
        is_valid, data, msg = tool.process(bad_schema_json)
        if not is_valid:
            print(f"Status: Failed ({msg})")
    

    How it works

    1. _remove_comments: Uses Regex to strip out // and /* */ style comments, which are illegal in standard JSON but common in config files.
    2. _fix_trailing_commas: Uses Regex loop-ahead to find commas at the end of lists or objects ([1, 2,] becomes [1, 2]).
    3. _convert_single_quotes: If the string uses Python-style single quotes ({'key': 'val'}), standard json.loads will fail. The tool attempts to fix this using Python's AST evaluator purely for parsing the structure, then converts it to standard JSON.
    4. process workflow:
      • It tries to load the data normally.
      • If that fails, it runs the cleaning functions and tries loading again.
      • If it parses successfully, it validates against the jsonschema (if provided).

    Integration

    You can use this in your own project simply by importing the class:

    from json_tool import JSONTool
    
    # Simple syntax check
    cleaner = JSONTool()
    valid, data, error = cleaner.process("{'foo': 'bar',}")
    
    if valid:
        print(data['foo']) # outputs: bar
    

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.