Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0125-preview· OpenAIAi2 WildChat-4.8M (AI2)

I have an image file (invoice) and a json where I have list of extracted entites. And I…

  1. User

    I have an image file (invoice) and a json where I have list of extracted entites. And I have a csv file containing OCR textract output containing these columns ((page_num,block_num,line_num,word_num,left,right,top,bottom,width,height,conf,text,image_height,image_width)) JSON Format: { “invoice_details”:{ “invoice_number “: “”, “invoice_date”: “”, “invoice_due_date”: “”, “order_id”: “”, “vendor_name”: “”, “buyer_name”: “”, “shipto_name”: “”, “billto_name”: “”, }, “Payment Details”: { “vendor_ifsccode” :””, “vendor_bankname” :“”, “account_number” :“”, “bank_payment_terms”:“”, }, “amounts_and_tax”: { “Subtotal_or_taxable_amount”: “”, "total_sgst_amount ": “”, "total_cgst_amount ": “”, "total_igst_amount ": “”, “total_amount_after_tax”: “” “billto_GSTIN”: “”, “vendor_GSTIN”: “”, “shipto_GSTIN”: “” } }

    I want to map the bounding boxes from the json to image. And print the bounding boxes on image Write a highly advance python code to map the bounding boxex. There are some important conditions. That code should handle effectively.

    1. I want separate function to handle single token entity and multi token entiy
    2. In case of multi token entity :- Check all the words in multi token entity are nearby to each other or not. They should in same line or only in next line always. You can also check the distance between words before picking them up from dataframe.
    3. Repetation of any entity is not allowed. If we have already got the bounding box for any entity. Then we should not consider that entity again.
    4. In case of numerical value or entity we have to remove the comma from dataframe and extracted entity then only we have to match both.
  2. Assistant

    To accomplish the stated requirements, we'll need a multi-step approach that involves reading the JSON and CSV data, processing it according to the specified conditions, and finally drawing bounding boxes on the image. Let's break down the solution into functions for clarity and modularity.

    First, ensure you have the necessary packages installed:

    pip install pandas opencv-python
    

    Here's a comprehensive Python script to achieve your goals:

    import json
    import pandas as pd
    import cv2
    
    def preprocess_entity(entity):
        """Preprocess entities by removing commas in numerical values."""
        return entity.replace(",", "")
    
    def is_nearby(current_box, next_box, max_line_difference=10, max_distance=100):
        """Check if the next bounding box is nearby to the current one.
           Both should be on the same line or the next line and within a certain distance."""
        same_line = abs(current_box['top'] - next_box['top']) <= max_line_difference
        close_enough = abs(current_box['right'] - next_box['left']) <= max_distance
        return same_line and close_enough
    
    def merge_boxes(boxes):
        """Merge bounding boxes of a multi-token entity."""
        if not boxes: return None
        min_left = min(box['left'] for box in boxes)
        max_right = max(box['right'] for box in boxes)
        min_top = min(box['top'] for box in boxes)
        max_bottom = max(box['bottom'] for box in boxes)
        return {
            'left': min_left, 'right': max_right,
            'top': min_top, 'bottom': max_bottom
        }
    
    def process_single_token_entity(dataframe, entity):
        """Process a single token entity to find its bounding box."""
        entity = preprocess_entity(entity)
        for _, row in dataframe.iterrows():
            if preprocess_entity(row['text']) == entity:
                return {
                    'left': row['left'], 'right': row['right'],
                    'top': row['top'], 'bottom': row['bottom']
                }
        # Return None if no match is found
        return None
    
    def process_multi_token_entity(dataframe, entity):
        """Process a multi-token entity to find and merge its bounding boxes."""
        tokens = entity.split()
        boxes = []
        for token in tokens:
            box = process_single_token_entity(dataarrayframe, token)
            if box is None:
                return None  # Unable to find all tokens
            boxes.append(box)
    
        # Verify if boxes are near to each other
        for i in range(len(boxes) - 1):
            if not is_nearby(boxes[i], boxes[i + 1]):
                return None  # Tokens are not closely positioned
    
        return merge_boxes(boxes)
    
    def draw_bounding_boxes(image_path, bounding_boxes):
        """Draw bounding boxes on the image."""
        image = cv2.imread(image_path)
        for box in bounding_boxes:
            if box is not None:
                top_left = (box['left'], box['top'])
                bottom_right = (box['right'], box['bottom'])
                cv2.rectangle(image, top_left, bottom_right, (0, 255, 0), 2)
        cv2.imwrite('output_image.png', image)
    
    def main(json_path, csv_path, image_path):
        with open(json_path, 'r') as f:
            data = json.load(f)
        
        dataframe = pd.read_csv(csv_path)
        bounding_boxes = []
    
        for key, entity in data.items():
            for sub_key, value in entity.items():
                if " " in preprocess_entity(value):  # Multi-token entity
                    box = process_multi_token_entity(dataframe, value)
                else:  # Single token entity
                    box = process_single_token_entity(dataframe, value)
                if box is not None and box not already in bounding_boxes:
                    bounding_boxes.append(box)
        
        draw_bounding_boxes(image_path, bounding_boxes)
    
    if __name__ == "__main__":
        main("path_to_json.json", "path_to_csv.csv", "path_to_image.jpg")
    

    This script achieves the following:

    • Preprocesses entities to handle numerical values correctly.
    • Processes single and multi-token entities as per the conditions mentioned, including checking if multi-token entities' words are nearby.
    • Ensures that repetitions of entities are not considered, by checking if a bounding box already exists in the list before adding it.
    • Finally, draws the bounding boxes on the image and saves the output.

    Please ensure to replace "path_to_json.json", "path_to_csv.csv", and "path_to_image.jpg" with the actual paths to your files. Also, this script assumes the DataFrame has been correctly read according to the CSV structure you provided, including the preprocessing step to remove commas from numerical values in both the entities and the DataFrame. The distance thresholds in is_nearby might need adjustments depending on the specific layout and scale of your documents.

  3. User

    there is some error in main function

  4. Assistant

    I see, let's correct the issues in the main function. The main issues might stem from how entities are traversed and the check for existing bounding boxes that was incorrectly mentioned in the previous snippet. We need to iterate through the nested structure of your JSON correctly and fix the issue with the bounding box uniqueness check. Let's adjust the implementation:

    def main(json_path, csv_path, image_path):
        with open(json_path, 'r') as f:
            data = json.load(f)
        
        dataframe = pd.read_csv(csv_path)
        processed_entities = set()  # To track processed entities, ensuring no repetition
        bounding_boxes = []
    
        for section, entities in data.items():
            # 'entity' refers to each key-value pair within the sections of the JSON
            for entity_name, value in entities.items():
                if value in processed_entities:
                    continue  # Skipping already processed entities
                
                value_no_comma = preprocess_entity(value)  # Handling numerical values
                if " " in value_no_comma:  # Multi-token entity
                    box = process_multi_token_entity(dataframe, value_no_comma)
                else:  # Single token entity
                    box = process_single_token_entity(dataframe, value_no_comma)
                
                # Check if the box is not None and not already added
                if box:
                    bounding_boxes.append(box)
                    processed_entities.add(value)  # Mark entity as processed
    
        draw_bounding_boxes(image_path, bounding_boxes)
    
    if __name__ == "__main__":
        main("path_to_json.json", "path_to_csv.csv", "path_to_image.jpg")
    

    This adjustment modifies the handling of the JSON entity lookup to ensure entities are only processed once. It also removes the incorrect bounding box uniqueness check based on the assumption that a box variable would directly contain the textual content of the entity, which was a mistake in the provided context.

    A few key points regarding changes and assumptions:

    • Entities are preprocessed (e.g., removing commas) before checking or processing, ensuring consistency with the OCR-outputted CSV content.
    • The processed_entities set tracks which entities have already been processed, preventing repetition based on the original textual values of the entities.
    • We ensure all values are treated as strings when performing operations such as splitting into tokens or comparing with CSV entries.
    • Error handling or special cases (e.g., very close but not exactly matching texts due to OCR errors) are not extensively covered here; depending on the quality of the OCR data, further adjustments might be necessary.

    Ensure the paths in the main call are correctly replaced with the paths to your actual files before running the script.

  5. User

    to match entity token in dataframe use fuzz match. Instead of simply matching string

  6. Assistant

    To incorporate fuzzy string matching in the process of finding entities within the DataFrame, we'll use the fuzzywuzzy library in Python, which allows us to compute the similarity between strings and find the best match. This is particularly useful in cases where the OCR text may not be perfectly accurate.

    First, ensure you have fuzzywuzzy installed:

    pip install fuzzywuzzy[speedup]
    

    Now, let's adjust the functions that process entities to use fuzzy string matching. We'll modify both process_single_token_entity and process_multi_token_entity functions:

    from fuzzywuzzy import fuzz
    from fuzzywuzzy import process
    
    def preprocess_entity(entity):
        """Preprocess entities by removing commas in numerical values and stripping."""
        return entity.replace(",", "").strip()
    
    def process_single_token_entity(dataframe, entity, threshold=85):
        """Process a single token entity to find its bounding box using fuzzy matching."""
        best_score = 0
        best_box = None
        entity = preprocess_entity(entity)
    
        for _, row in dataframe.iterrows():
            ocr_text = preprocess_entity(row['text'])
            score = fuzz.ratio(entity, ocr_text)
            
            if score > best_score and score > threshold:
                best_score = score
                best_box = {
                    'left': row['left'], 'right': row['right'],
                    'top': row['top'], 'bottom': row['bottom']
                }
        
        return best_box
    
    def process_multi_token_entity(dataframe, entity, threshold=85):
        """Process a multi-token entity to find and merge its bounding boxes using fuzzy matching."""
        tokens = entity.split()
        boxes = []
        for token in tokens:
            box = process_single_token_entity(dataframe, token, threshold)
            if box is None:
                return None  # Unable to find all tokens
            boxes.append(box)
    
        # Verify if boxes are near to each other
        if all(is_nearby(boxes[i], boxes[i + 1]) for i in range(len(boxes) - 1)):
            return merge_boxes(boxes)
        else:
            return None
    

    The process_single_token_entity function now searches through each row in the DataFrame and uses fuzz.ratio to calculate the similarity between the entity and the OCR text. It only considers the OCR text as a match if the similarity score is above a certain threshold, in this case, 85. You may need to adjust this threshold based on the accuracy of your OCR data and the nature of your entities.

    The process_multi_token_entity function has also been adjusted to use the fuzzy matching approach for each token of a multi-token entity. It continues to ensure that the bounding boxes for these tokens are near each other before considering them as parts of a single entity.

    With these adjustments, your code is now more resilient against OCR inaccuracies, using fuzzy string matching to improve the likelihood of correctly identifying entities in the text.

  7. User

    Modify the code to check if all the entity present in json file got their bounding boxes or not if any entity is left out you have to find bounding box for that entity as well any how

  8. Assistant

    To ensure all entities mentioned in the JSON file get their bounding boxes (even if they're not found in the initially scanned OCR text or through fuzzy matching), we will need to add a mechanism to track all entities and verify which ones were successfully matched. For entities that were not matched initially, we'll attempt a broader and more lenient search.

    We'll adjust the main function and introduce a lenient search mechanism into the entity processing steps to serve as a fallback for unmatched entities.

    First, ensure to import the necessary modules (some are already mentioned):

    import pandas as pd
    import cv2
    # Ensure you've installed fuzzywuzzy as previously mentioned
    from fuzzywuzzy import fuzz
    from fuzzywuzzy import process
    

    Now, let's enhance the entity processing functions and modify the main function to incorporate these changes:

    def find_bounding_box_for_unmatched_entity(dataframe, unmatched_entities):
        """Attempt to find bounding boxes for entities that were not matched in the initial search."""
        for entity in unmatched_entities:
            entity_no_comma = preprocess_entity(entity)
            
            # Attempt a broader, more lenient search for unmatched entities
            best_match = process.extractOne(entity_no_comma, dataframe['text'].apply(preprocess_entity), scorer=fuzz.partial_ratio)
            
            # If a match is found with a partial ratio score above a leniency threshold, return its bounding box
            if best_match and best_match[1] > 50:  # Adjust the threshold based on your requirements
                matched_text = best_match[0]
                row = dataframe.loc[dataframe['text'].apply(preprocess_entity) == matched_text].iloc[0]
                
                return {
                    'left': row['left'], 'right': row['right'],
                    'top': row['top'], 'bottom': row['bottom']
                }
        return None
    
    def main(json_path, csv_path, image_path):
        with open(json_path, 'r') as f:
            data = json.load(f)
        
        dataframe = pd.read_csv(csv_path)
        processed_entities = set()
        unmatched_entities = []
        bounding_boxes = []
    
        for section, entities in data.items():
            for entity_name, value in entities.items():
                value_no_comma = preprocess_entity(value) 
                box = None
                
                if " " in value_no_comma:  # Multi-token entity
                    box = process_multi_token_entity(dataframe, value_no_comma)
                else:  # Single token entity
                    box = process_single_token_entity(dataframe, value_no_comma)
                
                if box:
                    bounding_boxes.append(box)
                    processed_entities.add(value)
                else:
                    unmatched_entities.append(value)
    
        # Attempt to find bounding boxes for unmatched entities
        for entity in unmatched_entities:
            box = find_bounding_box_for_unmatched_entity(dataframe, [entity])
            if box:
                bounding_boxes.append(box)
            else:
                print(f"Warning: No bounding box found for '{entity}' even after lenient search.")
    
        draw_bounding_boxes(image_path, bounding_boxes)
    
    # Make sure to replace the paths with your actual file paths
    if __name__ == "__main__":
        main("path_to_json.json", "path_to_csv.csv", "path_to_image.jpg")
    

    What's Changed?

    1. A fallback mechanism:

      • find_bounding_box_for_unmatched_entity function is introduced to perform a lenient search for entities that initially couldn't be matched.
      • It uses process.extractOne with fuzz.partial_ratio to find the best match, even if it's only partially similar. This can help with entities that were significantly misread by OCR.
    2. Tracking Unmatched Entities:

      • Entities that were not successfully matched during the first pass are tracked.
      • After the initial process, the script attempts to find bounding boxes for these unmatched entities using the new lenient search mechanism.
    3. Adjustable Leniency:

      • The leniency threshold in find_bounding_box_for_unmatched_entity can be adjusted. A more lenient threshold (lower score) increases the chance of matching entities at the cost of potentially incorrect matches.

    Note:

    This solution increases the chances of finding bounding boxes for all entities but also increases the risk of incorrect matches, especially with very lenient thresholds. You might need to adjust the leniency threshold based on your OCR data quality and the desired balance between match coverage and accuracy.

  9. User

    Give the complete latest code

  10. Assistant

    Certainly! Below is the complete code integrating all the modifications discussed, including fuzzy matching for entity recognition, handling of both single and multi-token entities, attempts to find bounding boxes for all entities listed in the JSON, and the fallback mechanism for unmatched entities. Ensure you have the required libraries (pandas, opencv-python, and fuzzywuzzy) installed.

    import json
    import pandas as pd
    import cv2
    from fuzzywuzzy import fuzz
    from fuzzywuzzy import process
    
    def preprocess_entity(entity):
        """Preprocess entities by removing commas in numerical values and stripping."""
        return entity.replace(",", "").strip()
    
    def is_nearby(current_box, next_box, max_line_difference=10, max_distance=100):
        """Check if the next bounding box is nearby to the current one."""
        same_line = abs(current_box['top'] - next_box['top']) <= max_line_difference
        close_enough = abs(current_box['right'] - next_box['left']) <= max_distance
        return same_line and close_enough
    
    def merge_boxes(boxes):
        """Merge bounding boxes of a multi-token entity."""
        if not boxes: return None
        min_left = min(box['left'] for box in boxes)
        max_right = max(box['right'] for box in boxes)
        min_top = min(box['top'] for box in boxes)
        max_bottom = max(box['bottom'] for box in boxes)
        return {'left': min_left, 'right': max_right, 'top': min_top, 'bottom': max_bottom}
    
    def process_single_token_entity(dataframe, entity, threshold=85):
        """Process a single token entity to find its bounding box using fuzzy matching."""
        best_score = 0
        best_box = None
        entity = preprocess_entity(entity)
    
        for _, row in dataframe.iterrows():
            ocr_text = preprocess_entity(row['text'])
            score = fuzz.ratio(entity, ocr_text)
    
            if score > best_score and score > threshold:
                best_score = score
                best_box = {'left': row['left'], 'right': row['right'], 'top': row['top'], 'bottom': row['bottom']}
        return best_box
    
    def process_multi_token_entity(dataframe, entity, threshold=85):
        """Process a multi-token entity to find and merge its bounding boxes using fuzzy matching."""
        tokens = entity.split()
        boxes = []
        for token in tokens:
            box = process_single_token_entity(dataframe, token, threshold)
            if box is None:
                return None
            boxes.append(box)
    
        if all(is_nearby(boxes[i], boxes[i + 1]) for i in range(len(boxes) - 1)):
            return merge_boxes(boxes)
        else:
            return None
    
    def find_bounding_box_for_unmatched_entity(dataframe, unmatched_entities):
        """Attempt to find bounding boxes for entities that were not matched in the initial search."""
        for entity in unmatched_entities:
            entity_no_comma = preprocess_entity(entity)
            
            best_match = process.extractOne(entity_no_comma, dataframe['text'].apply(preprocess_entity), scorer=fuzz.partial_ratio)
            
            if best_match and best_match[1] > 50:
                matched_text = best_match[0]
                row = dataframe.loc[dataframe['text'].apply(preprocess_entity) == matched_text].iloc[0]
                
                return {'left': row['left'], 'right': row['right'], 'top': row['top'], 'bottom': row['bottom']}
        return None
    
    def draw_bounding_boxes(image_path, bounding_boxes):
        """Draw bounding boxes on the image."""
        image = cv2.imread(image_path)
        for box in bounding_boxes:
            if box is not None:
                top_left = (box['left'], box['top'])
                bottom_right = (box['right'], box['bottom'])
                cv2.rectangle(image, top_left, bottom_right, (0, 255, 0), 2)
        cv2.imwrite('output_image.png', image)
    
    def main(json_path, csv_path, image_path):
        with open(json_path, 'r') as f:
            data = json.load(f)
        
        dataframe = pd.read_csv(csv_path)
        bounding_boxes = []
    
        unmatched_entities = {value for entity in data.values() for value in entity.values() if value}
    
        for section, entities in data.items():
            for entity_name, value in entities.items():
                value_no_comma = preprocess_entity(value)
                box = process_multi_token_entity(dataframe, value_no_comma) if " " in value_no_comma else process_single_token_entity(dataframe, value_no_comma)
    
                if box:
                    bounding_boxes.append(box)
                    unmatched_entities.discard(value)
    
        # Attempt to find bounding boxes for unmatched entities
        for entity in unmatched_entities:
            box = find_bounding_box_for_unmatched_entity(dataframe, [entity])
            if box:
                bounding_boxes.append(box)
            else:
                print(f"Warning: No bounding box found for '{entity}' even after lenient search.")
    
        draw_bounding_boxes(image_path, bounding_boxes)
    
    if __name__ == "__main__":
        main("path_to_json.json", "path_to_csv.csv", "path_to_image.jpg")
    

    This code introduces a sophisticated approach to handle entity recognition and bounding box extraction from OCR text. It processes single and multi-token entities with fuzzy matching to enhance matching accuracy against OCR inaccuracies. If any entity from the JSON doesn't match initially, a lenient search is performed to try to find a bounding box anyhow, increasing the chances of capturing all entities mentioned in the JSON.

    Do replace "path_to_json.json", "path_to_csv.csv", and "path_to_image.jpg" with the actual paths to your files. This code provides a more robust solution for extracting and visualizing entity bounding boxes from OCR data while accounting for potential OCR inaccuracies and ensuring coverage for all entities listed in your JSON file.

  11. User

    Simply this peice of code unmatched_entities = {value for entity in data.values() for value in entity.values() if value}

  12. Assistant

    Certainly! The line you're referring to is a list comprehension that's aimed at extracting all unique, non-empty values from the nested dictionary data that you extracted from the JSON. Given the structure of your JSON, which involves nested dictionaries, this snippet iteratively goes through each nested dictionary and collects all non-empty values (entities) into a set named unmatched_entities.

    To simplify or rather to explain it:

    unmatched_entities = {value for entity in data.values() for value in entity.values() if value}
    

    Here's the breakdown:

    • data.values() iterates through the values of the data dictionary, where each value is itself a dictionary (representing sections like "invoice_details", "Payment Details", etc.).
    • for entity in data.values() takes each of these dictionaries (each entity is a dictionary).
    • for value in entity.values() iterates through the values of these nested dictionaries. These values are the actual entities you're interested in extracting (like "invoice_number", "vendor_name", etc.).
    • if value ensures that only non-empty values are considered. This filters out any case where an entity might not have a value assigned (e.g., an empty string).
    • {...} around the comprehensive loop indicates that the result is stored as a set. This automatically removes duplicate values ensuring every entity is unique.

    This approach efficiently collects all entities that you're going to attempt to find in your OCR data. After the initial attempt to find bounding boxes for these entities, if any are unmatched (not found or poorly matched), you're left with a set of entities for which you'll try to find bounding boxes again using a lenient search.

    The use of the set here is intentional and beneficial because:

    • It automatically eliminates duplicates, ensuring you don't process the same entity value more than once.
    • It allows efficient add/remove/check operations which you utilize when discarding matched entities and checking for unmatched ones.

    I hope this simplifies and clarifies the purpose and functionality of that particular line of code!

  13. User

    write in simple way this code unmatched_entities = {value for entity in data.values() for value in entity.values() if value}

  14. Assistant

    To simplify and clearly explain the operation of the line:

    unmatched_entities = {value for entity in data.values() for value in entity.values() if value}
    

    Let's break it down step by step in a more verbose manner:

    1. Objective: We aim to collect all non-empty values from a nested dictionary structure (data) where each key maps to another dictionary.

    2. Structure of data: It's a nested dictionary, meaning dictionaries within a dictionary. Here's a simplified example, similar to what your data might look like:

    {
      "invoice_details": { "invoice_number": "123", "vendor_name": "Abc Corp" },
      "payment_details": { "account_number": "XYZ456", "bank_name": "Bank XYZ" }
    }
    
    1. Breaking Down the One-Liner:

      a. data.values() gets all the values in the data dictionary. Each value is itself a dictionary.

      b. for entity in data.values() iterates over these dictionaries.

      c. entity.values() now, for each of those dictionaries, we get their values. These are the actual contents we're interested in.

      d. for value in entity.values() this iterates over the contents we're interested in.

      e. if value is a condition to only include non-empty values. An empty string ("") is considered False in a boolean context.

      f. {...} is the syntax for a set comprehension, which creates a set of all the values that pass the condition. A set only includes unique items.

    2. Verbose Alternative: Here's what the above comprehension could look like if we were to write it out in a more extended form, which might be easier to follow if you're not used to Python comprehensions:

    unmatched_entities = set()  # Initialize an empty set
    for entity in data.values():  # For each dictionary in our main dictionary
        for value in entity.values():  # For each value in these dictionaries
            if value:  # If the value is not empty
                unmatched_entities.add(value)  # Add it to our set
    

    In this extended form, it's more apparent what's happening: we're iterating over every value in each nested dictionary within data, and if that value is not empty, we're adding it to a set. The set automatically handles de-duplication, meaning if two entries have the same non-empty value, it will only be stored once in unmatched_entities.

  15. User

    Modify the complete code Because I just want to process these section Invoice_details, Payment_details, Amounts_and_tax. I don't wanna process address_details, Line_items and table_column_names_in_reading_order

  16. Assistant

    To modify the complete Python code to only process the sections "Invoice_details", "Payment_details", and "Amounts_and_tax" from your JSON structure, we can add a filter to only include these specific sections when processing your data. This way, the script will ignore any other sections like "address_details", "Line_items", and "table_column_names_in_reading_order".

    Below is your modified code with this specific filtering:

    import json
    import pandas as pd
    import cv2
    from fuzzywuzzy import fuzz
    from fuzzywuzzy import process
    
    def preprocess_entity(entity):
        return entity.replace(",", "").strip()
    
    def is_nearby(current_box, next_box, max_line_difference=10, max_distance=100):
        same_line = abs(current_box['top'] - next_box['top']) <= max_line_difference
        close_enough = abs(current_box['right'] - next_box['left']) <= max_distance
        return same_line and close_enough
    
    def merge_boxes(boxes):
        if not boxes:
            return None
        min_left = min(box['left'] for box in boxes)
        max_right = max(box['right'] for box in boxes)
        min_top = min(box['top'] for box in boxes)
        max_bottom = max(box['bottom'] for box in boxes)
        return {'left': min_left, 'right': max_right, 'top': min_top, 'bottom': max_bottom}
    
    def process_single_token_entity(dataframe, entity, threshold=85):
        best_score = 0
        best_box = None
        entity = preprocess_entity(entity)
        for _, row in dataframe.iterrows():
            ocr_text = preprocess_entity(row['text'])
            score = fuzz.ratio(entity, ocr_text)
            if score > best_score and score > threshold:
                best_score = score
                best_box = {'left': row['left'], 'right': row['right'], 'top': row['top'], 'bottom': row['bottom']}
        return best_box
    
    def process_multi_token_entity(dataframe, entity, threshold=85):
        tokens = entity.split()
        boxes = []
        for token in tokens:
            box = process_single_token_entity(dataframe, token, threshold)
            if box is None:
                return None
            boxes.append(box)
        if all(is_nearby(boxes[i], boxes[i+1]) for i in range(len(boxes)-1)):
            return merge_boxes(boxes)
        else:
            return None
    
    def find_bounding_box_for_unmatched_entity(dataframe, unmatched_entities):
        for entity in unmatched_entities:
            entity_no_comma = preprocess_entity(entity)
            best_match = process.extractOne(entity_no_comma, dataframe['text'].apply(preprocess_entity), scorer=fuzz.partial_ratio)
            if best_match and best_match[1] > 50:
                matched_text = best_match[0]
                row = dataframe.loc[dataframe['text'].apply(preprocess_entity) == matched_text].iloc[0]
                return {'left': row['left'], 'right': row['right'], 'top': row['top'], 'bottom': row['bottom']}
        return None
    
    def draw_bounding_boxes(image_path, bounding_boxes):
        image = cv2.imread(image_path)
        for box in bounding_boxes:
            if box is not None:
                top_left = (box['left'], box['top'])
                bottom_right = (box['right'], box['bottom'])
                cv2.rectangle(image, top_left, bottom_right, (0, 255, 0), 2)
        cv2.imwrite('output_image_with_boxes.png', image)
    
    def main(json_path, csv_path, image_path):
        with open(json_path, 'r') as f:
            data = json.load(f)
        
        dataframe = pd.read_csv(csv_path)
        bounding_boxes = []
    
        # Specify the sections you want to process
        sections_to_process = ['invoice_details', 'Payment_Details', 'amounts_and_tax']
    
        for section, entities in data.items():
            # Skip the sections not to be processed
            if section.lower() not in sections_to_process:
                continue
    
            for entity_name, value in entities.items():
                value_no_comma = preprocess_entity(value)
                box = None
    
                if " " in value_no_comma:
                    box = process_multi_token_entity(dataframe, value_no_comma)
                else:
                    box = process_single_token_entity(dataframe, value_no_comma)
    
                if box:
                    bounding_boxes.append(box)
    
        unmatched_entities = {value for entity in data.values() for key, value in entity.items() if key.lower() in sections_to_process and value}
        for entity in unmatched_entities:
            box = find_bounding_box_for_unmatched_entity(dataframe, [entity])
            if box:
                bounding_boxes.append(box)
            else:
                print(f"Warning: No bounding box found for '{entity}' even after lenient search.")
    
        draw_bounding_boxes(image_path, bounding_boxes)
    
    if __name__ == "__main__":
        main("path_to_json.json", "path_to_csv.csv", "path_to_image.jpg")
    

    What's Changed:

    • Added a sections_to_process list that specifies the sections of the JSON to work with. This list is used to filter out sections not mentioned before processing.
    • The unmatched entities are filtered based on whether their keys are within the sections specified in sections_to_process, considering case sensitivity by converting section to lowercase using section.lower().
    • The drawing of bounding boxes and the search for unmatched entities is constrained to the selected sections.

    This approach allows flexibility; you can easily adjust sections_to_process to include or exclude sections based on your requirements. Remember to replace "path_to_json.json", "path_to_csv.csv", and "path_to_image.jpg" with the actual file paths you are working with.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.