Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0125-preview· openaiHugging Face Suzhen/CodeChat

I have a code to map bounding boxes from json to image using ocr textract output which…

  1. User

    I have a code to map bounding boxes from json to image using ocr textract output which in csv format. But I am strugging to get correct bounding boxes for multi token entities. Multi token entity should be in one bounding box. I want some good clustering approch to address this issue. And a better solution to match the entity token from dataframe.

    Please modify my code according to my requirements. Code should avoid taking values from anywhere in dataframe. It should always take the nearest word of other token in case of multi token entity. Code should be robust.

    import json import pandas as pd import cv2 from thefuzz import fuzz from thefuzz import process import numpy as np

    def read_textract_output(csv_path): return pd.read_csv(csv_path)

    def read_json_entities(json_path): with open(json_path, 'r') as file: return json.load(file)

    def find_entity_bounding_boxes(entity_text, textract_df, image_size): entity_tokens = entity_text.split() results = pd.DataFrame()

    for token in entity_tokens:
        choices = textract_df["text"].dropna().tolist()
        best_match, score = process.extractOne(token, choices, scorer=fuzz.token_sort_ratio)
        if score > 70:
            best_matches_df = textract_df[textract_df["text"] == best_match]
            results = pd.concat([results, best_matches_df])
    
    bounding_boxes = []
    try:
        # Sort by line and word number to cluster words that are on the same line
        sorted_results = results.sort_values(by=["line_num", "word_num"])
    
        # Group the resulting bounding boxes by line number
        grouped_results = sorted_results.groupby("line_num")
    
        for _, group in grouped_results:
            # Calculate scaling factors based on image size and original image dimensions
            image_width, image_height = group.iloc[0]["image_width"], group.iloc[0]["image_height"]
            scale_x = image_size[0] / image_width
            scale_y = image_size[1] / image_height
    
            # Calculate the bounding box for the whole line
            min_left = np.min(group["left"])
            min_top = np.min(group["top"])
            max_right = np.max(group["left"] + group["width"])
            max_bottom = np.max(group["top"] + group["height"])
            bbox = (min_left * scale_x, min_top * scale_y, (max_right - min_left) * scale_x, (max_bottom - min_top) * scale_y)
            bounding_boxes.append(bbox)
            
    except Exception as e:
        print(f"An error occurred: {e}")
    
    return bounding_boxes
        
    

    def draw_bounding_boxes(image_path, entities, textract_df):

    image = cv2.imread(image_path)

    image_size = image.shape[1], image.shape[0]

    for category, details in entities.items():

    if category == "invoice_details" or category == "Payment Details" or category == "amounts_and_tax":

    for entity, value in details.items():

    if value:

    bounding_boxes = find_entity_bounding_boxes(value, textract_df, image_size)

    try:

    for bbox in bounding_boxes:

    x, y, w, h = map(int, bbox)

    cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)

    except:

    pass

    else:

    pass

    def draw_bounding_boxes(image_path, entities, textract_df): image = cv2.imread(image_path) image_size = image.shape[1], image.shape[0]

    font = cv2.FONT_HERSHEY_SIMPLEX
    font_scale = 0.5
    font_color = (0, 0, 0)
    line_type = 2
    padding = 5  # Padding between the start of bbox and text
    
    for category, details in entities.items():
        if category == "invoice_details" or category == "Payment Details" or category == "amounts_and_tax":
            for entity, value in details.items():
                if value:
                    bounding_boxes = find_entity_bounding_boxes(value, textract_df, image_size)
                    try:
                        for bbox in bounding_boxes:
                            x, y, w, h = map(int, bbox)
                            cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)
                            # Text placement calculation
                            text_position = (x, y - padding) if y - padding > 10 else (x, y + h + 20)
                            # Drawing the text
                            cv2.putText(image, entity, text_position, font, font_scale, font_color, line_type)
                    except:
                        pass
        else:
            pass
    cv2.imwrite('/home/ritik1s/Desktop/bbox_issues/temp_GPT/annotated_invoice.jpg', image)
    
    

    IMAGE_PATH = '/home/ritik1s/Desktop/bbox_issues/temp_GPT/check.jpeg' CSV_PATH = "/home/ritik1s/Desktop/bbox_issues/temp_GPT/check.csv" JSON_PATH = "/home/ritik1s/Desktop/bbox_issues/temp_GPT/row_skip.json"

    Read Textract output and JSON entities

    textract_df = read_textract_output(CSV_PATH) entities = read_json_entities(JSON_PATH)

    Draw the bounding boxes

    draw_bounding_boxes(IMAGE_PATH, entities, textract_df)

  2. Assistant

    Enhancing your code to better handle multi-token entities and ensure that bounding boxes are accurate and robust involves several steps. The following version of your code implements a more sophisticated approach to find and group bounding boxes for multi-token entities. It primarily focuses on clustering tokens that are spatially close to each other, using their position on the page:

    import json
    import pandas as pd
    import cv2
    from thefuzz import fuzz
    from thefuzz import process
    import numpy as np
    from sklearn.cluster import DBSCAN
    
    def read_textract_output(csv_path):
        return pd.read_csv(csv_path)
    
    def read_json_entities(json_path):
        with open(json_path, 'r') as file:
            return json.load(file)
        
    def find_entity_bounding_boxes(entity_text, textract_df, image_size):
        entity_tokens = entity_text.split()
        results = pd.DataFrame()
        
        # Update approach to minimize false matches by prioritizing nearest tokens
        for token in entity_tokens:
            choices = textract_df["text"].dropna().tolist()
            best_match, score = process.extractOne(token, choices, scorer=fuzz.token_sort_ratio)
            if score > 70:
                best_match_rows = textract_df[textract_df["text"] == best_match]
                results = pd.concat([results, best_match_rows], ignore_index=True)
    
        bounding_boxes = []
        try:
            if not results.empty:
                # Utilizing DBSCAN to cluster word bounding boxes based on spatial proximity.
                # Epsilon and min_samples need to be adjusted based on the average spacing and size of text in your documents.
                # dbscan = DBSCAN(eps=30, min_samples=1, metric='euclidean')
                dbscan = DBSCAN(eps=50, min_samples=1, metric='euclidean')  # Adjust params as needed.
                
                # Use the centroids of bounding boxes for clustering
                results["centroid_x"] = results["left"] + results["width"] / 2
                results["centroid_y"] = results["top"] + results["height"] / 2
                clusters = dbscan.fit_predict(results[["centroid_x", "centroid_y"]])
    
                results['cluster'] = clusters
                unique_clusters = results['cluster'].unique()
    
                for cluster in unique_clusters:
                    cluster_group = results[results['cluster'] == cluster]
                    # Continue with your scaling and bbox grouping as before
                    image_width, image_height = cluster_group.iloc[0]["image_width"], cluster_group.iloc[0]["image_height"]
                    scale_x = image_size[0] / image_width
                    scale_y = image_size[1] / image_height
    
                    min_left = np.min(cluster_group["left"])
                    min_top = np.min(cluster_group["top"])
                    max_right = np.max(cluster_group["left"] + cluster_group["width"])
                    max_bottom = np.max(cluster_group["top"] + cluster_group["height"])
                    
                    bbox = (min_left * scale_x, min_top * scale_y, (max_right - min_left) * scale_x, (max_bottom - min_top) * scale_y)
                    bounding_boxes.append(bbox)
                
        except Exception as e:
            print(f"An error occurred: {e}")
    
        return bounding_boxes
            
    def draw_bounding_boxes(image_path, entities, textract_df):
        image = cv2.imread(image_path)
        image_size = image.shape[1], image.shape[0]
    
        font = cv2.FONT_HERSHEY_SIMPLEX
        font_scale = 0.5
        font_color = (0, 0, 0)
        line_type = 2
        padding = 5  # Padding between the start of bbox and text
    
        for category, details in entities.items():
            for entity, value in details.items():
                if value:
                    bounding_boxes = find_entity_bounding_boxes(value, textract_df, image_size)
                    try:
                        for bbox in bounding_boxes:
                            x, y, w, h = map(int, bbox)
                            cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)
                            text_position = (x, y - padding) if y - padding > 10 else (x, y + h + 20)
                            cv2.putText(image, entity, text_position, font, font_scale, font_color, line_type)
                    except Exception as e:
                        print(f"Failed to draw bbox for {entity}: {e}")
        
        cv2.imwrite('/home/ritik1s/Desktop/bbox_issues/temp_GPT/annotated_invoice.jpg', image)
    
    IMAGE_PATH = '/path/to/your/image.jpeg'
    CSV_PATH = "/path/to/your/data.csv"
    JSON_PATH = "/path/to/your/entities.json"
    
    # Read Textract output and JSON entities
    textract_df = read_textract_output(CSV_PATH)
    entities = read_json_entities(JSON_PATH)
    
    # Draw bounding boxes
    draw_bounding_boxes(IMAGE_PATH, entities, textract_df)
    

    Key Enhancements:

    • Clustering: The most critical enhancement is utilizing DBSCAN clustering to group tokens that are spatially close to one another into a single entity. Adjust eps and min_samples parameters according to your documents' text distribution.
    • Minimizing false matches: Continued use of thefuzz to make best-effort matches but ensuring we do concatenation only with matched rows, preventing dataframe corruption.
    • Exception Handling: Better try-except usage for robust error handling, allowing the process to proceed even if a single entity's bounding box fails to generate.

    Always ensure you review the DBSCAN parameters (eps and min_samples) and adjust them according to your specific dataset to optimize its effectiveness.

  3. User

    cluster multitoken entity in one seprate separate bounding boxes

  4. Assistant

    To modify the approach for clustering multi-token entities into their separate bounding boxes, it's essential to refine how we handle the spatial relationships between words. The goal here is to ensure that even when we have multi-token entities, each token grouped together based on their spatial proximity forms a single bounding box around the entire phrase or entity. This involves slightly tweaking our previous strategy to ensure that the entire entity is captured as one bounding box rather than multiple.

    Below is an updated version of the relevant part of your code:

    def find_entity_bounding_boxes(entity_text, textract_df, image_size):
        entity_tokens = entity_text.split()
        results = pd.DataFrame()
        
        for token in entity_tokens:
            # Find a match in the textract output 
            choices = textract_df["text"].dropna().tolist()
            best_match, score = process.extractOne(token, choices, scorer=fuzz.token_sort_ratio)
            if score > 70:
                best_match_rows = textract_df[textract_df["text"] == best_match]
                results = pd.concat([results, best_match_rows], ignore_index=True)
        
        bounding_box = None
        try:
            if not results.empty:
                # Calculating the bounding box for the entire entity phrase
                image_width, image_height = results.iloc[0]["image_width"], results.iloc[0]["image_height"]
                scale_x = image_size[0] / image_width
                scale_y = image_size[1] / image_height
    
                min_left = np.min(results["left"])
                min_top = np.min(results["top"])
                max_right = np.max(results["left"] + results["width"])
                max_bottom = np.max(results["top"] + results["height"])
                
                bbox = (min_left * scale_x, min_top * scale_y, (max_right - min_left) * scale_x, (max_bottom - min_top) * scale_y)
                bounding_box = [bbox]  # Since we now want a single bounding box for the entity, wrap it in a list.
    
        except Exception as e:
            print(f"An error occurred: {e}")
    
        return bounding_box  # Return a single bounding box or None
    

    This version directly focuses on the requirement to cluster multi-token entities into a single bounding box. It does so by concatenating the best_match_rows for each token found by the fuzzy matching process and calculating the overall bounding box around all matches that are above the similarity score threshold.

    Important Notes:

    • Accuracy vs. Coverage: The confidence threshold (here score > 70) is critical. Adjusting it may be necessary to find the balance between false positives and misses.
    • Spatial Clustering: In this approach, spatial clustering (like DBSCAN previously suggested) is not explicitly required for the aggregation into a single bounding box, as this aggregation inherently assumes the entity tokens are near each other. However, in documents where tokens of the same entity might be spread apart (for example, with line breaks), a more complex spatial analysis might still be needed.
    • Scaling Factors: Calculation of scaling factors (scale_x and scale_y) ensures that the bounding box correctly maps onto the actual image dimensions. Ensure that your Textract output provides the necessary dimensions for accuracy.

    Remember to thoroughly test this approach with your specific documents, as fine-tuning (especially around fuzzy matching thresholds and handling varied document layouts) may be required to achieve optimal results.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.