I have a code to map bounding boxes from json to image using ocr textract output which…
- User
I have a code to map bounding boxes from json to image using ocr textract output which in csv format. But I am strugging to get correct bounding boxes for multi token entities. Multi token entity should be in one bounding box. I want some good clustering approch to address this issue. And a better solution to match the entity token from dataframe.
Please modify my code according to my requirements. Code should avoid taking values from anywhere in dataframe. It should always take the nearest word of other token in case of multi token entity. Code should be robust.
import json import pandas as pd import cv2 from thefuzz import fuzz from thefuzz import process import numpy as np
def read_textract_output(csv_path): return pd.read_csv(csv_path)
def read_json_entities(json_path): with open(json_path, 'r') as file: return json.load(file)
def find_entity_bounding_boxes(entity_text, textract_df, image_size): entity_tokens = entity_text.split() results = pd.DataFrame()
for token in entity_tokens: choices = textract_df["text"].dropna().tolist() best_match, score = process.extractOne(token, choices, scorer=fuzz.token_sort_ratio) if score > 70: best_matches_df = textract_df[textract_df["text"] == best_match] results = pd.concat([results, best_matches_df]) bounding_boxes = [] try: # Sort by line and word number to cluster words that are on the same line sorted_results = results.sort_values(by=["line_num", "word_num"]) # Group the resulting bounding boxes by line number grouped_results = sorted_results.groupby("line_num") for _, group in grouped_results: # Calculate scaling factors based on image size and original image dimensions image_width, image_height = group.iloc[0]["image_width"], group.iloc[0]["image_height"] scale_x = image_size[0] / image_width scale_y = image_size[1] / image_height # Calculate the bounding box for the whole line min_left = np.min(group["left"]) min_top = np.min(group["top"]) max_right = np.max(group["left"] + group["width"]) max_bottom = np.max(group["top"] + group["height"]) bbox = (min_left * scale_x, min_top * scale_y, (max_right - min_left) * scale_x, (max_bottom - min_top) * scale_y) bounding_boxes.append(bbox) except Exception as e: print(f"An error occurred: {e}") return bounding_boxesdef draw_bounding_boxes(image_path, entities, textract_df):
image = cv2.imread(image_path)
image_size = image.shape[1], image.shape[0]
for category, details in entities.items():
if category == "invoice_details" or category == "Payment Details" or category == "amounts_and_tax":
for entity, value in details.items():
if value:
bounding_boxes = find_entity_bounding_boxes(value, textract_df, image_size)
try:
for bbox in bounding_boxes:
x, y, w, h = map(int, bbox)
cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)
except:
pass
else:
pass
def draw_bounding_boxes(image_path, entities, textract_df): image = cv2.imread(image_path) image_size = image.shape[1], image.shape[0]
font = cv2.FONT_HERSHEY_SIMPLEX font_scale = 0.5 font_color = (0, 0, 0) line_type = 2 padding = 5 # Padding between the start of bbox and text for category, details in entities.items(): if category == "invoice_details" or category == "Payment Details" or category == "amounts_and_tax": for entity, value in details.items(): if value: bounding_boxes = find_entity_bounding_boxes(value, textract_df, image_size) try: for bbox in bounding_boxes: x, y, w, h = map(int, bbox) cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2) # Text placement calculation text_position = (x, y - padding) if y - padding > 10 else (x, y + h + 20) # Drawing the text cv2.putText(image, entity, text_position, font, font_scale, font_color, line_type) except: pass else: pass cv2.imwrite('/home/ritik1s/Desktop/bbox_issues/temp_GPT/annotated_invoice.jpg', image)IMAGE_PATH = '/home/ritik1s/Desktop/bbox_issues/temp_GPT/check.jpeg' CSV_PATH = "/home/ritik1s/Desktop/bbox_issues/temp_GPT/check.csv" JSON_PATH = "/home/ritik1s/Desktop/bbox_issues/temp_GPT/row_skip.json"
Read Textract output and JSON entities
textract_df = read_textract_output(CSV_PATH) entities = read_json_entities(JSON_PATH)
Draw the bounding boxes
draw_bounding_boxes(IMAGE_PATH, entities, textract_df)
- Assistant
Enhancing your code to better handle multi-token entities and ensure that bounding boxes are accurate and robust involves several steps. The following version of your code implements a more sophisticated approach to find and group bounding boxes for multi-token entities. It primarily focuses on clustering tokens that are spatially close to each other, using their position on the page:
import json import pandas as pd import cv2 from thefuzz import fuzz from thefuzz import process import numpy as np from sklearn.cluster import DBSCAN def read_textract_output(csv_path): return pd.read_csv(csv_path) def read_json_entities(json_path): with open(json_path, 'r') as file: return json.load(file) def find_entity_bounding_boxes(entity_text, textract_df, image_size): entity_tokens = entity_text.split() results = pd.DataFrame() # Update approach to minimize false matches by prioritizing nearest tokens for token in entity_tokens: choices = textract_df["text"].dropna().tolist() best_match, score = process.extractOne(token, choices, scorer=fuzz.token_sort_ratio) if score > 70: best_match_rows = textract_df[textract_df["text"] == best_match] results = pd.concat([results, best_match_rows], ignore_index=True) bounding_boxes = [] try: if not results.empty: # Utilizing DBSCAN to cluster word bounding boxes based on spatial proximity. # Epsilon and min_samples need to be adjusted based on the average spacing and size of text in your documents. # dbscan = DBSCAN(eps=30, min_samples=1, metric='euclidean') dbscan = DBSCAN(eps=50, min_samples=1, metric='euclidean') # Adjust params as needed. # Use the centroids of bounding boxes for clustering results["centroid_x"] = results["left"] + results["width"] / 2 results["centroid_y"] = results["top"] + results["height"] / 2 clusters = dbscan.fit_predict(results[["centroid_x", "centroid_y"]]) results['cluster'] = clusters unique_clusters = results['cluster'].unique() for cluster in unique_clusters: cluster_group = results[results['cluster'] == cluster] # Continue with your scaling and bbox grouping as before image_width, image_height = cluster_group.iloc[0]["image_width"], cluster_group.iloc[0]["image_height"] scale_x = image_size[0] / image_width scale_y = image_size[1] / image_height min_left = np.min(cluster_group["left"]) min_top = np.min(cluster_group["top"]) max_right = np.max(cluster_group["left"] + cluster_group["width"]) max_bottom = np.max(cluster_group["top"] + cluster_group["height"]) bbox = (min_left * scale_x, min_top * scale_y, (max_right - min_left) * scale_x, (max_bottom - min_top) * scale_y) bounding_boxes.append(bbox) except Exception as e: print(f"An error occurred: {e}") return bounding_boxes def draw_bounding_boxes(image_path, entities, textract_df): image = cv2.imread(image_path) image_size = image.shape[1], image.shape[0] font = cv2.FONT_HERSHEY_SIMPLEX font_scale = 0.5 font_color = (0, 0, 0) line_type = 2 padding = 5 # Padding between the start of bbox and text for category, details in entities.items(): for entity, value in details.items(): if value: bounding_boxes = find_entity_bounding_boxes(value, textract_df, image_size) try: for bbox in bounding_boxes: x, y, w, h = map(int, bbox) cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2) text_position = (x, y - padding) if y - padding > 10 else (x, y + h + 20) cv2.putText(image, entity, text_position, font, font_scale, font_color, line_type) except Exception as e: print(f"Failed to draw bbox for {entity}: {e}") cv2.imwrite('/home/ritik1s/Desktop/bbox_issues/temp_GPT/annotated_invoice.jpg', image) IMAGE_PATH = '/path/to/your/image.jpeg' CSV_PATH = "/path/to/your/data.csv" JSON_PATH = "/path/to/your/entities.json" # Read Textract output and JSON entities textract_df = read_textract_output(CSV_PATH) entities = read_json_entities(JSON_PATH) # Draw bounding boxes draw_bounding_boxes(IMAGE_PATH, entities, textract_df)Key Enhancements:
- Clustering: The most critical enhancement is utilizing
DBSCANclustering to group tokens that are spatially close to one another into a single entity. Adjustepsandmin_samplesparameters according to your documents' text distribution. - Minimizing false matches: Continued use of
thefuzzto make best-effort matches but ensuring we do concatenation only with matched rows, preventing dataframe corruption. - Exception Handling: Better try-except usage for robust error handling, allowing the process to proceed even if a single entity's bounding box fails to generate.
Always ensure you review the DBSCAN parameters (
epsandmin_samples) and adjust them according to your specific dataset to optimize its effectiveness. - Clustering: The most critical enhancement is utilizing
- User
cluster multitoken entity in one seprate separate bounding boxes
- Assistant
To modify the approach for clustering multi-token entities into their separate bounding boxes, it's essential to refine how we handle the spatial relationships between words. The goal here is to ensure that even when we have multi-token entities, each token grouped together based on their spatial proximity forms a single bounding box around the entire phrase or entity. This involves slightly tweaking our previous strategy to ensure that the entire entity is captured as one bounding box rather than multiple.
Below is an updated version of the relevant part of your code:
def find_entity_bounding_boxes(entity_text, textract_df, image_size): entity_tokens = entity_text.split() results = pd.DataFrame() for token in entity_tokens: # Find a match in the textract output choices = textract_df["text"].dropna().tolist() best_match, score = process.extractOne(token, choices, scorer=fuzz.token_sort_ratio) if score > 70: best_match_rows = textract_df[textract_df["text"] == best_match] results = pd.concat([results, best_match_rows], ignore_index=True) bounding_box = None try: if not results.empty: # Calculating the bounding box for the entire entity phrase image_width, image_height = results.iloc[0]["image_width"], results.iloc[0]["image_height"] scale_x = image_size[0] / image_width scale_y = image_size[1] / image_height min_left = np.min(results["left"]) min_top = np.min(results["top"]) max_right = np.max(results["left"] + results["width"]) max_bottom = np.max(results["top"] + results["height"]) bbox = (min_left * scale_x, min_top * scale_y, (max_right - min_left) * scale_x, (max_bottom - min_top) * scale_y) bounding_box = [bbox] # Since we now want a single bounding box for the entity, wrap it in a list. except Exception as e: print(f"An error occurred: {e}") return bounding_box # Return a single bounding box or NoneThis version directly focuses on the requirement to cluster multi-token entities into a single bounding box. It does so by concatenating the
best_match_rowsfor each token found by the fuzzy matching process and calculating the overall bounding box around all matches that are above the similarity score threshold.Important Notes:
- Accuracy vs. Coverage: The confidence threshold (here
score > 70) is critical. Adjusting it may be necessary to find the balance between false positives and misses. - Spatial Clustering: In this approach, spatial clustering (like DBSCAN previously suggested) is not explicitly required for the aggregation into a single bounding box, as this aggregation inherently assumes the entity tokens are near each other. However, in documents where tokens of the same entity might be spread apart (for example, with line breaks), a more complex spatial analysis might still be needed.
- Scaling Factors: Calculation of scaling factors (
scale_xandscale_y) ensures that the bounding box correctly maps onto the actual image dimensions. Ensure that your Textract output provides the necessary dimensions for accuracy.
Remember to thoroughly test this approach with your specific documents, as fine-tuning (especially around fuzzy matching thresholds and handling varied document layouts) may be required to achieve optimal results.
- Accuracy vs. Coverage: The confidence threshold (here
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.