Skip to content
TheChatArchiveThe Chat Archive
gpt-4-1106-preview· openaiHugging Face Suzhen/CodeChat

write a python script that's able to find a template picture in another picture even if…

  1. User

    write a python script that's able to find a template picture in another picture even if it's a little bit different angled

  2. Assistant

    To find a template picture within another picture, even if it's slightly rotated or angled, we can make use of the feature matching technique in computer vision. OpenCV is a popular library in Python that has the tools necessary to accomplish this task.

    We'll be using the SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF) algorithm to detect the keypoints and compute the descriptors. Then we'll use a matcher such as FLANN (Fast Library for Approximate Nearest Neighbors) or BFMatcher (Brute-Force Matcher) to match these features and find the best matches to locate the template in the image.

    Here's a sample script that does this using OpenCV:

    import cv2
    import numpy as np
    
    # Read the template and the image where the template will be searched
    template_image_path = 'template.jpg'
    search_image_path = 'search_image.jpg'
    template = cv2.imread(template_image_path, 0)  # Reading in grayscale
    search_image = cv2.imread(search_image_path, 0)  # Reading in grayscale
    
    # Initiate SIFT detector (or use ORB if SIFT is not available)
    sift = cv2.SIFT_create()
    # sift = cv2.ORB_create()
    
    # Find the keypoints and descriptors with SIFT or ORB
    keypoints_template, descriptors_template = sift.detectAndCompute(template, None)
    keypoints_search_image, descriptors_search_image = sift.detectAndCompute(search_image, None)
    
    # Create the matcher object
    # For SIFT or SURF, which are float descriptors, use FLANN with the following parameters
    FLANN_INDEX_KDTREE = 1
    index_params = dict(algorithm=FLANN_INDEX_KDTREE, trees=5)
    search_params = dict(checks=50)
    
    flann = cv2.FlannBasedMatcher(index_params, search_params)
    matches = flann.knnMatch(descriptors_template, descriptors_search_image, k=2)
    
    # Lowe's ratio test to filter good matches
    good_matches = []
    for m, n in matches:
        if m.distance < 0.75 * n.distance:
            good_matches.append(m)
    
    if len(good_matches) > 10:  # Arbitrary number, can adjust based on the use case
        # Extract location of good matches
        points_template = np.float32([keypoints_template[m.queryIdx].pt for m in good_matches])
        points_search_image = np.float32([keypoints_search_image[m.trainIdx].pt for m in good_matches])
    
        # Find homography
        matrix, mask = cv2.findHomography(points_template, points_search_image, cv2.RANSAC, 5.0)
        matches_mask = mask.ravel().tolist()
    
        # Get the dimensions of the template image for perspective transform
        h, w = template.shape
        points = np.float32([[0, 0], [0, h], [w, h], [w, 0]]).reshape(-1, 1, 2)
    
        # Apply perspective transform to the points
        transformed_points = cv2.perspectiveTransform(points, matrix)
        search_image_color = cv2.cvtColor(search_image, cv2.COLOR_GRAY2BGR)
        cv2.polylines(search_image_color, [np.int32(transformed_points)], True, (0, 255, 0), 3, cv2.LINE_AA)
        
        cv2.imshow('Detected', search_image_color)
        cv2.waitKey(0)
        cv2.destroyAllWindows()
    else:
        print("Not enough matches are found - {}/{}".format(len(good_matches), 10))
    
    

    Please note:

    1. You might need to install OpenCV first to run this script:

      pip install opencv-python
      pip install opencv-contrib-python
      
    2. SIFT and SURF are patented algorithms and are included in opencv-contrib-python. If you can't use them due to licensing issues, consider using ORB or other available feature detectors and descriptors.

    3. The parameters in SIFT/SURF/ORB and matcher can be tuned for your specific images and needs.

    4. The threshold used in Lowe's ratio test (0.75 in this case) is a standard heuristic, but it can be adjusted according to your requirements. The number '10' is an arbitrarily chosen threshold for the number of matches required to consider the object found – you might need a different value.

    5. The quality of matches and the result will significantly depend on the size, resolution, and angle of rotation of the template with respect to the target. In the real-world scenario, feature matching might need to be combined with other techniques for robustness.

  3. User

    crop the detectect window

  4. Assistant

    Certainly! We can add a few lines to the previous script to crop out the detected template area from the original (search) image and then save or display it.

    import cv2
    import numpy as np
    
    # Read the template and the image where the template will be searched
    template_image_path = 'template.jpg'
    search_image_path = 'search_image.jpg'
    template = cv2.imread(template_image_path, 0)  # Reading in grayscale
    search_image = cv2.imread(search_image_path, 0)  # Reading in grayscale
    
    # Initialize SIFT detector
    sift = cv2.SIFT_create()
    
    # Find the keypoints and descriptors with SIFT
    keypoints_template, descriptors_template = sift.detectAndCompute(template, None)
    keypoints_search_image, descriptors_search_image = sift.detectAndCompute(search_image, None)
    
    # FLANN parameters
    FLANN_INDEX_KDTREE = 1
    index_params = dict(algorithm=FLANN_INDEX_KDTREE, trees=5)
    search_params = dict(checks=50)
    
    flann = cv2.FlannBasedMatcher(index_params, search_params)
    matches = flann.knnMatch(descriptors_template, descriptors_search_image, k=2)
    
    # Lowe's ratio test
    good_matches = []
    for m, n in matches:
        if m.distance < 0.75 * n.distance:
            good_matches.append(m)
    
    if len(good_matches) > 10:
        points_template = np.float32([keypoints_template[m.queryIdx].pt for m in good_matches])
        points_search_image = np.float32([keypoints_search_image[m.trainIdx].pt for m in good_matches])
    
        # Find homography to get a proper perspective of the object
        matrix, mask = cv2.findHomography(points_template, points_search_image, cv2.RANSAC, 5.0)
    
        # Get the dimensions of the template image
        h, w = template.shape
        points = np.float32([[0, 0], [0, h], [w, h], [w, 0]]).reshape(-1, 1, 2)
    
        # Perform perspective transform
        transformed_points = cv2.perspectiveTransform(points, matrix)
        transformed_points = np.int32(transformed_points)
    
        # Crop detected window from the search image
        # We define a mask with the same dimensions as the search image, filled with zeros (black).
        mask = np.zeros(search_image.shape, dtype=np.uint8)
    
        # We then fill the mask with white color for the area of the detected window.
        channel_count = search_image.shape[2] if len(search_image.shape) > 2 else 1
        ignore_mask_color = (255,) * channel_count
        cv2.fillConvexPoly(mask, transformed_points, ignore_mask_color)
    
        # We apply the mask to the search image to get the window.
        cropped_image = cv2.bitwise_and(search_image, search_image, mask=mask)
    
        # Now, we need to find the bounding rectangle for the window to crop to.
        x, y, w, h = cv2.boundingRect(transformed_points)
        cropped_window = cropped_image[y:y+h, x:x+w]
    
        # We've now got the cropped window, we can save or display it:
        cv2.imwrite('cropped_window.jpg', cropped_window)
        cv2.imshow('Cropped Window', cropped_window)
        cv2.waitKey(0)
        cv2.destroyAllWindows()
    else:
        print("Not enough matches are found - {}/{}".format(len(good_matches), 10))
    

    This script will create a cropped image around the detected template within the search image. The fillConvexPoly function creates a mask based on the transformed points (which define the template's perspective in the search image), and then bitwise_and is used to extract only the parts of the image that are within the mask. We use boundingRect to find the minimal up-right bounding rectangle for the set of points (the detected window) so we can easily crop this particular region out of the search image. The cropped area is then saved to a file called cropped_window.jpg and displayed.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.