I have a working warp function, however I also want to be able to see the warps that…
- User
I have a working warp function, however I also want to be able to see the warps that happen to negative region. What I mean is after I apply the matrix, the region of interest gets placed to a fixed spot and aligned to left. However I know there is more content that falls to the left (negative region) because I can see the non processed image too. I want a solution where I will be able to get data of left part of the image, possibly extends the image or align the warped image to center so the left part is visible or move the warped image to the right so left part fits the warped image. Here is my code: "def warp_perspective(img, H, reverse=False): if reverse: H = np.linalg.inv(H) h, w = img.shape[0], img.shape[1] test = np.array([[0, w, w, 0], [0, 0, h, h], [1, 1, 1, 1]]) test_c = np.dot(H, test) test_w = test_c[0]/test_c[2] test_h = test_c[1]/test_c[2] print("Test W: ", test_w) print("Test H: ", test_h) max_y, min_y = test_w.max(), test_w.min() max_x, min_x = test_h.max(), test_h.min() print("x: ", max_x, min_x, " = ", max_x - min_x) print("y: ", max_y, min_y, " = ", max_y - min_y) w, h = int(max_y + np.abs(min_y)), int(max_x + np.abs(min_x))
target_y, target_x = np.meshgrid(np.arange(0, h), np.arange(0, w)) target_coordinates = np.stack([target_x.ravel(), target_y.ravel(), np.ones(target_x.size)]) source_coordinates = np.dot(np.linalg.inv(H), target_coordinates) source_coordinates /= source_coordinates[2, :] valid = np.logical_and(np.logical_and(0 <= source_coordinates[0, :], source_coordinates[0, :] < img.shape[1]), np.logical_and(0 <= source_coordinates[1, :], source_coordinates[1, :] < img.shape[0])) valid_source_coordinates = source_coordinates[:, valid].astype(int) valid_target_coordinates = target_coordinates[:, valid].astype(int) valid_source_coordinates[0] = np.clip(valid_source_coordinates[0], -100000, img.shape[1] - 1) valid_source_coordinates[1] = np.clip(valid_source_coordinates[1], -100000, img.shape[0] - 1) #valid_target_coordinates[0] -= int(min_x) #valid_target_coordinates[1] -= int(min_y) warped_image = np.zeros((h, w, 3), dtype=np.uint8) for i in range(3): warped_image[:,:, i][valid_target_coordinates[1], valid_target_coordinates[0]] = img[:,:, i][valid_source_coordinates[1], valid_source_coordinates[0]] display_image(img) display_image(warped_image) print(warped_image.shape) return warped_image" - Assistant
To ensure that the warped image is displayed entirely and ensure that the left part fits in the warped image, you can adjust the target coordinates after computing the new height and width of the warped image.
You can accomplish this by adding the absolute value of the minimum values in test_h and test_w to the valid_target_coordinates. This will ensure that the image is translated according to the left and up parts in the warp.
Replace this part of your code:
#valid_target_coordinates[0] -= int(min_x) #valid_target_coordinates[1] -= int(min_y)With the following modification:
valid_target_coordinates[0] += int(abs(min_y)) valid_target_coordinates[1] += int(abs(min_x))Here's the complete modified function:
def warp_perspective(img, H, reverse=False): if reverse: H = np.linalg.inv(H) h, w = img.shape[0], img.shape[1] test = np.array([[0, w, w, 0], [0, 0, h, h], [1, 1, 1, 1]]) test_c = np.dot(H, test) test_w = test_c[0]/test_c[2] test_h = test_c[1]/test_c[2] print("Test W: ", test_w) print("Test H: ", test_h) max_y, min_y = test_w.max(), test_w.min() max_x, min_x = test_h.max(), test_h.min() print("x: ", max_x, min_x, " = ", max_x - min_x) print("y: ", max_y, min_y, " = ", max_y - min_y) w, h = int(max_y + np.abs(min_y)), int(max_x + np.abs(min_x)) target_y, target_x = np.meshgrid(np.arange(0, h), np.arange(0, w)) target_coordinates = np.stack([target_x.ravel(), target_y.ravel(), np.ones(target_x.size)]) source_coordinates = np.dot(np.linalg.inv(H), target_coordinates) source_coordinates /= source_coordinates[2, :] valid = np.logical_and(np.logical_and(0 <= source_coordinates[0, :], source_coordinates[0, :] < img.shape[1]), np.logical_and(0 <= source_coordinates[1, :], source_coordinates[1, :] < img.shape[0])) valid_source_coordinates = source_coordinates[:, valid].astype(int) valid_target_coordinates = target_coordinates[:, valid].astype(int) valid_source_coordinates[0] = np.clip(valid_source_coordinates[0], -100000, img.shape[1] - 1) valid_source_coordinates[1] = np.clip(valid_source_coordinates[1], -100000, img.shape[0] - 1) valid_target_coordinates[0] += int(abs(min_y)) valid_target_coordinates[1] += int(abs(min_x)) warped_image = np.zeros((h, w, 3), dtype=np.uint8) for i in range(3): warped_image[:,:, i][valid_target_coordinates[1], valid_target_coordinates[0]] = img[:,:, i][valid_source_coordinates[1], valid_source_coordinates[0]] display_image(img) display_image(warped_image) print(warped_image.shape) return warped_imageNow the entire warped image should be visible, including the left part that was previously in the negative region.
- User
okay I had to do couple other fixes but got it working. So now that I have the full image warped, I have the next issue. My image merging code was taking warped images and adding them on top of each other. Since the region of interest was almost at the top most part of the image it was aligning perfectly. However in order to see negative regions and rest of the data, we also shifted the region of interest. So my image merging code, when it adds images on top of one another as they are, now they do not align. I need a solution which will account for the warped images shift and when constructing panorama, aligns them on top of one another. We can calculate where the region of interest simply by looking at the min_x and min_y value as they are used for shifting. However another side note is the panorama image at the end result should also show the negative regions. I will give you my full code, so look in to a solution please: "import numpy as np import cv2 import os import glob import matplotlib.pyplot as plt import time from kayla_tools import *
def compute_homography_matrix(src_pts, dst_pts): def normalize_points(pts): pts_homogeneous = np.hstack((pts, np.ones((pts.shape[0], 1)))) centroid = np.mean(pts, axis=0) scale = np.sqrt(2) / np.mean(np.linalg.norm(pts - centroid, axis=1)) T = np.array([[scale, 0, -scale * centroid[0]], [0, scale, -scale * centroid[1]], [0, 0, 1]]) normalized_pts = (T @ pts_homogeneous.T).T return normalized_pts[:, :2], T
src_pts_normalized, T1 = normalize_points(src_pts) dst_pts_normalized, T2 = normalize_points(dst_pts) A = [] for p1, p2 in zip(src_pts_normalized, dst_pts_normalized): x1, y1 = p1 x2, y2 = p2 A.append([0, 0, 0, -x1, -y1, -1, y2 * x1, y2 * y1, y2]) A.append([x1, y1, 1, 0, 0, 0, -x2 * x1, -x2 * y1, -x2]) A = np.array(A) try: _, _, VT = np.linalg.svd(A) except np.linalg.LinAlgError: return None h = VT[-1] H_normalized = h.reshape(3, 3) H = np.linalg.inv(T2) @ H_normalized @ T1 if np.abs(H[-1, -1]) > 1e-6: H = H / H[-1, -1] else: return None return Hdef filter_matches(matches, ratio_thres=0.7): filtered_matches = [] for match in matches: good_match = [] for m, n in match: if m.distance < ratio_thres * n.distance: good_match.append(m) filtered_matches.append(good_match)
return filtered_matchesdef find_homography(keypoints, filtered_matches): homographies = [] skipped_indices = [] # Keep track of skipped images and their indices
for i, matches in enumerate(filtered_matches): src_pts = np.float32([keypoints[0][m.queryIdx].pt for m in matches]).reshape(-1, 1, 2) dst_pts = np.float32([keypoints[i + 1][m.trainIdx].pt for m in matches]).reshape(-1, 1, 2) H = ransac_homography(src_pts, dst_pts) if H is not None: H = H.astype(np.float32) homographies.append(H) else: print(f"Warning: Homography computation failed for image pair (0, {i + 1}). Skipping.") skipped_indices.append(i + 1) # Add indices of skipped images to the list continue return homographies, skipped_indicesdef ransac_homography(src_pts, dst_pts, iterations=2000, threshold=3): best_inlier_count = 0 best_homography = None
if len(src_pts) != len(dst_pts) or len(src_pts) < 4: raise ValueError("The number of source and destination points must be equal and at least 4.") src_pts = np.array(src_pts) dst_pts = np.array(dst_pts) for _ in range(iterations): indices = np.random.choice(len(src_pts), 4, replace=False) src_subset = src_pts[indices].reshape(-1, 2) dst_subset = dst_pts[indices].reshape(-1, 2) homography = compute_homography_matrix(src_subset, dst_subset) if homography is None: continue inliers = 0 for i in range(len(src_pts)): projected_point = np.dot(homography, np.append(src_pts[i], 1)) if np.abs(projected_point[-1]) > 1e-6: projected_point = projected_point / projected_point[-1] else: continue distance = np.linalg.norm(projected_point[:2] - dst_pts[i]) if distance < threshold: inliers += 1 if inliers > best_inlier_count: best_inlier_count = inliers best_homography = homography if best_homography is None: raise RuntimeError("Failed to find a valid homography matrix.") return best_homographydef read_ground_truth_homographies(dataset_path): H_files = sorted(glob.glob(os.path.join(dataset_path, "H_*"))) ground_truth_homographies = []
for filename in H_files: H = np.loadtxt(filename) ground_truth_homographies.append(H) return ground_truth_homographiesdef warp_perspective(img, H, reverse=False): if reverse: H = np.linalg.inv(H) h, w = img.shape[0], img.shape[1] test = np.array([[0, w, w, 0], [0, 0, h, h], [1, 1, 1, 1]]) test_c = np.dot(H, test) test_w = test_c[0]/test_c[2] test_h = test_c[1]/test_c[2] print("Test W: ", test_w) print("Test H: ", test_h) max_y, min_y = test_w.max(), test_w.min() max_x, min_x = test_h.max(), test_h.min() print("x: ", max_x, min_x, " = ", max_x - min_x) print("y: ", max_y, min_y, " = ", max_y - min_y) w, h = int(max_y + np.abs(min_y)), int(max_x + np.abs(min_x))
target_y, target_x = np.meshgrid(np.arange(min_x, h), np.arange(min_y, w)) target_coordinates = np.stack([target_x.ravel(), target_y.ravel(), np.ones(target_x.size)]) source_coordinates = np.dot(np.linalg.inv(H), target_coordinates) source_coordinates /= source_coordinates[2, :] valid = np.logical_and(np.logical_and(0 <= source_coordinates[0, :], source_coordinates[0, :] < img.shape[1]), np.logical_and(0 <= source_coordinates[1, :], source_coordinates[1, :] < img.shape[0])) valid_source_coordinates = source_coordinates[:, valid].astype(int) valid_target_coordinates = target_coordinates[:, valid].astype(int) valid_source_coordinates[0] = np.clip(valid_source_coordinates[0], -100000, img.shape[1] - 1) valid_source_coordinates[1] = np.clip(valid_source_coordinates[1], -100000, img.shape[0] - 1) valid_target_coordinates[0] += int(abs(min_y)) valid_target_coordinates[1] += int(abs(min_x)) valid_target_coordinates[0] = np.clip(valid_target_coordinates[0], -100000, w - 1) valid_target_coordinates[1] = np.clip(valid_target_coordinates[1], -100000, h - 1) warped_image = np.zeros((h, w, 3), dtype=np.uint8) for i in range(3): warped_image[:,:, i][valid_target_coordinates[1], valid_target_coordinates[0]] = img[:,:, i][valid_source_coordinates[1], valid_source_coordinates[0]] display_image(img) display_image(warped_image) print(warped_image.shape) return warped_imagedef image_merging(images, homographies): if len(images) - 1 != len(homographies): raise ValueError("The number of homography matrices must be one less than the number of images.")
# Warp all images except the first one using their corresponding homography matrices warped_images = [images[0]] + [warp_perspective(img, H, True) for img, H in zip(images[1:], homographies)] # Compute the size of the merged image min_x, min_y, max_x, max_y = 0, 0, 0, 0 for img in warped_images: h, w = img.shape[:2] corners = np.array([[0, 0, 1], [h, 0, 1], [0, w, 1], [h, w, 1]]) min_x = min(min_x, corners[:, 1].min()) min_y = min(min_y, corners[:, 0].min()) max_x = max(max_x, corners[:, 1].max()) max_y = max(max_y, corners[:, 0].max()) merged_height = int(np.ceil(max_y - min_y)) merged_width = int(np.ceil(max_x - min_x)) # Initialize the merged image with zeros merged_image = np.zeros((merged_height, merged_width, 3), dtype=np.uint8) print("Merged Shape: ", merged_image.shape) # Merge the warped images by overlaying them on the merged image for img in warped_images: mask = np.all(img == 0, axis=-1) y_offset, x_offset = max(0, -min_y), max(0, -min_x) merged_image_slice = merged_image[y_offset:y_offset + img.shape[0], x_offset:x_offset + img.shape[1]] merged_image_slice[~mask] = img[~mask] print("Merged Shape 2: ", merged_image.shape) return merged_image" - Assistant
To account for the shift in the warped images and align them correctly when constructing the panorama, you can modify the
image_mergingfunction. You can calculate the offsets for each warped image based on the min_x and min_y values and apply these offsets when merging the images.Here's the modified
image_mergingfunction:def image_merging(images, homographies, min_xy_values): if len(images) - 1 != len(homographies) or len(images) - 1 != len(min_xy_values): raise ValueError("The number of homography matrices and min_xy_values must be one less than the number of images.") # Warp all images except the first one using their corresponding homography matrices warped_images = [images[0]] + [warp_perspective(img, H, True) for img, H in zip(images[1:], homographies)] # Compute the size of the merged image min_x, min_y, max_x, max_y = 0, 0, 0, 0 for img in warped_images: h, w = img.shape[:2] corners = np.array([[0, 0, 1], [h, 0, 1], [0, w, 1], [h, w, 1]]) min_x = min(min_x, corners[:, 1].min()) min_y = min(min_y, corners[:, 0].min()) max_x = max(max_x, corners[:, 1].max()) max_y = max(max_y, corners[:, 0].max()) merged_height = int(np.ceil(max_y - min_y)) merged_width = int(np.ceil(max_x - min_x)) # Initialize the merged image with zeros merged_image = np.zeros((merged_height, merged_width, 3), dtype=np.uint8) print("Merged Shape: ", merged_image.shape) # Merge the warped images by overlaying them on the merged image for i, (img, min_xy) in enumerate(zip(warped_images, [np.array([0, 0])] + min_xy_values)): min_x, min_y = min_xy mask = np.all(img == 0, axis=-1) y_offset, x_offset = max(0, -min_y), max(0, -min_x) merged_image_slice = merged_image[y_offset:y_offset + img.shape[0], x_offset:x_offset + img.shape[1]] merged_image_slice[~mask] = img[~mask] print("Merged Shape 2: ", merged_image.shape) return merged_imageNow, you need to store the min_x and min_y values for each warped image and pass them to the
image_mergingfunction. Modify thewarp_perspectivefunction to return the min_x and min_y values along with the warped image:def warp_perspective(img, H, reverse=False): # ... (rest of the code) return warped_image, np.array([min_x, min_y])Finally, update the part where you call the
warp_perspectivefunction and store the min_x and min_y values:# Warp all images except the first one using their corresponding homography matrices warped_images = [images[0]] min_xy_values = [] for img, H in zip(images[1:], homographies): warped_img, min_xy = warp_perspective(img, H, True) warped_images.append(warped_img) min_xy_values.append(min_xy)Now, when you call the
image_mergingfunction, pass themin_xy_valuesas well:merged_image = image_merging(images, homographies, min_xy_values)This should align the images correctly in the panorama and show the negative regions as well.