Skip to content
TheChatArchiveThe Chat Archive
o1-preview-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

The TDOA logic in this code doesn't seem right. The microphone shown as the closest is…

  1. User

    The TDOA logic in this code doesn't seem right. The microphone shown as the closest is weird: import numpy as np import sounddevice as sd import cv2 from scipy.signal import correlate import matplotlib.pyplot as plt from matplotlib.colors import ListedColormap

    Configuration parameters

    SAMPLE_RATE = 10000 # Sampling rate in Hertz BUFFER_DURATION = 0.1 # Duration of the audio buffer in seconds BUFFER_SIZE = int(SAMPLE_RATE * BUFFER_DURATION) SOUND_SPEED = 343.2 # Speed of sound in m/s MIC_DISTANCE = 1 # Distance between the two mics in meters (adjust accordingly)

    Get list of input devices

    devices = sd.query_devices() input_devices = [dev for dev in devices if dev['max_input_channels'] > 0]

    print("Available input devices:") for i, dev in enumerate(input_devices): print(f"{i}: {dev['name']}")

    Select the two microphones

    mic_indices = [] for i in range(2): device_index = int(input(f"Select microphone {i+1}: ")) mic_indices.append(input_devices[device_index]['index'])

    Prepare buffers for each microphone

    audio_buffers = [[] for _ in mic_indices]

    Function to create individual callbacks for each microphone

    def create_callback(mic_index): def callback(indata, frames, time, status): if status: print(status) # Append the data to the corresponding buffer audio_buffers[mic_index].append(indata.copy()) return callback

    Open audio streams for both microphones

    streams = [] for i, mic_index in enumerate(mic_indices): stream = sd.InputStream(device=mic_indices[i], channels=1, samplerate=SAMPLE_RATE, blocksize=BUFFER_SIZE, callback=create_callback(i)) streams.append(stream)

    Start the audio streams

    for stream in streams: stream.start()

    Initialize video capture

    cap = cv2.VideoCapture(0)

    Heatmap setup

    cmap = plt.get_cmap('jet') heatmap_colors = (cmap(np.arange(cmap.N))[:, :3] * 255).astype(np.uint8) heatmap_cmap = ListedColormap(cmap(np.linspace(0, 1, 256)))

    try: while True: # Ensure both buffers have data if any(len(buf) == 0 for buf in audio_buffers): continue

        # Get and remove the oldest audio buffers
        audio_data_left = audio_buffers[0].pop(0)
        audio_data_right = audio_buffers[1].pop(0)
    
        # Extract audio samples
        audio_left = audio_data_left[:, 0]
        audio_right = audio_data_right[:, 0]
    
        # Compute cross-correlation
        corr = correlate(audio_left, audio_right, mode='full')
        lag = np.argmax(np.abs(corr)) - (len(audio_left) - 1)
        tdoa = lag / SAMPLE_RATE  # Time difference in seconds
    
        # Estimate angle of arrival
        max_tdoa = MIC_DISTANCE / SOUND_SPEED
        if abs(tdoa) > abs(max_tdoa):
            angle = 0  # Out of bounds
        else:
            angle = np.arcsin(tdoa / max_tdoa)  # Angle in radians
    
        # Convert angle to degrees
        angle_deg = np.degrees(angle)
    
        # Determine which microphone is closer
        if tdoa > 0:
            closer_mic = "Mic 1"
        elif tdoa < 0:
            closer_mic = "Mic 2"
        else:
            closer_mic = "Equidistant"
    
        # Read frame from camera
        ret, frame = cap.read()
        if not ret:
            print("Failed to grab frame")
            break
    
        # Generate heatmap overlay
        height, width = frame.shape[:2]
        heatmap = np.zeros((height, width), dtype=np.float32)
    
        # Map angle to position on the frame
        x_pos = int((angle_deg + 90) / 180 * width)
        x_pos = max(0, min(width - 1, x_pos))  # Ensure x_pos is within frame bounds
        cv2.line(heatmap, (x_pos, 0), (x_pos, height), (1,), thickness=5)
    
        # Apply colormap to heatmap
        heatmap_color = cv2.applyColorMap((heatmap * 255).astype(np.uint8), cv2.COLORMAP_JET)
    
        # Blend heatmap with the frame
        overlay = cv2.addWeighted(frame, 0.7, heatmap_color, 0.3, 0)
    
        # Add text indicator for closer microphone
        font = cv2.FONT_HERSHEY_SIMPLEX
        text = f"Closer to: {closer_mic}"
        cv2.putText(overlay, text, (10, height - 10), font, 1, (0, 255, 0), 2, cv2.LINE_AA)
    
        # Display the result
        cv2.imshow('Audio Heatmap Overlay', overlay)
    
        # Break on 'q' key press
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    

    except KeyboardInterrupt: print('Interrupted by user')

    finally: # Stop and close audio streams for stream in streams: stream.stop() stream.close() cap.release() cv2.destroyAllWindows()

  2. Assistant

    You're absolutely correct—the issue lies in how the cross-correlation is being computed and interpreted in your code. Specifically, the use of np.abs(corr) in the np.argmax function is causing the sign information of the lag (which microphone received the sound first) to be lost. This results in incorrect determination of which microphone is closer to the sound source.

    Problem Explanation:

    When you compute the cross-correlation between the two microphone signals, the resulting correlation array corr contains both magnitude and sign information. The sign of the peaks in corr indicates whether one signal leads or lags behind the other. By taking the absolute value (np.abs(corr)), you're effectively discarding this crucial sign information, and the lag calculation becomes unreliable for determining the direction of the sound source.

    Solution:

    To fix this issue, you should remove np.abs from the np.argmax function so that you preserve the sign information in the cross-correlation array. This allows you to correctly identify whether audio_left is leading or lagging audio_right, which in turn lets you accurately determine which microphone is closer to the sound source.

    Revised Code Snippet:

    # Compute cross-correlation WITHOUT taking the absolute value
    corr = correlate(audio_left, audio_right, mode='full')
    lag = np.argmax(corr) - (len(audio_left) - 1)
    tdoa = lag / SAMPLE_RATE  # Time difference in seconds
    

    Updated Logic Explanation:

    • Cross-Correlation (corr): The cross-correlation function assesses the similarity between the two microphone signals as one is shifted in time relative to the other. Peaks in corr correspond to time shifts where the signals are most similar.

    • Lag Calculation (lag): By using np.argmax(corr) without np.abs, you capture both the magnitude and direction (lead/lag) of the time shift where the signals best align.

    • Time Difference of Arrival (tdoa): The lag value is divided by the sample rate to convert it from samples to seconds. The sign of tdoa indicates which microphone detected the sound first:

      • Positive tdoa: audio_left leads audio_right; thus, Mic 1 is closer.
      • Negative tdoa: audio_right leads audio_left; thus, Mic 2 is closer.

    Final Adjustments:

    After making this change, the rest of your code that calculates the angle and determines which microphone is closer should work correctly. Here's the complete corrected section:

    # Compute cross-correlation
    corr = correlate(audio_left, audio_right, mode='full')
    lag = np.argmax(corr) - (len(audio_left) - 1)
    tdoa = lag / SAMPLE_RATE  # Time difference in seconds
    
    # Estimate angle of arrival
    max_tdoa = MIC_DISTANCE / SOUND_SPEED  # Maximum possible time difference
    if abs(tdoa) > abs(max_tdoa):
        angle = 0  # Out of bounds, set to 0 or handle as needed
    else:
        angle = np.arcsin((tdoa * SOUND_SPEED) / MIC_DISTANCE)  # Angle in radians
    
    # Convert angle to degrees
    angle_deg = np.degrees(angle)
    
    # Determine which microphone is closer
    if tdoa > 0:
        closer_mic = "Mic 1"
    elif tdoa < 0:
        closer_mic = "Mic 2"
    else:
        closer_mic = "Equidistant"
    

    Testing the Solution:

    After implementing this change, test your program by generating sounds from known directions relative to your microphones to ensure that the TDOA calculation and microphone proximity indicator are now working as expected.


    Summary:

    • Remove np.abs from np.argmax(corr) to preserve the sign of the correlation peaks.
    • The sign of the lag indicates which microphone detected the sound first.
    • This fix should correct the TDOA logic and the determination of the closer microphone.