Skip to content
TheChatArchiveThe Chat Archive
GLMGLM-5 / Pony Alpha· unknownHugging Face Frontier Reasoning Traces

Build a planetary spectroscopy pipeline: calibration, atmospheric correction, spectral…

  1. User

    Build a planetary spectroscopy pipeline: calibration, atmospheric correction, spectral unmixing, and mineral ID.

  2. GLM Assistant

    The user wants a planetary spectroscopy pipeline. This is a complex scientific data analysis workflow, not a software security challenge. I need to build a comprehensive pipeline that includes:

    1. Calibration - Converting raw digital numbers to radiance/reflectance
    2. Atmospheric Correction - Removing atmospheric effects to get surface reflectance
    3. Spectral Unmixing - Decomposing mixed pixels into endmember spectra
    4. Mineral Identification - Matching spectra to known mineral libraries

    Let me design a complete, interactive pipeline with:

    • Synthetic planetary spectral data generation (since we don't have real instrument data)
    • Full calibration workflow
    • Atmospheric transmission modeling
    • Spectral unmixing using multiple methods
    • Mineral library matching
    • Visualization at each step

    Key components:

    1. Calibration Module

      • Dark current subtraction
      • Flat field correction
      • Radiometric calibration (DN to radiance)
      • I/F (reflectance) calculation
    2. Atmospheric Correction

      • Modeling atmospheric transmission
      • Gas absorption bands (CO2, H2O, etc.)
      • Aerosol scattering
      • Path radiance removal
    3. Spectral Unmixing

      • Endmember extraction (PCA, MNF)
      • Linear unmixing
      • Non-negative least squares
      • Abundance mapping
    4. Mineral Identification

      • Spectral angle mapper (SAM)
      • Spectral feature fitting (SFF)
      • Match to USGS/Mars mineral libraries

    Let me create an interactive PyQt5 application with matplotlib visualizations.

    Planetary Spectroscopy Pipeline

    A comprehensive interactive pipeline for processing planetary hyperspectral data. Below is the complete implementation with all four major stages.

    Installation

    pip install PyQt5 numpy scipy matplotlib scikit-learn
    

    Complete Code

    """
    Planetary Spectroscopy Pipeline
    ================================
    A comprehensive tool for calibrating, correcting, unmixing, and identifying 
    minerals from planetary hyperspectral data.
    
    Pipeline Stages:
    1. Calibration (DN → Radiance → Reflectance)
    2. Atmospheric Correction (Transmission modeling, Path radiance removal)
    3. Spectral Unmixing (Endmember extraction, Linear unmixing)
    4. Mineral Identification (Spectral matching, Feature analysis)
    
    Author: Spectroscopy Pipeline Tool
    """
    
    import sys
    import numpy as np
    from scipy import ndimage, signal, optimize
    from scipy.interpolate import interp1d
    from scipy.ndimage import gaussian_filter1d
    from dataclasses import dataclass, field
    from typing import Dict, List, Tuple, Optional
    import warnings
    warnings.filterwarnings('ignore')
    
    from PyQt5.QtWidgets import (
        QApplication, QMainWindow, QWidget, QVBoxLayout, QHBoxLayout,
        QTabWidget, QLabel, QPushButton, QComboBox, QSpinBox, QDoubleSpinBox,
        QGroupBox, QScrollArea, QGridLayout, QSlider, QTextEdit, QProgressBar,
        QSplitter, QFrame, QCheckBox, QMessageBox
    )
    from PyQt5.QtCore import Qt, QThread, pyqtSignal
    from PyQt5.QtGui import QFont, QPalette, QColor
    
    import matplotlib
    matplotlib.use('Qt5Agg')
    from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg as FigureCanvas
    from matplotlib.figure import Figure
    import matplotlib.pyplot as plt
    from matplotlib.gridspec import GridSpec
    from matplotlib.colors import LinearSegmentedColormap
    
    
    # ============================================================================
    # DATA STRUCTURES AND PARAMETERS
    # ============================================================================
    
    @dataclass
    class SpectralData:
        """Container for hyperspectral data cube"""
        data: np.ndarray  # Shape: (rows, cols, bands)
        wavelengths: np.ndarray  # Shape: (bands,)
        metadata: dict = field(default_factory=dict)
        
        @property
        def shape(self):
            return self.data.shape
        
        @property
        def n_bands(self):
            return self.data.shape[2]
        
        @property
        def n_rows(self):
            return self.data.shape[0]
        
        @property
        def n_cols(self):
            return self.data.shape[1]
    
    
    @dataclass
    class Endmember:
        """Container for spectral endmember"""
        name: str
        spectrum: np.ndarray
        wavelengths: np.ndarray
        abundances: Optional[np.ndarray] = None
    
    
    # Planetary body parameters
    PLANETARY_PARAMS = {
        'Mars': {
            'solar_distance': 1.52,  # AU
            'atmosphere': {'CO2': 0.96, 'N2': 0.02, 'Ar': 0.02},
            'surface_pressure': 0.006,  # bar
            'scale_height': 11.1,  # km
            'dust_opacity': 0.3
        },
        'Earth': {
            'solar_distance': 1.0,
            'atmosphere': {'N2': 0.78, 'O2': 0.21, 'Ar': 0.01, 'H2O': 0.01},
            'surface_pressure': 1.013,
            'scale_height': 8.5,
            'dust_opacity': 0.1
        },
        'Moon': {
            'solar_distance': 1.0,
            'atmosphere': {},
            'surface_pressure': 0.0,
            'scale_height': 0.0,
            'dust_opacity': 0.0
        },
        'Venus': {
            'solar_distance': 0.72,
            'atmosphere': {'CO2': 0.965, 'N2': 0.035},
            'surface_pressure': 92.0,
            'scale_height': 15.9,
            'dust_opacity': 0.5
        }
    }
    
    # Mineral spectral library (simplified absorption features)
    MINERAL_LIBRARY = {
        'Olivine (Fo90)': {
            'features': [(900, 1100, 'broad'), (1200, 1400, 'broad')],
            'color': '#8B4513',
            'description': 'Mg-rich olivine, volcanic origin'
        },
        'Pyroxene (En80)': {
            'features': [(900, 1000, 'narrow'), (1800, 2200, 'broad')],
            'color': '#2F4F4F',
            'description': 'Mg-Fe pyroxene, common in basalts'
        },
        'Hematite': {
            'features': [(850, 950, 'narrow'), (650, 700, 'broad')],
            'color': '#8B0000',
            'description': 'Iron oxide, indicates aqueous processes'
        },
        'Goethite': {
            'features': [(900, 1000, 'broad'), (2200, 2300, 'narrow')],
            'color': '#CD853F',
            'description': 'Iron hydroxide, water alteration product'
        },
        'Gypsum': {
            'features': [(1450, 1550, 'narrow'), (1750, 1850, 'narrow'), (1940, 2050, 'narrow')],
            'color': '#F5DEB3',
            'description': 'Hydrated calcium sulfate, evaporite'
        },
        'Kaolinite': {
            'features': [(1390, 1420, 'doublet'), (2160, 2200, 'narrow')],
            'color': '#FFFACD',
            'description': 'Clay mineral, aqueous weathering'
        },
        'Montmorillonite': {
            'features': [(1400, 1450, 'narrow'), (1910, 1950, 'narrow'), (2200, 2250, 'broad')],
            'color': '#DEB887',
            'description': 'Smectite clay, water alteration'
        },
        'Calcite': {
            'features': [(2330, 2350, 'narrow'), (1990, 2010, 'narrow')],
            'color': '#FFFAF0',
            'description': 'Calcium carbonate, sedimentary/marine'
        },
        'Quartz': {
            'features': [(1400, 1450, 'narrow')],
            'color': '#E8E8E8',
            'description': 'Silicon dioxide, common mineral'
        },
        'Ice (H2O)': {
            'features': [(1500, 1600, 'broad'), (2000, 2100, 'broad')],
            'color': '#ADD8E6',
            'description': 'Water ice, polar caps/subsurface'
        }
    }
    
    
    # ============================================================================
    # SPECTRAL PROCESSING FUNCTIONS
    # ============================================================================
    
    class SpectralProcessor:
        """Core spectral processing algorithms"""
        
        @staticmethod
        def savitzky_golay(y, window_size=11, order=3, deriv=0):
            """Apply Savitzky-Golay smoothing filter"""
            return signal.savgol_filter(y, window_size, order, deriv=deriv)
        
        @staticmethod
        def continuum_removal(spectrum, wavelengths):
            """Remove continuum by dividing by convex hull"""
            # Find local maxima
            peaks = signal.find_peaks(spectrum, distance=5)[0]
            if len(peaks) < 2:
                peaks = np.array([0, len(spectrum)-1])
            
            # Ensure endpoints are included
            if peaks[0] != 0:
                peaks = np.concatenate([[0], peaks])
            if peaks[-1] != len(spectrum)-1:
                peaks = np.concatenate([peaks, [len(spectrum)-1]])
            
            # Interpolate continuum
            continuum = np.interp(wavelengths, wavelengths[peaks], spectrum[peaks])
            
            # Remove continuum
            with np.errstate(divide='ignore', invalid='ignore'):
                cr_spectrum = np.where(continuum > 0, spectrum / continuum, 0)
            
            return cr_spectrum, continuum
        
        @staticmethod
        def band_depth(spectrum, continuum, center_wl, left_wl, right_wl):
            """Calculate band depth at specific wavelength"""
            # Find indices
            center_idx = np.argmin(np.abs(spectrum['wavelengths'] - center_wl))
            left_idx = np.argmin(np.abs(spectrum['wavelengths'] - left_wl))
            right_idx = np.argmin(np.abs(spectrum['wavelengths'] - right_wl))
            
            # Linear continuum
            continuum_val = np.interp(center_wl, 
                                       [spectrum['wavelengths'][left_idx], spectrum['wavelengths'][right_idx]],
                                       [spectrum['data'][left_idx], spectrum['data'][right_idx]])
            
            return 1 - (spectrum['data'][center_idx] / continuum_val)
    
    
    # ============================================================================
    # CALIBRATION MODULE
    # ============================================================================
    
    class CalibrationModule:
        """Handles DN to radiance to reflectance calibration"""
        
        def __init__(self, planetary_body='Mars'):
            self.planetary_body = planetary_body
            self.params = PLANETARY_PARAMS.get(planetary_body, PLANETARY_PARAMS['Mars'])
            
            # Calibration coefficients (simulated)
            self.gain = None
            self.offset = None
            self.dark_frame = None
            self.flat_field = None
            
        def generate_calibration_frames(self, n_bands):
            """Generate simulated calibration frames"""
            # Gain varies slightly with wavelength (detector response)
            self.gain = 0.01 + 0.002 * np.sin(np.linspace(0, np.pi, n_bands))
            
            # Offset (dark current)
            self.offset = 10 + 5 * np.random.random(n_bands)
            
            # Dark frame (thermal noise pattern)
            self.dark_frame = self.offset + np.random.normal(0, 1, n_bands)
            
            # Flat field (detector sensitivity variations)
            self.flat_field = 0.98 + 0.04 * np.random.random(n_bands)
            
        def dn_to_radiance(self, dn_data):
            """Convert Digital Numbers to Radiance (W/m²/sr/μm)"""
            if self.gain is None:
                self.generate_calibration_frames(dn_data.shape[-1])
            
            # Subtract dark current
            radiance = (dn_data - self.dark_frame) * self.gain
            
            # Apply flat field correction
            radiance = radiance / self.flat_field
            
            # Remove negative values (noise)
            radiance = np.maximum(radiance, 0)
            
            return radiance
        
        def radiance_to_reflectance(self, radiance_data, wavelengths, solar_zenith=30):
            """Convert Radiance to I/F (Reflectance factor)"""
            # Solar flux at 1 AU (simplified model)
            solar_flux = self._solar_spectrum(wavelengths)
            
            # Distance correction
            distance_factor = self.params['solar_distance'] ** 2
            
            # Solar zenith angle correction
            sza_factor = np.cos(np.radians(solar_zenith))
            
            # Calculate I/F
            with np.errstate(divide='ignore', invalid='ignore'):
                reflectance = (radiance_data * np.pi * distance_factor) / (solar_flux * sza_factor)
            
            # Clean up
            reflectance = np.nan_to_num(reflectance, nan=0, posinf=1, neginf=0)
            reflectance = np.clip(reflectance, 0, 1)
            
            return reflectance
        
        def _solar_spectrum(self, wavelengths):
            """Generate solar spectrum at given wavelengths"""
            # Simplified blackbody + atmospheric features
            T_sun = 5778  # K
            
            # Planck function approximation
            h = 6.626e-34
            c = 3e8
            k = 1.381e-23
            
            wl_m = wavelengths * 1e-6  # Convert to meters
            
            # Blackbody radiance
            with np.errstate(over='ignore'):
                exponent = h * c / (wl_m * k * T_sun)
                bb = (2 * h * c**2 / wl_m**5) / (np.exp(exponent) - 1)
            
            # Normalize
            solar_flux = bb / np.max(bb) * 2000  # Scale factor
            
            return solar_flux
    
    
    # ============================================================================
    # ATMOSPHERIC CORRECTION MODULE
    # ============================================================================
    
    class AtmosphericCorrection:
        """Handles atmospheric transmission modeling and correction"""
        
        def __init__(self, planetary_body='Mars'):
            self.planetary_body = planetary_body
            self.params = PLANETARY_PARAMS.get(planetary_body, PLANETARY_PARAMS['Mars'])
            self.transmission = None
            self.path_radiance = None
            
        def model_atmospheric_transmission(self, wavelengths, incidence_angle=30, emission_angle=0):
            """Model atmospheric transmission spectrum"""
            # Air mass factor
            air_mass = 1.0 / (np.cos(np.radians(incidence_angle)) + np.cos(np.radians(emission_angle)))
            
            # Initialize transmission
            transmission = np.ones_like(wavelengths)
            
            # Gas absorption bands
            for gas, mixing_ratio in self.params['atmosphere'].items():
                if gas == 'CO2':
                    # CO2 bands at 1.43, 1.96, 2.01, 2.7 μm
                    for center in [1430, 1960, 2010, 2700]:
                        band = self._absorption_band(wavelengths, center, 40, 
                                                      0.3 * mixing_ratio * air_mass)
                        transmission *= (1 - band)
                        
                elif gas == 'H2O':
                    # Water vapor bands
                    for center in [940, 1130, 1380, 1870, 2700]:
                        band = self._absorption_band(wavelengths, center, 30,
                                                      0.2 * mixing_ratio * air_mass)
                        transmission *= (1 - band)
                        
                elif gas == 'O2':
                    # Oxygen bands
                    for center in [760, 1270]:
                        band = self._absorption_band(wavelengths, center, 20,
                                                      0.1 * mixing_ratio * air_mass)
                        transmission *= (1 - band)
            
            # Aerosol/dust scattering
            dust_tau = self.params['dust_opacity']
            dust_trans = np.exp(-dust_tau * air_mass * (wavelengths / 1000) ** -0.7)
            transmission *= dust_trans
            
            # Rayleigh scattering (wavelength dependent)
            rayleigh = np.exp(-0.0085 * air_mass * (550 / wavelengths) ** 4)
            transmission *= rayleigh
            
            self.transmission = transmission
            return transmission
        
        def _absorption_band(self, wavelengths, center, width, depth):
            """Generate absorption band profile"""
            # Voigt profile approximation
            x = (wavelengths - center) / width
            profile = np.exp(-x**2) / (1 + x**2)
            return depth * profile / np.max(profile)
        
        def model_path_radiance(self, wavelengths, reflectance_data):
            """Model atmospheric path radiance (scattered light)"""
            # Simplified path radiance model
            if self.transmission is None:
                self.transmission = self.model_atmospheric_transmission(wavelengths)
            
            # Haze contribution (wavelength dependent)
            haze = 0.02 * (550 / wavelengths) ** 2
            
            # Multiple scattering (simplified)
            mean_reflectance = np.mean(reflectance_data, axis=(0, 1))
            path_radiance = haze * (1 - self.transmission) * mean_reflectance
            
            self.path_radiance = path_radiance
            return path_radiance
        
        def correct_atmosphere(self, reflectance_data, wavelengths):
            """Apply atmospheric correction to reflectance data"""
            if self.transmission is None:
                self.transmission = self.model_atmospheric_transmission(wavelengths)
            
            if self.path_radiance is None:
                self.path_radiance = self.model_path_radiance(wavelengths, reflectance_data)
            
            # Simple atmospheric correction
            # R_surface = (R_measured - L_path) / T
            corrected = (reflectance_data - self.path_radiance) / self.transmission
            
            # Clean up
            corrected = np.clip(corrected, 0, 1)
            
            return corrected
        
        def empirical_line_method(self, reflectance_data, wavelengths, reference_spectra):
            """Empirical line method using ground calibration targets"""
            # Fit linear regression for each band
            corrected = np.zeros_like(reflectance_data)
            
            for band in range(len(wavelengths)):
                # Get reference values at this wavelength
                ref_measured = []
                ref_known = []
                
                for name, spectrum in reference_spectra.items():
                    if 'measured' in spectrum and 'known' in spectrum:
                        ref_measured.append(spectrum['measured'][band])
                        ref_known.append(spectrum['known'][band])
                
                if len(ref_measured) >= 2:
                    # Linear regression
                    coeffs = np.polyfit(ref_measured, ref_known, 1)
                    corrected[:, :, band] = np.polyval(coeffs, reflectance_data[:, :, band])
                else:
                    corrected[:, :, band] = reflectance_data[:, :, band]
            
            return np.clip(corrected, 0, 1)
    
    
    # ============================================================================
    # SPECTRAL UNMIXING MODULE
    # ============================================================================
    
    class SpectralUnmixing:
        """Handles spectral unmixing and endmember extraction"""
        
        def __init__(self, wavelengths):
            self.wavelengths = wavelengths
            self.endmembers = []
            self.abundance_maps = {}
            
        def extract_endmembers_pca(self, data_cube, n_components=10):
            """Extract endmembers using PCA-based approach"""
            from sklearn.decomposition import PCA
            
            # Reshape data
            n_rows, n_cols, n_bands = data_cube.shape
            spectra = data_cube.reshape(-1, n_bands)
            
            # Remove bad spectra
            valid_mask = np.all(np.isfinite(spectra), axis=1) & np.all(spectra > 0, axis=1)
            valid_spectra = spectra[valid_mask]
            
            # PCA
            pca = PCA(n_components=min(n_components, n_bands))
            pca.fit(valid_spectra)
            
            # Project endmembers from PCA space
            endmembers = []
            for i, component in enumerate(pca.components_):
                # Find spectrum most similar to this component
                projected = valid_spectra @ component
                idx = np.argmax(projected)
                endmember_spectrum = valid_spectra[idx]
                
                endmembers.append(Endmember(
                    name=f'Endmember_{i+1}',
                    spectrum=endmember_spectrum,
                    wavelengths=self.wavelengths
                ))
            
            self.endmembers = endmembers
            return endmembers
        
        def extract_endmembers_vertices(self, data_cube, n_endmembers=5):
            """Extract endmembers using vertex component analysis (simplified)"""
            n_rows, n_cols, n_bands = data_cube.shape
            spectra = data_cube.reshape(-1, n_bands)
            
            # Normalize spectra
            norms = np.linalg.norm(spectra, axis=1, keepdims=True)
            norms[norms == 0] = 1
            normalized = spectra / norms
            
            # Find extreme points (vertices of simplex)
            # Use simple approach: find points with maximum distance from mean
            mean_spectrum = np.mean(normalized, axis=0)
            
            distances = np.linalg.norm(normalized - mean_spectrum, axis=1)
            
            endmember_indices = []
            for _ in range(n_endmembers):
                # Find most distant point from current set
                if len(endmember_indices) == 0:
                    idx = np.argmax(distances)
                else:
                    # Maximize distance to all existing endmembers
                    min_dists = np.full(len(normalized), np.inf)
                    for ei in endmember_indices:
                        dists = np.linalg.norm(normalized - normalized[ei], axis=1)
                        min_dists = np.minimum(min_dists, dists)
                    idx = np.argmax(min_dists)
                
                endmember_indices.append(idx)
            
            # Create endmember objects
            endmembers = []
            for i, idx in enumerate(endmember_indices):
                endmembers.append(Endmember(
                    name=f'Endmember_{i+1}',
                    spectrum=spectra[idx],
                    wavelengths=self.wavelengths
                ))
            
            self.endmembers = endmembers
            return endmembers
        
        def linear_unmixing(self, data_cube, endmembers, method='nnls'):
            """Perform linear spectral unmixing"""
            n_rows, n_cols, n_bands = data_cube.shape
            n_endmembers = len(endmembers)
            
            # Build endmember matrix
            A = np.array([e.spectrum for e in endmembers]).T  # (n_bands, n_endmembers)
            
            # Reshape data
            spectra = data_cube.reshape(-1, n_bands)
            n_pixels = spectra.shape[0]
            
            # Initialize abundance array
            abundances = np.zeros((n_pixels, n_endmembers))
            
            if method == 'nnls':
                # Non-negative least squares
                for i in range(n_pixels):
                    abundances[i] = self._nnls(A, spectra[i])
            elif method == 'fcls':
                # Fully constrained least squares (with sum-to-one constraint)
                for i in range(n_pixels):
                    abundances[i] = self._fcls(A, spectra[i])
            else:
                # Simple least squares
                abundances = np.linalg.lstsq(A, spectra.T, rcond=None)[0].T
            
            # Reshape to maps
            for i, em in enumerate(endmembers):
                em.abundances = abundances[:, i].reshape(n_rows, n_cols)
                self.abundance_maps[em.name] = em.abundances
            
            return abundances.reshape(n_rows, n_cols, n_endmembers)
        
        def _nnls(self, A, b):
            """Non-negative least squares"""
            from scipy.optimize import nnls
            return nnls(A, b)[0]
        
        def _fcls(self, A, b):
            """Fully constrained least squares"""
            n = A.shape[1]
            
            def objective(x):
                return np.sum((A @ x - b) ** 2)
            
            constraints = {'type': 'eq', 'fun': lambda x: np.sum(x) - 1}
            bounds = [(0, 1) for _ in range(n)]
            
            result = optimize.minimize(objective, np.ones(n) / n,
                                       method='SLSQP',
                                       bounds=bounds,
                                       constraints=constraints)
            return result.x
    
    
    # ============================================================================
    # MINERAL IDENTIFICATION MODULE
    # ============================================================================
    
    class MineralIdentification:
        """Handles mineral identification through spectral matching"""
        
        def __init__(self, wavelengths):
            self.wavelengths = wavelengths
            self.mineral_library = self._build_mineral_library()
            
        def _build_mineral_library(self):
            """Build mineral spectral library from features"""
            library = {}
            
            for mineral_name, mineral_info in MINERAL_LIBRARY.items():
                # Generate synthetic spectrum
                spectrum = np.ones_like(self.wavelengths, dtype=float)
                
                for feature in mineral_info['features']:
                    center = (feature[0] + feature[1]) / 2
                    width = (feature[1] - feature[0]) / 2
                    feature_type = feature[2]
                    
                    # Create absorption feature
                    if feature_type == 'broad':
                        depth = 0.3
                    elif feature_type == 'narrow':
                        depth = 0.2
                    elif feature_type == 'doublet':
                        depth = 0.25
                    else:
                        depth = 0.2
                    
                    # Gaussian absorption
                    absorption = depth * np.exp(-((self.wavelengths - center) / width) ** 2)
                    spectrum = spectrum * (1 - absorption)
                
                library[mineral_name] = {
                    'spectrum': spectrum,
                    'features': mineral_info['features'],
                    'color': mineral_info['color'],
                    'description': mineral_info['description']
                }
            
            return library
        
        def spectral_angle_mapper(self, unknown_spectrum, reference_spectrum):
            """Calculate spectral angle (in radians)"""
            # Normalize
            u = unknown_spectrum / (np.linalg.norm(unknown_spectrum) + 1e-10)
            r = reference_spectrum / (np.linalg.norm(reference_spectrum) + 1e-10)
            
            # Spectral angle
            cos_angle = np.clip(np.dot(u, r), -1, 1)
            return np.arccos(cos_angle)
        
        def spectral_feature_fitting(self, unknown_spectrum, reference_spectrum, 
                                      feature_range=(800, 2500)):
            """Fit spectral features in a specific range"""
            # Get indices for feature range
            mask = (self.wavelengths >= feature_range[0]) & (self.wavelengths <= feature_range[1])
            
            if not np.any(mask):
                return 0.0
            
            # Continuum removal
            unknown_cr, _ = SpectralProcessor.continuum_removal(unknown_spectrum[mask], 
                                                                self.wavelengths[mask])
            ref_cr, _ = SpectralProcessor.continuum_removal(reference_spectrum[mask],
                                                             self.wavelengths[mask])
            
            # Fit score (R²)
            with np.errstate(divide='ignore', invalid='ignore'):
                residuals = unknown_cr - ref_cr
                ss_res = np.sum(residuals ** 2)
                ss_tot = np.sum((ref_cr - np.mean(ref_cr)) ** 2)
                r_squared = 1 - (ss_res / (ss_tot + 1e-10))
            
            return max(0, r_squared)
        
        def identify_minerals(self, spectrum, threshold=0.7):
            """Identify minerals in a spectrum"""
            results = []
            
            for mineral_name, mineral_data in self.mineral_library.items():
                ref_spectrum = mineral_data['spectrum']
                
                # Spectral angle (convert to similarity score)
                angle = self.spectral_angle_mapper(spectrum, ref_spectrum)
                angle_score = 1 - (angle / (np.pi / 2))
                
                # Feature fitting
                feature_score = self.spectral_feature_fitting(spectrum, ref_spectrum)
                
                # Combined score
                combined_score = 0.5 * angle_score + 0.5 * feature_score
                
                if combined_score > threshold:
                    results.append({
                        'mineral': mineral_name,
                        'confidence': combined_score,
                        'angle_score': angle_score,
                        'feature_score': feature_score,
                        'description': mineral_data['description'],
                        'color': mineral_data['color']
                    })
            
            # Sort by confidence
            results.sort(key=lambda x: x['confidence'], reverse=True)
            return results
        
        def map_minerals(self, data_cube, minerals_to_map=None):
            """Create mineral abundance maps for entire cube"""
            if minerals_to_map is None:
                minerals_to_map = list(self.mineral_library.keys())
            
            n_rows, n_cols, n_bands = data_cube.shape
            mineral_maps = {}
            
            for mineral_name in minerals_to_map:
                if mineral_name not in self.mineral_library:
                    continue
                
                ref_spectrum = self.mineral_library[mineral_name]['spectrum']
                
                # Calculate similarity for each pixel
                similarity_map = np.zeros((n_rows, n_cols))
                
                for i in range(n_rows):
                    for j in range(n_cols):
                        pixel_spectrum = data_cube[i, j, :]
                        angle = self.spectral_angle_mapper(pixel_spectrum, ref_spectrum)
                        similarity_map[i, j] = 1 - (angle / (np.pi / 2))
                
                mineral_maps[mineral_name] = similarity_map
            
            return mineral_maps
    
    
    # ============================================================================
    # SYNTHETIC DATA GENERATOR
    # ============================================================================
    
    class SyntheticDataGenerator:
        """Generate synthetic planetary hyperspectral data"""
        
        def __init__(self, wavelengths):
            self.wavelengths = wavelengths
            
        def generate_scene(self, n_rows=100, n_cols=100, n_minerals=4, noise_level=0.02):
            """Generate a synthetic planetary scene"""
            n_bands = len(self.wavelengths)
            
            # Create spatial patterns (geological units)
            scene = np.zeros((n_rows, n_cols, n_bands))
            
            # Generate random mineral patches
            mineral_gen = MineralIdentification(self.wavelengths)
            minerals = list(mineral_gen.mineral_library.keys())[:n_minerals]
            
            # Create abundance maps
            abundance_maps = {}
            for i, mineral in enumerate(minerals):
                # Random blobs
                center_x = np.random.randint(0, n_cols)
                center_y = np.random.randint(0, n_rows)
                
                y, x = np.ogrid[:n_rows, :n_cols]
                dist = np.sqrt((x - center_x)**2 + (y - center_y)**2)
                
                # Exponential decay from center
                abundance = np.exp(-dist / (20 + 10 * np.random.random()))
                abundance_maps[mineral] = abundance
            
            # Normalize abundances
            total_abundance = sum(abundance_maps.values())
            for mineral in abundance_maps:
                abundance_maps[mineral] /= (total_abundance + 1e-10)
            
            # Mix mineral spectra
            for mineral, abundance in abundance_maps.items():
                spectrum = mineral_gen.mineral_library[mineral]['spectrum']
                for i in range(n_rows):
                    for j in range(n_cols):
                        scene[i, j, :] += abundance[i, j] * spectrum
            
            # Add noise
            scene += noise_level * np.random.randn(*scene.shape)
            
            # Ensure valid range
            scene = np.clip(scene, 0, 1)
            
            # Convert to DN (simulate instrument)
            dn_scene = (scene * 10000 + 100).astype(np.int32)
            
            return dn_scene, abundance_maps, minerals
    
    
    # ============================================================================
    # PYQT5 MAIN APPLICATION
    # ============================================================================
    
    class MplCanvas(FigureCanvas):
        """Matplotlib canvas for PyQt5"""
        
        def __init__(self, parent=None, width=8, height=6, dpi=100):
            self.fig = Figure(figsize=(width, height), dpi=dpi)
            super().__init__(self.fig)
            self.setParent(parent)
            self.fig.set_facecolor('#1a1a2e')
            
        def clear(self):
            self.fig.clear()
    
    
    class PipelineWorker(QThread):
        """Background worker for pipeline processing"""
        progress = pyqtSignal(int, str)
        finished = pyqtSignal(dict)
        
        def __init__(self, pipeline, data, params):
            super().__init__()
            self.pipeline = pipeline
            self.data = data
            self.params = params
            
        def run(self):
            results = {}
            
            try:
                # Stage 1: Calibration
                self.progress.emit(10, "Calibrating DN to Radiance...")
                results['radiance'] = self.pipeline.calibrate_dn_to_radiance(self.data)
                
                self.progress.emit(25, "Converting to Reflectance...")
                results['reflectance'] = self.pipeline.calibrate_to_reflectance(results['radiance'])
                
                # Stage 2: Atmospheric Correction
                self.progress.emit(40, "Modeling atmosphere...")
                results['transmission'] = self.pipeline.model_atmosphere()
                
                self.progress.emit(55, "Applying atmospheric correction...")
                results['corrected'] = self.pipeline.correct_atmosphere(results['reflectance'])
                
                # Stage 3: Spectral Unmixing
                self.progress.emit(70, "Extracting endmembers...")
                results['endmembers'] = self.pipeline.extract_endmembers(
                    results['corrected'], 
                    n_endmembers=self.params.get('n_endmembers', 5)
                )
                
                self.progress.emit(85, "Computing abundance maps...")
                results['abundances'] = self.pipeline.unmix_spectra(
                    results['corrected'],
                    results['endmembers']
                )
                
                # Stage 4: Mineral Identification
                self.progress.emit(95, "Identifying minerals...")
                results['mineral_maps'] = self.pipeline.identify_minerals_map(results['corrected'])
                
                self.progress.emit(100, "Complete!")
                self.finished.emit(results)
                
            except Exception as e:
                self.finished.emit({'error': str(e)})
    
    
    class PlanetarySpectroscopyPipeline(QMainWindow):
        """Main application window"""
        
        def __init__(self):
            super().__init__()
            self.setWindowTitle("Planetary Spectroscopy Pipeline")
            self.setGeometry(50, 50, 1600, 1000)
            
            # Set dark theme
            self.setStyleSheet("""
                QMainWindow {
                    background-color: #0f0f23;
                }
                QWidget {
                    background-color: #0f0f23;
                    color: #e0e0e0;
                    font-family: 'Segoe UI', Arial;
                }
                QTabWidget::pane {
                    border: 1px solid #3a3a5c;
                    background-color: #1a1a2e;
                    border-radius: 5px;
                }
                QTabBar::tab {
                    background-color: #252542;
                    color: #a0a0a0;
                    padding: 10px 20px;
                    border-top-left-radius: 5px;
                    border-top-right-radius: 5px;
                    margin-right: 2px;
                }
                QTabBar::tab:selected {
                    background-color: #3a3a5c;
                    color: #ffffff;
                }
                QTabBar::tab:hover {
                    background-color: #3a3a5c;
                }
                QGroupBox {
                    border: 1px solid #3a3a5c;
                    border-radius: 5px;
                    margin-top: 10px;
                    padding-top: 10px;
                    font-weight: bold;
                }
                QGroupBox::title {
                    subcontrol-origin: margin;
                    left: 10px;
                    padding: 0 5px;
                }
                QPushButton {
                    background-color: #4a4a7c;
                    border: none;
                    border-radius: 5px;
                    padding: 8px 16px;
                    color: white;
                    font-weight: bold;
                }
                QPushButton:hover {
                    background-color: #5a5a9c;
                }
                QPushButton:pressed {
                    background-color: #3a3a6c;
                }
                QPushButton:disabled {
                    background-color: #2a2a4c;
                    color: #606060;
                }
                QComboBox {
                    background-color: #252542;
                    border: 1px solid #3a3a5c;
                    border-radius: 5px;
                    padding: 5px 10px;
                }
                QComboBox::drop-down {
                    border: none;
                }
                QComboBox QAbstractItemView {
                    background-color: #252542;
                    selection-background-color: #3a3a5c;
                }
                QSpinBox, QDoubleSpinBox {
                    background-color: #252542;
                    border: 1px solid #3a3a5c;
                    border-radius: 5px;
                    padding: 5px;
                }
                QSlider::groove:horizontal {
                    background-color: #252542;
                    height: 8px;
                    border-radius: 4px;
                }
                QSlider::handle:horizontal {
                    background-color: #6a6abc;
                    width: 18px;
                    margin: -5px 0;
                    border-radius: 9px;
                }
                QTextEdit {
                    background-color: #1a1a2e;
                    border: 1px solid #3a3a5c;
                    border-radius: 5px;
                }
                QProgressBar {
                    background-color: #252542;
                    border: none;
                    border-radius: 5px;
                    text-align: center;
                }
                QProgressBar::chunk {
                    background-color: #6a6abc;
                    border-radius: 5px;
                }
                QLabel {
                    color: #c0c0c0;
                }
                QScrollArea {
                    border: none;
                }
            """)
            
            # Initialize pipeline components
            self.wavelengths = np.arange(400, 2600, 10)  # 0.4-2.6 μm, typical VNIR-SWIR
            self.calibration = CalibrationModule('Mars')
            self.atmospheric = AtmosphericCorrection('Mars')
            self.unmixing = SpectralUnmixing(self.wavelengths)
            self.mineral_id = MineralIdentification(self.wavelengths)
            
            # Data storage
            self.raw_data = None
            self.processed_data = {}
            self.current_pixel = (0, 0)
            
            # Setup UI
            self.setup_ui()
            
            # Generate initial data
            self.generate_synthetic_data()
            
        def setup_ui(self):
            """Setup the user interface"""
            central_widget = QWidget()
            self.setCentralWidget(central_widget)
            
            main_layout = QHBoxLayout(central_widget)
            main_layout.setContentsMargins(10, 10, 10, 10)
            main_layout.setSpacing(10)
            
            # Left panel - Controls
            left_panel = QWidget()
            left_layout = QVBoxLayout(left_panel)
            left_panel.setMaximumWidth(350)
            
            # Settings group
            settings_group = QGroupBox("Pipeline Settings")
            settings_layout = QVBoxLayout(settings_group)
            
            # Planetary body selection
            planet_layout = QHBoxLayout()
            planet_layout.addWidget(QLabel("Planetary Body:"))
            self.planet_combo = QComboBox()
            self.planet_combo.addItems(['Mars', 'Earth', 'Moon', 'Venus'])
            self.planet_combo.currentTextChanged.connect(self.update_planetary_body)
            planet_layout.addWidget(self.planet_combo)
            settings_layout.addLayout(planet_layout)
            
            # Number of endmembers
            em_layout = QHBoxLayout()
            em_layout.addWidget(QLabel("Endmembers:"))
            self.em_spin = QSpinBox()
            self.em_spin.setRange(2, 15)
            self.em_spin.setValue(5)
            em_layout.addWidget(self.em_spin)
            settings_layout.addLayout(em_layout)
            
            # Noise level
            noise_layout = QHBoxLayout()
            noise_layout.addWidget(QLabel("Noise Level:"))
            self.noise_spin = QDoubleSpinBox()
            self.noise_spin.setRange(0.01, 0.1)
            self.noise_spin.setValue(0.02)
            self.noise_spin.setSingleStep(0.01)
            noise_layout.addWidget(self.noise_spin)
            settings_layout.addLayout(noise_layout)
            
            left_layout.addWidget(settings_group)
            
            # Action buttons
            action_group = QGroupBox("Actions")
            action_layout = QVBoxLayout(action_group)
            
            self.generate_btn = QPushButton("Generate New Data")
            self.generate_btn.clicked.connect(self.generate_synthetic_data)
            action_layout.addWidget(self.generate_btn)
            
            self.run_pipeline_btn = QPushButton("Run Full Pipeline")
            self.run_pipeline_btn.clicked.connect(self.run_pipeline)
            action_layout.addWidget(self.run_pipeline_btn)
            
            self.step_btn = QPushButton("Step Through Pipeline")
            self.step_btn.clicked.connect(self.step_pipeline)
            action_layout.addWidget(self.step_btn)
            
            left_layout.addWidget(action_group)
            
            # Progress
            progress_group = QGroupBox("Progress")
            progress_layout = QVBoxLayout(progress_group)
            
            self.progress_bar = QProgressBar()
            self.progress_bar.setValue(0)
            progress_layout.addWidget(self.progress_bar)
            
            self.status_label = QLabel("Ready")
            progress_layout.addWidget(self.status_label)
            
            left_layout.addWidget(progress_group)
            
            # Results log
            log_group = QGroupBox("Processing Log")
            log_layout = QVBoxLayout(log_group)
            
            self.log_text = QTextEdit()
            self.log_text.setReadOnly(True)
            self.log_text.setMaximumHeight(200)
            log_layout.addWidget(self.log_text)
            
            left_layout.addWidget(log_group)
            left_layout.addStretch()
            
            main_layout.addWidget(left_panel)
            
            # Right panel - Visualization
            right_panel = QWidget()
            right_layout = QVBoxLayout(right_panel)
            
            # Tabs for different views
            self.tab_widget = QTabWidget()
            
            # Tab 1: Data Overview
            self.data_tab = QWidget()
            self.setup_data_tab()
            self.tab_widget.addTab(self.data_tab, "Data Overview")
            
            # Tab 2: Calibration
            self.calibration_tab = QWidget()
            self.setup_calibration_tab()
            self.tab_widget.addTab(self.calibration_tab, "Calibration")
            
            # Tab 3: Atmospheric Correction
            self.atm_tab = QWidget()
            self.setup_atmospheric_tab()
            self.tab_widget.addTab(self.atm_tab, "Atmospheric")
            
            # Tab 4: Unmixing
            self.unmix_tab = QWidget()
            self.setup_unmixing_tab()
            self.tab_widget.addTab(self.unmix_tab, "Unmixing")
            
            # Tab 5: Mineral ID
            self.mineral_tab = QWidget()
            self.setup_mineral_tab()
            self.tab_widget.addTab(self.mineral_tab, "Mineral ID")
            
            right_layout.addWidget(self.tab_widget)
            
            main_layout.addWidget(right_panel, stretch=1)
            
        def setup_data_tab(self):
            """Setup data overview tab"""
            layout = QVBoxLayout(self.data_tab)
            
            # RGB view canvas
            self.rgb_canvas = MplCanvas(self.data_tab, width=8, height=4)
            layout.addWidget(self.rgb_canvas)
            
            # Spectrum canvas
            self.spectrum_canvas = MplCanvas(self.data_tab, width=8, height=3)
            layout.addWidget(self.spectrum_canvas)
            
        def setup_calibration_tab(self):
            """Setup calibration tab"""
            layout = QVBoxLayout(self.calibration_tab)
            
            self.calibration_canvas = MplCanvas(self.calibration_tab, width=8, height=7)
            layout.addWidget(self.calibration_canvas)
            
        def setup_atmospheric_tab(self):
            """Setup atmospheric correction tab"""
            layout = QVBoxLayout(self.atm_tab)
            
            self.atm_canvas = MplCanvas(self.atm_tab, width=8, height=7)
            layout.addWidget(self.atm_canvas)
            
        def setup_unmixing_tab(self):
            """Setup spectral unmixing tab"""
            layout = QVBoxLayout(self.unmix_tab)
            
            self.unmix_canvas = MplCanvas(self.unmix_tab, width=8, height=7)
            layout.addWidget(self.unmix_canvas)
            
        def setup_mineral_tab(self):
            """Setup mineral identification tab"""
            layout = QVBoxLayout(self.mineral_tab)
            
            self.mineral_canvas = MplCanvas(self.mineral_tab, width=8, height=7)
            layout.addWidget(self.mineral_canvas)
            
        def log(self, message):
            """Add message to log"""
            self.log_text.append(f"• {message}")
            QApplication.processEvents()
            
        def update_planetary_body(self, body):
            """Update planetary body parameters"""
            self.calibration = CalibrationModule(body)
            self.atmospheric = AtmosphericCorrection(body)
            self.log(f"Updated planetary body to {body}")
            
        def generate_synthetic_data(self):
            """Generate synthetic hyperspectral data"""
            self.log("Generating synthetic data...")
            
            generator = SyntheticDataGenerator(self.wavelengths)
            self.raw_data, self.ground_truth, self.minerals_present = generator.generate_scene(
                n_rows=80,
                n_cols=80,
                n_minerals=4,
                noise_level=self.noise_spin.value()
            )
            
            self.log(f"Generated {self.raw_data.shape} data cube")
            self.log(f"Minerals present: {', '.join(self.minerals_present)}")
            
            # Update display
            self.update_data_display()
            
        def update_data_display(self):
            """Update data overview display"""
            if self.raw_data is None:
                return
                
            # RGB view
            self.rgb_canvas.clear()
            fig = self.rgb_canvas.fig
            gs = GridSpec(1, 3, figure=fig, wspace=0.3)
            
            # Select RGB bands
            r_band = np.argmin(np.abs(self.wavelengths - 2300))  # SWIR
            g_band = np.argmin(np.abs(self.wavelengths - 1650))  # NIR
            b_band = np.argmin(np.abs(self.wavelengths - 800))   # VNIR
            
            # Normalize for display
            def normalize(arr):
                arr = arr.astype(float)
                p2, p98 = np.percentile(arr[arr > 0], [2, 98])
                return np.clip((arr - p2) / (p98 - p2 + 1e-10), 0, 1)
            
            rgb_image = np.stack([
                normalize(self.raw_data[:, :, r_band]),
                normalize(self.raw_data[:, :, g_band]),
                normalize(self.raw_data[:, :, b_band])
            ], axis=-1)
            
            ax1 = fig.add_subplot(gs[0, 0])
            ax1.imshow(rgb_image)
            ax1.set_title('False Color RGB', color='white', fontsize=10)
            ax1.axis('off')
            
            # Single band view
            ax2 = fig.add_subplot(gs[0, 1])
            im2 = ax2.imshow(self.raw_data[:, :, b_band], cmap='viridis')
            ax2.set_title(f'Band {b_band} ({self.wavelengths[b_band]:.0f} nm)', color='white', fontsize=10)
            ax2.axis('off')
            fig.colorbar(im2, ax=ax2, fraction=0.046, pad=0.04)
            
            # Mean spectrum
            ax3 = fig.add_subplot(gs[0, 2])
            mean_spectrum = np.mean(self.raw_data, axis=(0, 1))
            ax3.plot(self.wavelengths, mean_spectrum, 'cyan', linewidth=1.5)
            ax3.set_xlabel('Wavelength (nm)', color='white', fontsize=9)
            ax3.set_ylabel('Mean DN', color='white', fontsize=9)
            ax3.set_title('Mean Spectrum', color='white', fontsize=10)
            ax3.tick_params(colors='white', labelsize=8)
            ax3.set_facecolor('#1a1a2e')
            for spine in ax3.spines.values():
                spine.set_color('#3a3a5c')
            ax3.grid(True, alpha=0.3)
            
            self.rgb_canvas.draw()
            
            # Ground truth abundance maps
            self.spectrum_canvas.clear()
            fig2 = self.spectrum_canvas.fig
            gs2 = GridSpec(1, 4, figure=fig2, wspace=0.3)
            
            for i, mineral in enumerate(self.minerals_present[:4]):
                ax = fig2.add_subplot(gs2[0, i])
                im = ax.imshow(self.ground_truth[mineral], cmap='inferno', vmin=0, vmax=1)
                ax.set_title(mineral[:10], color='white', fontsize=9)
                ax.axis('off')
                fig2.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
            
            self.spectrum_canvas.draw()
            
        def run_pipeline(self):
            """Run the complete pipeline"""
            if self.raw_data is None:
                QMessageBox.warning(self, "No Data", "Please generate data first!")
                return
                
            self.log("Starting pipeline...")
            self.progress_bar.setValue(0)
            
            # Create worker thread
            self.worker = PipelineWorker(
                self,
                self.raw_data,
                {'n_endmembers': self.em_spin.value()}
            )
            self.worker.progress.connect(self.update_progress)
            self.worker.finished.connect(self.pipeline_finished)
            self.worker.start()
            
        def calibrate_dn_to_radiance(self, data):
            """Calibration step 1"""
            return self.calibration.dn_to_radiance(data)
            
        def calibrate_to_reflectance(self, radiance):
            """Calibration step 2"""
            return self.calibration.radiance_to_reflectance(radiance, self.wavelengths)
            
        def model_atmosphere(self):
            """Atmospheric modeling"""
            return self.atmospheric.model_atmospheric_transmission(self.wavelengths)
            
        def correct_atmosphere(self, reflectance):
            """Atmospheric correction"""
            return self.atmospheric.correct_atmosphere(reflectance, self.wavelengths)
            
        def extract_endmembers(self, data, n_endmembers=5):
            """Endmember extraction"""
            return self.unmixing.extract_endmembers_vertices(data, n_endmembers)
            
        def unmix_spectra(self, data, endmembers):
            """Spectral unmixing"""
            return self.unmixing.linear_unmixing(data, endmembers, method='nnls')
            
        def identify_minerals_map(self, data):
            """Mineral identification mapping"""
            return self.mineral_id.map_minerals(data)
            
        def update_progress(self, value, message):
            """Update progress bar"""
            self.progress_bar.setValue(value)
            self.status_label.setText(message)
            self.log(message)
            QApplication.processEvents()
            
        def pipeline_finished(self, results):
            """Handle pipeline completion"""
            if 'error' in results:
                self.log(f"Error: {results['error']}")
                return
                
            self.processed_data = results
            self.log("Pipeline complete!")
            
            # Update all displays
            self.update_calibration_display()
            self.update_atmospheric_display()
            self.update_unmixing_display()
            self.update_mineral_display()
            
        def step_pipeline(self):
            """Step through pipeline stages"""
            if self.raw_data is None:
                QMessageBox.warning(self, "No Data", "Please generate data first!")
                return
                
            # Determine current stage
            if 'radiance' not in self.processed_data:
                self.log("Step 1: DN to Radiance...")
                self.processed_data['radiance'] = self.calibrate_dn_to_radiance(self.raw_data)
                self.update_calibration_display()
            elif 'reflectance' not in self.processed_data:
                self.log("Step 2: Radiance to Reflectance...")
                self.processed_data['reflectance'] = self.calibrate_to_reflectance(
                    self.processed_data['radiance']
                )
                self.update_calibration_display()
            elif 'transmission' not in self.processed_data:
                self.log("Step 3: Model Atmosphere...")
                self.processed_data['transmission'] = self.model_atmosphere()
                self.update_atmospheric_display()
            elif 'corrected' not in self.processed_data:
                self.log("Step 4: Atmospheric Correction...")
                self.processed_data['corrected'] = self.correct_atmosphere(
                    self.processed_data['reflectance']
                )
                self.update_atmospheric_display()
            elif 'endmembers' not in self.processed_data:
                self.log("Step 5: Extract Endmembers...")
                self.processed_data['endmembers'] = self.extract_endmembers(
                    self.processed_data['corrected'],
                    n_endmembers=self.em_spin.value()
                )
                self.update_unmixing_display()
            elif 'abundances' not in self.processed_data:
                self.log("Step 6: Spectral Unmixing...")
                self.processed_data['abundances'] = self.unmix_spectra(
                    self.processed_data['corrected'],
                    self.processed_data['endmembers']
                )
                self.update_unmixing_display()
            elif 'mineral_maps' not in self.processed_data:
                self.log("Step 7: Mineral Identification...")
                self.processed_data['mineral_maps'] = self.identify_minerals_map(
                    self.processed_data['corrected']
                )
                self.update_mineral_display()
            else:
                self.log("Pipeline already complete!")
                
        def update_calibration_display(self):
            """Update calibration visualization"""
            self.calibration_canvas.clear()
            fig = self.calibration_canvas.fig
            gs = GridSpec(2, 3, figure=fig, hspace=0.3, wspace=0.3)
            
            if 'radiance' in self.processed_data:
                # Radiance spectrum
                ax1 = fig.add_subplot(gs[0, 0])
                mean_radiance = np.mean(self.processed_data['radiance'], axis=(0, 1))
                ax1.plot(self.wavelengths, mean_radiance, 'orange', linewidth=1.5)
                ax1.set_xlabel('Wavelength (nm)', color='white', fontsize=9)
                ax1.set_ylabel('Radiance', color='white', fontsize=9)
                ax1.set_title('Radiance Spectrum', color='white', fontsize=10)
                ax1.tick_params(colors='white', labelsize=8)
                ax1.set_facecolor('#1a1a2e')
                ax1.grid(True, alpha=0.3)
                for spine in ax1.spines.values():
                    spine.set_color('#3a3a5c')
                    
                # Radiance image
                ax2 = fig.add_subplot(gs[0, 1])
                band_idx = len(self.wavelengths) // 2
                im2 = ax2.imshow(self.processed_data['radiance'][:, :, band_idx], cmap='hot')
                ax2.set_title(f'Radiance @ {self.wavelengths[band_idx]:.0f}nm', color='white', fontsize=10)
                ax2.axis('off')
                fig.colorbar(im2, ax=ax2, fraction=0.046, pad=0.04)
            
            if 'reflectance' in self.processed_data:
                # Reflectance spectrum
                ax3 = fig.add_subplot(gs[0, 2])
                mean_reflectance = np.mean(self.processed_data['reflectance'], axis=(0, 1))
                ax3.plot(self.wavelengths, mean_reflectance, 'lime', linewidth=1.5)
                ax3.set_xlabel('Wavelength (nm)', color='white', fontsize=9)
                ax3.set_ylabel('I/F', color='white', fontsize=9)
                ax3.set_title('Reflectance Spectrum (I/F)', color='white', fontsize=10)
                ax3.tick_params(colors='white', labelsize=8)
                ax3.set_facecolor('#1a1a2e')
                ax3.grid(True, alpha=0.3)
                for spine in ax3.spines.values():
                    spine.set_color('#3a3a5c')
                    
                # Reflectance RGB
                ax4 = fig.add_subplot(gs[1, 0])
                r_band = np.argmin(np.abs(self.wavelengths - 2300))
                g_band = np.argmin(np.abs(self.wavelengths - 1650))
                b_band = np.argmin(np.abs(self.wavelengths - 800))
                
                def norm(arr):
                    p2, p98 = np.percentile(arr[arr > 0], [2, 98])
                    return np.clip((arr - p2) / (p98 - p2 + 1e-10), 0, 1)
                    
                rgb = np.stack([
                    norm(self.processed_data['reflectance'][:, :, r_band]),
                    norm(self.processed_data['reflectance'][:, :, g_band]),
                    norm(self.processed_data['reflectance'][:, :, b_band])
                ], axis=-1)
                ax4.imshow(rgb)
                ax4.set_title('Reflectance RGB', color='white', fontsize=10)
                ax4.axis('off')
                
                # Calibration coefficients
                ax5 = fig.add_subplot(gs[1, 1])
                if self.calibration.gain is not None:
                    ax5.plot(self.wavelengths, self.calibration.gain * 1000, 'cyan', linewidth=1.5, label='Gain (×1000)')
                    ax5.plot(self.wavelengths, self.calibration.offset, 'magenta', linewidth=1.5, label='Offset')
                    ax5.legend(fontsize=8, loc='upper right')
                ax5.set_xlabel('Wavelength (nm)', color='white', fontsize=9)
                ax5.set_ylabel('Calibration Value', color='white', fontsize=9)
                ax5.set_title('Calibration Coefficients', color='white', fontsize=10)
                ax5.tick_params(colors='white', labelsize=8)
                ax5.set_facecolor('#1a1a2e')
                ax5.grid(True, alpha=0.3)
                for spine in ax5.spines.values():
                    spine.set_color('#3a3a5c')
                    
                # Histogram
                ax6 = fig.add_subplot(gs[1, 2])
                ax6.hist(self.processed_data['reflectance'].flatten(), bins=50, color='lime', alpha=0.7)
                ax6.set_xlabel('I/F', color='white', fontsize=9)
                ax6.set_ylabel('Frequency', color='white', fontsize=9)
                ax6.set_title('Reflectance Histogram', color='white', fontsize=10)
                ax6.tick_params(colors='white', labelsize=8)
                ax6.set_facecolor('#1a1a2e')
                ax6.grid(True, alpha=0.3)
                for spine in ax6.spines.values():
                    spine.set_color('#3a3a5c')
            
            self.calibration_canvas.draw()
            
        def update_atmospheric_display(self):
            """Update atmospheric correction visualization"""
            self.atm_canvas.clear()
            fig = self.atm_canvas.fig
            gs = GridSpec(2, 3, figure=fig, hspace=0.3, wspace=0.3)
            
            if 'transmission' in self.processed_data:
                # Transmission spectrum
                ax1 = fig.add_subplot(gs[0, 0])
                ax1.plot(self.wavelengths, self.processed_data['transmission'], 'yellow', linewidth=1.5)
                ax1.set_xlabel('Wavelength (nm)', color='white', fontsize=9)
                ax1.set_ylabel('Transmission', color='white', fontsize=9)
                ax1.set_title('Atmospheric Transmission', color='white', fontsize=10)
                ax1.set_ylim(0, 1.1)
                ax1.tick_params(colors='white', labelsize=8)
                ax1.set_facecolor('#1a1a2e')
                ax1.grid(True, alpha=0.3)
                for spine in ax1.spines.values():
                    spine.set_color('#3a3a5c')
                    
                # Mark absorption bands
                ax1.axvline(1430, color='red', alpha=0.5, linestyle='--', label='CO2')
                ax1.axvline(1960, color='red', alpha=0.5, linestyle='--')
                ax1.legend(fontsize=8)
            
            if 'transmission' in self.processed_data:
                # Path radiance
                ax2 = fig.add_subplot(gs[0, 1])
                if self.atmospheric.path_radiance is not None:
                    ax2.plot(self.wavelengths, self.atmospheric.path_radiance, 'coral', linewidth=1.5)
                ax2.set_xlabel('Wavelength (nm)', color='white', fontsize=9)
                ax2.set_ylabel('Path Radiance', color='white', fontsize=9)
                ax2.set_title('Path Radiance', color='white', fontsize=10)
                ax2.tick_params(colors='white', labelsize=8)
                ax2.set_facecolor('#1a1a2e')
                ax2.grid(True, alpha=0.3)
                for spine in ax2.spines.values():
                    spine.set_color('#3a3a5c')
            
            if 'corrected' in self.processed_data and 'reflectance' in self.processed_data:
                # Before/After comparison
                ax3 = fig.add_subplot(gs[0, 2])
                mean_before = np.mean(self.processed_data['reflectance'], axis=(0, 1))
                mean_after = np.mean(self.processed_data['corrected'], axis=(0, 1))
                ax3.plot(self.wavelengths, mean_before, 'red', linewidth=1.5, alpha=0.7, label='Before')
                ax3.plot(self.wavelengths, mean_after, 'green', linewidth=1.5, label='After')
                ax3.set_xlabel('Wavelength (nm)', color='white', fontsize=9)
                ax3.set_ylabel('I/F', color='white', fontsize=9)
                ax3.set_title('Before/After Correction', color='white', fontsize=10)
                ax3.legend(fontsize=8)
                ax3.tick_params(colors='white', labelsize=8)
                ax3.set_facecolor('#1a1a2e')
                ax3.grid(True, alpha=0.3)
                for spine in ax3.spines.values():
                    spine.set_color('#3a3a5c')
                    
                # Corrected RGB
                ax4 = fig.add_subplot(gs[1, 0])
                r_band = np.argmin(np.abs(self.wavelengths - 2300))
                g_band = np.argmin(np.abs(self.wavelengths - 1650))
                b_band = np.argmin(np.abs(self.wavelengths - 800))
                
                def norm(arr):
                    p2, p98 = np.percentile(arr[arr > 0], [2, 98])
                    return np.clip((arr - p2) / (p98 - p2 + 1e-10), 0, 1)
                    
                rgb = np.stack([
                    norm(self.processed_data['corrected'][:, :, r_band]),
                    norm(self.processed_data['corrected'][:, :, g_band]),
                    norm(self.processed_data['corrected'][:, :, b_band])
                ], axis=-1)
                ax4.imshow(rgb)
                ax4.set_title('Corrected RGB', color='white', fontsize=10)
                ax4.axis('off')
                
                # Difference map
                ax5 = fig.add_subplot(gs[1, 1])
                diff = np.mean(self.processed_data['corrected'] - self.processed_data['reflectance'], axis=2)
                im5 = ax5.imshow(diff, cmap='RdBu', vmin=-0.1, vmax=0.1)
                ax5.set_title('Correction Difference', color='white', fontsize=10)
                ax5.axis('off')
                fig.colorbar(im5, ax=ax5, fraction=0.046, pad=0.04)
                
                # Atmospheric parameters
                ax6 = fig.add_subplot(gs[1, 2])
                ax6.axis('off')
                
                params = self.atmospheric.params
                text = f"""Atmospheric Parameters
                
    Body: {self.planet_combo.currentText()}
    Surface Pressure: {params['surface_pressure']:.3f} bar
    Scale Height: {params['scale_height']:.1f} km
    Dust Opacity: {params['dust_opacity']:.2f}
    
    Composition:
    """
                for gas, frac in params['atmosphere'].items():
                    text += f"  {gas}: {frac*100:.1f}%\n"
                    
                ax6.text(0.1, 0.9, text, transform=ax6.transAxes, 
                        fontsize=10, color='white', verticalalignment='top',
                        fontfamily='monospace')
            
            self.atm_canvas.draw()
            
        def update_unmixing_display(self):
            """Update spectral unmixing visualization"""
            self.unmix_canvas.clear()
            fig = self.unmix_canvas.fig
            gs = GridSpec(2, 4, figure=fig, hspace=0.3, wspace=0.3)
            
            if 'endmembers' in self.processed_data:
                endmembers = self.processed_data['endmembers']
                
                # Plot endmember spectra
                colors = plt.cm.Set1(np.linspace(0, 1, len(endmembers)))
                
                for i, em in enumerate(endmembers[:4]):
                    ax = fig.add_subplot(gs[0, i])
                    ax.plot(self.wavelengths, em.spectrum, color=colors[i], linewidth=1.5)
                    ax.set_xlabel('Wavelength (nm)', color='white', fontsize=8)
                    ax.set_ylabel('I/F', color='white', fontsize=8)
                    ax.set_title(em.name, color='white', fontsize=10)
                    ax.tick_params(colors='white', labelsize=7)
                    ax.set_facecolor('#1a1a2e')
                    ax.grid(True, alpha=0.3)
                    for spine in ax.spines.values():
                        spine.set_color('#3a3a5c')
            
            if 'abundances' in self.processed_data and 'endmembers' in self.processed_data:
                endmembers = self.processed_data['endmembers']
                
                # Plot abundance maps
                for i, em in enumerate(endmembers[:4]):
                    ax = fig.add_subplot(gs[1, i])
                    if em.abundances is not None:
                        im = ax.imshow(em.abundances, cmap='inferno', vmin=0, vmax=1)
                        fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
                    ax.set_title(f'{em.name} Abundance', color='white', fontsize=10)
                    ax.axis('off')
            
            self.unmix_canvas.draw()
            
        def update_mineral_display(self):
            """Update mineral identification visualization"""
            self.mineral_canvas.clear()
            fig = self.mineral_canvas.fig
            gs = GridSpec(2, 4, figure=fig, hspace=0.3, wspace=0.3)
            
            if 'mineral_maps' in self.processed_data:
                mineral_maps = self.processed_data['mineral_maps']
                
                # Sort by mean abundance
                sorted_minerals = sorted(mineral_maps.items(), 
                                        key=lambda x: np.mean(x[1]), 
                                        reverse=True)
                
                # Plot top mineral maps
                for i, (mineral_name, abundance_map) in enumerate(sorted_minerals[:4]):
                    ax = fig.add_subplot(gs[0, i])
                    im = ax.imshow(abundance_map, cmap='viridis', vmin=0, vmax=1)
                    ax.set_title(f'{mineral_name}', color='white', fontsize=9)
                    ax.axis('off')
                    fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
                    
                # Mineral library comparison
                ax_lib = fig.add_subplot(gs[1, :2])
                ax_lib.axis('off')
                
                # Get pixel spectrum
                if 'corrected' in self.processed_data:
                    center_pixel = self.processed_data['corrected'][
                        self.processed_data['corrected'].shape[0]//2,
                        self.processed_data['corrected'].shape[1]//2,
                        :
                    ]
                    
                    # Identify minerals in this pixel
                    identifications = self.mineral_id.identify_minerals(center_pixel, threshold=0.5)
                    
                    text = "Pixel Mineral Identification:\n"
                    for ident in identifications[:5]:
                        text += f"\n{ident['mineral']}: {ident['confidence']*100:.1f}%"
                        text += f"\n  → {ident['description']}"
                        
                    ax_lib.text(0.05, 0.95, text, transform=ax_lib.transAxes,
                               fontsize=9, color='white', verticalalignment='top',
                               fontfamily='monospace')
                    
                    # Plot pixel spectrum with mineral matches
                    ax_spec = fig.add_subplot(gs[1, 2:])
                    ax_spec.plot(self.wavelengths, center_pixel, 'white', linewidth=2, label='Observed')
                    
                    for ident in identifications[:3]:
                        mineral_name = ident['mineral']
                        ref_spectrum = self.mineral_id.mineral_library[mineral_name]['spectrum']
                        color = self.mineral_id.mineral_library[mineral_name]['color']
                        ax_spec.plot(self.wavelengths, ref_spectrum, color=color, 
                                   linewidth=1.5, alpha=0.7, linestyle='--',
                                   label=f"{mineral_name}")
                    
                    ax_spec.set_xlabel('Wavelength (nm)', color='white', fontsize=9)
                    ax_spec.set_ylabel('I/F', color='white', fontsize=9)
                    ax_spec.set_title('Spectral Matching', color='white', fontsize=10)
                    ax_spec.legend(fontsize=7, loc='upper right')
                    ax_spec.tick_params(colors='white', labelsize=8)
                    ax_spec.set_facecolor('#1a1a2e')
                    ax_spec.grid(True, alpha=0.3)
                    for spine in ax_spec.spines.values():
                        spine.set_color('#3a3a5c')
            
            self.mineral_canvas.draw()
    
    
    # ============================================================================
    # MAIN ENTRY POINT
    # ============================================================================
    
    def main():
        app = QApplication(sys.argv)
        
        # Set application properties
        app.setStyle('Fusion')
        
        # Dark palette
        palette = QPalette()
        palette.setColor(QPalette.Window, QColor(15, 15, 35))
        palette.setColor(QPalette.WindowText, QColor(224, 224, 224))
        palette.setColor(QPalette.Base, QColor(26, 26, 46))
        palette.setColor(QPalette.AlternateBase, QColor(37, 37, 66))
        palette.setColor(QPalette.ToolTipBase, QColor(224, 224, 224))
        palette.setColor(QPalette.ToolTipText, QColor(224, 224, 224))
        palette.setColor(QPalette.Text, QColor(224, 224, 224))
        palette.setColor(QPalette.Button, QColor(37, 37, 66))
        palette.setColor(QPalette.ButtonText, QColor(224, 224, 224))
        palette.setColor(QPalette.BrightText, QColor(255, 0, 0))
        palette.setColor(QPalette.Highlight, QColor(106, 106, 188))
        palette.setColor(QPalette.HighlightedText, QColor(255, 255, 255))
        app.setPalette(palette)
        
        window = PlanetarySpectroscopyPipeline()
        window.show()
        
        sys.exit(app.exec_())
    
    
    if __name__ == '__main__':
        main()
    

    Pipeline Overview

    ┌─────────────────────────────────────────────────────────────────────────────┐
    │                    PLANETARY SPECTROSCOPY PIPELINE                          │
    ├─────────────────────────────────────────────────────────────────────────────┤
    │                                                                             │
    │  ┌──────────────┐   ┌──────────────┐   ┌──────────────┐   ┌──────────────┐ │
    │  │  RAW DATA    │──▶│ CALIBRATION  │──▶│  ATMOSPHERIC │──▶│   UNMIXING   │ │
    │  │  (DN Cube)   │   │              │   │  CORRECTION  │   │              │ │
    │  └──────────────┘   └──────────────┘   └──────────────┘   └──────────────┘ │
    │        │                   │                   │                   │        │
    │        ▼                   ▼                   ▼                   ▼        │
    │   ┌─────────┐        ┌─────────┐        ┌─────────┐        ┌─────────┐     │
    │   │Dark     │        │DN→Rad   │        │Trans-   │        │End-     │     │
    │   │Subtract │        │→I/F     │        │mission  │        │member   │     │
    │   │Flat     │        │Solar    │        │Model    │        │Extract  │     │
    │   │Field    │        │Flux     │        │         │        │         │     │
    │   └─────────┘        └─────────┘        └─────────┘        └─────────┘     │
    │                                                                    │        │
    │                                                                    ▼        │
    │                                                            ┌──────────────┐ │
    │                                                            │  MINERAL ID  │ │
    │                                                            │              │ │
    │                                                            │ Spectral     │ │
    │                                                            │ Angle Mapper │ │
    │                                                            │ Feature      │ │
    │                                                            │ Fitting      │ │
    │                                                            └──────────────┘ │
    └─────────────────────────────────────────────────────────────────────────────┘
    

    Key Features

    ModuleFunctionsOutput
    CalibrationDark current, Flat field, DN→Radiance→ReflectanceI/F spectra
    AtmosphericGas absorption, Aerosol scattering, Path radianceSurface reflectance
    UnmixingEndmember extraction (VCA/PCA), Linear unmixing (NNLS/FCLS)Abundance maps
    Mineral IDSpectral Angle Mapper, Feature fitting, Library matchingMineral maps & IDs

    Supported Minerals

    • Silicates: Olivine, Pyroxene, Quartz
    • Oxides: Hematite, Goethite
    • Sulfates: Gypsum
    • Phyllosilicates: Kaolinite, Montmorillonite
    • Carbonates: Calcite
    • Volatiles: Water Ice
Build a planetary spectroscopy pipeline: calibration… · The Chat Archive