Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

// ==UserScript== // @name Advanced Web Crawler Enhanced Pro Optimized v6.4.4 //…

  1. User

    // ==UserScript== // @name Advanced Web Crawler Enhanced Pro Optimized v6.4.4 // @namespace http://tampermonkey.net/ // @version 6.4.5 // @description Further optimized and enhanced version with representative photos display, comment window for crawling strategies, responsive design, loading indicators, enhanced scheduling, data analysis tools integration, automated testing, improved UI management, and multiple crawling strategies selectable from settings. // @author
    // @match https://www.google.com/search* // @match https://www.bing.com/search* // @match https://search.yahoo.com/search* // @match https://duckduckgo.com/* // @match https://www.baidu.com/s* // @match https://www.yandex.com/search/* // @match https://search.naver.com/search* // @match https://search.daum.net/search* // @match https://www.ask.com/* // @match https://www.ecosia.org/search* // @match https://www.startpage.com/* // @match https://searx.me/* // @match https://www.dogpile.com/* // @match https://www.bing.com/images* // @grant GM_xmlhttpRequest // @grant GM_setValue // @grant GM_getValue // @grant GM_deleteValue // @grant GM_addStyle // @connect * // @require https://cdnjs.cloudflare.com/ajax/libs/jquery/3.6.0/jquery.min.js // @require https://cdn.jsdelivr.net/npm/chart.js // @require https://cdnjs.cloudflare.com/ajax/libs/xlsx/0.18.5/xlsx.full.min.js // @require https://cdnjs.cloudflare.com/ajax/libs/mocha/10.2.0/mocha.min.js // @require https://cdnjs.cloudflare.com/ajax/libs/chai/4.3.7/chai.min.js // @require https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.8.0/highlight.min.js // ==/UserScript==

    (() => { 'use strict';

    const Config = (() => {
        const DEFAULT_SETTINGS = {
            delay: 2000,
            delayRandom: 500,
            maxPages: 5,
            userAgents: [
                'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)',
                'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15',
                'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)',
                'Mozilla/5.0 (iPhone; CPU iPhone OS 14_0 like Mac OS X) AppleWebKit/605.1.15',
                'Mozilla/5.0 (iPad; CPU OS 14_0 like Mac OS X) AppleWebKit/605.1.15',
                'AdvancedWebCrawler/6.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
            ],
            userAgentIndex: 0,
            blacklist: [],
            whitelist: [],
            contentTypes: {
                include: ['text/html'],
                exclude: ['application/pdf', 'image/jpeg', 'image/png', 'video/mp4']
            },
            timeout: 8000,
            includeKeywords: [],
            excludeKeywords: [],
            language: [],
            minWordCount: 100,
            maxWordCount: 10000,
            maxDepth: 2,
            proxy: '',
            regexInclude: [],
            regexExclude: [],
            dataFields: {
                name: true,
                url: true,
                description: true,
                html: true,
                category: true,
                quality: true,
                language: true,
                wordCount: true,
                metaTags: false,
                headings: false,
                image: true // Added image field
            },
            schedule: {
                enabled: false,
                startTime: '00:00',
                interval: 3600000,
                specificDays: []
            },
            searchEngines: {
                google: true,
                bing: true,
                yahoo: true,
                duckduckgo: true,
                baidu: true,
                yandex: true,
                naver: true,
                daum: true,
                ask: true,
                ecosia: true,
                startpage: true,
                searx: true,
                dogpile: true,
                bingimages: true
            },
            retryAttempts: 3,
            retryDelay: 2000,
            respectRobots: true,
            maxConcurrentRequests: 3,
            dataVisualization: true,
            exportFormats: {
                csv: true,
                json: true,
                xml: true,
                xls: true
            },
            advancedLogging: true,
            enableCaptchas: false,
            customKeywords: {},
            templates: {},
            currentTab: 'logs',
            isCrawling: false,
            darkMode: false,
            crawlingStrategies: [
                'Basic Crawling',
                'Focused Crawling',
                'Incremental Crawling',
                'Parallel Crawling',
                'Distributed Crawling',
                'Adaptive Crawling',
                'Semantic Crawling',
                'Deep Crawling',
                'Shallow Crawling',
                'Politeness Crawling',
                'Reinforcement Learning-based Crawling',
                'Hybrid Crawling',
                'Parallel-incremental Crawling',
                'Data-driven Crawling',
                'Constraint-based Crawling',
                'URL Frontier-based Crawling',
                'Distributed Queue Frontier',
                'Reverse Crawling',
                'Sitemap-based Crawling',
                'Incremental Adaptive Crawling'
            ],
            selectedCrawlingStrategies: ['Basic Crawling'],
            crawlingStrategyComments: {} // Added comments for crawling strategies
        };
    
        let settings = { ...DEFAULT_SETTINGS, ...getSavedSettings() };
    
        function getSavedSettings() {
            return JSON.parse(GM_getValue('settings', JSON.stringify(DEFAULT_SETTINGS)));
        }
    
        function saveSettings(newSettings) {
            settings = { ...settings, ...newSettings };
            GM_setValue('settings', JSON.stringify(settings));
        }
    
        function resetSettings() {
            settings = { ...DEFAULT_SETTINGS };
            GM_setValue('settings', JSON.stringify(settings));
        }
    
        function saveTemplate(name) {
            settings.templates[name] = { ...settings };
            saveSettings({});
            Logger.log(`설정 템플릿 '${name}'이(가) 저장되었습니다.`, 'success');
        }
    
        function loadTemplate(name) {
            if (name in settings.templates) {
                settings = { ...settings, ...settings.templates[name] };
                saveSettings({});
                Logger.log(`설정 템플릿 '${name}'이(가) 불러와졌습니다.`, 'success');
            } else {
                Logger.log(`설정 템플릿 '${name}'을(를) 찾을 수 없습니다.`, 'error');
            }
        }
    
        function validateProxy(proxyUrl) {
            try {
                const url = new URL(proxyUrl);
                return ['http:', 'https:'].includes(url.protocol);
            } catch {
                return false;
            }
        }
    
        return {
            DEFAULT_SETTINGS,
            get settings() { return settings; },
            saveSettings,
            resetSettings,
            saveTemplate,
            loadTemplate,
            validateProxy
        };
    })();
    
    const Logger = (() => {
        let logs = JSON.parse(GM_getValue('logs', '[]'));
    
        function log(message, type = 'info') {
            const timestamp = new Date().toLocaleString();
            logs.push({ timestamp, message, type });
            if (logs.length > 1000) logs.shift();
            GM_setValue('logs', JSON.stringify(logs));
            UI.updateLogs();
            if (Config.settings.advancedLogging) {
                const styles = {
                    info: 'color: blue;',
                    warning: 'color: orange;',
                    error: 'color: red;',
                    success: 'color: green; font-weight: bold;',
                    debug: 'color: purple;'
                };
                console.log(`%c[${timestamp}] [${type.toUpperCase()}] ${message}`, styles[type] || '');
            }
        }
    
        function getLogs() {
            return logs;
        }
    
        function clearLogs() {
            logs = [];
            GM_setValue('logs', JSON.stringify(logs));
            UI.updateLogs();
        }
    
        return {
            log,
            getLogs,
            clearLogs
        };
    })();
    
    const Utils = (() => {
        const escapeMap = {
            html: { '&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;', "'": '&#039;' },
            csv: { '"': '""' },
            xml: { '&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;', "'": '&#039;' }
        };
    
        function escape(str, type = 'html') {
            return String(str).replace(new RegExp(`[${Object.keys(escapeMap[type]).join('')}]`, 'g'), match => escapeMap[type][match]);
        }
    
        function capitalize(str) {
            return str.charAt(0).toUpperCase() + str.slice(1);
        }
    
        function delay(ms) {
            return new Promise(res => setTimeout(res, ms));
        }
    
        function generateColors(count) {
            return Array.from({ length: count }, (_, i) => `hsl(${i * (360 / count)}, 70%, 50%)`);
        }
    
        function parseCustomKeywords(text) {
            const lines = text.split('\n').map(l => l.trim()).filter(l => l);
            return lines.reduce((acc, line) => {
                const [cat, kws] = line.split(':').map(s => s.trim());
                if (cat && kws) acc[cat] = kws.split(',').map(kw => kw.trim()).filter(Boolean);
                return acc;
            }, {});
        }
    
        function formatCustomKeywords(ck) {
            return Object.entries(ck).map(([c, kws]) => `${c}:${kws.join(',')}`).join('\n');
        }
    
        function isValidURL(url) {
            try { new URL(url); return true; }
            catch { return false; }
        }
    
        return {
            escape,
            capitalize,
            delay,
            generateColors,
            parseCustomKeywords,
            formatCustomKeywords,
            isValidURL
        };
    })();
    
    const UI = (() => {
        const selectors = {
            container: '#crawler-container',
            header: '#crawler-header',
            tabs: '#crawler-tabs',
            content: '#crawler-content',
            minimizeBtn: '#crawler-minimize',
            darkModeToggle: '#dark-mode-toggle',
            tabsButtons: '.crawler-tab',
            logTab: '#tab-logs',
            dataTab: '#tab-data',
            settingsTab: '#tab-settings',
            previousCrawlsTab: '#tab-previous-crawls',
            templatesTab: '#tab-templates',
            testsTab: '#tab-tests',
            aiAssistantTab: '#tab-ai-assistant',
            strategiesTab: '#tab-strategies',
            logMessages: '#log-messages',
            dataTableBody: '#data-table-body',
            previousCrawlsBody: '#previous-crawls-body',
            templateSelect: '#template-select',
            newTemplateName: '#new-template-name',
            searchFilter: '#search-filter',
            errorMessage: '#error-message',
            successMessage: '#success-message',
            htmlModal: '#html-modal',
            modalHtmlContent: '#modal-html-content-rendered',
            testModal: '#test-modal',
            mochaContainer: '#mocha',
            dataChart: '#data-chart'
        };
    
        function createUI() {
            GM_addStyle(`
                /* Highlight.js default style */
                .hljs {
                    display: block;
                    overflow-x: auto;
                    padding: 0.5em;
                    background: #f0f0f0;
                }
                /* Dark mode adjustments for Highlight.js */
                ${selectors.container}.dark .hljs {
                    background: #2b2b2b;
                    color: #f1f1f1;
                }
                /* Container */
                ${selectors.container} {
                    position: fixed;
                    top: 50px;
                    right: 20px;
                    width: 500px; /* Increased width for image display */
                    max-width: 95%;
                    height: 75vh; /* Increased height */
                    background: var(--bg-color, #fff);
                    border: 1px solid #ccc;
                    z-index: 10000;
                    font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
                    box-shadow: 0 4px 12px rgba(0,0,0,0.3);
                    overflow: hidden;
                    resize: both;
                    border-radius: 10px;
                    min-width: 350px;
                    min-height: 350px;
                    display: flex;
                    flex-direction: column;
                }
                ${selectors.container}.dark {
                    --bg-color: #2b2b2b;
                    color: #f1f1f1;
                    border-color: #555;
                }
                /* Header */
                ${selectors.header} {
                    display: flex;
                    justify-content: space-between;
                    align-items: center;
                    background: var(--header-bg, #f1f1f1);
                    border-bottom: 1px solid #ccc;
                    padding: 10px 15px;
                    cursor: move;
                    user-select: none;
                }
                ${selectors.container}.dark ${selectors.header} {
                    --header-bg: #3c3c3c;
                    border-color: #555;
                }
                /* Tabs */
                ${selectors.tabs} {
                    display: flex;
                    flex-wrap: wrap;
                    gap: 5px;
                }
                .crawler-tab {
                    padding: 8px 12px;
                    cursor: pointer;
                    font-weight: bold;
                    border-radius: 4px;
                    background: #e0e0e0;
                    transition: background 0.3s;
                }
                .crawler-tab:hover { background: #d5d5d5; }
                .crawler-tab.active { background: var(--active-tab-bg, #2196F3); color: #fff; }
                ${selectors.container}.dark .crawler-tab.active { background: #1E88E5; }
                ${selectors.container}.dark .crawler-tab:hover { background: #3c3c3c; }
                /* Control Buttons and Content */
                ${selectors.content} {
                    padding: 15px;
                    overflow-y: auto;
                    flex: 1;
                    background: var(--content-bg, #fff);
                }
                ${selectors.container}.dark ${selectors.content} {
                    --content-bg: #2b2b2b;
                }
                /* Tables */
                table {
                    width: 100%;
                    border-collapse: collapse;
                    table-layout: fixed;
                    word-wrap: break-word;
                }
                th, td {
                    padding: 8px;
                    text-align: left;
                    border-bottom: 1px solid #ddd;
                }
                th {
                    background-color: var(--th-bg, #f2f2f2);
                    cursor: pointer;
                    position: sticky;
                    top: 0;
                    z-index: 1;
                }
                ${selectors.container}.dark th { background-color: #444; }
                tr:hover { background-color: #f5f5f5; }
                ${selectors.container}.dark tr:hover { background-color: #555; }
                /* Progress Bar */
                .progress-bar {
                    width: 100%;
                    background-color: #f3f3f3;
                    border: 1px solid #ccc;
                    border-radius: 5px;
                    overflow: hidden;
                    margin-bottom: 10px;
                }
                .progress-bar-fill {
                    height: 20px;
                    background-color: #4caf50;
                    width: 0%;
                    transition: width 0.2s;
                }
                /* Spinner */
                .spinner {
                    border: 4px solid #f3f3f3;
                    border-top: 4px solid #3498db;
                    border-radius: 50%;
                    width: 24px;
                    height: 24px;
                    animation: spin 2s linear infinite;
                    margin-right: 10px;
                }
                @keyframes spin {
                    0% { transform: rotate(0deg); }
                    100% { transform: rotate(360deg); }
                }
                /* Modal */
                .modal {
                    display: none;
                    position: fixed;
                    z-index: 10001;
                    left: 0;
                    top: 0;
                    width: 100%;
                    height: 100%;
                    overflow: auto;
                    background-color: rgba(0,0,0,0.6);
                }
                .modal-content {
                    background-color: var(--modal-bg, #fff);
                    margin: 5% auto;
                    padding: 20px;
                    border: 1px solid #888;
                    width: 90%;
                    max-width: 800px;
                    max-height: 80%;
                    overflow-y: auto;
                    border-radius: 8px;
                    position: relative;
                }
                ${selectors.container}.dark .modal-content {
                    --modal-bg: #3c3c3c;
                    color: #f1f1f1;
                    border-color: #555;
                }
                .close {
                    color: #aaa;
                    position: absolute;
                    top: 10px;
                    right: 20px;
                    font-size: 28px;
                    font-weight: bold;
                    cursor: pointer;
                }
                .close:hover, .close:focus {
                    color: #000;
                    text-decoration: none;
                    cursor: pointer;
                }
                /* Control Buttons */
                .control-buttons {
                    margin: 10px 0;
                    display: flex;
                    flex-wrap: wrap;
                    gap: 10px;
                }
                .control-buttons button {
                    padding: 10px 15px;
                    cursor: pointer;
                    border: none;
                    background-color: #2196F3;
                    color: white;
                    border-radius: 4px;
                    font-size: 14px;
                    flex: 1 1 auto;
                    min-width: 80px;
                    transition: background-color 0.3s;
                }
                .control-buttons button:hover { background-color: #1976D2; }
                .control-buttons button:disabled {
                    background-color: #a5d6a7;
                    cursor: not-allowed;
                }
                /* Status Text */
                .status-text {
                    margin-top: 10px;
                    font-weight: bold;
                    font-size: 14px;
                }
                /* Settings Sections */
                .settings-section {
                    margin-bottom: 15px;
                }
                .settings-section label {
                    display: block;
                    margin-bottom: 5px;
                    font-weight: bold;
                }
                .settings-section input[type="text"],
                .settings-section input[type="number"],
                .settings-section textarea,
                .settings-section select {
                    width: 100%;
                    padding: 8px;
                    box-sizing: border-box;
                    border: 1px solid #ccc;
                    border-radius: 4px;
                    font-size: 14px;
                    background-color: var(--input-bg, #fff);
                    color: var(--input-color, #000);
                    transition: background-color 0.3s, color 0.3s;
                }
                ${selectors.container}.dark .settings-section input,
                ${selectors.container}.dark .settings-section select,
                ${selectors.container}.dark .settings-section textarea {
                    background-color: #555;
                    color: #f1f1f1;
                    border-color: #777;
                }
                .checkbox-group, .radio-group {
                    display: flex;
                    flex-wrap: wrap;
                    gap: 10px;
                }
                .checkbox-group label, .radio-group label {
                    font-size: 14px;
                }
                /* Tooltip */
                .tooltip {
                    position: relative;
                    display: inline-block;
                    cursor: help;
                    font-size: 14px;
                    margin-left: 5px;
                }
                .tooltip .tooltiptext {
                    visibility: hidden;
                    width: 220px;
                    background-color: #555;
                    color: #fff;
                    text-align: center;
                    border-radius: 6px;
                    padding: 5px;
                    position: absolute;
                    z-index: 1;
                    bottom: 125%;
                    left: 50%;
                    margin-left: -110px;
                    opacity: 0;
                    transition: opacity 0.3s;
                }
                .tooltip:hover .tooltiptext {
                    visibility: visible;
                    opacity: 1;
                }
                /* Resizer */
                ${selectors.container}::-webkit-resizer {
                    background: #f1f1f1;
                    border: none;
                }
                /* Full URL Display */
                td a.full-url {
                    display: block;
                    word-break: break-all;
                    position: relative;
                }
                td a.full-url:hover::after {
                    content: attr(href);
                    position: absolute;
                    background: #333;
                    color: #fff;
                    padding: 5px;
                    border-radius: 5px;
                    white-space: nowrap;
                    z-index: 10002;
                    top: 100%;
                    left: 0;
                }
                /* AI Assistant */
                #tab-ai-assistant {
                    display: none;
                }
                #ai-assistant-iframe {
                    width: 100%;
                    height: 600px;
                    border: none;
                    border-radius: 4px;
                }
                /* Responsive Design */
                @media (max-width: 768px) {
                    ${selectors.container} {
                        width: 95%;
                        height: 80vh;
                        top: 10px;
                        right: 2.5%;
                    }
                    .control-buttons {
                        flex-direction: column;
                    }
                }
                @media (max-width: 480px) {
                    ${selectors.header} {
                        flex-direction: column;
                        align-items: flex-start;
                    }
                    ${selectors.tabs} {
                        flex-wrap: nowrap;
                        overflow-x: auto;
                        scrollbar-width: thin;
                    }
                    .crawler-tab {
                        flex: 0 0 auto;
                    }
                }
                /* Enhanced Scrollbars */
                #crawler-tabs, #log-messages, #data-table-body, #previous-crawls-body, #mocha {
                    scrollbar-width: thin;
                    scrollbar-color: #888 #f1f1f1;
                }
                #crawler-tabs::-webkit-scrollbar,
                #log-messages::-webkit-scrollbar,
                #data-table-body::-webkit-scrollbar,
                #previous-crawls-body::-webkit-scrollbar,
                #mocha::-webkit-scrollbar {
                    width: 8px;
                    height: 8px;
                }
                #crawler-tabs::-webkit-scrollbar-track,
                #log-messages::-webkit-scrollbar-track,
                #data-table-body::-webkit-scrollbar-track,
                #previousCrawlsBody::-webkit-scrollbar-track,
                #mocha::-webkit-scrollbar-track {
                    background: #f1f1f1;
                    border-radius: 4px;
                }
                #crawler-tabs::-webkit-scrollbar-thumb,
                #log-messages::-webkit-scrollbar-thumb,
                #data-table-body::-webkit-scrollbar-thumb,
                #previousCrawlsBody::-webkit-scrollbar-thumb,
                #mocha::-webkit-scrollbar-thumb {
                    background: #888;
                    border-radius: 4px;
                }
                #crawler-tabs::-webkit-scrollbar-thumb:hover,
                #log-messages::-webkit-scrollbar-thumb:hover,
                #data-table-body::-webkit-scrollbar-thumb:hover,
                #previousCrawlsBody::-webkit-scrollbar-thumb:hover,
                #mocha::-webkit-scrollbar-thumb:hover {
                    background: #555;
                }
                /* Image Display in Data Table */
                .site-image {
                    max-width: 100px;
                    max-height: 100px;
                    border-radius: 4px;
                }
                /* Comment Window Style */
                .comment-window {
                    margin-top: 10px;
                }
                .comment-window textarea {
                    width: 100%;
                    height: 100px;
                    padding: 8px;
                    border: 1px solid #ccc;
                    border-radius: 4px;
                    resize: vertical;
                }
                /* Additional Styles for Image Column */
                th.image-column, td.image-column {
                    width: 120px;
                }
            `);
    
            const container = $(`
                <div id="crawler-container">
                    <div id="crawler-header">
                        <div id="crawler-tabs">
                            <div class="crawler-tab" data-tab="logs" title="로그 보기">📜 로그</div>
                            <div class="crawler-tab" data-tab="data" title="수집된 데이터 보기">📊 데이터</div>
                            <div class="crawler-tab" data-tab="settings" title="설정 변경">⚙️ 설정</div>
                            <div class="crawler-tab" data-tab="previous-crawls" title="이전 크롤링 기록">📁 기록</div>
                            <div class="crawler-tab" data-tab="templates" title="설정 템플릿">📂 템플릿</div>
                            <div class="crawler-tab" data-tab="tests" title="테스트 실행">🧪 테스트</div>
                            <div class="crawler-tab" data-tab="ai-assistant" title="AI 어시스턴트">🧠 AI</div>
                            <div class="crawler-tab" data-tab="strategies" title="크롤링 전략">🌀 전략</div>
                        </div>
                        <div>
                            <button id="dark-mode-toggle" class="dark-mode-toggle" title="테마 전환">🌙</button>
                            <button id="crawler-minimize" title="창 최소화">━</button>
                        </div>
                    </div>
                    <div id="crawler-content">
                        <div id="tab-logs">
                            <div class="progress-bar">
                                <div class="progress-bar-fill" style="width: 0%;"></div>
                            </div>
                            <div class="status-text">진행률: 0% (0/0 페이지)</div>
                            <div class="control-buttons">
                                <button id="start-crawling" title="크롤링을 시작합니다.">▶ 시작</button>
                                <button id="stop-crawling" title="크롤링을 중지합니다." disabled>■ 중지</button>
                                ${Config.settings.exportFormats.csv ? `<button id="download-csv" title="CSV 다운로드">📥 CSV</button>` : ''}
                                ${Config.settings.exportFormats.json ? `<button id="download-json" title="JSON 다운로드">📥 JSON</button>` : ''}
                                ${Config.settings.exportFormats.xml ? `<button id="download-xml" title="XML 다운로드">📥 XML</button>` : ''}
                                ${Config.settings.exportFormats.xls ? `<button id="download-xls" title="XLS 다운로드">📥 XLS</button>` : ''}
                                <button id="reset-data" title="모든 크롤링 데이터를 초기화합니다.">🗑️ 초기화</button>
                            </div>
                            <div id="log-messages" style="height: 200px; overflow-y: auto; background: var(--log-bg, #f9f9f9); padding: 10px; border: 1px solid #ccc; border-radius: 4px;">
                                로그 내용이 여기에 표시됩니다.
                            </div>
                            <div id="error-message" class="error-message" style="display: none;"></div>
                            <div id="success-message" class="success-message" style="display: none;"></div>
                        </div>
                        <div id="tab-data" style="display: none;">
                            <input type="text" id="search-filter" placeholder="검색 필터..." style="width: 100%; padding: 8px; margin-bottom: 10px; border: 1px solid #ccc; border-radius: 4px;">
                            <table>
                                <thead>
                                    <tr>
                                        ${Config.settings.dataFields.image ? '<th class="image-column">이미지</th>' : ''}
                                        ${Config.settings.dataFields.name ? '<th data-sort="name">사이트 이름</th>' : ''}
                                        ${Config.settings.dataFields.url ? '<th data-sort="url">주소</th>' : ''}
                                        ${Config.settings.dataFields.description ? '<th data-sort="description">설명</th>' : ''}
                                        ${Config.settings.dataFields.html ? '<th>📄 HTML 내용</th>' : ''}
                                        ${Config.settings.dataFields.category ? '<th data-sort="category">분류</th>' : ''}
                                        ${Config.settings.dataFields.quality ? '<th data-sort="quality">품질점수</th>' : ''}
                                        ${Config.settings.dataFields.language ? '<th data-sort="language">언어</th>' : ''}
                                        ${Config.settings.dataFields.wordCount ? '<th data-sort="wordCount">단어 수</th>' : ''}
                                        ${Config.settings.dataFields.metaTags ? '<th>메타 태그</th>' : ''}
                                        ${Config.settings.dataFields.headings ? '<th>헤딩</th>' : ''}
                                    </tr>
                                </thead>
                                <tbody id="data-table-body"></tbody>
                            </table>
                            ${Config.settings.dataVisualization ? `
                                <div id="data-visualization">
                                    <h3>데이터 시각화</h3>
                                    <canvas id="data-chart"></canvas>
                                </div>
                            ` : ''}
                        </div>
                        <div id="tab-settings" style="display: none;">
                            <h3>설정</h3>
                            ${createSettingsSections()}
                            <div class="settings-section" style="display: flex; gap: 10px;">
                                <button id="save-settings" style="padding: 10px 20px; background-color: #4caf50; color: white; border-radius: 4px; font-size: 14px;">저장</button>
                                <button id="reset-settings" style="padding: 10px 20px; background-color: #f44336; color: white; border-radius: 4px; font-size: 14px;">초기화</button>
                            </div>
                        </div>
                        <div id="tab-templates" style="display: none;">
                            <h3>설정 템플릿</h3>
                            <div class="settings-section">
                                <label for="new-template-name">새 템플릿 이름:</label>
                                <input type="text" id="new-template-name" placeholder="템플릿 이름을 입력하세요">
                                <button id="save-template" style="margin-top: 10px; padding: 8px 12px; background-color: #4caf50; color: white; border-radius: 4px; font-size: 14px;">📁 템플릿 저장</button>
                            </div>
                            <div class="settings-section">
                                <label for="template-select">기존 템플릿:</label>
                                <select id="template-select" style="width: 100%; padding: 8px; margin-top: 10px; border: 1px solid #ccc; border-radius: 4px;">
                                    <option value="" disabled selected>템플릿 선택</option>
                                </select>
                                <button id="load-template" style="margin-top: 10px; padding: 8px 12px; background-color: #2196F3; color: white; border-radius: 4px; font-size: 14px;">📂 템플릿 불러오기</button>
                                <button id="delete-template" style="margin-top: 5px; padding: 8px 12px; background-color: #f44336; color: white; border-radius: 4px; font-size: 14px;">템플릿 삭제</button>
                            </div>
                        </div>
                        <div id="tab-previous-crawls" style="display: none;">
                            <h3>이전 크롤링 기록</h3>
                            <button id="clear-previous-crawls" style="padding: 8px 12px; background-color: #f44336; color: white; border-radius: 4px; font-size: 14px; margin-bottom: 10px;">기록 초기화</button>
                            <table>
                                <thead>
                                    <tr>
                                        <th>날짜</th>
                                        <th>크롤링 수</th>
                                        <th>상태</th>
                                    </tr>
                                </thead>
                                <tbody id="previous-crawls-body"></tbody>
                            </table>
                        </div>
                        <div id="tab-tests" style="display: none;">
                            <button id="run-tests" style="padding: 12px 24px; background-color: #FFC107; color: white; border-radius: 4px; font-size: 16px;">테스트 실행</button>
                        </div>
                        <div id="tab-ai-assistant" style="display: none;">
                            <h3>AI 어시스턴트</h3>
                            <iframe id="ai-assistant-iframe" src="https://huggingface.co/spaces/gpt4all/chat"></iframe>
                        </div>
                        <div id="tab-strategies" style="display: none;">
                            <h3>크롤링 전략</h3>
                            <div class="settings-section">
                                <label for="crawling-strategies-select">사용할 크롤링 전략 선택:</label>
                                <select id="crawling-strategies-select" multiple size="10">
                                </select>
                                <button id="update-strategies" style="margin-top: 10px; padding: 8px 12px; background-color: #4caf50; color: white; border-radius: 4px; font-size: 14px;">전략 업데이트</button>
                            </div>
                            <div class="comment-window">
                                <label for="strategy-comment">선택한 전략에 대한 댓글:</label>
                                <textarea id="strategy-comment" placeholder="전략에 대한 의견이나 메모를 입력하세요..."></textarea>
                                <button id="save-strategy-comment" style="margin-top: 10px; padding: 8px 12px; background-color: #2196F3; color: white; border-radius: 4px; font-size: 14px;">댓글 저장</button>
                            </div>
                            <p>선택한 크롤링 전략에 따라 크롤링 방식을 조정합니다. 여러 전략을 조합할 수 있습니다.</p>
                        </div>
                        <div id="html-modal" class="modal">
                            <div class="modal-content">
                                <span class="close">&times;</span>
                                <iframe id="modal-html-content-rendered"></iframe>
                            </div>
                        </div>
                        <div id="test-modal" class="modal">
                            <div class="modal-content">
                                <span class="close">&times;</span>
                                <div id="mocha"></div>
                            </div>
                        </div>
                    </div>
                </div>
            `);
            $('body').append(container);
            populateCrawlingStrategies();
            bindEvents();
            makeDraggable();
        }
    
        function createSettingsSections() {
            const sections = [
                // ... [Other settings sections remain unchanged] ...
                // Add image field section if needed
                // ...
    
                // Crawling strategies comment section
                {
                    label: '크롤링 전략에 대한 댓글',
                    id: 'strategy-comment',
                    type: 'textarea',
                    value: Config.settings.crawlingStrategyComments[Config.settings.selectedCrawlingStrategies.join(', ')] || '',
                    tooltip: '선택한 크롤링 전략에 대한 의견이나 메모를 입력하세요.'
                }
            ];
    
            // The existing settings sections are created here, plus any new sections
            // For brevity, only including the relevant part for the comment window
            return `
                <!-- Existing settings sections -->
    
                <!-- Representative Photos Settings (Optional if needed) -->
                <!-- You can add specific settings for representative photos if needed -->
    
                <!-- Crawling Strategies Comment Section -->
                <div class="settings-section">
                    <label for="strategy-comment">크롤링 전략에 대한 댓글:
                        <span class="tooltip">ℹ️
                            <span class="tooltiptext">선택한 크롤링 전략에 대한 의견이나 메모를 입력하세요.</span>
                        </span>
                    </label>
                    <textarea id="strategy-comment" rows="4">${Config.settings.crawlingStrategyComments[Config.settings.selectedCrawlingStrategies.join(', ')] || ''}</textarea>
                </div>
            `;
        }
    
        function bindEvents() {
            $(selectors.minimizeBtn).on('click', toggleMinimize);
            $(selectors.tabsButtons).on('click', switchTab);
            $('#save-settings').on('click', saveSettingsHandler);
            $('#reset-settings').on('click', resetSettingsHandler);
            $('#add-user-agent').on('click', addUserAgent);
            $('#remove-user-agent').on('click', removeUserAgent);
            $('#download-csv').on('click', Download.downloadCSV);
            $('#download-json').on('click', Download.downloadJSON);
            $('#download-xml').on('click', Download.downloadXML);
            $('#download-xls').on('click', Download.downloadXLS);
            $(selectors.searchFilter).on('input', filterData);
            $('#schedule-enabled').on('change', toggleScheduleOptions);
            $('#data-visualization-checkbox').on('change', toggleDataVisualization);
            $('#enable-captchas-checkbox').on('change', toggleCaptchaHandling);
            $('#save-template').on('click', saveTemplateHandler);
            $('#load-template').on('click', loadTemplateHandler);
            $('#delete-template').on('click', deleteTemplateHandler);
            $(selectors.templateSelect).on('change', selectTemplateHandler);
            $('#run-tests').on('click', Testing.runTests);
            $('#clear-previous-crawls').on('click', clearPreviousCrawls);
            $('#reset-data').on('click', resetData);
            $(selectors.darkModeToggle).on('click', toggleDarkMode);
            $('.close').on('click', () => $('.modal').hide());
            $(window).on('click', e => { if ($(e.target).hasClass('modal')) $('.modal').hide(); });
            $('#start-crawling').on('click', Crawling.startCrawling);
            $('#stop-crawling').on('click', Crawling.stopCrawling);
            $('#update-strategies').on('click', updateCrawlingStrategies);
            $('#save-strategy-comment').on('click', saveStrategyComment); // New event for saving comments
            $(document).on('keydown', handleShortcuts);
            $('#proxy-input').on('input', validateProxyInput);
            $('input[type="checkbox"], input[type="radio"]').on('change', toggleDataField);
            $('.view-html').off('click').on('click', function() {
                const htmlContent = decodeURIComponent($(this).data('html'));
                $('#modal-html-content-rendered').attr('srcdoc', htmlContent);
                $('#html-modal').show();
            });
        }
    
        function toggleMinimize() {
            $(selectors.content).toggle();
            $(selectors.minimizeBtn).text($(selectors.content).is(':hidden') ? '☐' : '━');
            Logger.log(`크롤러 창이 ${$(selectors.content).is(':hidden') ? '최소화' : '확대'}되었습니다.`, 'info');
        }
    
        function switchTab() {
            const tab = $(this).data('tab');
            $(selectors.tabsButtons).removeClass('active');
            $(this).addClass('active');
            $(selectors.content).children('div').hide();
            $(`#tab-${tab.replace(/\s+/g, '-').toLowerCase()}`).show();
            Config.saveSettings({ currentTab: tab.replace(/\s+/g, '-').toLowerCase() });
        }
    
        function populateCrawlingStrategies() {
            Config.settings.crawlingStrategies.forEach(strategy => {
                $('#crawling-strategies-select').append(`<option value="${strategy}" ${Config.settings.selectedCrawlingStrategies.includes(strategy) ? 'selected' : ''}>${strategy}</option>`);
            });
        }
    
        function updateUI() {
            const currentTab = Config.settings.currentTab || 'logs';
            $(`.crawler-tab[data-tab="${currentTab.replace(/-/g, ' ')}"]`).addClass('active');
            $(selectors.content).children('div').hide();
            $(`#tab-${currentTab}`).show();
            UI.updateLogs();
            UI.updateDataTable();
            UI.updatePreviousCrawlsUI();
            UI.updateTemplatesUI();
            populateCrawlingStrategies();
            if (Config.settings.dataVisualization) DataVisualization.renderChart();
            applyDarkMode();
        }
    
        function updateLogs() {
            const logs = Logger.getLogs();
            $(selectors.logMessages).html(logs.map(l => `<div>[${l.timestamp}] [${l.type.toUpperCase()}] ${Utils.escape(l.message)}</div>`).join(''));
            const progress = GM_getValue('progress', 0);
            const currentPage = GM_getValue('currentPage', 0);
            const totalPages = Config.settings.maxPages;
            $('.progress-bar-fill').css('width', `${progress.toFixed(2)}%`);
            $('.status-text').text(`진행률: ${progress.toFixed(2)}% (${currentPage}/${totalPages} 페이지)`);
            const latestLog = logs[logs.length -1];
            if (latestLog) {
                if (['error', 'warning'].includes(latestLog.type)) {
                    $(selectors.errorMessage).text(latestLog.message).show().delay(5000).fadeOut();
                }
                if (['success', 'info'].includes(latestLog.type)) {
                    $(selectors.successMessage).text(latestLog.message).show().delay(5000).fadeOut();
                }
            }
        }
    
        function updateDataTable() {
            const data = JSON.parse(GM_getValue('data', '[]'));
            const filter = $(selectors.searchFilter).val().toLowerCase();
            const filtered = filter ? data.filter(item => Object.keys(Config.settings.dataFields).some(f => Config.settings.dataFields[f] && String(item[f]).toLowerCase().includes(filter))) : data;
            $(selectors.dataTableBody).html(filtered.map(item => `
                <tr>
                    ${Config.settings.dataFields.image ? `<td class="image-column">${item.image ? `<img src="${Utils.escape(item.image)}" alt="Image" class="site-image">` : 'No Image'}</td>` : ''}
                    ${Config.settings.dataFields.name ? `<td>${Utils.escape(item.name)}</td>` : ''}
                    ${Config.settings.dataFields.url ? `<td><a href="${Utils.escape(item.url)}" target="_blank" class="full-url">${Utils.escape(item.url)}</a></td>` : ''}
                    ${Config.settings.dataFields.description ? `<td>${Utils.escape(item.description)}</td>` : ''}
                    ${Config.settings.dataFields.html ? `<td><button class="view-html" data-html="${encodeURIComponent(item.html)}">📄 보기</button></td>` : ''}
                    ${Config.settings.dataFields.category ? `<td>${Utils.escape(item.category)}</td>` : ''}
                    ${Config.settings.dataFields.quality ? `<td>${Utils.escape(item.quality.toString())}</td>` : ''}
                    ${Config.settings.dataFields.language ? `<td>${Utils.escape(item.language)}</td>` : ''}
                    ${Config.settings.dataFields.wordCount ? `<td>${Utils.escape(item.wordCount.toString())}</td>` : ''}
                    ${Config.settings.dataFields.metaTags ? `<td>${Utils.escape(JSON.stringify(item.metaTags))}</td>` : ''}
                    ${Config.settings.dataFields.headings ? `<td>${Utils.escape(item.headings.join(', '))}</td>` : ''}
                </tr>
            `).join(''));
            $('.view-html').off('click').on('click', function() {
                const htmlContent = decodeURIComponent($(this).data('html'));
                $(selectors.modalHtmlContent).attr('srcdoc', htmlContent);
                $(selectors.htmlModal).show();
            });
            if (Config.settings.dataVisualization) DataVisualization.renderChart();
        }
    
        function updatePreviousCrawlsUI() {
            const previousCrawls = JSON.parse(GM_getValue('previousCrawls', '[]'));
            $(selectors.previousCrawlsBody).html(previousCrawls.map(crawl => `
                <tr>
                    <td>${Utils.escape(crawl.date)}</td>
                    <td>${Utils.escape(crawl.count.toString())}</td>
                    <td>${Utils.escape(crawl.status)}</td>
                </tr>
            `).join(''));
        }
    
        function updateTemplatesUI() {
            $(selectors.templateSelect).empty().append(`<option value="" disabled selected>템플릿 선택</option>`);
            Object.keys(Config.settings.templates).forEach(t => {
                $(selectors.templateSelect).append(`<option value="${t}">${t}</option>`);
            });
        }
    
        function addUserAgent() {
            const ua = prompt("새 사용자 에이전트를 입력하세요:");
            if (ua) {
                Config.saveSettings({ userAgents: [...Config.settings.userAgents, ua] });
                updateUserAgentSelect();
                Logger.log('새 사용자 에이전트가 추가되었습니다.', 'success');
            }
        }
    
        function removeUserAgent() {
            const selected = $('#user-agent-select').val();
            if (selected && selected.length) {
                selected.sort((a, b) => b - a).forEach(idx => Config.settings.userAgents.splice(idx, 1));
                Config.saveSettings({ userAgents: Config.settings.userAgents, userAgentIndex: 0 });
                updateUserAgentSelect();
                Logger.log('선택한 사용자 에이전트가 제거되었습니다.', 'success');
            } else {
                alert('제거할 사용자 에이전트를 선택하세요.');
            }
        }
    
        function updateUserAgentSelect() {
            $('#user-agent-select').empty();
            Config.settings.userAgents.forEach((ua, idx) => {
                $('#user-agent-select').append(`<option value="${idx}" ${idx === Config.settings.userAgentIndex ? 'selected' : ''}>${ua}</option>`);
            });
        }
    
        function saveSettingsHandler() {
            const newSettings = {
                delay: parseInt($('#delay-input').val(), 10) || Config.DEFAULT_SETTINGS.delay,
                delayRandom: parseInt($('#delay-random-input').val(), 10) || Config.DEFAULT_SETTINGS.delayRandom,
                maxPages: parseInt($('#max-pages-input').val(), 10) || Config.DEFAULT_SETTINGS.maxPages,
                userAgentIndex: parseInt($('#user-agent-select').val(), 10) || 0,
                contentTypes: {
                    include: $('#content-include-input').val().split(',').map(ct => ct.trim()).filter(Boolean),
                    exclude: $('#content-exclude-input').val().split(',').map(ct => ct.trim()).filter(Boolean)
                },
                timeout: parseInt($('#timeout-input').val(), 10) || Config.DEFAULT_SETTINGS.timeout,
                includeKeywords: $('#include-keywords-input').val().split('\n').map(k => k.trim()).filter(Boolean),
                excludeKeywords: $('#exclude-keywords-input').val().split('\n').map(k => k.trim()).filter(Boolean),
                regexInclude: $('#regex-include-input').val().split('\n').map(r => r.trim()).filter(Boolean),
                regexExclude: $('#regex-exclude-input').val().split('\n').map(r => r.trim()).filter(Boolean),
                language: $('#language-select').val() || [],
                minWordCount: parseInt($('#min-word-count-input').val(), 10) || Config.DEFAULT_SETTINGS.minWordCount,
                maxWordCount: parseInt($('#max-word-count-input').val(), 10) || Config.DEFAULT_SETTINGS.maxWordCount,
                maxDepth: parseInt($('#max-depth-input').val(), 10) || Config.DEFAULT_SETTINGS.maxDepth,
                proxy: $('#proxy-input').val().trim(),
                dataFields: Object.fromEntries(['name','url','description','html','category','quality','language','wordCount','metaTags','headings','image'].map(f => [`${f}`, $(`#field-${f}`).is(':checked') ])),
                retryAttempts: parseInt($('#retry-attempts-input').val(), 10) || Config.DEFAULT_SETTINGS.retryAttempts,
                retryDelay: parseInt($('#retry-delay-input').val(), 10) || Config.DEFAULT_SETTINGS.retryDelay,
                schedule: {
                    enabled: $('#schedule-enabled').is(':checked'),
                    startTime: $('#start-time-input').val() || Config.DEFAULT_SETTINGS.schedule.startTime,
                    interval: (parseInt($('#interval-input').val(), 10) || (Config.DEFAULT_SETTINGS.schedule.interval / 60000)) * 60000,
                    specificDays: $('#specific-days-input').val() ? $('#specific-days-input').val().map(d => parseInt(d, 10)) : []
                },
                selectedCrawlingStrategies: $('#crawling-strategies-select').val() || Config.DEFAULT_SETTINGS.selectedCrawlingStrategies,
                searchEngines: Object.fromEntries(Object.keys(Config.settings.searchEngines).map(engine => [`${engine}`, $(`#engine-${engine}`).is(':checked') ])),
                blacklist: $('#blacklist-input').val().split('\n').map(d => d.trim()).filter(Boolean),
                whitelist: $('#whitelist-input').val().split('\n').map(d => d.trim()).filter(Boolean),
                respectRobots: $('#respect-robots-checkbox').is(':checked'),
                maxConcurrentRequests: parseInt($('#max-concurrent-requests-input').val(), 10) || Config.DEFAULT_SETTINGS.maxConcurrentRequests,
                dataVisualization: $('#data-visualization-checkbox').is(':checked'),
                exportFormats: Object.fromEntries(['csv','json','xml','xls'].map(fmt => [`${fmt}`, $(`#export-${fmt}`).is(':checked') ])),
                advancedLogging: $('#advanced-logging-checkbox').is(':checked'),
                enableCaptchas: $('#enable-captchas-checkbox').is(':checked'),
                customKeywords: Utils.parseCustomKeywords($('#custom-keywords-input').val()),
                crawlingStrategyComments: {
                    ...Config.settings.crawlingStrategyComments,
                    [Config.settings.selectedCrawlingStrategies.join(', ')]: $('#strategy-comment').val().trim()
                }
            };
    
            if (newSettings.proxy && !Config.validateProxy(newSettings.proxy)) {
                $(selectors.errorMessage).text('유효하지 않은 프록시 URL입니다.').show();
                setTimeout(() => $(selectors.errorMessage).fadeOut(), 5000);
                return;
            }
    
            Config.saveSettings(newSettings);
            Crawling.updateCrawlSettings();
            Logger.log('설정이 저장되었습니다.', 'success');
            if (newSettings.schedule.enabled) Scheduler.scheduleCrawling();
            updateDataTable();
            if (newSettings.dataVisualization) DataVisualization.renderChart();
        }
    
        function resetSettingsHandler() {
            if (confirm("기본 설정으로 초기화하시겠습니까?")) {
                Config.resetSettings();
                location.reload();
            }
        }
    
        function toggleScheduleOptions() {
            $('#start-time-input, #interval-input, #specific-days-input').toggle($('#schedule-enabled').is(':checked'));
        }
    
        function toggleDataVisualization() {
            const enabled = $('#data-visualization-checkbox').is(':checked');
            Config.saveSettings({ dataVisualization: enabled });
            if (enabled) {
                $('#data-visualization').show();
                DataVisualization.renderChart();
            } else {
                $('#data-visualization').hide();
            }
            Logger.log(`데이터 시각화가 ${enabled ? '활성화' : '비활성화'}되었습니다.`, 'success');
        }
    
        function toggleCaptchaHandling() {
            const enabled = $('#enable-captchas-checkbox').is(':checked');
            Config.saveSettings({ enableCaptchas: enabled });
            Logger.log(`CAPTCHA 처리 ${enabled ? '활성화' : '비활성화'}되었습니다.`, 'success');
        }
    
        function filterData() {
            updateDataTable();
        }
    
        function clearPreviousCrawls() {
            if (confirm("이전 크롤링 기록을 모두 초기화하시겠습니까?")) {
                GM_setValue('previousCrawls', JSON.stringify([]));
                updatePreviousCrawlsUI();
                Logger.log('이전 크롤링 기록이 초기화되었습니다.', 'success');
            }
        }
    
        function resetData() {
            if (confirm("모든 크롤링 데이터를 초기화하시겠습니까?")) {
                GM_setValue('data', JSON.stringify([]));
                GM_setValue('visitedUrls', JSON.stringify([]));
                updateDataTable();
                Logger.log('크롤링 데이터가 초기화되었습니다.', 'success');
            }
        }
    
        function handleShortcuts(e) {
            if (e.ctrlKey) {
                const actions = {
                    '1': 'logs',
                    '2': 'data',
                    '3': 'settings',
                    '4': 'previous-crawls',
                    '5': 'templates',
                    '6': 'tests',
                    '7': 'ai-assistant',
                    '8': 'strategies',
                    's': 'save',
                    'r': 'reset',
                    'p': 'start',
                    'q': 'stop',
                    'c': 'csv',
                    'j': 'json',
                    'x': 'xml',
                    'l': 'xls',
                    'd': 'dark-mode'
                };
                const action = actions[e.key];
                if (action) {
                    e.preventDefault();
                    switch(action) {
                        case 'logs': $('.crawler-tab[data-tab="logs"]').click(); break;
                        case 'data': $('.crawler-tab[data-tab="data"]').click(); break;
                        case 'settings': $('.crawler-tab[data-tab="settings"]').click(); break;
                        case 'previous-crawls': $('.crawler-tab[data-tab="previous-crawls"]').click(); break;
                        case 'templates': $('.crawler-tab[data-tab="templates"]').click(); break;
                        case 'tests': $('.crawler-tab[data-tab="tests"]').click(); break;
                        case 'ai-assistant': $('.crawler-tab[data-tab="ai-assistant"]').click(); break;
                        case 'strategies': $('.crawler-tab[data-tab="strategies"]').click(); break;
                        case 'save': saveSettingsHandler(); break;
                        case 'reset': resetSettingsHandler(); break;
                        case 'start': if (!Crawling.isCrawling()) Crawling.startCrawling(); break;
                        case 'stop': if (Crawling.isCrawling()) Crawling.stopCrawling(); break;
                        case 'csv': Config.settings.exportFormats.csv && Download.downloadCSV(); break;
                        case 'json': Config.settings.exportFormats.json && Download.downloadJSON(); break;
                        case 'xml': Config.settings.exportFormats.xml && Download.downloadXML(); break;
                        case 'xls': Config.settings.exportFormats.xls && Download.downloadXLS(); break;
                        case 'dark-mode': toggleDarkMode(); break;
                    }
                }
            }
        }
    
        function makeDraggable() {
            const container = document.getElementById('crawler-container');
            const header = document.getElementById('crawler-header');
            let isDragging = false, startX, startY, initialX, initialY;
            header.onmousedown = function(e) {
                isDragging = true;
                startX = e.clientX;
                startY = e.clientY;
                initialX = container.offsetLeft;
                initialY = container.offsetTop;
                document.onmousemove = drag;
                document.onmouseup = stopDrag;
                return false;
            };
            function drag(e) {
                if (isDragging) {
                    container.style.left = `${initialX + e.clientX - startX}px`;
                    container.style.top = `${initialY + e.clientY - startY}px`;
                }
            }
            function stopDrag() {
                isDragging = false;
                document.onmousemove = null;
                document.onmouseup = null;
            }
        }
    
        function getFieldName(field) {
            const names = {
                name: '이름',
                url: '주소',
                description: '설명',
                html: 'HTML 내용',
                category: '분류',
                quality: '품질점수',
                language: '언어',
                wordCount: '단어 수',
                metaTags: '메타 태그',
                headings: '헤딩',
                image: '이미지'
            };
            return names[field] || field;
        }
    
        function saveTemplateHandler() {
            const templateName = $(selectors.newTemplateName).val().trim();
            if (!templateName) {
                alert('템플릿 이름을 입력하세요.');
                return;
            }
            Config.saveTemplate(templateName);
            updateTemplatesUI();
            $(selectors.newTemplateName).val('');
        }
    
        function loadTemplateHandler() {
            const selected = $(selectors.templateSelect).val();
            if (!selected) {
                alert('불러올 템플릿을 선택하세요.');
                return;
            }
            Config.loadTemplate(selected);
            updateUI();
            $('.crawler-tab[data-tab="settings"]').click();
        }
    
        function deleteTemplateHandler() {
            const selected = $(selectors.templateSelect).val();
            if (!selected) {
                alert('삭제할 템플릿을 선택하세요.');
                return;
            }
            delete Config.settings.templates[selected];
            Config.saveSettings({ templates: Config.settings.templates });
            Logger.log(`템플릿 ${selected}이(가) 삭제되었습니다.`, 'success');
            updateTemplatesUI();
        }
    
        function selectTemplateHandler() {
            const selected = $(this).val();
            if (selected) {
                Config.loadTemplate(selected);
                updateUI();
                $('.crawler-tab[data-tab="settings"]').click();
            }
        }
    
        function toggleDarkMode() {
            $(selectors.container).toggleClass('dark');
            const isDark = $(selectors.container).hasClass('dark');
            $(selectors.darkModeToggle).text(isDark ? '☀️' : '🌙');
            Config.saveSettings({ darkMode: isDark });
            Logger.log(`다크 모드가 ${isDark ? '활성화' : '비활성화'}되었습니다.`, 'success');
        }
    
        function applyDarkMode() {
            if (Config.settings.darkMode) {
                $(selectors.container).addClass('dark');
                $(selectors.darkModeToggle).text('☀️');
            } else {
                $(selectors.container).removeClass('dark');
                $(selectors.darkModeToggle).text('🌙');
            }
        }
    
        function toggleDataField() {
            const field = $(this).attr('id').replace('field-', '');
            Config.settings.dataFields[field] = $(this).is(':checked');
            Config.saveSettings({ dataFields: Config.settings.dataFields });
            Download.updateExportButtons();
        }
    
        function validateProxyInput() {
            const proxy = $('#proxy-input').val().trim();
            if (proxy && !Config.validateProxy(proxy)) {
                $(selectors.errorMessage).text('유효하지 않은 프록시 URL입니다.').show();
            } else {
                $(selectors.errorMessage).hide();
            }
        }
    
        function updateCrawlingStrategies() {
            const selected = $('#crawling-strategies-select').val() || [];
            Config.saveSettings({ selectedCrawlingStrategies: selected });
            Logger.log('크롤링 전략이 업데이트되었습니다.', 'success');
        }
    
        function saveStrategyComment() {
            const comment = $('#strategy-comment').val().trim();
            const strategies = Config.settings.selectedCrawlingStrategies.join(', ');
            Config.saveSettings({
                crawlingStrategyComments: {
                    ...Config.settings.crawlingStrategyComments,
                    [strategies]: comment
                }
            });
            Logger.log('크롤링 전략에 대한 댓글이 저장되었습니다.', 'success');
        }
    
        return {
            createUI,
            updateUI,
            updateLogs,
            updateDataTable,
            updatePreviousCrawlsUI,
            updateTemplatesUI,
            applyDarkMode
        };
    })();
    
    const Download = (() => {
        function downloadCSV() {
            const data = JSON.parse(GM_getValue('data', '[]'));
            if (!data.length) { Logger.log('다운로드할 데이터가 없습니다.', 'warning'); return; }
            const headers = Object.keys(Config.settings.dataFields).filter(f => Config.settings.dataFields[f]).map(f => Utils.capitalize(getFieldName(f)));
            const csvContent = [headers.join(",")]
                .concat(data.map(item => Object.keys(Config.settings.dataFields).filter(f => Config.settings.dataFields[f]).map(f => `"${Utils.escape(item[f], 'csv')}"`).join(",")))
                .join("\n");
            const blob = new Blob([csvContent], { type: "text/csv;charset=utf-8;" });
            const link = document.createElement("a");
            link.href = URL.createObjectURL(blob);
            link.download = "data.csv";
            document.body.appendChild(link);
            link.click();
            document.body.removeChild(link);
            Logger.log('CSV 파일이 다운로드되었습니다.', 'success');
        }
    
        function downloadJSON() {
            const data = JSON.parse(GM_getValue('data', '[]'));
            if (!data.length) { Logger.log('다운로드할 데이터가 없습니다.', 'warning'); return; }
            const jsonContent = JSON.stringify(data, null, 2);
            const blob = new Blob([jsonContent], { type: "application/json" });
            const link = document.createElement("a");
            link.href = URL.createObjectURL(blob);
            link.download = "data.json";
            document.body.appendChild(link);
            link.click();
            document.body.removeChild(link);
            Logger.log('JSON 파일이 다운로드되었습니다.', 'success');
        }
    
        function downloadXML() {
            const data = JSON.parse(GM_getValue('data', '[]'));
            if (!data.length) { Logger.log('다운로드할 데이터가 없습니다.', 'warning'); return; }
            let xmlContent = '<?xml version="1.0" encoding="UTF-8"?><data>';
            data.forEach(item => {
                xmlContent += '<entry>';
                if (Config.settings.dataFields.name) xmlContent += `<name>${Utils.escape(item.name || 'No Title', 'xml')}</name>`;
                if (Config.settings.dataFields.url) xmlContent += `<url>${Utils.escape(item.url || '', 'xml')}</url>`;
                if (Config.settings.dataFields.description) xmlContent += `<description>${Utils.escape(item.description || 'No Description', 'xml')}</description>`;
                if (Config.settings.dataFields.html) xmlContent += `<html><![CDATA[${item.html || ''}]]></html>`;
                if (Config.settings.dataFields.category) xmlContent += `<category>${Utils.escape(item.category || '', 'xml')}</category>`;
                if (Config.settings.dataFields.quality) xmlContent += `<quality>${Utils.escape(item.quality)}</quality>`;
                if (Config.settings.dataFields.language) xmlContent += `<language>${Utils.escape(item.language || 'unknown', 'xml')}</language>`;
                if (Config.settings.dataFields.wordCount) xmlContent += `<wordCount>${Utils.escape(item.wordCount)}</wordCount>`;
                if (Config.settings.dataFields.metaTags) xmlContent += `<metaTags>${Utils.escape(JSON.stringify(item.metaTags), 'xml')}</metaTags>`;
                if (Config.settings.dataFields.headings) xmlContent += `<headings>${Utils.escape(item.headings.join('; '), 'xml')}</headings>`;
                if (Config.settings.dataFields.image) xmlContent += `<image>${Utils.escape(item.image || '', 'xml')}</image>`;
                xmlContent += '</entry>';
            });
            xmlContent += '</data>';
            const blob = new Blob([xmlContent], { type: "application/xml" });
            const link = document.createElement("a");
            link.href = URL.createObjectURL(blob);
            link.download = "data.xml";
            document.body.appendChild(link);
            link.click();
            document.body.removeChild(link);
            Logger.log('XML 파일이 다운로드되었습니다.', 'success');
        }
    
        function downloadXLS() {
            const data = JSON.parse(GM_getValue('data', '[]'));
            if (!data.length) { Logger.log('다운로드할 데이터가 없습니다.', 'warning'); return; }
            const worksheetData = data.map(item => {
                const entry = {};
                Object.keys(Config.settings.dataFields).forEach(f => {
                    if (Config.settings.dataFields[f]) {
                        if (f === 'metaTags') entry['메타 태그'] = JSON.stringify(item[f]);
                        else if (f === 'headings') entry['헤딩'] = item[f].join('; ');
                        else if (f === 'image') entry['이미지'] = item[f] ? `[Image: ${item[f]}]` : 'No Image';
                        else entry[Utils.capitalize(getFieldName(f))] = item[f];
                    }
                });
                return entry;
            });
            const worksheet = XLSX.utils.json_to_sheet(worksheetData);
            const workbook = XLSX.utils.book_new();
            XLSX.utils.book_append_sheet(workbook, worksheet, "Data");
            XLSX.writeFile(workbook, "data.xlsx");
            Logger.log('XLS 파일이 다운로드되었습니다.', 'success');
        }
    
        function getFieldName(field) {
            const names = {
                name: '이름',
                url: '주소',
                description: '설명',
                html: 'HTML 내용',
                category: '분류',
                quality: '품질점수',
                language: '언어',
                wordCount: '단어 수',
                metaTags: '메타 태그',
                headings: '헤딩',
                image: '이미지'
            };
            return names[field] || field;
        }
    
        function updateExportButtons() {
            // Placeholder for dynamic enabling/disabling of export buttons based on data fields
        }
    
        return {
            downloadCSV,
            downloadJSON,
            downloadXML,
            downloadXLS,
            updateExportButtons
        };
    })();
    
    const DataVisualization = (() => {
        let chartInstance = null;
    
        function renderChart() {
            const ctx = document.getElementById('data-chart').getContext('2d');
            if (chartInstance) chartInstance.destroy();
            const data = JSON.parse(GM_getValue('data', '[]'));
            const categories = data.reduce((acc, item) => {
                const cat = item.category || '기타';
                acc[cat] = (acc[cat] || 0) + 1;
                return acc;
            }, {});
            const chartData = {
                labels: Object.keys(categories),
                datasets: [{
                    label: '카테고리별 데이터 수',
                    data: Object.values(categories),
                    backgroundColor: Utils.generateColors(Object.keys(categories).length)
                }]
            };
            chartInstance = new Chart(ctx, { type: 'pie', data: chartData, options: { responsive: true } });
        }
    
        return {
            renderChart
        };
    })();
    
    const Crawling = (() => {
        let isCrawlingFlag = Config.settings.isCrawling;
        let currentPage = parseInt(GM_getValue('currentPage', '1'), 10);
        let totalPages = Config.settings.maxPages;
        let progress = parseFloat(GM_getValue('progress', '0'));
        let currentDepth = parseInt(GM_getValue('currentDepth', '0'), 10);
        let activeRequests = 0;
        let requestQueue = [];
        let previousCrawls = JSON.parse(GM_getValue('previousCrawls', '[]'));
        let visitedUrls = new Set(JSON.parse(GM_getValue('visitedUrls', '[]')));
        let data = JSON.parse(GM_getValue('data', '[]'));
        let navigationObserver = null;
    
        const searchEngineHandlers = {
            google: {
                extractResults: doc => {
                    const results = [];
                    doc.querySelectorAll('div.g, div[data-hveid]').forEach(item => {
                        const name = item.querySelector('h3')?.innerText.trim() || 'No Title';
                        const url = item.querySelector('a')?.href || '';
                        const desc = item.querySelector('.aCOpRe, .VwiC3b')?.innerText.trim() || 'No Description';
                        const image = item.querySelector('img')?.src || ''; // Extract image
                        if (Utils.isValidURL(url)) results.push({ name, url, description: desc, image });
                    });
                    return results;
                },
                getNextPageUrl: doc => doc.querySelector('a#pnnext')?.href || null,
                getContentContainer: () => '#search'
            },
            bing: {
                extractResults: doc => {
                    const results = [];
                    doc.querySelectorAll('.b_algo').forEach(item => {
                        const name = item.querySelector('h2')?.innerText.trim() || 'No Title';
                        const url = item.querySelector('h2 a')?.href || '';
                        const desc = item.querySelector('.b_caption p')?.innerText.trim() || 'No Description';
                        const image = item.querySelector('img')?.src || ''; // Extract image
                        if (Utils.isValidURL(url)) results.push({ name, url, description: desc, image });
                    });
                    return results;
                },
                getNextPageUrl: doc => doc.querySelector('a.sb_pagN')?.href || null,
                getContentContainer: () => '#b_content'
            },
            yahoo: {
                extractResults: doc => {
                    const results = [];
                    doc.querySelectorAll('.dd.algo').forEach(item => {
                        const name = item.querySelector('h3')?.innerText.trim() || 'No Title';
                        const url = item.querySelector('a')?.href || '';
                        const desc = item.querySelector('.compText a')?.innerText.trim() || 'No Description';
                        const image = item.querySelector('img')?.src || ''; // Extract image
                        if (Utils.isValidURL(url)) results.push({ name, url, description: desc, image });
                    });
                    return results;
                },
                getNextPageUrl: doc => doc.querySelector('a.next')?.href || null,
                getContentContainer: () => '#web'
            },
            duckduckgo: {
                extractResults: doc => {
                    const results = [];
                    doc.querySelectorAll('.result').forEach(item => {
                        const name = item.querySelector('h2')?.innerText.trim() || 'No Title';
                        const url = item.querySelector('a.result__a')?.href || '';
                        const desc = item.querySelector('.result__snippet')?.innerText.trim() || 'No Description';
                        const image = item.querySelector('img.result__image')?.src || ''; // Extract image
                        if (Utils.isValidURL(url)) results.push({ name, url, description: desc, image });
                    });
                    return results;
                },
                getNextPageUrl: doc => doc.querySelector('a.result--more__btn')?.href || null,
                getContentContainer: () => '#links'
            },
            // Additional search engines can be defined here
        };
    
        async function startCrawling() {
            if (isCrawlingFlag) {
                Logger.log('이미 크롤링이 진행 중입니다.', 'warning');
                return;
            }
            if (!Object.values(Config.settings.searchEngines).some(v => v)) {
                Logger.log('사용할 검색 엔진이 하나도 선택되지 않았습니다.', 'error');
                return;
            }
            isCrawlingFlag = true;
            Config.saveSettings({ isCrawling: isCrawlingFlag });
            Logger.log('크롤링을 시작합니다.', 'info');
            UI.updateLogs();
            if (!previousCrawls.some(crawl => crawl.status === '진행 중')) {
                previousCrawls.push({ date: new Date().toLocaleString(), count: 0, status: '진행 중' });
                GM_setValue('previousCrawls', JSON.stringify(previousCrawls));
                UI.updatePreviousCrawlsUI();
            }
            setButtonStates();
            await crawlPage(currentDepth);
        }
    
        function stopCrawling() {
            if (!isCrawlingFlag) {
                Logger.log('크롤링이 진행 중이지 않습니다.', 'warning');
                return;
            }
            isCrawlingFlag = false;
            Config.saveSettings({ isCrawling: isCrawlingFlag });
            Logger.log('크롤링이 중지되었습니다.', 'warning');
            finalizeCrawl('중지됨');
            UI.updateLogs();
            setButtonStates();
        }
    
        function setButtonStates() {
            $('#start-crawling').prop('disabled', isCrawlingFlag);
            $('#stop-crawling').prop('disabled', !isCrawlingFlag);
        }
    
        async function crawlPage(depth) {
            try {
                while (isCrawlingFlag && currentPage <= totalPages && depth <= Config.settings.maxDepth) {
                    Logger.log(`페이지 ${currentPage} 크롤링 시작. (깊이: ${depth})`, 'info');
                    if (Config.settings.respectRobots && !(await Robots.checkRobotsTxt(window.location.href))) {
                        Logger.log(`robots.txt에 의해 크롤링이 금지된 페이지: ${window.location.href}`, 'warning');
                        break;
                    }
                    const searchEngine = identifySearchEngine();
                    if (!searchEngine || !searchEngineHandlers[searchEngine]) {
                        Logger.log(`지원되지 않는 검색 엔진: ${window.location.hostname}`, 'error');
                        break;
                    }
                    const doc = document;
                    const handler = searchEngineHandlers[searchEngine];
                    const results = handler.extractResults(doc);
                    if (!results) { Logger.log('검색 결과를 찾을 수 없습니다.', 'error'); break; }
                    results.forEach(result => {
                        if (!isCrawlingFlag) return;
                        const { name, url, description, image } = result;
                        if (shouldSkip(url, description)) return;
                        requestQueue.push({ url, depth, image }); // Pass image to queue
                        if (activeRequests >= Config.settings.maxConcurrentRequests) return;
                    });
                    if (previousCrawls.length && previousCrawls[previousCrawls.length -1].status === '진행 중') {
                        previousCrawls[previousCrawls.length -1].count += results.length;
                        GM_setValue('previousCrawls', JSON.stringify(previousCrawls));
                        UI.updatePreviousCrawlsUI();
                    }
                    await processQueue();
                    progress = (currentPage / totalPages) * 100;
                    Config.saveSettings({ progress });
                    UI.updateLogs();
                    if (currentPage < totalPages && isCrawlingFlag) {
                        const nextUrl = handler.getNextPageUrl(doc);
                        if (nextUrl) {
                            currentPage++;
                            Config.saveSettings({ currentPage });
                            depth++;
                            Config.saveSettings({ currentDepth: depth });
                            Logger.log(`다음 페이지로 이동: ${currentPage}`, 'info');
                            await Utils.delay(Config.settings.delay + Math.random() * Config.settings.delayRandom);
                            const nextPageContent = await Fetch.fetchPageContent(nextUrl);
                            if (nextPageContent) {
                                // Parse the fetched content without altering the current page's DOM
                                const parser = new DOMParser();
                                const fetchedDoc = parser.parseFromString(nextPageContent, 'text/html');
                                // Temporarily replace document's body with fetched content for extraction
                                const originalBody = document.body.innerHTML;
                                document.body.innerHTML = fetchedDoc.body.innerHTML;
                                continue;
                            } else {
                                Logger.log('다음 페이지 내용을 가져올 수 없습니다.', 'error');
                                break;
                            }
                        } else { Logger.log('다음 페이지 링크를 찾을 수 없습니다.', 'error'); break; }
                    } else { break; }
                }
                if (isCrawlingFlag && currentPage > totalPages && activeRequests === 0 && requestQueue.length === 0) {
                    Logger.log('크롤링이 완료되었습니다.', 'success');
                    finalizeCrawl('완료됨');
                    isCrawlingFlag = false;
                    Config.saveSettings({ isCrawling: isCrawlingFlag });
                    setButtonStates();
                    Config.saveSettings({ progress: 100 });
                    UI.updateLogs();
                }
            } catch (error) {
                Logger.log(`크롤링 중 오류 발생: ${error.message}`, 'error');
                finalizeCrawl('오류 발생');
            }
        }
    
        function shouldSkip(url, desc) {
            if (visitedUrls.has(url) || !Utils.isValidURL(url)) return true;
            if (Config.settings.whitelist.length && !Config.settings.whitelist.some(d => url.includes(d))) return true;
            if (Config.settings.blacklist.some(d => url.includes(d))) return true;
            if (Config.settings.includeKeywords.length && !Config.settings.includeKeywords.some(k => url.includes(k) || desc.includes(k))) return true;
            if (Config.settings.excludeKeywords.some(k => url.includes(k) || desc.includes(k))) return true;
            if (Config.settings.regexInclude.length && !Config.settings.regexInclude.some(r => new RegExp(r).test(url) || new RegExp(r).test(desc))) return true;
            if (Config.settings.regexExclude.length && Config.settings.regexExclude.some(r => new RegExp(r).test(url) || new RegExp(r).test(desc))) return true;
            return false;
        }
    
        async function processQueue() {
            while (requestQueue.length && activeRequests < Config.settings.maxConcurrentRequests && isCrawlingFlag) {
                const { url, depth, image } = requestQueue.shift();
                activeRequests++;
                handleRequest(url, depth, image).finally(() => { activeRequests--; processQueue(); });
            }
        }
    
        async function handleRequest(url, depth, image) {
            if (!isCrawlingFlag || visitedUrls.has(url)) return;
            try {
                const response = await Fetch.fetchWithRetry(url, Config.settings.retryAttempts, Config.settings.retryDelay);
                const contentType = Fetch.getContentType(response);
                if (!Fetch.isContentTypeAllowed(contentType, Config.settings.contentTypes)) {
                    Logger.log(`허용되지 않은 콘텐츠 유형(${contentType})의 URL: ${url}`, 'warning');
                    return;
                }
                const parser = new DOMParser();
                const doc = parser.parseFromString(response.responseText, 'text/html');
                const text = doc.body?.innerText || '';
                if (Config.settings.enableCaptchas && Robots.detectCaptcha(doc)) {
                    await Robots.handleCaptcha(url);
                    return;
                }
                const wc = text.split(/\s+/).length;
                if (wc < Config.settings.minWordCount || wc > Config.settings.maxWordCount) {
                    Logger.log(`단어 수 조건에 맞지 않는 URL: ${url} (${wc} 단어)`, 'warning');
                    return;
                }
                const quality = Evaluate.evaluateContentQuality(text);
                const lang = doc.documentElement.lang || 'unknown';
                const category = Classifier.classifyKeywords(text);
                const meta = Extractor.extractMetaTags(doc);
                const headings = Extractor.extractHeadings(doc);
                let siteImage = image; // Use extracted image from search results
                if (!siteImage) {
                    // Try to extract og:image
                    siteImage = doc.querySelector('meta[property="og:image"]')?.getAttribute('content') || '';
                }
                if (category !== '기타' || quality > 100) {
                    const entry = { url, quality, language: lang, wordCount: wc, category, image: siteImage };
                    if (Config.settings.dataFields.name) entry.name = Extractor.extractName(doc) || 'No Title';
                    if (Config.settings.dataFields.description) entry.description = Extractor.extractDescription(doc) || 'No Description';
                    if (Config.settings.dataFields.html) entry.html = response.responseText;
                    if (Config.settings.dataFields.metaTags) entry.metaTags = meta;
                    if (Config.settings.dataFields.headings) entry.headings = headings;
                    data.push(entry);
                    GM_setValue('data', JSON.stringify(data));
                    UI.updateDataTable();
                    Logger.log(`데이터 수집 완료: ${url}`, 'success');
                    if (Config.settings.dataVisualization) DataVisualization.renderChart();
                }
                if (depth < Config.settings.maxDepth) {
                    Extractor.extractLinks(doc).forEach(link => {
                        if (!visitedUrls.has(link) && Utils.isValidURL(link)) requestQueue.push({ url: link, depth: depth + 1, image: '' });
                    });
                }
            } catch (error) {
                Logger.log(`데이터 수집 실패: ${url} - ${error.message}`, 'error');
            }
            visitedUrls.add(url);
            GM_setValue('visitedUrls', JSON.stringify([...visitedUrls]));
        }
    
        function finalizeCrawl(status) {
            if (previousCrawls.length) {
                previousCrawls[previousCrawls.length -1].status = status;
                GM_setValue('previousCrawls', JSON.stringify(previousCrawls));
                UI.updatePreviousCrawlsUI();
            }
            if (navigationObserver) {
                navigationObserver.disconnect();
                navigationObserver = null;
            }
        }
    
        function updateCrawlSettings() {
            // 필요한 경우 크롤링 설정 업데이트 로직을 여기에 추가하세요.
        }
    
        function identifySearchEngine() {
            const host = window.location.hostname;
            if (host.includes('google.com')) return 'google';
            if (host.includes('bing.com') && !window.location.pathname.includes('/images')) return 'bing';
            if (host.includes('yahoo.com')) return 'yahoo';
            if (host.includes('duckduckgo.com')) return 'duckduckgo';
            // Additional search engine identification logic can be added here
            return null;
        }
    
        return {
            startCrawling,
            stopCrawling,
            isCrawling: () => isCrawlingFlag,
            updateCrawlSettings
        };
    })();
    
    const Fetch = (() => {
        async function fetchWithRetry(url, attempts, delayMs) {
            for (let i = 1; i <= attempts; i++) {
                try {
                    return await GM_xmlhttpRequestPromise(url);
                } catch (e) {
                    Logger.log(`요청 실패 (${i}/${attempts}): ${url} - ${e.message}`, 'warning');
                    if (i < attempts) await Utils.delay(delayMs);
                }
            }
            throw new Error('모든 재시도 시도가 실패했습니다.');
        }
    
        function GM_xmlhttpRequestPromise(url) {
            return new Promise((resolve, reject) => {
                const reqUrl = Config.settings.proxy ? `${Config.settings.proxy}?url=${encodeURIComponent(url)}` : url;
                GM_xmlhttpRequest({
                    method: "GET",
                    url: reqUrl,
                    headers: { 'User-Agent': getCurrentUserAgent(), 'Accept': '*/*' },
                    timeout: Config.settings.timeout,
                    onload: res => (res.status >= 200 && res.status < 300) ? resolve(res) : reject(new Error(`HTTP ${res.status}`)),
                    onerror: () => reject(new Error('Request failed')),
                    ontimeout: () => reject(new Error('Request timed out'))
                });
            });
        }
    
        function getCurrentUserAgent() {
            const index = Config.settings.userAgentIndex % Config.settings.userAgents.length;
            return Config.settings.userAgents[index];
        }
    
        function getContentType(res) {
            return res.responseHeaders.split('\n').find(h => h.toLowerCase().startsWith('content-type'))?.split(':')[1].trim().split(';')[0] || '';
        }
    
        function isContentTypeAllowed(ct, contentTypes) {
            if (contentTypes.include.length && !contentTypes.include.includes(ct)) return false;
            if (contentTypes.exclude.includes(ct)) return false;
            return true;
        }
    
        async function fetchPageContent(url) {
            try {
                const response = await fetchWithRetry(url, Config.settings.retryAttempts, Config.settings.retryDelay);
                return response.responseText;
            } catch (error) {
                Logger.log(`페이지 내용 가져오기 실패: ${url} - ${error.message}`, 'error');
                return null;
            }
        }
    
        return {
            fetchWithRetry,
            getCurrentUserAgent,
            getContentType,
            isContentTypeAllowed,
            fetchPageContent
        };
    })();
    
    const Robots = (() => {
        async function checkRobotsTxt(url) {
            try {
                const parsed = new URL(url);
                const robotsUrl = `${parsed.origin}/robots.txt`;
                const res = await Fetch.fetchWithRetry(robotsUrl, 2, 2000);
                return new RobotsParser(res.responseText).isAllowed(url, Fetch.getCurrentUserAgent());
            } catch {
                Logger.log('robots.txt 확인 실패, 기본 허용.', 'warning');
                return true;
            }
        }
    
        function detectCaptcha(doc) {
            return !!doc.querySelector('iframe[src*="captcha"], form[action*="captcha"]');
        }
    
        async function handleCaptcha(url) {
            Logger.log(`CAPTCHA 감지됨: ${url}`, 'warning');
            await Utils.delay(5000);
            Logger.log(`CAPTCHA 처리 완료: ${url}`, 'info');
        }
    
        class RobotsParser {
            constructor(text) { this.rules = []; this.parse(text); }
            parse(text) {
                let agents = [];
                text.split('\n').forEach(line => {
                    const [key, ...val] = line.split(':').map(s => s.trim());
                    if (!key) return;
                    if (key.toLowerCase() === 'user-agent') agents = [val.join(':').toLowerCase()];
                    else if (['disallow','allow'].includes(key.toLowerCase())) this.rules.push({ agents, type: key.toLowerCase(), path: val.join(':') });
                });
            }
            isAllowed(url, ua) {
                const parsedUrl = new URL(url);
                const path = parsedUrl.pathname;
                let allowed = true;
                this.rules.forEach(rule => {
                    if (rule.agents.includes('*') || rule.agents.includes(ua.toLowerCase())) {
                        if (rule.type === 'disallow' && path.startsWith(rule.path)) allowed = false;
                        if (rule.type === 'allow' && path.startsWith(rule.path)) allowed = true;
                    }
                });
                return allowed;
            }
        }
    
        return {
            checkRobotsTxt,
            detectCaptcha,
            handleCaptcha
        };
    })();
    
    const Extractor = (() => {
        function extractMetaTags(doc) {
            const metas = {};
            doc.querySelectorAll('meta[name], meta[property]').forEach(meta => {
                const name = meta.getAttribute('name') || meta.getAttribute('property');
                const content = meta.getAttribute('content');
                if (name && content) metas[name.toLowerCase()] = content;
            });
            return metas;
        }
    
        function extractHeadings(doc) {
            const hs = [];
            for (let i = 1; i <= 6; i++) doc.querySelectorAll(`h${i}`).forEach(h => hs.push(h.innerText.trim()));
            return hs;
        }
    
        function extractName(doc) { return doc.querySelector('title')?.innerText || 'No Title'; }
        function extractDescription(doc) { return doc.querySelector('meta[name="description"]')?.getAttribute('content') || 'No Description'; }
        function extractLinks(doc) { return Array.from(doc.querySelectorAll('a[href]')).map(a => a.href).filter(Utils.isValidURL); }
    
        return {
            extractMetaTags,
            extractHeadings,
            extractName,
            extractDescription,
            extractLinks
        };
    })();
    
    const Classifier = (() => {
        function classifyKeywords(text) {
            const kws = combineKeywords();
            for (const [cat, keywords] of Object.entries(kws)) {
                if (keywords.some(kw => text.includes(kw))) return cat;
            }
            return '기타';
        }
    
        function combineKeywords() {
            const standard = {
                '교육': ['학교', '교육', '학생', '교수', '강의'],
                '기술': ['기술', '소프트웨어', '하드웨어', '프로그래밍', '개발'],
                '뉴스': ['뉴스', '속보', '기사', '보도'],
                '엔터테인먼트': ['영화', '음악', '연예', '게임', '드라마'],
                '스포츠': ['축구', '야구', '농구', '경기', '토너먼트'],
                '비즈니스': ['비즈니스', '경제', '주식', '투자', '시장'],
                '건강': ['건강', '의료', '헬스', '운동', '영양'],
                '과학': ['과학', '연구', '실험', '발견', '이론'],
                '문학': ['소설', '시', '문학', '작가', '책'],
                '여행': ['여행', '관광', '숙박', '비행기', '여권']
            };
            const custom = Config.settings.customKeywords || {};
            for (const [cat, kws] of Object.entries(custom)) {
                standard[cat] = standard[cat] ? [...new Set([...standard[cat], ...kws])] : kws;
            }
            return standard;
        }
    
        return {
            classifyKeywords
        };
    })();
    
    const Evaluate = (() => {
        function evaluateContentQuality(text) { return text.split(/\s+/).length; }
    
        return {
            evaluateContentQuality
        };
    })();
    
    const Scheduler = (() => {
        let scheduleInterval = null;
    
        function scheduleCrawling() {
            clearInterval(scheduleInterval);
            const now = new Date();
            const [startHour, startMinute] = Config.settings.schedule.startTime.split(':').map(Number);
            const startTime = new Date(now.getFullYear(), now.getMonth(), now.getDate(), startHour, startMinute, 0);
            if (startTime < now) startTime.setDate(startTime.getDate() + 1);
            const initialDelay = startTime - now;
            setTimeout(() => {
                if (Config.settings.schedule.specificDays.length === 0 || Config.settings.schedule.specificDays.includes(startTime.getDay())) {
                    Crawling.startCrawling();
                }
                scheduleInterval = setInterval(() => {
                    const currentDay = new Date().getDay();
                    if (Config.settings.schedule.specificDays.length === 0 || Config.settings.schedule.specificDays.includes(currentDay)) {
                        Crawling.startCrawling();
                    }
                }, Config.settings.schedule.interval);
            }, initialDelay);
            Logger.log(`크롤링 스케줄이 설정되었습니다. 시작 시간: ${startTime.toLocaleString()}, 간격: ${Config.settings.schedule.interval / 60000}분`, 'info');
        }
    
        return {
            scheduleCrawling
        };
    })();
    
    const Testing = (() => {
        function runTests() {
            $(selectors.testModal).show();
            $(selectors.mochaContainer).html('');
            $('head').append('<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/mocha/10.2.0/mocha.min.css" />');
            mocha.setup('bdd');
            const assert = chai.assert;
    
            describe('Utils Module', function() {
                it('should escape HTML correctly', function() {
                    const input = '<div>Hello & "World"!</div>';
                    const expected = '&lt;div&gt;Hello &amp; &quot;World&quot;!&lt;/div&gt;';
                    assert.equal(Utils.escape(input, 'html'), expected);
                });
                it('should capitalize strings correctly', function() {
                    const input = 'test';
                    const expected = 'Test';
                    assert.equal(Utils.capitalize(input), expected);
                });
                it('should validate URLs correctly', function() {
                    assert.isTrue(Utils.isValidURL('https://www.example.com'));
                    assert.isFalse(Utils.isValidURL('htp:/invalid-url'));
                });
            });
    
            describe('Config Module', function() {
                it('should save and load settings correctly', function() {
                    const originalDelay = Config.settings.delay;
                    Config.saveSettings({ delay: 3000 });
                    assert.equal(Config.settings.delay, 3000);
                    Config.saveSettings({ delay: originalDelay });
                });
                it('should add and remove templates correctly', function() {
                    const templateName = 'TestTemplate';
                    Config.saveTemplate(templateName);
                    assert.property(Config.settings.templates, templateName);
                    Config.loadTemplate(templateName);
                    Config.settings.templates[templateName] = undefined;
                    delete Config.settings.templates[templateName];
                    assert.notProperty(Config.settings.templates, templateName);
                });
            });
    
            describe('Logger Module', function() {
                it('should log messages correctly', function() {
                    Logger.log('Test log message', 'info');
                    const logs = Logger.getLogs();
                    assert.isAtLeast(logs.length, 1);
                    assert.equal(logs[logs.length -1].message, 'Test log message');
                });
                it('should clear logs correctly', function() {
                    Logger.clearLogs();
                    assert.equal(Logger.getLogs().length, 0);
                });
            });
    
            describe('Crawling Module', function() {
                it('should start and stop crawling correctly', function(done) {
                    Crawling.startCrawling();
                    assert.isTrue(Crawling.isCrawling());
                    Crawling.stopCrawling();
                    setTimeout(() => {
                        assert.isFalse(Crawling.isCrawling());
                        done();
                    }, 1000);
                });
                // Additional integration tests can be added here
            });
    
            mocha.run();
        }
    
        return {
            runTests
        };
    })();
    
    function init() {
        UI.createUI();
        UI.updateUI();
        if (Config.settings.dataVisualization) DataVisualization.renderChart();
        if (Config.settings.schedule.enabled) Scheduler.scheduleCrawling();
        if (Config.settings.isCrawling) {
            Logger.log('크롤링 재개됨.', 'info');
            Crawling.startCrawling();
        } else {
            Logger.log('크롤러 초기화됨.', 'info');
        }
    }
    
    window.addEventListener('load', init);
    

    })();

  2. Assistant

    Your Tampermonkey userscript, "Advanced Web Crawler Enhanced Pro Optimized v6.4.4", is an extensive and feature-rich tool designed to enhance web crawling across multiple search engines. Below is a detailed overview of its functionalities, structure, and some recommendations for further improvement.

    Overview of Features

    1. Multi-Search Engine Support:

      • Compatible with Google, Bing, Yahoo, DuckDuckGo, Baidu, Yandex, Naver, Daum, Ask, Ecosia, Startpage, Searx, Dogpile, and Bing Images.
    2. User Interface (UI):

      • Tabs: Logs, Data, Settings, Previous Crawls, Templates, Tests, AI Assistant, and Strategies.
      • Dark Mode: Toggle between light and dark themes.
      • Responsive Design: Adapts to different screen sizes.
      • Draggable and Resizable Window: Users can move and adjust the crawler window as needed.
    3. Crawling Strategies:

      • A variety of strategies like Basic, Focused, Incremental, Parallel, Distributed, Adaptive, Semantic, Deep, Shallow, Politeness, Reinforcement Learning-based, Hybrid, and more.
      • Users can select multiple strategies and add comments for each.
    4. Scheduling:

      • Set up scheduled crawls with specific start times, intervals, and designated days.
    5. Data Management:

      • Data Fields: Configurable fields such as name, URL, description, HTML content, category, quality score, language, word count, meta tags, headings, and images.
      • Export Formats: CSV, JSON, XML, and XLS.
      • Data Visualization: Integration with Chart.js for visual representation of crawled data.
    6. Logging and Testing:

      • Advanced logging with different log types (info, warning, error, success, debug).
      • Integrated testing framework using Mocha and Chai for unit and integration tests.
    7. AI Assistant:

      • Embeds an AI assistant from Hugging Face for additional support.
    8. Additional Functionalities:

      • Proxy Support: Configure proxy URLs for requests.
      • User-Agent Rotation: Multiple user-agent strings to mimic different browsers/devices.
      • Robots.txt Compliance: Respects robots.txt rules to avoid crawling restricted pages.
      • CAPTCHA Handling: Optional CAPTCHA detection and handling.
      • Automated Testing and Templates: Save and load different configuration templates.

    Code Structure and Modules

    1. Config Module:

      • Manages settings with default values and provides methods to save, load, and reset configurations.
      • Handles templates for different configurations.
    2. Logger Module:

      • Manages logging of events, errors, and other messages.
      • Displays logs in the UI and logs them to the console with styling based on log type.
    3. Utils Module:

      • Provides utility functions like escaping strings, capitalizing text, delaying execution, generating colors, parsing custom keywords, and validating URLs.
    4. UI Module:

      • Handles the creation and updating of the user interface.
      • Manages UI interactions, such as switching tabs, handling button clicks, and rendering data tables.
    5. Download Module:

      • Facilitates downloading of crawled data in various formats (CSV, JSON, XML, XLS).
    6. DataVisualization Module:

      • Integrates Chart.js to render visual charts based on the crawled data.
    7. Crawling Module:

      • Core module responsible for performing the web crawling based on selected search engines and strategies.
      • Manages the crawling process, including handling concurrency, depth, and queuing of crawl requests.
    8. Fetch Module:

      • Handles HTTP requests with retry mechanisms.
      • Manages user-agent headers and proxy configurations.
    9. Robots Module:

      • Parses and respects robots.txt directives.
      • Detects and handles CAPTCHAs.
    10. Extractor Module:

      • Extracts relevant information from crawled pages, such as meta tags, headings, names, descriptions, and links.
    11. Classifier Module:

      • Classifies crawled content based on keywords into predefined categories.
      • Merges standard and custom keywords for classification.
    12. Evaluate Module:

      • Evaluates the quality of the crawled content based on word count.
    13. Scheduler Module:

      • Schedules crawling tasks based on user-defined intervals and specific days.
    14. Testing Module:

      • Runs unit and integration tests using Mocha and Chai frameworks inside a modal window.

    Recommendations for Improvement

    1. Modularization:

      • While the current structure is modular, further separation into individual files or leveraging ES6 modules could enhance maintainability. Tools like Webpack or Rollup can bundle modules if necessary.
    2. Error Handling Enhancements:

      • Implement more granular error handling within each module to capture and log specific issues.
      • Provide user feedback for different error types, not just a general error message.
    3. Performance Optimization:

      • Throttle Requests: Introduce more sophisticated request throttling mechanisms to comply with target websites' rate limits and avoid being blocked.
      • Caching: Implement caching strategies for visited URLs to prevent redundant processing.
    4. Security Considerations:

      • Sanitize Inputs: Ensure all user inputs are properly sanitized to prevent injection attacks, especially when dealing with dynamic content rendering.
      • CORS Handling: Be cautious with cross-origin requests when using proxies to avoid security vulnerabilities.
    5. User Experience Enhancements:

      • Progress Indicators: Provide more detailed progress indicators, such as estimated time remaining.
      • Notifications: Implement in-browser notifications for important events (e.g., crawl completion, errors).
      • Customization: Allow users to customize UI elements, such as the position, size, and which tabs to display.
    6. Extensibility:

      • Plugin Support: Design the script to support plugins or extensions, enabling users to add custom functionalities without altering the core script.
      • API Integration: Provide hooks or APIs for integrating external services or databases for storing crawled data.
    7. Documentation and Comments:

      • Inline Comments: Add more inline comments explaining complex logic blocks to aid future maintenance and enhancements.
      • User Documentation: Create comprehensive user documentation outlining how to use various features, configure settings, and troubleshoot common issues.
    8. Testing Enhancements:

      • Automated Testing: Expand the testing suite to cover more modules and edge cases.
      • Continuous Integration: Integrate CI tools to automatically run tests on code changes, ensuring stability.
    9. Localization:

      • While the script already includes Korean translations, consider adding support for multiple languages to cater to a broader user base.
    10. Accessibility:

      • Ensure that the UI components are accessible, adhering to ARIA standards, and providing keyboard navigation for improved usability.

    Potential Issues and Fixes

    1. Concurrency Management:

      • Ensure that activeRequests accurately reflects ongoing requests, especially when handling retries or errors. Implement safeguards to prevent the queue from being overwhelmed.
    2. DOM Manipulation:

      • Temporarily replacing the document.body.innerHTML during crawling could lead to unexpected behaviors or UI glitches. Consider using virtual DOM manipulation or separate worker threads if possible.
    3. Resource Cleanup:

      • Ensure that all event listeners and observers (like navigationObserver) are properly cleaned up when stopping the crawl to prevent memory leaks.
    4. Data Persistence:

      • Regularly update and backup crawled data to prevent data loss in case of script interruption or crashes.
    5. Robots.txt Parsing:

      • The current implementation of RobotsParser is basic. Consider using more robust robots.txt parsing libraries to handle complex directives accurately.

    Conclusion

    Your userscript is a comprehensive tool that offers a wide range of functionalities for advanced web crawling. By implementing the above recommendations and addressing potential issues, you can enhance its performance, usability, and reliability. If you have specific questions or need assistance with particular aspects of the script, feel free to ask!