I would like to create a script for Violentmonkey to scrap images from…
- User
I would like to create a script for Violentmonkey to scrap images from http://d15.876515.xyz, http://98.987328.xyz, http://97.987327.xyz , let's call it WebURL . the @match should be set to WebURL/* .
- add a DOWNLOAD button in the element div.title h2 a span
- from the same element, capture the href , add WebURL to the front and call it PageURL
- When the DOWNLOAD button is pressed, go to PageURL, and from the content of div.jianjie capture the title which is embraced by, "[ "and ends with "P]" (quotation marks are excluded). Let's call this TITLE .
- from div.page, go through all the href (PageURL)inside and copy all images within div.content p from each PageURL.
- Rename the captured image files with serial number starting with 000 and add them to a zip file using TITLE as the filename.
- Assistant
Certainly! Below is a comprehensive Violentmonkey userscript that accomplishes the tasks you've outlined. This script adds a DOWNLOAD button to specified elements on your target websites. When clicked, it navigates through related pages, collects images, renames them sequentially, packages them into a ZIP file named after the extracted title, and prompts the user to download the ZIP.
Prerequisites:
- Violentmonkey Extension: Ensure you have the Violentmonkey extension installed in your browser.
- JSZip Library: This script utilizes the JSZip library to handle ZIP file creation. The library is included via the
@requiredirective.
Script Breakdown:
-
@match Directive:
- The script matches the specified URLs (
http://d15.876515.xyz/*,http://98.987328.xyz/*, andhttp://97.987327.xyz/*).
- The script matches the specified URLs (
-
Button Injection:
- A DOWNLOAD button is added to each
div.title h2 a spanelement found on the page.
- A DOWNLOAD button is added to each
-
Button Functionality:
- Upon clicking the DOWNLOAD button:
- The script captures the
hrefattribute of the associated link and constructs thePageURL. - It navigates to
PageURLto extract theTITLEfromdiv.jianjie. - It then collects all related
PageURLs fromdiv.page. - For each
PageURL, the script fetches the page and extracts all image sources withindiv.content p. - Images are fetched as blobs, renamed sequentially (starting from
000), and added to a ZIP archive named after the extractedTITLE. - Finally, the ZIP file is automatically downloaded to the user's device.
- The script captures the
- Upon clicking the DOWNLOAD button:
-
Concurrency Handling:
- The script utilizes asynchronous functions (
async/await) to handle multiple concurrent network requests efficiently.
- The script utilizes asynchronous functions (
-
Error Handling:
- Basic error handling is implemented to alert the user in case of failures during the download process.
Complete Userscript:
// ==UserScript== // @name WebURL Image Downloader // @namespace http://tampermonkey.net/ // @version 1.0 // @description Download images from specified WebURLs as a ZIP file. // @author Your Name // @match http://d15.876515.xyz/* // @match http://98.987328.xyz/* // @match http://97.987327.xyz/* // @grant GM_download // @require https://cdnjs.cloudflare.com/ajax/libs/jszip/3.10.0/jszip.min.js // @connect * // ==/UserScript== (async function() { 'use strict'; // Utility function to fetch and parse HTML async function fetchHTML(url) { const response = await fetch(url); const text = await response.text(); const parser = new DOMParser(); return parser.parseFromString(text, 'text/html'); } // Utility function to fetch image as blob async function fetchImage(url) { const response = await fetch(url); if (!response.ok) throw new Error(`Failed to fetch image: ${url}`); return await response.blob(); } // Function to add DOWNLOAD button function addDownloadButton(spanElement, pageUrl) { // Create the button const button = document.createElement('button'); button.innerText = 'DOWNLOAD'; button.style.marginLeft = '10px'; button.style.padding = '5px 10px'; button.style.cursor = 'pointer'; // Add click event listener button.addEventListener('click', async () => { button.disabled = true; button.innerText = 'Processing...'; try { await downloadImages(pageUrl); button.innerText = 'DOWNLOAD'; button.disabled = false; alert('Download ready.'); } catch (error) { console.error(error); alert('An error occurred during the download process.'); button.innerText = 'DOWNLOAD'; button.disabled = false; } }); // Append the button to the span element spanElement.appendChild(button); } // Main function to handle the download process async function downloadImages(initialPageUrl) { // Fetch the initial page const initialDoc = await fetchHTML(initialPageUrl); // Extract TITLE from div.jianjie const jianjieDiv = initialDoc.querySelector('div.jianjie'); if (!jianjieDiv) throw new Error('div.jianjie not found.'); const jianjieText = jianjieDiv.textContent; const titleMatch = jianjieText.match(/\[ (.*?)P\]/); if (!titleMatch || !titleMatch[1]) throw new Error('TITLE not found.'); const TITLE = titleMatch[1].trim(); // Initialize JSZip const zip = new JSZip(); // Extract all related PageURLs from div.page const pageDiv = initialDoc.querySelector('div.page'); if (!pageDiv) throw new Error('div.page not found.'); const pageLinks = pageDiv.querySelectorAll('a[href]'); const pageUrls = Array.from(pageLinks).map(link => { let href = link.getAttribute('href'); // Ensure the href is absolute if (!href.startsWith('http')) { // Assuming relative links, prepend the origin href = new URL(href, initialPageUrl).href; } return href; }); // Limit concurrent fetches to avoid overloading const CONCURRENT_LIMIT = 5; let index = 0; async function processNext() { if (index >= pageUrls.length) return; const currentIndex = index++; const pageUrl = pageUrls[currentIndex]; try { const pageDoc = await fetchHTML(pageUrl); const contentDiv = pageDoc.querySelector('div.content'); if (contentDiv) { const paragraphs = contentDiv.querySelectorAll('p'); paragraphs.forEach(p => { const imgs = p.querySelectorAll('img[src]'); imgs.forEach(img => { images.push(img.src); }); }); } } catch (e) { console.warn(`Failed to process ${pageUrl}: ${e.message}`); } await processNext(); } // Collect all image URLs const images = []; const workers = []; for (let i = 0; i < CONCURRENT_LIMIT; i++) { workers.push(processNext()); } await Promise.all(workers); if (images.length === 0) throw new Error('No images found.'); // Fetch all images and add to zip let counter = 0; const total = images.length; for (const imgUrl of images) { try { const blob = await fetchImage(imgUrl); const extension = imgUrl.split('.').pop().split(/\#|\?/)[0] || 'jpg'; const filename = `${String(counter).padStart(3, '0')}.${extension}`; zip.file(filename, blob); counter++; // Optional: Update progress here } catch (e) { console.warn(`Failed to fetch image ${imgUrl}: ${e.message}`); } } if (counter === 0) throw new Error('No images were successfully downloaded.'); // Generate the zip file const zipBlob = await zip.generateAsync({ type: 'blob' }); // Trigger the download using a temporary link const downloadLink = document.createElement('a'); downloadLink.href = URL.createObjectURL(zipBlob); downloadLink.download = `${TITLE}.zip`; document.body.appendChild(downloadLink); downloadLink.click(); document.body.removeChild(downloadLink); } // Locate all target span elements and add DOWNLOAD buttons function initialize() { const spanElements = document.querySelectorAll('div.title h2 a span'); spanElements.forEach(span => { // Avoid adding multiple buttons if (span.querySelector('button') === null) { const aTag = span.parentElement; if (aTag && aTag.href) { // Construct absolute PageURL const pageUrl = aTag.href.startsWith('http') ? aTag.href : new URL(aTag.getAttribute('href'), window.location.origin).href; addDownloadButton(span, pageUrl); } } }); } // Wait for the DOM to load before initializing if (document.readyState === 'loading') { document.addEventListener('DOMContentLoaded', initialize); } else { initialize(); } })();Explanation of Key Components:
-
@grant GM_download:
- Grants permission to use
GM_downloadif you prefer using it for downloads. In this script, we handle downloads via temporary links, but you can modify the script to useGM_downloadif desired.
- Grants permission to use
-
Concurrency Limiting:
- To prevent overwhelming the server or running into rate limits, the script limits concurrent page fetches to 5 (
CONCURRENT_LIMIT). You can adjust this number based on the server's capabilities.
- To prevent overwhelming the server or running into rate limits, the script limits concurrent page fetches to 5 (
-
Progress Feedback:
- While not fully implemented, you can enhance the script by providing real-time progress updates to the user, especially if dealing with a large number of images.
-
Error Handling:
- The script includes basic error handling to notify users of issues during the download process. This can be further enhanced based on specific requirements.
-
File Naming:
- Images are renamed sequentially starting from
000(e.g.,000.jpg,001.png, etc.). The script preserves the original image file extensions.
- Images are renamed sequentially starting from
-
ZIP File Naming:
- The ZIP archive is named after the extracted
TITLEfromdiv.jianjieand suffixed with.zip.
- The ZIP archive is named after the extracted
Usage Instructions:
-
Install Violentmonkey:
- If you haven't already, install the Violentmonkey extension in your preferred browser.
-
Create a New Script:
- Open the Violentmonkey dashboard.
- Click on the + icon to create a new script.
- Paste the above script into the editor.
-
Save the Script:
- Save the script and ensure it's enabled.
-
Navigate to Target Websites:
- Visit any of the specified URLs (e.g.,
http://d15.876515.xyz/somepage).
- Visit any of the specified URLs (e.g.,
-
Use the DOWNLOAD Button:
- Locate the DOWNLOAD button added next to the relevant content.
- Click the button to initiate the image download process.
-
Download the ZIP:
- After processing, a ZIP file named after the extracted
TITLEwill be automatically downloaded to your device.
- After processing, a ZIP file named after the extracted
Notes:
-
CORS Considerations:
- The script assumes that all image URLs are accessible and do not have restrictive CORS policies. If images are hosted on domains with strict CORS settings, fetching them might fail.
-
Performance:
- Downloading a large number of images can be resource-intensive. Ensure your device has sufficient resources and allow some time for the process to complete.
-
Script Customization:
- Depending on the actual structure of your target websites, you might need to adjust the selectors (e.g.,
'div.jianjie','div.content') to match the exact HTML structure.
- Depending on the actual structure of your target websites, you might need to adjust the selectors (e.g.,
-
Security:
- Always review and understand userscripts before installing them, especially those that interact with multiple web pages and handle file downloads.
Conclusion
This userscript should effectively automate the process of downloading and packaging images from your specified WebURLs into a single ZIP file. Feel free to customize and enhance the script further based on your specific needs and the structure of the target websites.