what is this
- User
what is this
- Assistant
Today is Tuesday, April 15, 2025. This date holds significance for several reasons:
Tax Day: In the United States, April 15 is traditionally the deadline for filing federal income tax returns. For 2025, taxpayers can file for an automatic six-month extension by submitting Form 4868 by today. It's important to note that failing to file or pay on time may result in penalties[1].
World Art Day: Celebrated globally, World Art Day promotes the importance of art and its transformative impact on culture and society. Events include art exhibitions, workshops, and cultural activities that celebrate diversity and the power of artistic expression[2].
Titanic Remembrance Day: This day commemorates the sinking of the RMS Titanic on April 15, 1912, a tragic event that resulted in the loss of over 1,500 lives[3].
Jackie Robinson Day: Major League Baseball honors Jackie Robinson, who broke the baseball color barrier on April 15, 1947, by making his debut with the Brooklyn Dodgers[4].
National Anime Day: Fans of Japanese animation celebrate National Anime Day by attending conventions, connecting with fellow enthusiasts, or enjoying their favorite anime series[4].
National ASL Day: This day celebrates American Sign Language (ASL), the predominant sign language of Deaf communities in the United States and parts of Canada. It commemorates the opening of the American School for the Deaf on April 15, 1817[5].
Moon Phase: On April 15, 2025, the Moon is in its Waning Gibbous phase, with approximately 93% illumination. This phase is associated with reflection and gratitude, encouraging individuals to review accomplishments and plan for future endeavors[6].
Additionally, April 15 marks the birthdays of notable individuals such as actress Emma Watson, actor Seth Rogen, and actress Maisie Williams[3].
In summary, April 15, 2025, is a day marked by significant historical events, cultural celebrations, and personal milestones.
- User
Here is a list of files in the repository that may help you answer the query:
import { Response } from "express"; import { v4 as uuidv4 } from "uuid"; import { CrawlRequest, crawlRequestSchema, CrawlResponse, RequestWithAuth, toLegacyCrawlerOptions, } from "./types"; import { crawlToCrawler, saveCrawl, StoredCrawl } from "../../lib/crawl-redis"; import { logCrawl } from "../../services/logging/crawl_log"; import { _addScrapeJobToBullMQ } from "../../services/queue-jobs"; import { logger as _logger } from "../../lib/logger"; export async function crawlController( req: RequestWithAuth<{}, CrawlResponse, CrawlRequest>, res: Response<CrawlResponse>, ) { const preNormalizedBody = req.body; req.body = crawlRequestSchema.parse(req.body); const id = uuidv4(); const logger = _logger.child({ crawlId: id, module: "api/v1", method: "crawlController", teamId: req.auth.team_id, }); logger.debug("Crawl " + id + " starting", { request: req.body, originalRequest: preNormalizedBody, account: req.account, }); await logCrawl(id, req.auth.team_id); let { remainingCredits } = req.account!; const useDbAuthentication = process.env.USE_DB_AUTHENTICATION === "true"; if (!useDbAuthentication) { remainingCredits = Infinity; } const crawlerOptions = { ...req.body, url: undefined, scrapeOptions: undefined, }; const scrapeOptions = req.body.scrapeOptions; // TODO: @rafa, is this right? copied from v0 if (Array.isArray(crawlerOptions.includePaths)) { for (const x of crawlerOptions.includePaths) { try { new RegExp(x); } catch (e) { return res.status(400).json({ success: false, error: e.message }); } } } if (Array.isArray(crawlerOptions.excludePaths)) { for (const x of crawlerOptions.excludePaths) { try { new RegExp(x); } catch (e) { return res.status(400).json({ success: false, error: e.message }); } } } const originalLimit = crawlerOptions.limit; crawlerOptions.limit = Math.min(remainingCredits, crawlerOptions.limit); logger.debug("Determined limit: " + crawlerOptions.limit, { remainingCredits, bodyLimit: originalLimit, originalBodyLimit: preNormalizedBody.limit, }); const sc: StoredCrawl = { originUrl: req.body.url, crawlerOptions: toLegacyCrawlerOptions(crawlerOptions), scrapeOptions, internalOptions: { disableSmartWaitCache: true, teamId: req.auth.team_id }, // NOTE: smart wait disabled for crawls to ensure contentful scrape, speed does not matter team_id: req.auth.team_id, createdAt: Date.now(), }; const crawler = crawlToCrawler(id, sc); try { sc.robots = await crawler.getRobotsTxt(scrapeOptions.skipTlsVerification); } catch (e) { logger.debug("Failed to get robots.txt (this is probably fine!)", { error: e, }); } await saveCrawl(id, sc); await _addScrapeJobToBullMQ( { url: req.body.url, mode: "kickoff" as const, team_id: req.auth.team_id, crawlerOptions, scrapeOptions: sc.scrapeOptions, internalOptions: sc.internalOptions, origin: req.body.origin, crawl_id: id, webhook: req.body.webhook, v1: true, }, {}, crypto.randomUUID(), 10, ); const protocol = process.env.ENV === "local" ? req.protocol : "https"; return res.status(200).json({ success: true, id, url: `${protocol}://${req.get("host")}/v1/crawl/${id}`, }); }import { InternalOptions } from "../scraper/scrapeURL"; import { ScrapeOptions } from "../controllers/v1/types"; import { WebCrawler } from "../scraper/WebScraper/crawler"; import { redisConnection } from "../services/queue-service"; import { logger as _logger } from "./logger"; import { getAdjustedMaxDepth } from "../scraper/WebScraper/utils/maxDepthUtils"; export type StoredCrawl = { originUrl?: string; crawlerOptions: any; scrapeOptions: Omit<ScrapeOptions, "timeout">; internalOptions: InternalOptions; team_id: string; robots?: string; cancelled?: boolean; createdAt: number; }; export async function saveCrawl(id: string, crawl: StoredCrawl) { _logger.debug("Saving crawl " + id + " to Redis...", { crawl, module: "crawl-redis", method: "saveCrawl", crawlId: id, teamId: crawl.team_id, }); await redisConnection.set("crawl:" + id, JSON.stringify(crawl)); await redisConnection.expire("crawl:" + id, 24 * 60 * 60); } export async function getCrawl(id: string): Promise<StoredCrawl | null> { const x = await redisConnection.get("crawl:" + id); if (x === null) { return null; } await redisConnection.expire("crawl:" + id, 24 * 60 * 60); return JSON.parse(x); } export async function getCrawlExpiry(id: string): Promise<Date> { const d = new Date(); const ttl = await redisConnection.pttl("crawl:" + id); d.setMilliseconds(d.getMilliseconds() + ttl); d.setMilliseconds(0); return d; } export async function addCrawlJob(id: string, job_id: string) { _logger.debug("Adding crawl job " + job_id + " to Redis...", { jobId: job_id, module: "crawl-redis", method: "addCrawlJob", crawlId: id, }); await redisConnection.sadd("crawl:" + id + ":jobs", job_id); await redisConnection.expire("crawl:" + id + ":jobs", 24 * 60 * 60); } export async function addCrawlJobs(id: string, job_ids: string[]) { if (job_ids.length === 0) return true; _logger.debug("Adding crawl jobs to Redis...", { jobIds: job_ids, module: "crawl-redis", method: "addCrawlJobs", crawlId: id, }); await redisConnection.sadd("crawl:" + id + ":jobs", ...job_ids); await redisConnection.expire("crawl:" + id + ":jobs", 24 * 60 * 60); } export async function addCrawlJobDone( id: string, job_id: string, success: boolean, ) { _logger.debug("Adding done crawl job to Redis...", { jobId: job_id, module: "crawl-redis", method: "addCrawlJobDone", crawlId: id, }); await redisConnection.sadd("crawl:" + id + ":jobs_done", job_id); await redisConnection.expire( "crawl:" + id + ":jobs_done", 24 * 60 * 60, ); if (success) { await redisConnection.rpush("crawl:" + id + ":jobs_done_ordered", job_id); } else { // in case it's already been pushed, make sure it's removed await redisConnection.lrem( "crawl:" + id + ":jobs_done_ordered", -1, job_id, ); } await redisConnection.expire( "crawl:" + id + ":jobs_done_ordered", 24 * 60 * 60, ); } export async function getDoneJobsOrderedLength(id: string): Promise<number> { await redisConnection.expire("crawl:" + id + ":jobs_done_ordered", 24 * 60 * 60); return await redisConnection.llen("crawl:" + id + ":jobs_done_ordered"); } export async function getDoneJobsOrdered( id: string, start = 0, end = -1, ): Promise<string[]> { await redisConnection.expire("crawl:" + id + ":jobs_done_ordered", 24 * 60 * 60); return await redisConnection.lrange( "crawl:" + id + ":jobs_done_ordered", start, end, ); } export async function isCrawlFinished(id: string) { await redisConnection.expire("crawl:" + id + ":kickoff:finish", 24 * 60 * 60); return ( (await redisConnection.scard("crawl:" + id + ":jobs_done")) === (await redisConnection.scard("crawl:" + id + ":jobs")) && (await redisConnection.get("crawl:" + id + ":kickoff:finish")) !== null ); } export async function isCrawlKickoffFinished(id: string) { await redisConnection.expire("crawl:" + id + ":kickoff:finish", 24 * 60 * 60); return ( (await redisConnection.get("crawl:" + id + ":kickoff:finish")) !== null ); } export async function isCrawlFinishedLocked(id: string) { return await redisConnection.exists("crawl:" + id + ":finish"); } export async function finishCrawlKickoff(id: string) { await redisConnection.set( "crawl:" + id + ":kickoff:finish", "yes", "EX", 24 * 60 * 60, ); } export async function finishCrawlPre(id: string) { if (await isCrawlFinished(id)) { _logger.debug("Marking crawl as pre-finished.", { module: "crawl-redis", method: "finishCrawlPre", crawlId: id, }); const set = await redisConnection.setnx("crawl:" + id + ":finished_pre", "yes"); await redisConnection.expire("crawl:" + id + ":finished_pre", 24 * 60 * 60); return set === 1; } else { _logger.debug("Crawl can not be pre-finished yet, not marking as finished.", { module: "crawl-redis", method: "finishCrawlPre", crawlId: id, jobs_done: await redisConnection.scard("crawl:" + id + ":jobs_done"), jobs: await redisConnection.scard("crawl:" + id + ":jobs"), kickoff_finished: (await redisConnection.get("crawl:" + id + ":kickoff:finish")) !== null, }); } } export async function finishCrawl(id: string) { _logger.debug("Marking crawl as finished.", { module: "crawl-redis", method: "finishCrawl", crawlId: id, }); await redisConnection.set("crawl:" + id + ":finish", "yes"); await redisConnection.expire("crawl:" + id + ":finish", 24 * 60 * 60); } export async function getCrawlJobs(id: string): Promise<string[]> { return await redisConnection.smembers("crawl:" + id + ":jobs"); } export async function getCrawlJobCount(id: string): Promise<number> { return await redisConnection.scard("crawl:" + id + ":jobs"); } export function normalizeURL(url: string, sc: StoredCrawl): string { const urlO = new URL(url); if (!sc.crawlerOptions || sc.crawlerOptions.ignoreQueryParameters) { urlO.search = ""; } urlO.hash = ""; return urlO.href; } export function generateURLPermutations(url: string | URL): URL[] { const urlO = new URL(url); // Construct two versions, one with www., one without const urlWithWWW = new URL(urlO); const urlWithoutWWW = new URL(urlO); if (urlO.hostname.startsWith("www.")) { urlWithoutWWW.hostname = urlWithWWW.hostname.slice(4); } else { urlWithWWW.hostname = "www." + urlWithoutWWW.hostname; } let permutations = [urlWithWWW, urlWithoutWWW]; // Construct more versions for http/https permutations = permutations.flatMap((urlO) => { if (!["http:", "https:"].includes(urlO.protocol)) { return [urlO]; } const urlWithHTTP = new URL(urlO); const urlWithHTTPS = new URL(urlO); urlWithHTTP.protocol = "http:"; urlWithHTTPS.protocol = "https:"; return [urlWithHTTP, urlWithHTTPS]; }); // Construct more versions for index.html/index.php permutations = permutations.flatMap((urlO) => { const urlWithHTML = new URL(urlO); const urlWithPHP = new URL(urlO); const urlWithBare = new URL(urlO); const urlWithSlash = new URL(urlO); if (urlO.pathname.endsWith("/")) { urlWithBare.pathname = urlWithBare.pathname.length === 1 ? urlWithBare.pathname : urlWithBare.pathname.slice(0, -1); urlWithHTML.pathname += "index.html"; urlWithPHP.pathname += "index.php"; } else if (urlO.pathname.endsWith("/index.html")) { urlWithPHP.pathname = urlWithPHP.pathname.slice(0, -"index.html".length) + "index.php"; urlWithSlash.pathname = urlWithSlash.pathname.slice(0, -"index.html".length); urlWithBare.pathname = urlWithBare.pathname.slice(0, -"/index.html".length); } else if (urlO.pathname.endsWith("/index.php")) { urlWithHTML.pathname = urlWithHTML.pathname.slice(0, -"index.php".length) + "index.html"; urlWithSlash.pathname = urlWithSlash.pathname.slice(0, -"index.php".length); urlWithBare.pathname = urlWithBare.pathname.slice(0, -"/index.php".length); } else { urlWithSlash.pathname += "/"; urlWithHTML.pathname += "/index.html"; urlWithPHP.pathname += "/index.php"; } return [urlWithHTML, urlWithPHP, urlWithSlash, urlWithBare]; }); return [...new Set(permutations.map(x => x.href))].map(x => new URL(x)); } export async function lockURL( id: string, sc: StoredCrawl, url: string, ): Promise<boolean> { let logger = _logger.child({ crawlId: id, module: "crawl-redis", method: "lockURL", preNormalizedURL: url, teamId: sc.team_id, }); if (typeof sc.crawlerOptions?.limit === "number") { if ( (await redisConnection.scard("crawl:" + id + ":visited_unique")) >= sc.crawlerOptions.limit ) { logger.debug( "Crawl has already hit visited_unique limit, not locking URL.", ); return false; } } url = normalizeURL(url, sc); logger = logger.child({ url }); let res: boolean; if (!sc.crawlerOptions?.deduplicateSimilarURLs) { res = (await redisConnection.sadd("crawl:" + id + ":visited", url)) !== 0; } else { const permutations = generateURLPermutations(url).map((x) => x.href); // logger.debug("Adding URL permutations for URL " + JSON.stringify(url) + "...", { permutations }); const x = await redisConnection.sadd( "crawl:" + id + ":visited", ...permutations, ); res = x === permutations.length; } await redisConnection.expire("crawl:" + id + ":visited", 24 * 60 * 60); if (res) { await redisConnection.sadd("crawl:" + id + ":visited_unique", url); await redisConnection.expire( "crawl:" + id + ":visited_unique", 24 * 60 * 60, ); } // logger.debug("Locking URL " + JSON.stringify(url) + "... result: " + res, { // res, // }); return res; } /// NOTE: does not check limit. only use if limit is checked beforehand e.g. with sitemap export async function lockURLs( id: string, sc: StoredCrawl, urls: string[], ): Promise<boolean> { if (urls.length === 0) return true; urls = urls.map((url) => normalizeURL(url, sc)); const logger = _logger.child({ crawlId: id, module: "crawl-redis", method: "lockURL", teamId: sc.team_id, }); // Add to visited_unique set logger.debug("Locking " + urls.length + " URLs..."); await redisConnection.sadd("crawl:" + id + ":visited_unique", ...urls); await redisConnection.expire( "crawl:" + id + ":visited_unique", 24 * 60 * 60, ); let res: boolean; if (!sc.crawlerOptions?.deduplicateSimilarURLs) { const x = await redisConnection.sadd("crawl:" + id + ":visited", ...urls); res = x === urls.length; } else { const allPermutations = urls.flatMap((url) => generateURLPermutations(url).map((x) => x.href), ); logger.debug("Adding " + allPermutations.length + " URL permutations..."); const x = await redisConnection.sadd( "crawl:" + id + ":visited", ...allPermutations, ); res = x === allPermutations.length; } await redisConnection.expire("crawl:" + id + ":visited", 24 * 60 * 60); logger.debug("lockURLs final result: " + res, { res }); return res; } export async function lockURLsIndividually( id: string, sc: StoredCrawl, jobs: { id: string; url: string }[], ) { const out: typeof jobs = []; for (const job of jobs) { if (await lockURL(id, sc, job.url)) { out.push(job); } } return out; } export function crawlToCrawler( id: string, sc: StoredCrawl, newBase?: string, crawlerOptions?: any, ): WebCrawler { const crawler = new WebCrawler({ jobId: id, initialUrl: sc.originUrl!, baseUrl: newBase ? new URL(newBase).origin : undefined, includes: (sc.crawlerOptions?.includes ?? []).filter(x => x.trim().length > 0), excludes: (sc.crawlerOptions?.excludes ?? []).filter(x => x.trim().length > 0), maxCrawledLinks: sc.crawlerOptions?.maxCrawledLinks ?? 1000, maxCrawledDepth: getAdjustedMaxDepth( sc.originUrl!, sc.crawlerOptions?.maxDepth ?? 10, ), limit: sc.crawlerOptions?.limit ?? 10000, generateImgAltText: sc.crawlerOptions?.generateImgAltText ?? false, allowBackwardCrawling: sc.crawlerOptions?.allowBackwardCrawling ?? false, allowExternalContentLinks: sc.crawlerOptions?.allowExternalContentLinks ?? false, allowSubdomains: sc.crawlerOptions?.allowSubdomains ?? false, ignoreRobotsTxt: sc.crawlerOptions?.ignoreRobotsTxt ?? false, regexOnFullURL: sc.crawlerOptions?.regexOnFullURL ?? false, maxDiscoveryDepth: sc.crawlerOptions?.maxDiscoveryDepth, currentDiscoveryDepth: crawlerOptions?.currentDiscoveryDepth ?? 0, }); if (sc.robots !== undefined) { try { crawler.importRobotsTxt(sc.robots); } catch (_) {} } return crawler; }import axios, { AxiosError } from "axios"; import { load } from "cheerio"; // rustified import { URL } from "url"; import { getLinksFromSitemap } from "./sitemap"; import robotsParser, { Robot } from "robots-parser"; import { getURLDepth } from "./utils/maxDepthUtils"; import { axiosTimeout } from "../../lib/timeout"; import { logger as _logger } from "../../lib/logger"; import https from "https"; import { redisConnection } from "../../services/queue-service"; import { extractLinks } from "../../lib/html-transformer"; import { TimeoutSignal } from "../../controllers/v1/types"; export class WebCrawler { private jobId: string; private initialUrl: string; private baseUrl: string; private includes: string[]; private excludes: string[]; private maxCrawledLinks: number; private maxCrawledDepth: number; private visited: Set<string> = new Set(); private crawledUrls: Map<string, string> = new Map(); private limit: number; private robotsTxtUrl: string; public robots: Robot; private generateImgAltText: boolean; private allowBackwardCrawling: boolean; private allowExternalContentLinks: boolean; private allowSubdomains: boolean; private ignoreRobotsTxt: boolean; private regexOnFullURL: boolean; private logger: typeof _logger; private sitemapsHit: Set<string> = new Set(); private maxDiscoveryDepth: number | undefined; private currentDiscoveryDepth: number; constructor({ jobId, initialUrl, baseUrl, includes, excludes, maxCrawledLinks = 10000, limit = 10000, generateImgAltText = false, maxCrawledDepth = 10, allowBackwardCrawling = false, allowExternalContentLinks = false, allowSubdomains = false, ignoreRobotsTxt = false, regexOnFullURL = false, maxDiscoveryDepth, currentDiscoveryDepth, }: { jobId: string; initialUrl: string; baseUrl?: string; includes?: string[]; excludes?: string[]; maxCrawledLinks?: number; limit?: number; generateImgAltText?: boolean; maxCrawledDepth?: number; allowBackwardCrawling?: boolean; allowExternalContentLinks?: boolean; allowSubdomains?: boolean; ignoreRobotsTxt?: boolean; regexOnFullURL?: boolean; maxDiscoveryDepth?: number; currentDiscoveryDepth?: number; }) { this.jobId = jobId; this.initialUrl = initialUrl; this.baseUrl = baseUrl ?? new URL(initialUrl).origin; this.includes = Array.isArray(includes) ? includes : []; this.excludes = Array.isArray(excludes) ? excludes : []; this.limit = limit; this.robotsTxtUrl = `${this.baseUrl}${this.baseUrl.endsWith("/") ? "" : "/"}robots.txt`; this.robots = robotsParser(this.robotsTxtUrl, ""); // Deprecated, use limit instead this.maxCrawledLinks = maxCrawledLinks ?? limit; this.maxCrawledDepth = maxCrawledDepth ?? 10; this.generateImgAltText = generateImgAltText ?? false; this.allowBackwardCrawling = allowBackwardCrawling ?? false; this.allowExternalContentLinks = allowExternalContentLinks ?? false; this.allowSubdomains = allowSubdomains ?? false; this.ignoreRobotsTxt = ignoreRobotsTxt ?? false; this.regexOnFullURL = regexOnFullURL ?? false; this.logger = _logger.child({ crawlId: this.jobId, module: "WebCrawler" }); this.maxDiscoveryDepth = maxDiscoveryDepth; this.currentDiscoveryDepth = currentDiscoveryDepth ?? 0; } public filterLinks( sitemapLinks: string[], limit: number, maxDepth: number, fromMap: boolean = false, ): string[] { if (this.currentDiscoveryDepth === this.maxDiscoveryDepth) { this.logger.debug("Max discovery depth hit, filtering off all links", { currentDiscoveryDepth: this.currentDiscoveryDepth, maxDiscoveryDepth: this.maxDiscoveryDepth }); return []; } // If the initial URL is a sitemap.xml, skip filtering if (this.initialUrl.endsWith("sitemap.xml") && fromMap) { return sitemapLinks.slice(0, limit); } return sitemapLinks .filter((link) => { let url: URL; try { url = new URL(link.trim(), this.baseUrl); } catch (error) { this.logger.debug(`Error processing link: ${link}`, { link, error, method: "filterLinks", }); return false; } const path = url.pathname; const depth = getURLDepth(url.toString()); // Check if the link exceeds the maximum depth allowed if (depth > maxDepth) { if (process.env.FIRECRAWL_DEBUG_FILTER_LINKS) { this.logger.debug(`${link} DEPTH FAIL`); } return false; } const excincPath = this.regexOnFullURL ? link : path; // Check if the link should be excluded if (this.excludes.length > 0 && this.excludes[0] !== "") { if ( this.excludes.some((excludePattern) => new RegExp(excludePattern).test(excincPath), ) ) { if (process.env.FIRECRAWL_DEBUG_FILTER_LINKS) { this.logger.debug(`${link} EXCLUDE FAIL`); } return false; } } // Check if the link matches the include patterns, if any are specified if (this.includes.length > 0 && this.includes[0] !== "") { if ( !this.includes.some((includePattern) => new RegExp(includePattern).test(excincPath), ) ) { if (process.env.FIRECRAWL_DEBUG_FILTER_LINKS) { this.logger.debug(`${link} INCLUDE FAIL`); } return false; } } // Normalize the initial URL and the link to account for www and non-www versions const normalizedInitialUrl = new URL(this.initialUrl); let normalizedLink; try { normalizedLink = new URL(link); } catch (_) { if (process.env.FIRECRAWL_DEBUG_FILTER_LINKS) { this.logger.debug(`${link} URL PARSE FAIL`); } return false; } const initialHostname = normalizedInitialUrl.hostname.replace( /^www\./, "", ); const linkHostname = normalizedLink.hostname.replace(/^www\./, ""); // Ensure the protocol and hostname match, and the path starts with the initial URL's path // commented to able to handling external link on allowExternalContentLinks // if (linkHostname !== initialHostname) { // return false; // } if (!this.allowBackwardCrawling) { if ( !normalizedLink.pathname.startsWith(normalizedInitialUrl.pathname) ) { if (process.env.FIRECRAWL_DEBUG_FILTER_LINKS) { this.logger.debug(`${link} BACKWARDS FAIL ${normalizedLink.pathname} ${normalizedInitialUrl.pathname}`); } return false; } } const isAllowed = this.ignoreRobotsTxt ? true : ((this.robots.isAllowed(link, "FireCrawlAgent") || this.robots.isAllowed(link, "FirecrawlAgent")) ?? true); // Check if the link is disallowed by robots.txt if (!isAllowed) { this.logger.debug(`Link disallowed by robots.txt: ${link}`, { method: "filterLinks", link, }); if (process.env.FIRECRAWL_DEBUG_FILTER_LINKS) { this.logger.debug(`${link} ROBOTS FAIL`); } return false; } if (this.isFile(link)) { if (process.env.FIRECRAWL_DEBUG_FILTER_LINKS) { this.logger.debug(`${link} FILE FAIL`); } return false; } if (process.env.FIRECRAWL_DEBUG_FILTER_LINKS) { this.logger.debug(`${link} OK`); } return true; }) .slice(0, limit); } public async getRobotsTxt(skipTlsVerification = false, abort?: AbortSignal): Promise<string> { let extraArgs = {}; if (skipTlsVerification) { extraArgs["httpsAgent"] = new https.Agent({ rejectUnauthorized: false, }); } const response = await axios.get(this.robotsTxtUrl, { timeout: axiosTimeout, signal: abort, ...extraArgs, }); return response.data; } public importRobotsTxt(txt: string) { this.robots = robotsParser(this.robotsTxtUrl, txt); } public async tryGetSitemap( urlsHandler: (urls: string[]) => unknown, fromMap: boolean = false, onlySitemap: boolean = false, timeout: number = 120000, abort?: AbortSignal, mock?: string, ): Promise<number> { this.logger.debug(`Fetching sitemap links from ${this.initialUrl}`, { method: "tryGetSitemap", }); let leftOfLimit = this.limit; const normalizeUrl = (url: string) => { url = url.replace(/^https?:\/\//, "").replace(/^www\./, ""); if (url.endsWith("/")) { url = url.slice(0, -1); } return url; }; const _urlsHandler = async (urls: string[]) => { if (fromMap && onlySitemap) { return urlsHandler(urls); } else { let filteredLinks = this.filterLinks( [...new Set(urls)].filter(x => this.filterURL(x, this.initialUrl) !== null), leftOfLimit, this.maxCrawledDepth, fromMap, ); leftOfLimit -= filteredLinks.length; let uniqueURLs: string[] = []; for (const url of filteredLinks) { if ( await redisConnection.sadd( "sitemap:" + this.jobId + ":links", normalizeUrl(url), ) ) { uniqueURLs.push(url); } } await redisConnection.expire( "sitemap:" + this.jobId + ":links", 3600, "NX", ); if (uniqueURLs.length > 0) { return urlsHandler(uniqueURLs); } } }; const timeoutPromise = new Promise((_, reject) => { setTimeout(() => reject(new Error("Sitemap fetch timeout")), timeout); }); try { let count = (await Promise.race([ Promise.all([ this.tryFetchSitemapLinks(this.initialUrl, _urlsHandler, abort, mock), ...this.robots .getSitemaps() .map((x) => this.tryFetchSitemapLinks(x, _urlsHandler, abort, mock)), ]).then((results) => results.reduce((a, x) => a + x, 0)), timeoutPromise, ])) as number; if (count > 0) { if ( await redisConnection.sadd( "sitemap:" + this.jobId + ":links", normalizeUrl(this.initialUrl), ) ) { urlsHandler([this.initialUrl]); } count++; } return count; } catch (error) { if (error.message === "Sitemap fetch timeout") { this.logger.warn("Sitemap fetch timed out", { method: "tryGetSitemap", timeout, }); return 0; } this.logger.error("Error fetching sitemap", { method: "tryGetSitemap", error, }); return 0; } } public filterURL(href: string, url: string): string | null { let fullUrl = href; if (!href.startsWith("http")) { try { fullUrl = new URL(href, url).toString(); } catch (_) { return null; } } let urlObj; try { urlObj = new URL(fullUrl); } catch (_) { return null; } const path = urlObj.pathname; if (this.isInternalLink(fullUrl)) { // INTERNAL LINKS if ( this.isInternalLink(fullUrl) && this.noSections(fullUrl) && !this.matchesExcludes(path) && this.isRobotsAllowed(fullUrl, this.ignoreRobotsTxt) ) { return fullUrl; } else if ( this.isInternalLink(fullUrl) && this.noSections(fullUrl) && !this.matchesExcludes(path) && !this.isRobotsAllowed(fullUrl, this.ignoreRobotsTxt) ) { (async () => { await redisConnection.sadd( "crawl:" + this.jobId + ":robots_blocked", fullUrl, ); await redisConnection.expire( "crawl:" + this.jobId + ":robots_blocked", 24 * 60 * 60, ); })(); } } else { // EXTERNAL LINKS if ( this.isInternalLink(url) && this.allowExternalContentLinks && !this.isSocialMediaOrEmail(fullUrl) && !this.matchesExcludes(fullUrl, true) && !this.isExternalMainPage(fullUrl) ) { return fullUrl; } } if ( this.allowSubdomains && !this.isSocialMediaOrEmail(fullUrl) && this.isSubdomain(fullUrl) ) { return fullUrl; } return null; } private async extractLinksFromHTMLRust(html: string, url: string) { return (await extractLinks(html)).filter(x => this.filterURL(x, url)); } private extractLinksFromHTMLCheerio(html: string, url: string) { let links: string[] = []; const $ = load(html); $("a").each((_, element) => { let href = $(element).attr("href"); if (href) { if (href.match(/^https?:\/[^\/]/)) { href = href.replace(/^https?:\//, "$&/"); } const u = this.filterURL(href, url); if - Assistant
The provided code snippets are part of the Firecrawl project, which appears to be a web crawling and scraping service. Here's an overview of the key components and their roles:
1.
crawlControllerFunction (crawl.ts): This function handles incoming crawl requests. It validates the request, assigns a unique identifier to the crawl, logs the initiation, and sets up the crawl parameters. It then saves the crawl details to Redis and adds a scrape job to the queue for processing.2. Redis Operations (
crawl-redis.ts): This module manages the storage and retrieval of crawl data in Redis. It provides functions to save crawl details, retrieve them, manage job statuses, and handle URL locking to prevent duplicate processing.3.
WebCrawlerClass (crawler.ts): This class defines the core crawling logic. It initializes with crawl parameters, processes sitemaps, filters links based on specified criteria, and manages the crawling process, including handling robots.txt directives and depth limitations.In summary, these components work together to handle web crawling requests by validating inputs, managing crawl data in Redis, and executing the crawling process while adhering to specified constraints and rules.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
mpt-30b-chatCompare side by side →
- vicuna-7bCompare side by side →
- guanaco-33bCompare side by side →
claude-1Compare side by side →
llama-2-13b-chatCompare side by side →
zephyr-7b-alphaCompare side by side →
deepseek-r1-0528Compare side by side →
gemini-2.5-proCompare side by side →
- gpt-4o-mini-2024-07-18Compare side by side →
mistral-medium-2505Compare side by side →
qwq-32bCompare side by side →
command-a-03-2025Compare side by side →