help me to fetch a page and get js property from that page
- User
help me to fetch a page and get js property from that page
- Assistant
Here are a few approaches you can take, depending on whether the JavaScript property you need is already in the HTML source (e.g. a global
var foo = …;) or is only exposed once the page’s JS runs (e.g. after AJAX calls or client‐side rendering).
1. If the JS property lives in a
<script>tag in the HTMLSay the page contains somewhere:
<script> window.MY_CONFIG = { apiEndpoint: "https://api.example.com", version: "1.2.3" }; </script>You can fetch the raw HTML and use a regex (or a small parser) to extract it.
Node.js + node-fetch + regex
// npm install node-fetch const fetch = require('node-fetch'); async function fetchConfig(url) { const res = await fetch(url); const text = await res.text(); // crude regex to pull out the JSON in window.MY_CONFIG = { … }; const match = text.match(/window\.MY_CONFIG\s*=\s*(\{[\s\S]*?\});/); if (!match) throw new Error("MY_CONFIG not found"); const config = JSON.parse(match[1]); return config; } (async () => { const cfg = await fetchConfig('https://www.example.com'); console.log(cfg.apiEndpoint, cfg.version); })();
2. If the property is set only after the page’s JavaScript runs
In that case you need a headless browser so the JS will execute, then you can read
window.whateverviaevaluate().A. Node.js + Puppeteer
// npm install puppeteer const puppeteer = require('puppeteer'); async function getProp(url, propPath) { const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto(url, { waitUntil: 'networkidle0' }); // propPath could be 'MY_CONFIG.apiEndpoint' or 'myGlobalVar' const value = await page.evaluate(p => { // safely dig into window return p.split('.').reduce((o, key) => o && o[key], window); }, propPath); await browser.close(); return value; } (async () => { const apiEndpoint = await getProp('https://www.example.com', 'MY_CONFIG.apiEndpoint'); console.log('API endpoint is', apiEndpoint); })();B. Python + Selenium
# pip install selenium webdriver-manager from selenium import webdriver from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager def get_js_property(url, prop_path): service = Service(ChromeDriverManager().install()) options = webdriver.ChromeOptions() options.add_argument('--headless') driver = webdriver.Chrome(service=service, options=options) driver.get(url) # e.g. prop_path = 'MY_CONFIG.apiEndpoint' script = f"return {prop_path};" value = driver.execute_script(script) driver.quit() return value if __name__ == '__main__': ep = get_js_property('https://www.example.com', 'window.MY_CONFIG.apiEndpoint') print('API endpoint =', ep)
3. A middle‐ground: jsdom (for simple pages)
If there’s no AJAX or dynamic rendering, you can pull HTML and let jsdom simulate
window. jsdom won’t run external scripts by default, but you can inline them:// npm install node-fetch jsdom const fetch = require('node-fetch'); const { JSDOM } = require('jsdom'); async function getPropViaJsdom(url, propPath) { const res = await fetch(url); const html = await res.text(); const dom = new JSDOM(html, { runScripts: "dangerously", resources: "usable" }); await new Promise(r => dom.window.addEventListener("load", r)); // access nested property const value = propPath.split('.').reduce((o, k) => o && o[k], dom.window); return value; } (async () => { const v = await getPropViaJsdom('https://www.example.com', 'MY_CONFIG.version'); console.log(v); })();
Which should you pick?
- Simple static extraction (JS object literal in HTML): use the regex approach.
- Full JS execution needed (SPA or dynamic data): use Puppeteer (Node) or Selenium (Python).
- Minimal “DOM” without a full browser: try jsdom, knowing it may not handle all dynamic scripts.
Let me know which scenario matches yours or if you’d like code tuned further!
- User
i mean provide console snippet
- Assistant
Here’s a single‐paste snippet you can drop into your browser DevTools Console. It will fetch any URL, scrape out a JS object literal assigned to a global, and log it:
;(async ()=> { // 1) Edit these two values: const url = 'https://www.example.com'; // ← the page you want to fetch const varName = 'window.MY_CONFIG'; // ← the global you’re after // 2) Fetch the HTML const res = await fetch(url); const text = await res.text(); // 3) Build a regex to grab the “= { … };” that follows your varName const re = new RegExp(varName.replace('.','\\.') + '\\s*=\\s*(\\{[\\s\\S]*?\\});'); const m = text.match(re); if (!m) { console.error(`❌ ${varName} not found in the HTML of ${url}`); return; } // 4) Parse & log let obj; try { obj = JSON.parse(m[1]); } catch (err) { console.error('❌ Failed to JSON.parse:', err); return; } console.log(`✅ ${varName} =`, obj); return obj; // returns it too, in case you want to assign it })();How it works:
- You
fetch()the raw HTML of the target page. - A quick regex looks for e.g.
window.MY_CONFIG = { … }; - We
JSON.parsethe{ … }chunk and spit it back to you.
—
If the JS property you need isn’t inlined (i.e. it only exists after the page’s own scripts run), you’d have to spin up a headless browser (Puppeteer/Selenium). But for simple inline globals, the above console snippet should do the trick! - You
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.