My js script get uncompleted text, generated by ai. How to fix it? <Begin of…
- User
My js script get uncompleted text, generated by ai. How to fix it? #!/usr/bin/env node
const { chromium } = require('playwright-extra'); const stealth = require('puppeteer-extra-plugin-stealth')(); chromium.use(stealth);
const url = 'https://www.perplexity.ai/search?focus=internet&copilot=false&q=' + process.argv[2]; const buttonCopy = 'svg[data-icon="clipboard"]'; // copy icon on answer const buttonMenu = 'button[data-testid="thread-dropdown-menu"]'; // "..." icon on top right const buttonDelete = 'div[data-testid="thread-delete"]'; // "Delete Thread" button const buttonConfirm = 'css=button:has-text("Confirm")'; const textMessage = 'div[dir="auto"]'; const totalLoopCount = 80; const printIntervalTime = 4000; function containsLetters(text) { return /[a-zA-Zа-яА-ЯЁё]/.test(text); } chromium.launch({ headless: true, timeout: 30000 }).then(async browser => { // start session const page = await browser.newPage(); await page.goto(url, { waitUntil: 'domcontentloaded' }); answer = ""; // get answer for (let i = 0; i < totalLoopCount; i++) { await new Promise((resolve) => setTimeout(resolve, printIntervalTime)); const result = await page.locator(textMessage).textContent(); if (containsLetters(result) && answer < result) { answer = result; } if (await page.locator(buttonCopy).isVisible() && i != (totalLoopCount - 1)){ i = (totalLoopCount - 1); } } console.log(answer); // delete thread await page.click(buttonMenu, {focus: true}); await page.waitForTimeout(300); await page.click(buttonDelete); await page.waitForTimeout(300); await page.click(buttonConfirm);
// close browser await browser.close(); });
- Assistant
It seems like you're trying to extract text generated by an AI from a web page, but you're facing an issue with getting incomplete text.
Here are some suggestions to address the problem of incomplete text extraction:
-
Increase Waiting Time: The current waiting time of
printIntervalTime = 4000milliseconds might not be sufficient for all responses. Consider increasing this delay to allow more time for the AI’s response to fully load. -
Check if Text is Complete: Instead of using a fixed number of iterations (
totalLoopCount), implement a check for whether the text has changed or if it seems to be done loading. One way to achieve this is by looking for a specific phrase that indicates the text is complete or by checking if no new text appears after a certain period. -
Use
waitUntilon Navigation: When you navigate to the page or interact with it, thewaitUntiloption might be modified to include'networkidle'or similar, to ensure that all resources are fully loaded. -
Check for Text Changes: If you are continuously polling for new text, track the previous result and only update when the text actually changes. Additionally, implement a condition that detects when the new text is essentially the same as the previous text after some time has passed.
-
Modify the Answer Accumulation Logic: Instead of simply checking if
answer < result, consider modifying this logic to check whether theresultis truly a new, more complete response compared to what you've previously captured. -
Improving the Selector for Text: Ensure that the CSS selector used to get the AI response is accurate. If more text is available but not captured, it could be due to the selector targeting an element that isn't fully representative of the complete text.
Here’s an updated version of your script integrating some of these suggestions:
#!/usr/bin/env node const { chromium } = require('playwright-extra'); const stealth = require('puppeteer-extra-plugin-stealth')(); chromium.use(stealth); const url = 'https://www.perplexity.ai/search?focus=internet&copilot=false&q=' + process.argv[2]; const buttonCopy = 'svg[data-icon="clipboard"]'; const buttonMenu = 'button[data-testid="thread-dropdown-menu"]'; const buttonDelete = 'div[data-testid="thread-delete"]'; const buttonConfirm = 'css=button:has-text("Confirm")'; const textMessage = 'div[dir="auto"]'; const printIntervalTime = 5000; // Increased interval const maxWaitCycles = 5; // Max cycles of waiting for new text function containsLetters(text) { return /[a-zA-Zа-яА-ЯЁё]/.test(text); } chromium.launch({ headless: true, timeout: 30000 }).then(async browser => { const page = await browser.newPage(); await page.goto(url, { waitUntil: 'domcontentloaded' }); let answer = ""; let lastResult = ""; let waitCycles = 0; while (waitCycles < maxWaitCycles) { const result = await page.locator(textMessage).textContent(); if (containsLetters(result) && lastResult !== result) { answer = result; // Update answer if new text is found lastResult = result; waitCycles = 0; // Reset wait cycles since we found new text } else if (lastResult === result) { waitCycles++; // Increment wait cycles since no new text was detected } await new Promise((resolve) => setTimeout(resolve, printIntervalTime)); // Check if the clipboard button is visible to see if the answer is complete if (await page.locator(buttonCopy).isVisible()) { break; // Exit the loop if the clipboard button is visible } } console.log(answer); // Delete thread await page.click(buttonMenu, { focus: true }); await page.waitForTimeout(300); await page.click(buttonDelete); await page.waitForTimeout(300); await page.click(buttonConfirm); // Close browser await browser.close(); });In this updated version, I added a waiting mechanism that allows the script to stop when no new text appears for a few cycles, and I increased the print interval. Adjust the settings according to the behavior of the page you're working with to optimize for your use case.
-