Skip to content
TheChatArchiveThe Chat Archive
gpt-3.5-turbo-0301· openaiHugging Face Suzhen/CodeChat

Hi how can I make requests with axios and async.queue in node.js?

  1. User

    Hi how can I make requests with axios and async.queue in node.js?

  2. Assistant

    Here's an example code on how you can make requests using axios and async.queue in Node.js:

    const axios = require('axios');
    const async = require('async');
    
    // Create a queue with concurrency of 5
    const queue = async.queue(async (url) => {
      const response = await axios.get(url);
      console.log(`${url} - Status code: ${response.status}`);
    }, 5);
    
    // Add some URLs to the queue
    queue.push('https://www.google.com');
    queue.push('https://www.yahoo.com');
    queue.push('https://www.bing.com');
    queue.push('https://www.github.com');
    
    // Drain event will be triggered when all tasks are completed
    queue.drain(() => {
      console.log('All requests completed');
    });
    

    In this example, we import the axios and async modules, create an async queue with a concurrency of 5, and add some URLs to it.

    Then, we define the function that will be executed by each task in the queue. It receives a URL as a parameter and makes a GET request using axios. Once the response is received, we print the URL and the HTTP status code to the console.

    Finally, we listen to the drain event of the queue, which is emitted when all tasks have been completed. In our case, we use it to print a message to the console indicating that all requests have been completed.

  3. User

    then How can I return the responses?

  4. Assistant

    To return the responses, you can modify the function executed by each task in the queue to return the response data instead of printing it to the console. You can then use a callback function to capture the response data and do something with it.

    Here's an example code:

    const axios = require('axios');
    const async = require('async');
    
    // Create a queue with concurrency of 5
    const queue = async.queue(async (url, callback) => {
      try {
        const response = await axios.get(url);
        callback(null, response.data); // Call the callback with the response data
      } catch(error) {
        callback(error); // Call the callback with the error object
      }
    }, 5);
    
    // Add some URLs to the queue
    queue.push('https://www.google.com');
    queue.push('https://www.yahoo.com');
    queue.push('https://www.bing.com');
    queue.push('https://www.github.com');
    
    // Drain event will be triggered when all tasks are completed
    queue.drain(() => {
      console.log('All requests completed');
    });
    
    // Listen to the completion of each task
    queue.drain('taskComplete', (taskId, responseData) => {
      console.log(`Response data for task ${taskId}: ${responseData}`);
    });
    
    // Listen to the errors of each task
    queue.drain('taskError', (taskId, error) => {
      console.log(`Error for task ${taskId}: ${error.message}`);
    });
    

    In this example, we modified the function executed by each task to call the callback function with the response.data object if the request was successful, or with the error object if it failed. We also added two event handlers to listen to the completion and errors of each task.

    The drain event is triggered when all tasks have been completed. In our case, we use it to print a message to the console indicating that all requests have been completed.

    The taskComplete event is triggered when a task is completed successfully. In our case, we use it to print the response data to the console along with the task ID.

    The taskError event is triggered when a task encounters an error. In our case, we use it to print the error message to the console along with the task ID.