Skip to content
The Chat Archive
gpt-3.5-turbo-0301· OpenAI WildChat-4.8M (AI2)

Write a python program to read 50000 records from csv file and make POST rest API call…

  1. User

    Write a python program to read 50000 records from csv file and make POST rest API call with JSON request body to process all the records batch wise. at least 36000 records per minute should get uploaded via API utilizing 8 CPU core.

  2. Assistant

    As an AI language model, I don't have access to local files and resources. However, I would suggest the following approach to fulfill your requirements.

    1. Use the Pandas library to read the CSV file and create a DataFrame. You can use the read_csv() function to read the file.

    2. Divide the DataFrame into batches of 1000 records each.

    3. Create a function to make a POST request to the REST API with JSON request body. You can use the requests library to make HTTP requests.

    4. Use the concurrent.futures module to execute the API requests in parallel, utilizing all CPUs available. You can use the ThreadPoolExecutor class to create a thread pool with 8 workers.

    5. Use a timer to measure the time taken to upload 36000 records using the API. If the time taken is more than a minute, reduce the batch size and repeat the process until the required throughput is achieved.

    Here's some code that shows how you can implement the above steps:

    import pandas as pd
    import requests
    import concurrent.futures
    import time
    
    # Read the CSV file into a Pandas DataFrame
    df = pd.read_csv('filename.csv')
    
    batch_size = 1000
    records_per_min = 36000
    num_batches = len(df) // batch_size
    
    # Divide the DataFrame into batches
    batches = [df.iloc[i*batch_size:(i+1)*batch_size] for i in range(num_batches)]
    
    # Function to make POST request to the REST API
    def make_api_call(data):
        url = 'https://api.example.com/upload'
        headers = {'Content-Type': 'application/json'}
        response = requests.post(url, json=data, headers=headers)
        return response.text
    
    # Function to process a batch of data
    def process_batch(data):
        with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
            futures = [executor.submit(make_api_call, row.to_dict()) for _, row in data.iterrows()]
            results = [future.result() for future in futures]
        return results
    
    # Measure the time taken to upload 36000 records
    start_time = time.time()
    uploaded = 0
    for batch in batches:
        results = process_batch(batch)
        uploaded += len(results)
        if uploaded >= records_per_min:
            break
    elapsed_time = time.time() - start_time
    
    # Adjust the batch size if needed
    while elapsed_time > 60 and batch_size > 100:
        batch_size //= 2
        num_batches = len(df) // batch_size
        batches = [df.iloc[i*batch_size:(i+1)*batch_size] for i in range(num_batches)]
        start_time = time.time()
        uploaded = 0
        for batch in batches:
            results = process_batch(batch)
            uploaded += len(results)
            if uploaded >= records_per_min:
                break
        elapsed_time = time.time() - start_time
    
    print(f"Batch size: {batch_size}\nTime taken to upload {records_per_min} records: {elapsed_time:.2f}s")
    

    In this code, we divide the DataFrame into batches of 1000 records each and use the process_batch() function to process each batch. The make_api_call() function makes a POST request to the REST API with the JSON request body.

    We use the concurrent.futures.ThreadPoolExecutor() class to create a thread pool with 8 workers and execute the API requests in parallel. We also use a timer to measure the time taken to upload 36000 records via the API.

    If the time taken is more than a minute, we reduce the batch size by half and repeat the process until the required throughput is achieved. Finally, we print the batch size and the time taken to upload 36000 records.