Write a python program to read 50000 records from csv file and make POST rest API call…
- User
Write a python program to read 50000 records from csv file and make POST rest API call with JSON request body to process all the records batch wise. at least 36000 records per minute should get uploaded via API utilizing 8 CPU core.
- Assistant
As an AI language model, I don't have access to local files and resources. However, I would suggest the following approach to fulfill your requirements.
-
Use the Pandas library to read the CSV file and create a DataFrame. You can use the
read_csv()function to read the file. -
Divide the DataFrame into batches of 1000 records each.
-
Create a function to make a POST request to the REST API with JSON request body. You can use the
requestslibrary to make HTTP requests. -
Use the
concurrent.futuresmodule to execute the API requests in parallel, utilizing all CPUs available. You can use theThreadPoolExecutorclass to create a thread pool with 8 workers. -
Use a timer to measure the time taken to upload 36000 records using the API. If the time taken is more than a minute, reduce the batch size and repeat the process until the required throughput is achieved.
Here's some code that shows how you can implement the above steps:
import pandas as pd import requests import concurrent.futures import time # Read the CSV file into a Pandas DataFrame df = pd.read_csv('filename.csv') batch_size = 1000 records_per_min = 36000 num_batches = len(df) // batch_size # Divide the DataFrame into batches batches = [df.iloc[i*batch_size:(i+1)*batch_size] for i in range(num_batches)] # Function to make POST request to the REST API def make_api_call(data): url = 'https://api.example.com/upload' headers = {'Content-Type': 'application/json'} response = requests.post(url, json=data, headers=headers) return response.text # Function to process a batch of data def process_batch(data): with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor: futures = [executor.submit(make_api_call, row.to_dict()) for _, row in data.iterrows()] results = [future.result() for future in futures] return results # Measure the time taken to upload 36000 records start_time = time.time() uploaded = 0 for batch in batches: results = process_batch(batch) uploaded += len(results) if uploaded >= records_per_min: break elapsed_time = time.time() - start_time # Adjust the batch size if needed while elapsed_time > 60 and batch_size > 100: batch_size //= 2 num_batches = len(df) // batch_size batches = [df.iloc[i*batch_size:(i+1)*batch_size] for i in range(num_batches)] start_time = time.time() uploaded = 0 for batch in batches: results = process_batch(batch) uploaded += len(results) if uploaded >= records_per_min: break elapsed_time = time.time() - start_time print(f"Batch size: {batch_size}\nTime taken to upload {records_per_min} records: {elapsed_time:.2f}s")In this code, we divide the DataFrame into batches of 1000 records each and use the
process_batch()function to process each batch. Themake_api_call()function makes a POST request to the REST API with the JSON request body.We use the
concurrent.futures.ThreadPoolExecutor()class to create a thread pool with 8 workers and execute the API requests in parallel. We also use a timer to measure the time taken to upload 36000 records via the API.If the time taken is more than a minute, we reduce the batch size by half and repeat the process until the required throughput is achieved. Finally, we print the batch size and the time taken to upload 36000 records.
-