in the given code i am getting codes for test code asynchronously and then joining them…
- User
in the given code i am getting codes for test code asynchronously and then joining them together.
now i want to generate all test codes test at without async and at a single time by passing all scenarios at once. I've modified scenario_code_generator prompt to accept all scenarios at once and generate the code for all of them and give output in the format of {name: code for name, code in zip([scenario['scenario_name'] for scenario in scenarios], generated_scenario_codes)}
def test_code_generator(state): """Test Code Generator will generate the test code based on the provided context and the scenarios.""" logger.info("---START TestCodeGenerator---") language = state["metadata"]["language"] object_definition = state["object_definition"] dependency_descriptions = state["dependency_descriptions"] test_scenarios_json = state["test_scenarios_json"] test_scenarios_str = state["test_scenarios_str"] fmc_code = state["fmc_code"] clarifier_info = state["clarifier_info"] global_variables_and_custom_objects = state["global_variables_and_custom_objects"] file_name = state["metadata"]["file_name"] object_name = state["metadata"]["object_name"] import_statements = state["import_statements"] scenario_codes = state["scenario_codes"]
scenario_to_process = [] for scenario in test_scenarios_json["test_scenarios"]: if scenario_codes[scenario["scenario_name"]] == "": scenario_to_process.append(scenario) generated_scenario_codes = asyncio.run( generate_all_scenario_codes( scenario_to_process, object_definition, dependency_descriptions, clarifier_info, global_variables_and_custom_objects, fmc_code, file_name, object_name, language, ) ) combined_scenario_code = "\n\n".join(generated_scenario_codes.values()) for scenario_name in generated_scenario_codes: scenario_codes[scenario_name] = generated_scenario_codes[scenario_name] results_logger.save_to_file( path="codes/combined_scenario_functions.txt", data=combined_scenario_code ) context = { 'scenario_codes': scenario_codes, "test_scenarios_json": test_scenarios_json, "file_name": file_name, "object_name": object_name, "import_statements":import_statements } llm_response = compile_scenario(context) generated_text = process_code(llm_response, language) results_logger.save_to_file( path="codes/1_test_code_generator.py", data=generated_text ) return {"test_code": generated_text, "scenario_codes":scenario_codes}async def generate_all_scenario_codes( scenarios: List[Dict[str, str]], # Correctly specify the type of scenarios object_definition, dependency_descriptions, clarifier_info, global_variables_and_custom_objects, fmc_code, file_name, object_name, language, ) -> Dict[str, str]: """Generate test code for all scenarios in parallel.""" tasks = [] for scenario in scenarios: scenario_name = scenario['scenario_name'] context = { 'object_definition': object_definition, 'dependency_descriptions': dependency_descriptions, "test_scenario": scenario, "clarifier_info": clarifier_info, 'global_variables_and_custom_objects': global_variables_and_custom_objects, "fmc_code": fmc_code, "file_name": file_name, "object_name": object_name, }
system_prompt, input_prompt = prompt_builder.get_prompt("scenario_code_generator", language=language, context=context) task = generate_scenario_code(system_prompt, input_prompt) tasks.append(task) prompts_logger.save_to_file( f"scenario_code_generator_input_{scenario_name}.txt", input_prompt ) generated_scenario_codes = await asyncio.gather(*tasks) return {name: code for name, code in zip([scenario['scenario_name'] for scenario in scenarios], generated_scenario_codes)} - Assistant
Got it! You want to change the approach from generating code per scenario asynchronously to generating code for all scenarios at once synchronously, by modifying the prompt to accept all scenarios together and produce all their codes in one shot.
You already changed the prompt
scenario_code_generatorto accept all scenarios at once and generate a dictionary-like output:{name: code for name, code in zip([scenario['scenario_name'] for scenario in scenarios], generated_scenario_codes)}
How to modify the code accordingly?
1. Change
generate_all_scenario_codesfrom async to sync, remove asyncio parts.Since you no longer parallelize per scenario but pass all at once, you only need a single prompt with all scenarios, no tasks.
2. Pass all scenarios in the context to prompt builder.
Here is a minimal working adapted example based on your code style:
def generate_all_scenario_codes( scenarios: List[Dict[str, str]], object_definition, dependency_descriptions, clarifier_info, global_variables_and_custom_objects, fmc_code, file_name, object_name, language, ) -> Dict[str, str]: """ Generate test code for all scenarios at once (synchronously). """ # Prepare context with all scenarios context = { 'object_definition': object_definition, 'dependency_descriptions': dependency_descriptions, "test_scenarios": scenarios, # pass all scenarios at once "clarifier_info": clarifier_info, 'global_variables_and_custom_objects': global_variables_and_custom_objects, "fmc_code": fmc_code, "file_name": file_name, "object_name": object_name, } # Get prompt for all scenarios at once system_prompt, input_prompt = prompt_builder.get_prompt("scenario_code_generator", language=language, context=context) # Save prompt input for debugging prompts_logger.save_to_file( f"scenario_code_generator_input_all_scenarios.txt", input_prompt ) # Generate the combined scenario codes dict as a string generated_code_str = generate_scenario_code(system_prompt, input_prompt) # Process the generated string to convert it into a python dict # assuming the output is something like "{name1: code1, name2: code2, ...}" try: # Use ast.literal_eval for safer eval import ast generated_scenario_codes = ast.literal_eval(generated_code_str) except Exception as e: logger.error(f"Failed to parse generated scenario codes: {e}") # fallback: return empty codes for all scenarios generated_scenario_codes = {scenario['scenario_name']: "" for scenario in scenarios} return generated_scenario_codes
3. Update
test_code_generatorto call the syncgenerate_all_scenario_codes, noasyncio.run()needed:def test_code_generator(state): # ... existing assignments and logging scenario_to_process = [ scenario for scenario in test_scenarios_json["test_scenarios"] if scenario_codes[scenario["scenario_name"]] == "" ] if scenario_to_process: generated_scenario_codes = generate_all_scenario_codes( scenario_to_process, object_definition, dependency_descriptions, clarifier_info, global_variables_and_custom_objects, fmc_code, file_name, object_name, language, ) combined_scenario_code = "\n\n".join(generated_scenario_codes.values()) for scenario_name in generated_scenario_codes: scenario_codes[scenario_name] = generated_scenario_codes[scenario_name] results_logger.save_to_file( path="codes/combined_scenario_functions.txt", data=combined_scenario_code ) else: logger.info("No new scenarios to generate code for.") context = { 'scenario_codes': scenario_codes, "test_scenarios_json": test_scenarios_json, "file_name": file_name, "object_name": object_name, "import_statements": import_statements } llm_response = compile_scenario(context) generated_text = process_code(llm_response, language) results_logger.save_to_file( path="codes/1_test_code_generator.py", data=generated_text ) return {"test_code": generated_text, "scenario_codes": scenario_codes}
Summary of changes:
generate_all_scenario_codesis now synchronous and sends all scenarios at once.- It runs a single code generation call to get the dictionary with codes.
- Output is parsed (e.g., via
ast.literal_eval) from string to dict. test_code_generatorcalls the sync function directly withoutasyncio.run.- Prompt and prompt builder context changed to expect all scenarios at once in
"test_scenarios".
Additional notes:
- Make sure your
prompt_builder.get_prompt("scenario_code_generator", ...)and the underlying LLM prompt template matches this new input structure. - Ensure the LLM outputs the dictionary exactly in a Python-parsable format (e.g., single quotes instead of double quotes, consistent brackets).
- You might want to add some validation/parsing robustness around this step if output varies.
- Logging and prompt saving help debug any issues.
If you want, I can also help with an example prompt template for the all-at-once scenario or a helper function to safely parse the dict from LLM output. Let me know!
- User
this are the old system and use prompt for asynchronous code generation, convert this for the single generation. make sure the output we will get is correct for the code base
system prompt
You are the `ScenarioGenerator` agent, a crucial component in a multi-agent LLM system designed to generate comprehensive test scenarios for unit testing. Your task is to analyze function definition along with the related information to create a diverse set of test cases. Your responsibilities: 1. Analyze the given function definition and its dependency descriptions. 2. Understand the context, input/output types, and potential edge cases. 3. Take into account, the user provided information about the questions you might have. 4. Generate a list of test scenarios covering various aspects of the function's behavior. 5. Aim for a comprehensive set of scenarios that would lead to high code coverage when implemented as tests. Guidelines: 1. First, examine the dependency descriptions and other given info to understand the context and relationships between functions. 2. Then, carefully analyze the main function definition along with the user clarified questions. 3. Generate scenarios that cover normal cases, edge cases, and potential error conditions. 4. Ensure each scenario is unique and tests a specific aspect of the function. 5. Strictly adhere to the required output format for easy parsing. 6. Make sure you take user provided info into consideration while generating scenarios. Output: 1. Present your scenarios in the provided clear, structured string format for easy understanding by other agents. 2. Consider edge cases, boundary conditions, and potential error scenarios. Generate scenarios that test both normal operation and exceptional cases. 3. Ensure your scenarios cover different combinations of inputs, including valid and invalid inputs. 4. Consider scenarios that test the interaction between the main function and its dependencies. 5. Remember, your output will be used to generate actual test code, so be precise and thorough in your scenario descriptions. IMPORTANT: 1. Do Not Hallucinate.use prompt
Generate a list of test scenarios for the following function, taking into account its dependencies and context.
**Function Definition:** ``` {{object_definition}} ``` **Dependencies Descriptions:** ``` {{dependency_descriptions}} ``` {% if clarifier_info|length > 0 %} **User Provided Info** ``` {{clarifier_info}} ``` {% endif %} {% if global_variables_and_custom_objects|length > 0 %} **Global Variables and Custom Objects:** ```python {{global_variables_and_custom_objects}} ``` {% endif %} Please provide a list of test scenarios, considering the following: 1. The function's purpose and expected behavior 2. Various input combinations, including edge cases and boundary conditions 3. Potential interactions with dependency functions 4. Error handling and exceptional cases 5. Coverage of different code paths within the function 6. Use only the provided format as it will be parsed to get a JSON structure. Please provide a list of test scenarios in the following format(do not add backticks): ``` Scenario Name: [First scenario name] Description: [Brief description of the scenario] Input: [Example input or input description] Expected Output: [Expected output or behavior] Reasoning: [Brief explanation of why this scenario is important] test_tags: [A single label that best describes the test, out of: ['happy path', 'edge case','other']"] Scenario Name: [Second scenario name] ... ``` Remember: 1. Ensure that each scenario is separated by a blank line. 2. Generate at least 2-5 diverse scenarios per tags ['happy path', 'edge case','other'] that cover different aspects of the function's behavior, including normal cases, edge cases, and potential error conditions. 3. Stick strictly to the above format as the output will be parsed to generate a JSON structure. - Assistant
To migrate from the per-scenario asynchronous generation to single-shot generation of all scenarios at once, you should combine the previous individual scenario prompt into one prompt that passes all scenarios together and expect the LLM to output a dictionary-like object mapping scenario names to their generated code.
Given that your old system prompt and user prompt were tailored for async per-scenario generation, here is a converted version for single-shot generation of all scenarios at once, preserving the style and ensuring output can be correctly parsed.
1. Revised SYSTEM PROMPT (for all scenarios at once)
You are the `ScenarioCodeGenerator` agent, a core component in a multi-agent LLM system designed to generate test code for a list of provided test scenarios. Your task is to analyze each scenario in the list, along with the function definition and related dependency information, and then generate test code for each scenario. Responsibilities: 1. Carefully analyze the supplied function definition and its dependencies. 2. For each test scenario provided, generate corresponding test code that implements the test. 3. Consider the clarifier info, global variables and custom objects while generating code. 4. Ensure code for each scenario is complete, correct, and idiomatic for the given programming language. 5. Do not hallucinate or add code beyond what is required for the scenario. Output format: - Return a Python dictionary mapping scenario names to their generated test code strings. - The dictionary should be formatted as a Python literal, for example: { "Scenario A": "def test_scenario_a():\n ...", "Scenario B": "def test_scenario_b():\n ..." } - Ensure proper escaping and formatting so the output can be safely parsed. IMPORTANT: - Strictly adhere to the output format. - Do not add any extra text before or after the dictionary. - If you cannot generate code for a scenario, map it to an empty string.
2. Revised USER (INPUT) PROMPT
You will be given: - The function definition:{{object_definition}}
- The dependency descriptions:{{dependency_descriptions}}
{% if clarifier_info|length > 0 %} - User-provided clarifier information:{{clarifier_info}}
{% endif %} {% if global_variables_and_custom_objects|length > 0 %} - Global variables and custom objects: ```python {{global_variables_and_custom_objects}}{% endif %}
- A list of test scenarios (length {{test_scenarios|length}}):
For each scenario, you have:
Scenario Name: {{test_scenarios[0].scenario_name}} Description: {{test_scenarios[0].description}} Input: {{test_scenarios[0].input}} Expected Output: {{test_scenarios[0].expected_output}} Reasoning: {{test_scenarios[0].reasoning}} test_tags: {{test_scenarios[0].test_tags}}
... (repeat for each scenario)
Your task is to generate Python test code implementations for all these scenarios in a single response.
Return a Python dictionary mapping each scenario name to its generated test code string, as shown:
{ "Scenario Name 1": "<test code for scenario 1>", "Scenario Name 2": "<test code for scenario 2>", ... }
Notes:
- Ensure the test code matches the expected behavior described.
- Use appropriate assertion statements, setup, and teardown if necessary.
- Each scenario's test code should be a self-contained function.
- If you cannot generate code for a scenario, set its value to an empty string.
Do not add any additional text outside the dictionary.
--- ## Summary - You pass **all scenarios** at once in `test_scenarios` list. - All function and dependency info passed once. - LLM outputs **one Python dict** mapping scenario_name → generated_code. - Your parser can safely use `ast.literal_eval()` on the output string returned by the LLM. --- ## Example snippet showing usage in your code: ```python # context passed to prompt_builder context = { "object_definition": object_definition, "dependency_descriptions": dependency_descriptions, "clarifier_info": clarifier_info, "global_variables_and_custom_objects": global_variables_and_custom_objects, "test_scenarios": scenario_to_process, "language": language, } system_prompt, input_prompt = prompt_builder.get_prompt("scenario_code_generator", language=language, context=context) generated_code_str = generate_scenario_code(system_prompt, input_prompt) import ast try: generated_scenario_codes = ast.literal_eval(generated_code_str) except Exception as e: logger.error("Failed to parse generated scenario codes: %s", e) generated_scenario_codes = {s["scenario_name"]: "" for s in scenario_to_process}
If you want, I can also help update the prompt builder template files to reflect these changes in code — just ask!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.