Skip to content
TheChatArchiveThe Chat Archive
gpt-4.1-mini-2025-04-14· OpenAIAi2 WildChat-4.8M (AI2)

in the given code i am getting codes for test code asynchronously and then joining them…

  1. User

    in the given code i am getting codes for test code asynchronously and then joining them together.

    now i want to generate all test codes test at without async and at a single time by passing all scenarios at once. I've modified scenario_code_generator prompt to accept all scenarios at once and generate the code for all of them and give output in the format of {name: code for name, code in zip([scenario['scenario_name'] for scenario in scenarios], generated_scenario_codes)}

    def test_code_generator(state): """Test Code Generator will generate the test code based on the provided context and the scenarios.""" logger.info("---START TestCodeGenerator---") language = state["metadata"]["language"] object_definition = state["object_definition"] dependency_descriptions = state["dependency_descriptions"] test_scenarios_json = state["test_scenarios_json"] test_scenarios_str = state["test_scenarios_str"] fmc_code = state["fmc_code"] clarifier_info = state["clarifier_info"] global_variables_and_custom_objects = state["global_variables_and_custom_objects"] file_name = state["metadata"]["file_name"] object_name = state["metadata"]["object_name"] import_statements = state["import_statements"] scenario_codes = state["scenario_codes"]

    scenario_to_process = []
    for scenario in test_scenarios_json["test_scenarios"]:
        if scenario_codes[scenario["scenario_name"]] == "":
            scenario_to_process.append(scenario)
    
    generated_scenario_codes = asyncio.run(
        generate_all_scenario_codes(
            scenario_to_process,
            object_definition,
            dependency_descriptions,
            clarifier_info,
            global_variables_and_custom_objects,
            fmc_code,
            file_name,
            object_name,
            language,
        )
    )
    combined_scenario_code = "\n\n".join(generated_scenario_codes.values())
    for scenario_name in generated_scenario_codes:
        scenario_codes[scenario_name] = generated_scenario_codes[scenario_name]
    results_logger.save_to_file(
        path="codes/combined_scenario_functions.txt", data=combined_scenario_code
    )
    
    context = {
        'scenario_codes': scenario_codes,
        "test_scenarios_json": test_scenarios_json,
        "file_name": file_name,
        "object_name": object_name,
        "import_statements":import_statements
    }
    
    llm_response = compile_scenario(context)
    generated_text = process_code(llm_response, language)
    
    results_logger.save_to_file(
        path="codes/1_test_code_generator.py", data=generated_text
    )
    
    return {"test_code": generated_text, "scenario_codes":scenario_codes}
    

    async def generate_all_scenario_codes( scenarios: List[Dict[str, str]], # Correctly specify the type of scenarios object_definition, dependency_descriptions, clarifier_info, global_variables_and_custom_objects, fmc_code, file_name, object_name, language, ) -> Dict[str, str]: """Generate test code for all scenarios in parallel.""" tasks = [] for scenario in scenarios: scenario_name = scenario['scenario_name'] context = { 'object_definition': object_definition, 'dependency_descriptions': dependency_descriptions, "test_scenario": scenario, "clarifier_info": clarifier_info, 'global_variables_and_custom_objects': global_variables_and_custom_objects, "fmc_code": fmc_code, "file_name": file_name, "object_name": object_name, }

        system_prompt, input_prompt = prompt_builder.get_prompt("scenario_code_generator", language=language, context=context)
        
        task = generate_scenario_code(system_prompt, input_prompt)
        tasks.append(task)
        
        prompts_logger.save_to_file(
            f"scenario_code_generator_input_{scenario_name}.txt", input_prompt
        )
    
    generated_scenario_codes = await asyncio.gather(*tasks)
    return {name: code for name, code in zip([scenario['scenario_name'] for scenario in scenarios], generated_scenario_codes)}
    
  2. Assistant

    To generate all test codes at once (not asynchronously per scenario), and by passing all scenarios at once to your LLM in a single prompt, you should modify the generate_all_scenario_codes() to:

    • Accept the entire list of scenarios.
    • Construct a single system & user prompt covering all scenarios at once (not one per scenario).
    • Call a single generate_scenario_code() (or equivalent LLM invocation) with that combined prompt.
    • Parse the output into the dictionary {scenario_name: generated_code} and return that.

    Key points in your current setup:

    • You're currently building a prompt for each scenario and running many async tasks in parallel.
    • You want instead one prompt with all scenarios passed at once.
    • Your scenario_code_generator prompt is already modified to accept all scenarios at once and produce output as {name: code}.

    How to rewrite generate_all_scenario_codes for single prompt, no async concurrency

    def generate_all_scenario_codes(
        scenarios: List[Dict[str, str]],
        object_definition,
        dependency_descriptions,
        clarifier_info,
        global_variables_and_custom_objects,
        fmc_code,
        file_name,
        object_name,
        language,
    ) -> Dict[str, str]:
        """
        Generate test code for all scenarios in one go, passing all scenarios at once.
        """
        context = {
            'object_definition': object_definition,
            'dependency_descriptions': dependency_descriptions,
            'test_scenarios': scenarios,  # pass all scenarios together!
            'clarifier_info': clarifier_info,
            'global_variables_and_custom_objects': global_variables_and_custom_objects,
            'fmc_code': fmc_code,
            'file_name': file_name,
            'object_name': object_name,
        }
    
        system_prompt, input_prompt = prompt_builder.get_prompt(
            "scenario_code_generator", language=language, context=context
        )
    
        # For logging all prompt input at once
        prompts_logger.save_to_file(
            f"scenario_code_generator_input_all.txt", input_prompt
        )
    
        # Here your generate_scenario_code is assumed to be a synchronous function now:
        generated_output = generate_scenario_code(system_prompt, input_prompt)
    
        # The generated_output is expected to be a string representing a python dict {name: code}.
        # You need to parse it safely. (Assuming the LLM returns a valid dict literal)
        import ast
    
        try:
            generated_scenario_codes = ast.literal_eval(generated_output)
        except Exception as e:
            logger.error(f"Failed to parse generated scenario codes: {e}")
            raise
    
        return generated_scenario_codes
    

    Adjustments in test_code_generator

    Since generate_all_scenario_codes is no longer async, you can:

    def test_code_generator(state):
        # ... all your unchanged code ...
    
        scenario_to_process = [
            scenario for scenario in test_scenarios_json["test_scenarios"]
            if scenario_codes.get(scenario["scenario_name"], "") == ""
        ]
    
        if scenario_to_process:
            generated_scenario_codes = generate_all_scenario_codes(
                scenario_to_process,
                object_definition,
                dependency_descriptions,
                clarifier_info,
                global_variables_and_custom_objects,
                fmc_code,
                file_name,
                object_name,
                language,
            )
    
            # Combine and update scenario_codes dict
            combined_scenario_code = "\n\n".join(generated_scenario_codes.values())
            for scenario_name, code in generated_scenario_codes.items():
                scenario_codes[scenario_name] = code
    
            results_logger.save_to_file(
                path="codes/combined_scenario_functions.txt", data=combined_scenario_code
            )
    
        # rest of your code unchanged ...
    

    Notes:

    • generate_scenario_code should now be synchronous (or wrapped with asyncio.run() inside if needed).
    • You must carefully validate and possibly sanitize the output from LLM as it returns the dict string.
    • If your LLM sometimes produces invalid Python dict literals, consider returning JSON and using json.loads() instead, changing the prompt to produce JSON.
    • The important change is to pass all scenarios at once, so the prompt and LLM invocation are done a single time.
    • Remove async/await and asyncio.gather on per-scenario tasks.

    If you want, I can help you convert generate_scenario_code from async to sync and/or help you rewrite your prompt for JSON output to avoid unsafe Python eval parsing. Just ask!

  3. User

    this are the old system and use prompt for asynchronous code generation, convert this for the single generation. make sure the output we will get is correct for the code base

    system prompt

    You are the ScenarioCodeGenerator agent, a critical component in a multi-agent LLM system tasked with generating robust, comprehensive unit tests for given Python functions or classes, with the objective of maximizing the number of passing test cases.

    Your responsibilities:
    
      1. Carefully analyze the given function definition and all related dependencies.
      2. Thoroughly understand the provided test scenario and any additional user information.
      3. Utilize pytest best practices by incorporating:
        - **pytest-mock** to mock external dependencies or parts of the system that need isolation,
        - **pytest fixtures** to setup reusable test data or context,
        - **pytest parametrize** to cover multiple input-output combinations within the same test function when applicable.
      4. Generate only the test function code for the specific scenario that can be seamlessly integrated into a final consolidated test suite.
      5. Maximize meaningful code coverage and correctness, ensuring tests will run successfully under pytest with coverage measurement.
      6. Use the correct and consistent object names and file references as provided.
    
    Guidelines:
    
      1. Follow pytest naming conventions for test functions (e.g., `test_scenario_name`).
      2. Use assert statements to verify expected behavior explicitly.
      3. Apply mocking via the `mocker` fixture from pytest-mock when dependencies need to be stubbed or intercepted.
      4. Define and use reusable pytest fixtures for common setup actions or test data.
      5. Use `@pytest.mark.parametrize` to efficiently test multiple cases/scenarios within one test function if the scenario or user info implies multiple inputs or edge cases.
      6. Include error handling and edge case tests as described in the scenario.
      7. Avoid comments unless necessary for complex logic or user requests.
      8. Ensure final test functions are executable by pytest for generating coverage reports.
      9. Do not hallucinate or add information beyond what is provided.
    
    Example Test Function Structure:
    
      ```python
      import pytest
    
      @pytest.mark.parametrize("input1, input2, expected", [
          (val1, val2, expected_output1),
          (val3, val4, expected_output2),
      ])
      def test_scenario_name(fixture1, mocker, input1, input2, expected):
          # Arrange
          mocked_dep = mocker.patch("module.dependency")
          mocked_dep.return_value = some_mocked_value
          # set up fixtures or necessary test data if required
    
          # Act
          result = file_name.object_name(input1, input2)
    
          # Assert
          assert result == expected, f"Expected {expected}, but got {result}"
      ```
    
    IMPORTANT:
    
      1. Provide only the complete, standalone test function code as the output.
      2. Ensure the test function is ready to run in pytest, incorporating fixtures, parametrize, and mocking as appropriate.
      3. Use fully qualified object names with `{file_name}.{object_name}` as specified.
      4. Focus on maximizing the number of passing, meaningful test cases.
    

    use prompt

    Generate a complete pytest test function for the following Python function, using the provided scenario. Your generated test function should:

      - Use **pytest-mock** (`mocker` fixture) to mock dependencies as necessary.
      - Utilize pytest **fixtures** for setup or reusable test data if applicable.
      - Apply pytest **parametrize** to handle multiple inputs or edge cases within one test function if appropriate.
      - Follow pytest best practices for naming, assertions, and coverage.
      - Use fully qualified calls to objects as `{file_name}.{object_name}`.
      - Only output the test function code, ready to be combined into a larger suite.
    
    **Function Definition:**
      ```python
      {{object_definition}}
      ```
    
    **Dependency Descriptions:**
      ``` 
      {{dependency_descriptions}}
      ```
    
    **Test Scenario:**
      ```
      {{test_scenario}}
      ```
    
    {% if clarifier_info|length > 0 %}
    **User Provided Info:**
      ```
      {{clarifier_info}}
      ```
    {% endif %}
    
    {% if global_variables_and_custom_objects|length > 0 %}
    **Global Variables and Custom Objects:**
      ```python
      {{global_variables_and_custom_objects}}
      ```
    {% endif %}
    
    Please provide **ONLY** the complete test function code in the following format:
    
      ```python
      def test_scenario_name(fixture1, fixture2, mocker):
          # Arrange
          # Use fixtures, parametrize inputs, and set up mocks as required
    
          # Act
          result = file_name.object_name(param1, param2)
    
          # Assert
          assert result == expected_output, f"Expected {expected_output}, but got {result}"
      ```
    
    Ensure the test function:
    
      1. Uses pytest fixtures and/or parametrize appropriately.
      2. Uses pytest-mock via the `mocker` fixture to mock external dependencies.
      3. Is ready to be directly executed by pytest and included in coverage reports.
      4. Covers edge cases and error scenarios described or implied in the inputs.
      5. Respects the object and file names as provided.
    
  4. Assistant

    To convert your multi-prompt async scenario generation into a single-prompt batch generation (for all scenarios at once), you need a new combined system and user prompt that instructs the model to generate the test code for all scenarios together as a dictionary of scenario_name → test_code.


    How to rewrite the prompts for batch scenario code generation

    1) Combined System Prompt (replace your original single-scenario prompt):

    You are the `ScenarioCodeGenerator` agent, a critical component in a multi-agent LLM system tasked with generating robust, comprehensive unit tests for given Python functions or classes, aiming to maximize passing test cases coverage.
    
    Your responsibilities:
    
      1. Carefully analyze the given function definition and all related dependencies.
      2. Thoroughly understand each provided test scenario and any additional user information.
      3. Utilize pytest best practices by incorporating:
        - **pytest-mock** to mock external dependencies or parts needing isolation,
        - **pytest fixtures** to setup reusable test data or context,
        - **pytest parametrize** to cover multiple inputs or edge cases in the same test function where applicable.
      4. For *all* scenarios provided, generate only the test function code for each scenario, ready to be seamlessly integrated into a final consolidated test suite.
      5. Maximize meaningful code coverage and correctness; all tests should run successfully under pytest with coverage measurement.
      6. Use the correct and consistent object names and file references as provided.
    
    Guidelines and constraints:
    
      1. Follow pytest naming conventions for test functions (e.g., `test_scenario_name`).
      2. Use assert statements explicitly verifying expected behavior.
      3. Apply mocking via pytest-mock's `mocker` fixture where dependencies are stubbed.
      4. Define and use reusable pytest fixtures where helpful.
      5. Use `@pytest.mark.parametrize` to efficiently test multiple cases if applicable.
      6. Include error and edge case tests as per the scenarios.
      7. Avoid comments unless for complex logic or explicitly requested.
      8. Ensure final test functions are executable by pytest.
      9. Do not hallucinate or add information beyond provided data.
      10. **Produce a python dictionary** (mapping `scenario_name` string keys to corresponding full test function code string values) exactly in this format, valid Python syntax:
    
    ```python
    {
        "scenario_name_1": "def test_scenario_name_1(...):\n    <code>",
        "scenario_name_2": "def test_scenario_name_2(...):\n    <code>",
        ...
    }
    

    Only output this dictionary, no explanations or extraneous text.


    Example output snippet:

    {
      "scenario_one": "def test_scenario_one(mocker):\n    # test code ...",
      "scenario_two": "def test_scenario_two(fixture1, mocker):\n    # test code ..."
    }
    
    
    ---
    
    ### 2) Combined **User Prompt** (passing all scenarios at once):
    
    ```text
    Generate complete pytest test functions for the following Python function based on *all* provided test scenarios.
    
    Requirements:
    
      - Use **pytest-mock** (`mocker` fixture) to mock dependencies as needed.
      - Utilize pytest **fixtures** for reusable setup or test data where applicable.
      - Apply pytest **parametrize** to handle multiple inputs or edge cases in one test function, if relevant.
      - Follow pytest best practices for naming, assertions, and coverage.
      - Use fully qualified calls as `{file_name}.{object_name}`.
      - Only output a Python dict mapping each `scenario_name` to its test function code string as per the system prompt.
    
    **Function Definition:**
    ```python
    {{object_definition}}
    

    Dependency Descriptions:

    {{dependency_descriptions}}
    

    Test Scenarios:
    Provide all scenarios in a single combined list format:

    {{ test_scenarios }}
    

    {% if clarifier_info|length > 0 %} User Provided Info:

    {{ clarifier_info }}
    

    {% endif %}

    {% if global_variables_and_custom_objects|length > 0 %} Global Variables and Custom Objects:

    {{ global_variables_and_custom_objects }}
    

    {% endif %}

    Remember, output exactly one Python dict mapping scenario names to their corresponding test function code strings, ready to be composed into a final test suite.

    
    ---
    
    # How to change your code accordingly
    
    - Your prompt builder should be changed to create *one* prompt pair (`system_prompt`, `input_prompt`) passing **all the scenarios** in a structure like `context["test_scenarios"] = scenario_list` instead of one scenario at a time.
    
    - Your generation call should be synchronous (or wrapped in `asyncio.run()` once) to call the LLM **once**.
    
    - The output (string) from the LLM should be `ast.literal_eval`-ed or `json.loads`-ed (if you switched to strict JSON in prompt) to a dictionary mapping scenario names to their generated test code strings.
    
    - Return this dictionary to the caller; update your existing `scenario_codes` dict with these results.
    
    ---
    
    # Summary
    
    Your new `generate_all_scenario_codes()` becomes:
    
    ```python
    def generate_all_scenario_codes(
        scenarios: List[Dict[str, str]],
        object_definition,
        dependency_descriptions,
        clarifier_info,
        global_variables_and_custom_objects,
        fmc_code,
        file_name,
        object_name,
        language,
    ) -> Dict[str, str]:
        # Prepare a context with all scenarios at once
        context = {
            'object_definition': object_definition,
            'dependency_descriptions': dependency_descriptions,
            'test_scenarios': scenarios,  # pass all scenarios in the list
            'clarifier_info': clarifier_info,
            'global_variables_and_custom_objects': global_variables_and_custom_objects,
            'fmc_code': fmc_code,
            'file_name': file_name,
            'object_name': object_name,
        }
    
        system_prompt, input_prompt = prompt_builder.get_prompt(
            "scenario_code_generator", language=language, context=context
        )
    
        prompts_logger.save_to_file("scenario_code_generator_input_all.txt", input_prompt)
    
        generated_output = generate_scenario_code(system_prompt, input_prompt)
    
        import ast
        try:
            generated_scenario_codes = ast.literal_eval(generated_output)
        except Exception as e:
            logger.error(f"Error parsing scenario codes dict: {e}")
            raise
    
        return generated_scenario_codes
    

    Replace your old per-scenario async logic with this.


    If you want, I can help you with:

    • Prompt template text files for your codebase
    • Parsing or validation helpers for the returned dict
    • Modifying your prompt builder to handle combined prompts
    • Switching from Python dict literal parsing to JSON safer parsing

    Just tell me!

  5. User

    create a user prompt like the exact format but will return the dictionary kind of output

  6. Assistant

    Certainly! Below is a user prompt formatted similarly to your original style, but rewritten to instruct the model to output a Python dictionary mapping scenario names to their complete test function code strings.

    Use this prompt in your combined scenario code generation call where you pass all test scenarios at once:


    Generate complete pytest test functions **for all** of the following test scenarios, based on the given Python function.
    
    Your output **must be a single Python dictionary** where each key is a scenario name and each value is the corresponding complete test function code as a string.
    
    Each test function should:
    
      - Use **pytest-mock** (`mocker` fixture) to mock dependencies as needed.
      - Utilize pytest **fixtures** for reusable setup or test data if applicable.
      - Apply pytest **parametrize** to handle multiple inputs or edge cases within one test function if appropriate.
      - Follow pytest best practices for naming, assertions, and coverage.
      - Use fully qualified calls as `{file_name}.{object_name}`.
      - Be fully executable by pytest for generating coverage reports.
      - Cover all edge cases and error scenarios described or implied.
      - Avoid extraneous text or explanation — output only the Python dictionary as described.
    
    ---
    
    **Function Definition:**
    ```python
    {{object_definition}}
    

    Dependency Descriptions:

    {{dependency_descriptions}}
    

    Test Scenarios:
    Provide all scenarios in a single list:

    {{test_scenarios}}
    

    {% if clarifier_info|length > 0 %} User Provided Info:

    {{clarifier_info}}
    

    {% endif %}

    {% if global_variables_and_custom_objects|length > 0 %} Global Variables and Custom Objects:

    {{global_variables_and_custom_objects}}
    

    {% endif %}


    Example expected output:

    {
      "scenario_name_1": "def test_scenario_name_1(mocker):\n    # test code here\n    ...",
      "scenario_name_2": "def test_scenario_name_2(fixture1, mocker):\n    # test code here\n    ..."
    }
    
    
    ---
    
    ### Important:
    
    - Output *only* the Python dictionary literal above.
    - Use valid Python string escaping and indentation inside the test function strings.
    - Do not add any explanation or commentary outside the dict.
    - This output will be parsed directly by the codebase to extract individual scenario test functions.
    
    ---
    
    If you want me to help you generate the system prompt or combine this into your prompt builder, just ask!
    

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.