Skip to content
The Chat Archive
gemini-2.5-flash· Google LMArena Human Preference 140K

I'm writing an in-house Specification by example (SBE) spec document that will define…

  1. User

    I'm writing an in-house Specification by example (SBE) spec document that will define how my SBE test data files will be structured. The structuring part I've done. But there are two options that come to mind in how to break up the data.

    Firstly, the data is for specific functions and modules. This allows you to have a SBE for each function.

    The "normal" way

    src/my_project/

    my_file.py

    specs/my_project/

    my_file.sbe.yaml

    It would contain:

    
    - function: <function_name_1>
    
      examples:
    
      - params:
    
          <param_name_1>: <param_value_1>
    
          ...
    
        returns: <return_value>
    
        ...
    
    ...    
    
    

    For example:

    
    - function: add
    
      examples:
    
      - params:
    
          a: 1
    
          b: 2
    
        returns: 3
    
      - params:
    
          a: 5
    
          b: 10
    
        returns: 15
    
    - function: subtract
    
      examples:
    
      - params:
    
          minuend: 5
    
          subtrahend: 3
    
        returns: 2
    
      - params:
    
          minuend: 10
    
          subtrahend: 7
    
        returns: 3
    
    

    The alternative is:

    src/my_project/

    my_file.py

    specs/my_project/

    my_file.add.sbe.yaml

    my_file.subtract.sbe.yaml

    In the second example, we separate out the functions. So each file contains only data for that function (and the YAML wouldn't be a list of functions, just the one)

    One spec file per source file

    • For a given source file, it's immediately obvious where the spec data will live. Mirrored location, matching filename

    • Fewer spec files

    • If you move a function to another module, to keep parity you must move all of the spec examples to a new file

    On spec file per source file function

    • For a given source file, it's immediately obvious where the spec data will live. Mirror location, matching filename followed by matching function name

    • More spec files

    • If you move a function to another module, to keep parity you simply rename file: module1.add.sbe.yaml => module2.add.sbe.yaml

    Which do you think is the better approach?

  2. Assistant

    This is an excellent question that gets to the heart of maintainability and developer experience for your SBE system. Both approaches have valid arguments, but one tends to scale better with a growing codebase and team.

    Let's break down the pros and cons a bit more deeply for each, and then I'll provide a recommendation.


    Option 1: One Spec File Per Source File (my_file.sbe.yaml)

    Pros:

    1. Fewer Files: Less file system clutter, simpler directory structure.
    2. Module Overview: When you open my_file.sbe.yaml, you get an immediate overview of all the specified behaviors for that module/file. This can be useful for understanding the module's overall responsibilities.
    3. Unified Context: All examples for a module's functions are in one place, which might sometimes reveal implicit relationships or shared concerns between functions that are harder to spot when separated.
    4. Simpler Naming Convention: Just my_file.sbe.yaml.

    Cons:

    1. File Size and Complexity: As my_file.py grows and gains more functions, my_file.sbe.yaml can become very long and difficult to navigate. Scrolling through hundreds or thousands of lines to find a specific function's examples is tedious.
    2. Increased Merge Conflicts: If multiple developers are simultaneously working on different functions within the same my_file.py, they are highly likely to cause merge conflicts in my_file.sbe.yaml when adding or modifying examples. Resolving YAML merge conflicts can be painful.
    3. Refactoring Pain (as you noted): Moving a function from my_file.py to another_file.py requires manually cutting and pasting the function's examples from my_file.sbe.yaml into another_file.sbe.yaml. This is error-prone and tedious.
    4. Granularity of Testing/Reporting: If your SBE runner allows running tests for specific functions, it might still need to parse the entire large YAML file to find the relevant section. (Though this is usually a minor performance concern).
    5. Difficulty in isolating changes: When looking at git diff for my_file.sbe.yaml, it might be harder to immediately discern which function's specification has changed without careful examination.

    Option 2: One Spec File Per Source File Function (my_file.add.sbe.yaml)

    Pros:

    1. Granularity and Focus: Each spec file is small, focused, and directly describes the behavior of a single function. This makes them much easier to read, understand, and maintain.
    2. Reduced Merge Conflicts: This is a major advantage. If developers work on different functions, they will be touching different .sbe.yaml files, drastically reducing the likelihood of merge conflicts.
    3. Simplified Refactoring (as you noted): Moving a function like add from my_file.py to another_file.py simply involves moving/renaming my_file.add.sbe.yaml to another_file.add.sbe.yaml. This is clean, safe, and efficient.
    4. Clear Discoverability (Function Level): If you're looking for the spec for my_file.add(), you know exactly where to go: specs/my_project/my_file.add.sbe.yaml.
    5. Easier Review and Auditing: When reviewing a change to my_file.add.sbe.yaml, you know immediately that only the specification for add is being affected.
    6. Scalability: This approach scales better as your codebase grows and functions are added or refactored.

    Cons:

    1. More Files: This is the most obvious drawback. A module with 20 functions will result in 20 .sbe.yaml files, potentially cluttering the specs/my_project/ directory.
    2. Module Overview Difficulty: Getting a holistic view of a module's behavior requires opening multiple spec files.
    3. Navigation (Initial): While finding a specific function's spec is easy, initially browsing the specs directory might feel overwhelming if there are many files. (However, good IDEs and search tools mitigate this).
    4. Module-Level Specs: What if you have a specification that applies to the entire module rather than a specific function? You'd need a convention for this (e.g., my_file.module.sbe.yaml or my_file.init.sbe.yaml).

    Recommendation

    I strongly recommend Option 2: One spec file per source file function (my_file.function_name.sbe.yaml).

    While the "more files" argument is valid, the benefits in terms of maintainability, reduced merge conflicts, and simplified refactoring far outweigh the perceived clutter, especially in a team environment or for a codebase that will evolve over time.

    Why it's better for SBE specifically:

    SBE is about creating unambiguous, executable specifications. Each function represents a distinct unit of behavior that you want to specify. Giving each function its own dedicated specification file makes that unit of behavior incredibly clear and self-contained. It aligns well with the "single responsibility principle" applied to documentation.

    How to mitigate the "more files" con:

    • IDE Power: Modern IDEs (VS Code, PyCharm, etc.) are excellent at navigating large file structures. "Go to file" search (Ctrl+P, Cmd+P) makes finding my_file.add.sbe.yaml trivial.
    • Directory Structure (Optional Refinement): If the number of files becomes truly overwhelming, you could consider an additional subdirectory layer, e.g.: specs/my_project/my_file/add.sbe.yaml specs/my_project/my_file/subtract.sbe.yaml This keeps all specs for my_file.py within their own subdirectory.

    In summary:

    • Option 1 (One file per source): Good for small, stable modules; bad for large modules, active development teams, or frequent refactoring.
    • Option 2 (One file per function): Excellent for clarity, maintainability, team collaboration, and code refactoring; potentially more files, but manageable with good tooling.

    Go with Option 2. Your future self and your teammates will thank you when refactoring or resolving merges.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.