I'm writing an in-house Specification by example (SBE) spec document that will define…
- User
I'm writing an in-house
Specification by example (SBE)spec document that will define how my SBE test data files will be structured. The structuring part I've done. But there are two options that come to mind in how to break up the data.Firstly, the data is for specific functions and modules. This allows you to have a SBE for each function.
The "normal" way
src/my_project/
my_file.py
specs/my_project/
my_file.sbe.yaml
It would contain:
- function: <function_name_1> examples: - params: <param_name_1>: <param_value_1> ... returns: <return_value> ... ...For example:
- function: add examples: - params: a: 1 b: 2 returns: 3 - params: a: 5 b: 10 returns: 15 - function: subtract examples: - params: minuend: 5 subtrahend: 3 returns: 2 - params: minuend: 10 subtrahend: 7 returns: 3The alternative is:
src/my_project/
my_file.py
specs/my_project/
my_file.add.sbe.yaml
my_file.subtract.sbe.yaml
In the second example, we separate out the functions. So each file contains only data for that function (and the YAML wouldn't be a list of functions, just the one)
One spec file per source file
-
For a given source file, it's immediately obvious where the spec data will live. Mirrored location, matching filename
-
Fewer spec files
-
If you move a function to another module, to keep parity you must move all of the spec examples to a new file
On spec file per source file function
-
For a given source file, it's immediately obvious where the spec data will live. Mirror location, matching filename followed by matching function name
-
More spec files
-
If you move a function to another module, to keep parity you simply rename file: module1.add.sbe.yaml => module2.add.sbe.yaml
Which do you think is the better approach?
-
- Assistant
This is an excellent question that gets to the heart of maintainability and developer experience for your SBE system. Both approaches have valid arguments, but one tends to scale better with a growing codebase and team.
Let's break down the pros and cons a bit more deeply for each, and then I'll provide a recommendation.
Option 1: One Spec File Per Source File (
my_file.sbe.yaml)Pros:
- Fewer Files: Less file system clutter, simpler directory structure.
- Module Overview: When you open
my_file.sbe.yaml, you get an immediate overview of all the specified behaviors for that module/file. This can be useful for understanding the module's overall responsibilities. - Unified Context: All examples for a module's functions are in one place, which might sometimes reveal implicit relationships or shared concerns between functions that are harder to spot when separated.
- Simpler Naming Convention: Just
my_file.sbe.yaml.
Cons:
- File Size and Complexity: As
my_file.pygrows and gains more functions,my_file.sbe.yamlcan become very long and difficult to navigate. Scrolling through hundreds or thousands of lines to find a specific function's examples is tedious. - Increased Merge Conflicts: If multiple developers are simultaneously working on different functions within the same
my_file.py, they are highly likely to cause merge conflicts inmy_file.sbe.yamlwhen adding or modifying examples. Resolving YAML merge conflicts can be painful. - Refactoring Pain (as you noted): Moving a function from
my_file.pytoanother_file.pyrequires manually cutting and pasting the function's examples frommy_file.sbe.yamlintoanother_file.sbe.yaml. This is error-prone and tedious. - Granularity of Testing/Reporting: If your SBE runner allows running tests for specific functions, it might still need to parse the entire large YAML file to find the relevant section. (Though this is usually a minor performance concern).
- Difficulty in isolating changes: When looking at
git diffformy_file.sbe.yaml, it might be harder to immediately discern which function's specification has changed without careful examination.
Option 2: One Spec File Per Source File Function (
my_file.add.sbe.yaml)Pros:
- Granularity and Focus: Each spec file is small, focused, and directly describes the behavior of a single function. This makes them much easier to read, understand, and maintain.
- Reduced Merge Conflicts: This is a major advantage. If developers work on different functions, they will be touching different
.sbe.yamlfiles, drastically reducing the likelihood of merge conflicts. - Simplified Refactoring (as you noted): Moving a function like
addfrommy_file.pytoanother_file.pysimply involves moving/renamingmy_file.add.sbe.yamltoanother_file.add.sbe.yaml. This is clean, safe, and efficient. - Clear Discoverability (Function Level): If you're looking for the spec for
my_file.add(), you know exactly where to go:specs/my_project/my_file.add.sbe.yaml. - Easier Review and Auditing: When reviewing a change to
my_file.add.sbe.yaml, you know immediately that only the specification foraddis being affected. - Scalability: This approach scales better as your codebase grows and functions are added or refactored.
Cons:
- More Files: This is the most obvious drawback. A module with 20 functions will result in 20
.sbe.yamlfiles, potentially cluttering thespecs/my_project/directory. - Module Overview Difficulty: Getting a holistic view of a module's behavior requires opening multiple spec files.
- Navigation (Initial): While finding a specific function's spec is easy, initially browsing the
specsdirectory might feel overwhelming if there are many files. (However, good IDEs and search tools mitigate this). - Module-Level Specs: What if you have a specification that applies to the entire module rather than a specific function? You'd need a convention for this (e.g.,
my_file.module.sbe.yamlormy_file.init.sbe.yaml).
Recommendation
I strongly recommend Option 2: One spec file per source file function (
my_file.function_name.sbe.yaml).While the "more files" argument is valid, the benefits in terms of maintainability, reduced merge conflicts, and simplified refactoring far outweigh the perceived clutter, especially in a team environment or for a codebase that will evolve over time.
Why it's better for SBE specifically:
SBE is about creating unambiguous, executable specifications. Each function represents a distinct unit of behavior that you want to specify. Giving each function its own dedicated specification file makes that unit of behavior incredibly clear and self-contained. It aligns well with the "single responsibility principle" applied to documentation.
How to mitigate the "more files" con:
- IDE Power: Modern IDEs (VS Code, PyCharm, etc.) are excellent at navigating large file structures. "Go to file" search (
Ctrl+P,Cmd+P) makes findingmy_file.add.sbe.yamltrivial. - Directory Structure (Optional Refinement): If the number of files becomes truly overwhelming, you could consider an additional subdirectory layer, e.g.:
specs/my_project/my_file/add.sbe.yamlspecs/my_project/my_file/subtract.sbe.yamlThis keeps all specs formy_file.pywithin their own subdirectory.
In summary:
- Option 1 (One file per source): Good for small, stable modules; bad for large modules, active development teams, or frequent refactoring.
- Option 2 (One file per function): Excellent for clarity, maintainability, team collaboration, and code refactoring; potentially more files, but manageable with good tooling.
Go with Option 2. Your future self and your teammates will thank you when refactoring or resolving merges.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.