Polyglot repo toolchain: shared config, code style, linters, consistency checks.
- User
Polyglot repo toolchain: shared config, code style, linters, consistency checks.
- Assistant
We are going to design a polyglot repository toolchain that promotes shared configuration, code style, linters, and consistency checks. Since the repository is polyglot (multiple programming languages), we need to set up a configuration that can handle different languages but maintain consistency where possible.
Steps:
- Use a top-level configuration directory (e.g.,
.config/) or files at the root for tools that support it (like .editorconfig, .gitattributes, etc.) - For each language, we can have language-specific configurations, but we try to share as much as possible (e.g., same indentation, line endings, etc.)
- Use pre-commit hooks to run linters and formatters for each language.
- Use a continuous integration (CI) pipeline to run the same checks on every push.
Let's break down:
a. EditorConfig: to share basic coding styles (indentation, charset, etc.) across editors and languages.
b. Pre-commit: a framework for managing and maintaining multi-language pre-commit hooks.
c. Linters and formatters per language:
- For Python: black (formatter), flake8 (linter) - For JavaScript/TypeScript: prettier (formatter), eslint (linter) - For Go: gofmt (formatter), go vet (linter) - For Rust: rustfmt (formatter), clippy (linter) - For Java: spotless (formatter), checkstyle (linter) or similar - For C/C++: clang-format (formatter), cppcheck (linter) or clang-tidy But note: we want consistency. So we might choose a common formatter style where possible (like using the same indentation and line length settings) and configure each formatter accordingly.d. We can also have a top-level configuration for pre-commit that runs the appropriate hooks for the files that changed.
e. We can use a tool like
toxfor Python ornpmfor JavaScript to run tests, but note that we are polyglot so we might have multiple test runners.However, note that pre-commit can run any script, so we can write a script that, given the changed files, runs the appropriate linters/formatters for those files.
But pre-commit already has hooks for many languages and can be configured to run only on certain file types.
Approach:
-
Create a
.pre-commit-config.yamlat the root that includes hooks for each language, but with the condition that they only run on files of that language. -
For each language, we set up:
- A formatter hook (if available) that runs on staged files of that language.
- A linter hook that runs on staged files of that language.
-
We also want to run these checks in CI for all files, not just staged ones.
-
We can also set up a top-level
Makefileor similar to run all checks for the entire codebase.
Example for pre-commit:
We can use existing hooks from the pre-commit framework:
- For Python:
blackandflake8 - For JavaScript/TypeScript:
prettierandeslint - For Go:
gofmtandgo vet(but note: go vet might require building, so we might need to run it in CI for all files)
However, note that pre-commit runs on staged files, so for formatters that's fine. For linters, we might want to run on all files in CI.
Let's design:
In
.pre-commit-config.yaml:repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.4.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - id: check-yaml - id: check-added-large-files
Python
-
repo: https://github.com/psf/black rev: 23.7.0 hooks:
- id: black language_version: python3.10
-
repo: https://github.com/pycqa/flake8 rev: 6.0.0 hooks:
- id: flake8
JavaScript/TypeScript
-
repo: https://github.com/pre-commit/mirrors-prettier rev: v3.0.3 hooks:
- id: prettier
-
repo: https://github.com/pre-commit/mirrors-eslint rev: v8.45.0 hooks:
- id: eslint files: .[jt]sx?$ # .js, .jsx, .ts, .tsx
Go
- repo: https://github.com/dnephin/pre-commit-golang
rev: v0.5.1
hooks:
- id: gofmt
- id: go vet
Note: go vet might require building, and pre-commit runs on staged files. But we can also run it on all files in CI.
Rust: we can use
cargo fmtandcargo clippybut note that these might require the entire project to be buildable.We might not want to run these on every commit if the code doesn't build? But that's the point of pre-commit: to catch issues early.
However, note: pre-commit runs in isolated environments, so for Rust we might need to install rustup and then run cargo.
Alternatively, we can use:
- repo: local
hooks:
-
id: cargo-fmt name: cargo fmt entry: cargo fmt --all -- --check language: system types: [rust]
-
id: cargo-clippy name: cargo clippy entry: cargo clippy --all-targets --all-features -- -D warnings language: system types: [rust]
-
But then we require the developer to have Rust installed and the project to build.
Alternatively, we can use a hook that sets up Rust:
- repo: https://github.com/rust-lang/rust
rev: stable
hooks:
- id: rustfmt
- id: clippy
But the rust-lang/rust repo doesn't have a pre-commit hook. So we might have to write our own or use a third-party.
Actually, there is a community hook: https://github.com/rust-lang/rust-pre-commit
But let's avoid external dependencies that might break. We can use:
- repo: local
hooks:
-
id: rustfmt name: rustfmt entry: rustfmt --check language: system types: [rust]
-
id: clippy name: clippy entry: cargo clippy -- -D warnings language: system types: [rust]
-
But again, requires Rust.
-
We must also consider that some linters might not be safe to run on only staged files (like go vet on a package that depends on other changed files). So for some languages, we might want to run the linter on the entire project in CI, and in pre-commit only run the formatter and maybe very fast linters.
-
Another idea: use pre-commit to run formatters (which are safe and fast) and then in CI run the linters and tests.
-
We can also have a top-level script that runs all linters for the entire codebase, and then have pre-commit run the formatters and maybe the linters only on staged files (if safe).
-
We want to share code style: so we set up EditorConfig to have:
root = true
[*] charset = utf-8 indent_style = space indent_size = 4 end_of_line = lf insert_final_newline = true trim_trailing_whitespace = true
But note: some languages (like Go) have a different indentation style (tabs, but actually gofmt uses tabs and sets indent_style to tab). So we might have to override per language.
We can do:
[*.go] indent_style = tab
But then we break the consistency? Maybe we decide that for Go we follow the Go style (tabs) and for others spaces.
Alternatively, we can set a standard for the whole repo (spaces) and then configure gofmt to use spaces? But gofmt does not allow that: it uses tabs.
So we have to decide: either we break consistency for Go, or we use tabs for everything? But other languages might not like tabs.
Another option: use EditorConfig to set common rules and then per language override only what is necessary.
But note: EditorConfig is supported by many editors and tools, but not all. And some linters/formatters have their own configuration.
So maybe we don't rely on EditorConfig for the formatters, but use the formatters' own configuration, and use EditorConfig only for the basic ones that are widely supported.
-
We can have a top-level configuration for each formatter (like
.black.tomlfor black,.prettierrcfor prettier, etc.) and then configure them to have the same line length, etc. -
We also want to check for consistency across the entire codebase, so we might want to run the formatters in CI on all files, not just staged.
-
We can set up a CI pipeline that:
- Runs the formatters in check mode on all files (and fails if there are changes)
- Runs the linters on all files
- Runs tests for each language
-
We can have a
Makefilethat has:make fmt-check # runs all formatters in check mode on the entire codebase make lint # runs all linters on the entire codebase make test # runs all tests
-
We must be cautious with the formatters: some of them (like black) are opinionated and might change code that we don't want to change? But that's the point: we want consistency.
-
We also want to handle multiple languages in the same repository, so we might have directories for each language.
Example structure:
repo/ .editorconfig .pre-commit-config.yaml .github/workflows/ci.yml Makefile python/ pyproject.toml (for black and flake8) requirements.txt javascript/ .prettierrc .eslintrc.js go/ go.mod rust/ Cargo.toml java/ # ...
-
For the CI, we can set up a matrix to test on different operating systems? But that might be heavy. Alternatively, we can test on one OS (like ubuntu) but for all languages.
-
We might also want to set up a code coverage report? But that's language-specific.
-
We also want to check for dependency vulnerabilities? That's language-specific too.
Given the complexity, we might want to start with:
- EditorConfig for basic consistency.
- Pre-commit for formatters and fast linters on staged files.
- CI for full formatting check, linters, and tests.
-
We also want to make it easy for contributors: they can run
pre-commit installand then get the hooks. -
We can also set up a
CONTRIBUTING.mdthat explains the toolchain.
Let's draft the files:
a. .editorconfig:
root = true [*] charset = utf-8 end_of_line = lf insert_final_newline = true trim_trailing_whitespace = true indent_style = space indent_size = 4 [*.go] indent_style = tab [*.md] trim_trailing_whitespace = false [*.{yml,yaml,json}] indent_size = 2 [*.{js,jsx,ts,tsx}] indent_size = 2b. .pre-commit-config.yaml:
We'll include: - pre-commit-hooks for basic stuff - black (for Python) - flake8 (for Python) - prettier (for JS, TS, JSON, MD, etc.) - eslint (for JS, TS) - gofmt (for Go) - go vet (for Go) -> but note: go vet might not be safe on staged files? Actually, it's safe because it doesn't build the whole project? But go vet does require the package to build. If we have multiple changed files in the same package, then it's okay. But if we have only one file changed in a package that doesn't build because of other unchanged files? Then it would fail. So we might want to run go vet on the entire project in CI. So in pre-commit, we might skip go vet and only run gofmt? Or run go vet only on the changed package? That's complex. Alternatively, we can run go vet on the entire project in CI and in pre-commit only run gofmt. Similarly for Rust: we run cargo fmt in pre-commit and cargo clippy in CI. So let's adjust: In pre-commit: - For Go: only gofmt. - For Rust: only cargo fmt. - For Python: black and flake8 (flake8 is safe on a single file? Actually, flake8 can be run on a single file, but it might not catch all errors that require the whole project. So we might want to run flake8 on the entire project in CI as well? But then we duplicate. We can run flake8 in pre-commit on the staged files and in CI on all files. But note: flake8 can be run per file and it will only check that file, but it might miss errors that are across files. So for Python, we might want to run flake8 on the entire project in CI. However, pre-commit's flake8 hook runs on the staged files only, so we might miss errors that are in unstaged files? But the idea is that the entire codebase should be linted. So we need to run on all files in CI. So we do: Pre-commit: - Formatters for all languages (on staged files) - Quick linters that are safe on a per-file basis? Or we can run the linters on the staged files and assume that if the staged files are correct and the rest of the codebase is already clean, then it's okay. But if the rest of the codebase has errors, then we might not catch them until CI. So we have two layers: Layer 1 (pre-commit): format staged files and run linters on staged files (fast, but might miss some). Layer 2 (CI): format all files (to ensure consistency) and run linters on all files (to catch any existing issues). So in pre-commit, we run: - black on staged Python files? But black can format multiple files at once. We can run black on the staged files, but note that black might change files that are not staged? No, because we only pass the staged files. But black's default is to format the files we give it. So it's safe. However, for gofmt, we can run on the staged files. For prettier, we can run on the staged files. For eslint, we can run on the staged files (but note: eslint might have rules that require the entire project? Some rules do, but we can configure eslint to only check the staged files? Actually, eslint by default will look at the entire project configuration and might use type information that requires the whole project. So running on a single file might not be safe. So we might have to run eslint on the entire project in CI, and in pre-commit we might skip it? Or run it only on the staged files and hope that the configuration doesn't require the whole project? We can set up eslint to be a "flat" config and not use type information that requires the whole project? That's a trade-off. Alternatively, we can run eslint in pre-commit on the staged files and then in CI on the entire project, and if the CI fails because of an error in an unstaged file, then the developer must fix it. But then the pre-commit hook passed, but the CI fails because of an error in an unstaged file? That's okay because we want the entire codebase to be clean. So we do: Pre-commit: formatters and linters on staged files (fast, and ensures that the changes we are committing are clean). CI: formatters and linters on the entire codebase (to ensure the entire codebase is clean). So the pre-commit hooks might be a subset of the CI checks. We can write the CI to run the formatters in check mode on all files and the linters on all files. And for the formatters, we can run them in check mode and fail if they would change any file. Example for black: black --check . For prettier: prettier --check . For gofmt: gofmt -l . # lists files that would be changed For cargo fmt: cargo fmt --all -- --check And then for linters: flake8 . eslint . # for the entire project? But that might be heavy. We can run eslint on the entire project but only on the files that are part of the project? Actually, eslint by default looks at the current directory and subdirectories. But note: we have multiple languages in one repo, so we have to run the linters in the appropriate directories. So in CI, we can: - Run black on the python directory - Run flake8 on the python directory - Run prettier on the javascript directory (and maybe other directories that have JS/TS) - Run eslint on the javascript directory - Run gofmt and go vet on the go directory - Run cargo fmt and cargo clippy on the rust directory So we need a CI script that knows the structure. Alternatively, we can use the pre-commit framework in CI by running: pre-commit run --all-files This will run all the hooks on all files. But note: some hooks might not be safe to run on all files? Actually, the hooks are defined to run on certain file types, so it should be safe. And for hooks that are dangerous? We have to trust the hooks. So we can use the same pre-commit configuration for CI. Then we don't need separate commands. But note: some hooks might require building (like go vet) and might fail if the entire project doesn't build? But in CI we are running on the entire project, so that's okay. So we can do: In CI: - Install pre-commit and the required languages/tools. - Run `pre-commit run --all-files` But then we are running formatters in check mode? The hooks we defined for formatters (black, prettier, gofmt) are set to run in check mode? Actually, the black hook in pre-commit by default runs black in check mode? Let's check: The black hook in pre-commit-hooks (or the one from psf/black) runs black with `--diff` and `--check`? Looking at the pre-commit black hook: https://github.com/psf/black/blob/main/.pre-commit-hooks.yaml - id: black args: [--safe, --quiet] # or: args: [--line-length=88] It doesn't have `--check` by default. Actually, the black hook from psf/black runs black in place? No, by default it runs black and stages the changes? Wait, the pre-commit black hook runs black with `--diff` and then applies the changes? Actually, the hook entry point is `black` and it passes the files. Without `--check`, black will reformat the files and then pre-commit will stage the changes. But we want to check if the files are formatted correctly without changing them? In CI we want to check, not fix. So we have two options: Option 1: In pre-commit, we let the hook modify the files (so it fixes formatting) and then the developer commits the fixed files. Option 2: In pre-commit, we run in check mode and if it fails, the commit is aborted and the developer must fix. The standard for pre-commit is to fix the files automatically for formatters. So the black hook in pre-commit runs without `--check` and then stages the changes. But then in CI, we want to run the same hook but in check mode? We can set up the hook with `args: [--check]` for CI? But we use the same configuration. Alternatively, we can have two different hooks: one for pre-commit (with modify) and one for CI (with check). But that's duplication. Actually, the pre-commit black hook by default runs in check mode? Let me check: The hook from psf/black: - id: black args: [--safe, --quiet] # or: args: [--line-length=88] The `--safe` is for black, and `--quiet` is for black. There's no `--check`. So by default, black will reformat the files. But wait, the pre-commit framework runs hooks in a way that if the hook modifies a file, then pre-commit will stage the changes and the commit will include the formatted files. So that's the intended behavior: the formatter fixes the files. Then in CI, we want to run the same hook but in check mode? We can set an environment variable or a flag to change the behavior. Alternatively, we can have a separate hook for CI that runs in check mode. But note: we want the same configuration. So we can set: In .pre-commit-config.yaml for black: - id: black args: [--safe, --quiet] # This will reformat in pre-commit. Then in CI, we run: pre-commit run --all-files --hook-stage manual But that runs the same hook with the same args, so it will reformat? We don't want that in CI. So we need a way to run the hook in check mode in CI. We can define two hooks for black: - id: black args: [--safe, --quiet] # for pre-commit - id: black-check name: black (check) entry: black args: [--safe, --quiet, --check] language: system types: [python] But then we have two hooks and we have to maintain both. Alternatively, we can use the same hook and set an environment variable to change the behavior? But black doesn't have an environment variable for that. So we have two options: 1. In pre-commit, we let the hook reformat. In CI, we run the same hook but with `--check` by overriding the args. 2. In pre-commit, we run in check mode and then have a separate script to fix the formatting? But then the developer has to run that script. The common practice is to have the pre-commit hook fix the files, and then in CI we run the same hook in check mode? Actually, if the pre-commit hook fixed the files, then in CI the check should pass. But what if someone bypasses pre-commit? Then in CI the check will fail and we can fix it in CI? But then the commit is already made. So we want CI to fail if the code is not formatted, and then we can fix it in a separate commit? That's not ideal. The best practice is to have the pre-commit hook fix the files and then commit the fixed files. Then CI will pass. But if someone bypasses pre-commit and commits unformatted code, then CI will fail and they have to fix it. So we can set up: In pre-commit: formatters that fix the code (so the commit includes the fixed code). In CI: we run the same formatters in check mode to ensure that the code is formatted. But if we run the same hook without `--check` in CI, it will try to fix the files and might fail because the files are read-only? Actually, in CI we are in a clean workspace, so it can fix? But we don't want to modify the files in CI because we are just checking. So we need to run the formatters in check mode in CI. How about we set up the hooks in .pre-commit-config.yaml to have two different configurations for the same tool? We can have: - repo: https://github.com/psf/black rev: 23.7.0 hooks: - id: black args: [--safe, --quiet] # for pre-commit, this will reformat - id: black name: black (check) args: [--safe, --quiet, --check] pass_filenames: false # because we want to run on the entire project? Actually, we want to run on all files. But then we have two black hooks. We can set one to run in pre-commit (by default) and the other only in CI? But pre-commit runs all hooks by default. Alternatively, we can use the `stages` config in pre-commit to have one hook run only in the `manual` stage? But then we have to run `pre-commit run --stage manual` in CI. Actually, pre-commit has `--hook-stage`: We can set: - id: black stages: [commit] # default is commit and push? Actually, default is commit. - id: black-check stages: [manual] # then we run in CI with `pre-commit run --stage manual` Then in CI we run: pre-commit run --stage manual --all-files And in pre-commit, we run the normal black hook (without --check) which will fix the files. But wait, the black-check hook is the same tool but with `--check`. We have to define two hooks. Alternatively, we can have one hook that runs in both stages but with different args? No, because the args are fixed in the config. So we define two hooks. Let's do: For each formatter, we define two hooks: - The normal one (for pre-commit) that fixes the files. - A check one (for CI) that runs in check mode. But then we have to maintain two entries for each formatter. Alternatively, we can write a wrapper script that checks an environment variable to decide whether to fix or check. Example for black: entry: bash -c 'if [ "$PRE_COMMIT_CHECK" = "true" ]; then black --safe --quiet --check "$@"; else black --safe --quiet "$@"; fi' -- Then in pre-commit we don't set the environment variable, so it fixes. In CI we set PRE_COMMIT_CHECK=true. But then we have to set the environment variable in CI. We can do: In .pre-commit-config.yaml: - id: black entry: bash -c 'if [ -n "$PRE_COMMIT_CHECK" ]; then black --safe --quiet --check "$@"; else black --safe --quiet "$@"; fi' -- language: system types: [python] Then in CI: export PRE_COMMIT_CHECK=true pre-commit run --all-files But note: the hook is defined as `language: system` and we are using bash. That might not be portable? But in CI we control the environment. Alternatively, we can write a small script and put it in the repo, and then use that as the entry point. But let's keep it simple: define two hooks. However, note that the pre-commit framework already has a `hook` for black that is designed to fix the code. We don't want to change that. So we can define: - repo: https://github.com/psf/black rev: 23.7.0 hooks: - id: black args: [--safe, --quiet] # This will fix in pre-commit. - repo: local hooks: - id: black-check name: black (check) entry: black args: [--safe, --quiet, --check] language: system types: [python] stages: [manual] Then in CI, we run: pre-commit run --stage manual --all-files This will run the black-check hook on all files. And for other formatters, we do the same. But then we have to do this for every formatter. Alternatively, we can have a single hook that runs all formatters in check mode? But then we lose the per-language configuration. Given the complexity, maybe we don't use pre-commit for the CI check? Instead, we write a script that runs: black --check . prettier --check . gofmt -l . etc. But then we duplicate the configuration (the arguments, the file patterns, etc.) So we decide to use pre-commit for both, but with two sets of hooks. However, note that the pre-commit framework is designed to run on the changed files (by default) and in CI we run on all files. So we can run the same hooks on all files in CI. But the problem is the formatter hooks that modify files. We don't want them to modify in CI. So we have to have a way to run the formatters in check mode in CI. Let's see how other projects do it. Example: the `pre-commit` repo itself uses: In .pre-commit-config.yaml, they have: - repo: https://github.com/psf/black rev: 22.3.0 hooks: - id: black Then in their CI, they run: pre-commit run --all-files And that runs black in check mode? Wait, no. The black hook from psf/black runs black without --check, so it will try to modify the files. But in their CI, they run: - name: Run pre-commit run: | pip install pre-commit pre-commit run --all-files And that runs black without --check? Then it would try to modify the files in the CI workspace, and then the next steps might see the modified files? That's not what we want. Actually, the pre-commit framework runs the hooks and if the hook modifies a file, then pre-commit will not stage the changes (because we are not in a git repository? or it does stage the changes?). Wait, pre-commit by default runs on the unstaged changes and then stages the changes. But when we run `pre-commit run --all-files`, it runs on all files and does not stage the changes? Actually, it runs on all files and if a hook modifies a file, then the file is changed in the working directory. But then the CI step would see the changed files? That's not good. So we must run the formatters in check mode in CI. Therefore, we must have a way to run the formatters in check mode. So we go back to having two hooks per formatter. But then we have to maintain two sets of arguments. Alternatively, we can have a script that runs the formatter in check mode if an environment variable is set, and then use that script in both hooks. Let's create a script `scripts/format-check.sh`: #!/bin/bash set -e TOOL=$1 shift if [ -n "$FORMAT_CHECK" ]; then exec "$TOOL" --check "$@" else exec "$TOOL" "$@" fi Then for black: entry: scripts/format-check.sh black And in pre-commit, we don't set FORMANT_CHECK, so it runs black without --check (and fixes). In CI, we set FORMAT_CHECK=true. But then we have to write a script for each formatter? Because the command line arguments are different. Alternatively, we can have a script for each formatter? That's too many. Or we can have a script that takes the tool and then passes all arguments, and then conditionally adds --check. But the position of --check might matter? For black, --check must be at the end? Actually, black's --check is an option that can be anywhere? It's a flag. We can do: scripts/format-check.sh: #!/bin/bash set -e TOOL=$1 shift if [ -n "$FORMAT_CHECK" ]; then exec "$TOOL" --check "$@" else exec "$TOOL" "$@" fi Then in .pre-commit-config.yaml: - id: black entry: scripts/format-check.sh black language: system types: [python] And then we have to make sure the script is executable and present. But then we have to write a script for each tool? Because the tool might be a command that is not just one word? For example, `cargo fmt` is two words. We can do: scripts/format-check.sh: #!/bin/bash set -e TOOL=$1 shift if [ -n "$FORMAT_CHECK" ]; then exec "$TOOL" --check "$@" else exec "$TOOL" "$@" fi But for `cargo fmt`, we would set: entry: scripts/format-check.sh cargo fmt Then $1 is `cargo`, and $2 is `fmt`, and then we shift and then we have the file arguments. But then we are executing: cargo fmt --check file1 file2 ... That works because `cargo fmt` takes `--check` and then the files. But for `gofmt`, it doesn't have a `--check` option? Actually, gofmt has `-l` to list files that would be changed. So we need a different script. So we have to have per-tool scripts? That's too complex. Alternatively, we can have two separate hooks and maintain the arguments separately. Given the complexity, maybe we don't use the same configuration for pre-commit and CI. We can have: In pre-commit: use the standard hooks that fix the code. In CI: run the formatters in check mode with a separate script that doesn't use pre-commit. Then we duplicate the list of formatters and their arguments. But we can generate the CI script from the pre-commit config? That's too complex. Given the time, let's do: We'll use pre-commit for the developer experience (auto-fixing on commit) and then in CI we run the same tools in check mode, but with a separate script that we maintain. We can have a `scripts/run-formatters-check.sh` that runs: black --check . prettier --check . gofmt -l . # and then fail if there are any files listed cargo fmt --all -- --check # and then fail if there are changes? etc. But then we have to duplicate the arguments. Alternatively, we can have a `pyproject.toml` for black, a `.prettierrc` for prettier, etc. and then the commands are just the tool name. So we can have: black --check . prettier --check . gofmt -l . | grep -v '^$' && exit 1 || exit 0 But gofmt's `-l` lists the files that are not formatted. So if there are any, we fail. For cargo fmt, we can do: cargo fmt --all -- --check | grep -v '^$' && exit 1 || exit 0 But note: cargo fmt --check will output the files that would be changed? Actually, it doesn't output anything if there are no changes? We can use: if cargo fmt --all -- --check | grep -q .; then echo "Some files were not formatted" exit 1 fi But that might be brittle. Alternatively, we can run `cargo fmt --all -- --check` and check the exit code? According to the cargo fmt documentation, it returns 0 if no changes, 1 if changes. So: cargo fmt --all -- --check And then check the exit code. So we can write a script that runs each formatter and checks the exit code. But then we have to know which formatters to run for which directories? We can run them on the entire repo, but some formatters might fail on files they don't understand? They should ignore non-applicable files. So we can run: black --check . prettier --check . gofmt -l . | grep -v '^$' && exit 1 || true cargo fmt --all -- --check But note: black and prettier might fail if there are syntax errors? That's okay, we want to catch that. So we can write a `scripts/run-formatters-check.sh`: #!/bin/bash set -e echo "Running black..." black --check . echo "Running prettier..." prettier --check . echo "Running gofmt..." if gofmt -l . | grep -q .; then echo "Some go files are not formatted" exit 1 fi echo "Running cargo fmt..." cargo fmt --all -- --check But then we have to have the tools installed in the CI environment. And we have to run this script in the appropriate directories? Actually, we run it from the root and the tools will traverse the directory tree. But note: black will format Python files, but we might have a `python/` directory. We can run black from the root and it will find all Python files? Yes. But what if we have a `node_modules` directory? We should ignore that. The tools usually ignore by default? Black will format all .py files, but we can have a `pyproject.toml` that excludes directories? Actually, black by default will ignore directories listed in `.gitignore`? Not exactly, it uses the git index? When running `black .`, it will format all tracked files? Actually, black formats all files in the given directory that match the pattern, but it does not respect .gitignore by default? Wait, black does respect .gitignore? According to the black documentation, when given a directory, it will format all files that are not ignored by git? Actually, black uses `git ls-files` to get the list of files? Not exactly, it uses `pathlib` to walk the directory and then uses `pathspec` to match the patterns from `.gitignore`? I think black does respect `.gitignore` by default. Similarly, prettier uses its own ignore file. So we can run these from the root. But then we have to install all the tools in the CI environment. This is getting complex. Given the time, let's stick with: - Use pre-commit for developer experience (formatters fix and linters on staged files). - In CI, run `pre-commit run --all-files` but with a configuration that runs the formatters in check mode? We can't because the same hook is used. So we have to have two hooks for formatters. Let's define: For each formatter: - A hook that runs in the `commit` stage (default) and fixes the code. - A hook that runs in the `manual` stage and checks the code. Then in CI, we run: pre-commit run --stage manual --all-files And in pre-commit, we run the default hooks (commit stage) which fix the code. How do we set the stage for the check hooks? In .pre-commit-config.yaml: - id: black # This is the fixing hook, runs on commit stage (default) - id: black-check name: black (check) entry: black args: [--safe, --quiet, --check] language: system types: [python] stages: [manual] Then in CI, we run the manual stage. But note: the black-check hook is defined as `language: system` and we are using the system's black. But in pre-commit, the black hook uses the black from the repo (via `repo: https://github.com/psf/black`). So we have two different black versions? That's not good. So we have to use the same black version for both. We can define the black-check hook using the same repo? But then we are installing black twice? That's okay. Alternatively, we can define the black-check hook as: - repo: https://github.com/psf/black rev: 23.7.0 hooks: - id: black args: [--safe, --quiet, --check] stages: [manual] Then we have two hooks from the same repo, but with different args and stages. That's acceptable. So let's change: For black: - repo: https://github.com/psf/black rev: 23.7.0 hooks: - id: black args: [--safe, --quiet] - id: black name: black (check) args: [--safe, --quiet, --check] stages: [manual] But wait, the hook id is the same? That might cause issues. Actually, the hook id is used to identify the hook. We can have the same id? The pre-commit config requires unique ids? Actually, the id is the hook id, and we are listing two hooks with the same id? That's not allowed. So we have to give the second hook a different id? But then we are using the same repo and the same hook, but with different args. We can do: - repo: https://github.com/psf/black rev: 23.7.0 hooks: - id: black args: [--safe, --quiet] # This is the fixing hook, runs on commit. - id: black-check name: black (check) args: [--safe, --quiet, --check] stages: [manual] But the `id: black-check` is not a hook in the repo. We are creating a new hook with a custom name, but the entry point is still the same (black). That's okay. Actually, the `id` in the hook is the hook id from the repo? No, we can override the `id`? The `id` in the hook is the hook id from the repo, but we can change the `name`? Actually, the `id` is the identifier for the hook in our configuration. We can set it to anything? The `id` in the hook is the hook id from the repo, but we can change the `name`? Let me check the pre-commit documentation: The `id` is the hook id from the repository's `.pre-commit-hooks.yaml`. We cannot change it? Actually, we can set a custom `id`? No, the `id` is the hook id from the repository. We can set a custom `name` but not a custom `id`. So we have to use the same `id` for both? That would be two hooks with the same id? That's not allowed. So we have to use two different repositories? Or we use a local hook for the check. Let's use: For the fixing hook: - repo: https://github.com/psf/black rev: 23.7.0 hooks: - id: black args: [--safe, --quiet] For the check hook: - repo: local hooks: - id: black-check name: black (check) entry: black args: [--safe, --quiet, --check] language: system types: [python] stages: [manual] Then we are using the system's black for the check. But we want to use the same version as in the fixing hook? We can install black in the CI environment to the same version? We can specify the version in the CI setup. Alternatively, we can use the same black from the repo for both? But the local hook uses system, so we have to install black in the system. Given the complexity, maybe we don't use the same tool for both. We can have: In pre-commit: use the black hook from psf/black (which will install black in a virtualenv and run it). In CI: we install black in the system and run it with --check. But then we have two versions of black? We can pin the version in the CI. But we want to use the same version. So we can have a `requirements.txt` or `pyproject.toml` that specifies the version of black, and then in pre-commit we use the black from the repo (which is a specific version) and in CI we install that version. But then we are duplicating the version. Alternatively, we can have the pre-commit black hook use the black from the project's virtualenv? That's possible with `language: python` and `additional_dependencies`? Actually, the black hook from psf/black installs black in its own environment. So we cannot share the version. Given the trade-offs, I think we should: - Use pre-commit for fixing and quick checks on staged files. - In CI, run the same tools in check mode, but we accept that we might have a different version? We can pin the version in CI to the same as in pre-commit by reading the rev from .pre-commit-config.yaml? That's too complex. Alternatively, we can use the pre-commit framework in CI and run the same hooks, but we have to set up the environment exactly as pre-commit does? That's what pre-commit does: it installs the hooks in isolated environments. So if we run `pre-commit run --all-files` in CI, it will install the hooks (black, etc.) in isolated environments and run them. And for the formatters, they will try to modify the files? We don't want that. So we are back to the problem. Given the time, I think we should: - Use pre-commit for the developer experience (auto-fixing formatters and running linters on staged files). - In CI, we run the same tools but in check mode, and we accept that we might have to install the tools in the CI environment and pin the versions. We can have a `scripts/ci-checks.sh` that runs: # Formatters in check mode black --check . prettier --check . if gofmt -l . | grep -q .; then echo "go files not formatted" exit 1 fi cargo fmt --all -- --check # Linters flake8 . eslint . go vet ./... cargo clippy --all-targets --all-features -- -D warnings And then we pin the versions in the CI setup. But then we duplicate the configuration (like line length, etc.) because the formatters use their own configuration files (pyproject.toml for black, .prettierrc for prettier, etc.). So as long as we have those configuration files, the tools will use the same configuration. So we don't duplicate the configuration, we just duplicate the commands. This seems acceptable. So we do: - Use pre-commit for auto-fixing and quick linters on staged files. - Use a CI script that runs the formatters in check mode and the linters on the entire codebase. And we maintain the configuration for each tool in the language-specific directories. Then we don't have to worry about the pre-commit hooks trying to modify files in CI. So we remove the idea of having two sets of hooks in pre-commit. We'll have in .pre-commit-config.yaml: - pre-commit-hooks for basic stuff - black (without --check) for Python formatting (fixing) - flake8 for Python linting (on staged files) - prettier for formatting (fixing) for JS/TS - eslint for linting (on staged files) - gofmt for Go formatting (fixing) - for Go linting, we might not run in pre-commit because go vet requires the whole project? We can run go vet on the staged package? But if the staged package depends on other packages that are not staged, it might fail. So we skip go vet in pre-commit and only run it in CI. Similarly for Rust: we run cargo fmt in pre-commit (fixing) and cargo clippy in CI. So in pre-commit: - Python: black (fixing) and flake8 (on staged files) - JavaScript/TypeScript: prettier (fixing) and eslint (on staged files) - Go: gofmt (fixing) -> no go vet in pre-commit - Rust: cargo fmt (fixing) -> no clippy in pre-commit Then in CI, we run: - All formatters in check mode (black, prettier, gofmt, cargo fmt) - All linters (flake8, eslint, go vet, clippy) on the entire codebase. This way, the pre-commit hooks are fast and safe (only on staged files) and the CI does the full check. But then we might have a situation where the staged files are clean but the entire codebase is not? For example, if someone commits unformatted code and then the next person runs pre-commit, the pre-commit will fix the staged files, but the unformatted code from the previous commit is still there. Then CI will fail for the entire codebase? Yes, because we run the formatters on the entire codebase. So the CI will catch it and the developer must fix the entire codebase? That's okay. So let's go with that. Now, the .pre-commit-config.yaml: repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.4.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - id: check-yaml - id: check-added-large-files - repo: https://github.com/psf/black rev: 23.7.0 hooks: - id: black args: [--safe, --quiet] - repo: https://github.com/pycqa/flake8 rev: 6.0.0 hooks: - id: flake8 - repo: https://github.com/pre-commit/mirrors-prettier rev: v3.0.3 hooks: - id: prettier - repo: https://github.com/pre-commit/mirrors-eslint rev: v8.45.0 hooks: - id: eslint files: \.[jt]sx?$ - repo: https://github.com/dnephin/pre-commit-golang rev: v0.5.1 hooks: - id: gofmt - repo: local hooks: - id: cargo-fmt name: cargo fmt entry: cargo fmt --all -- --check # Wait, we want to fix in pre-commit? Then we don't use --check. # Actually, for pre-commit we want to fix, so we remove --check. entry: cargo fmt --all language: system types: [rust] pass_filenames: false # because cargo fmt doesn't take filenames? It formats the entire project. But note: cargo fmt formats the entire project, so we run it on the entire project even if only one file is staged? That's okay. But then we are formatting the entire project in pre-commit? That might be slow and also might format files that are not staged? But that's the intention: we want the entire project to be formatted. Alternatively, we can run cargo fmt only on the staged Rust files? But cargo fmt doesn't take a list of files? Actually, it does: `cargo fmt -- file1.rs file2.rs` but then it only formats those files? But note: cargo fmt might need to format the entire project because of module dependencies? I think it's safe to format only the changed files? But the Rust style is such that the entire project should be formatted consistently. So we run on the entire project. So we set `pass_filenames: false` and run `cargo fmt --all`. But then in pre-commit, we are formatting the entire project on every commit? That might be heavy. Alternatively, we can run `cargo fmt` on the staged files? But then we might have inconsistent formatting. So we run on the entire project. Similarly for gofmt: we run on the entire project? But gofmt can take a directory or a list of files. We can run `gofmt -l .` to list the files that are not formatted, and then fix them? But the pre-commit gofmt hook from dnephin runs `gofmt -l` and then `gofmt -w` on the listed files? Actually, the hook runs `gofmt -l` and then `gofmt -w` on the files that are not formatted? Let me check the hook: https://github.com/dnephin/pre-commit-golang The gofmt hook runs: gofmt -l $@ gofmt -w $@ So it formats only the files passed (the staged files). But then we might have inconsistent formatting? We want the entire project to be formatted. So we change the gofmt hook to run on the entire project? We can override the entry: - id: gofmt entry: gofmt -w . language: system types: [go] But then we are formatting the entire project. Alternatively, we can leave it as is and rely on the CI to format the entire project? But then the staged files are formatted, but the rest might not be. That's okay because the CI will fail if the entire project is not formatted, and then the developer must run `cargo fmt` and `gofmt` on the entire project and commit. So we can have: In pre-commit: - gofmt: run on staged files only (so it's fast and only touches the files we are committing). - cargo fmt: run on entire project (because it's fast? Actually, cargo fmt might be slow for large projects). But then we have inconsistency: gofmt only formats the staged files, cargo fmt formats the entire project. We want consistency: either both format only staged files, or both format the entire project. Given that gofmt is fast and formatting the entire project is acceptable, we can change the gofmt hook to format the entire project. But then we are formatting the entire project on every commit? That might be heavy for large projects. Alternatively, we can have a script that runs gofmt on the entire project and then only stages the changed files? But pre-commit will stage the changes anyway. So we can change the gofmt hook to: entry: gofmt -w . But then we are writing all go files in the project, even if they are not staged? That might be acceptable. But then we are modifying files that are not staged? And then pre-commit will stage those changes? That's not what we want. Actually, pre-commit runs the hook and then stages the changes that the hook made? Yes, if the hook modifies files, pre-commit will stage those changes. So if we run `gofmt -w .`, it will format all go files in the project (including untracked? and ignored?) and then pre-commit will stage all the changes? That's not good. So we must run the formatter only on the staged files. Therefore, we keep the gofmt hook as is (runs on staged files) and for cargo fmt, we run on the entire project? That's inconsistent. Alternatively, we can run cargo fmt only on the staged Rust files? But cargo fmt doesn't support a list of files? Actually, it does: `cargo fmt -- file1.rs file2.rs` but then it only formats those files? And it might not format the entire project consistently? The Rust style is per-file? Actually, rustfmt formats each file independently? I think it's per-file, so it's safe to format only the changed files. So we can change the cargo fmt hook to: entry: cargo fmt --all -- # and then pre-commit will pass the staged files? But we set pass_filenames: false, so no files are passed. We can set `pass_filenames: true` and then: entry: cargo fmt --all But then cargo fmt will ignore the passed files? Actually, `cargo fmt --all` formats the entire project, ignoring the arguments. So we want `cargo fmt` without `--all` and with the files? But then we have to pass the files. According to cargo fmt, if you pass files, it only formats those files. So we can do: entry: cargo fmt And set `pass_filenames: true` (default). Then pre-commit will pass the staged files. But then we are only formatting the staged Rust files. So we can have: - gofmt: runs on staged files (as provided by the hook) - cargo fmt: runs on staged files (by default) But then we might have inconsistent formatting? For Rust, if we only format the changed files, then the entire project might have inconsistent formatting? But rustfmt is deterministic and per-file? I think it's per-file, so it's safe. But what about imports? rustfmt can sort imports, and if we only format one file, the imports might be sorted differently than in other files? But that's okay because each file is independent. So we can run both on staged files. Then in CI, we run on the entire codebase. So we change the cargo fmt hook to: - id: cargo-fmt name: cargo fmt entry: cargo fmt language: system types: [rust] And remove `--all` and `pass_filenames: false`. Then it will format the staged Rust files. But note: cargo fmt might require the project to build? It can format without building? Yes, it only needs the source files. So we do: In pre-commit: - gofmt: formats staged Go files. - cargo fmt: formats staged Rust files. In CI: - gofmt -l . # and fail if any - cargo fmt --all --check # check the entire project But then in pre-commit, we are only formatting the staged files, so the entire project might be unformatted? But that's okay because the CI will catch it. So we are consistent: we only format the staged files in pre-commit, and in CI we check the entire project. This is acceptable. Now, for the linters: In pre-commit: - flake8 on staged Python files. - eslint on staged JS/TS files. - skip go vet and clippy in pre-commit. In CI: - flake8 on all Python files. - eslint on all JS/TS files. - go vet on the entire Go project. - cargo clippy on the entire Rust project. This way, pre-commit is fast and CI is comprehensive. Now, we write the .pre-commit-config.yaml accordingly. We also need to consider that some linters might not be safe to run on a single file? For example, flake8 can run on a single file, but it might not catch all errors. But that's acceptable for pre-commit. Let's write the .pre-commit-config.yaml: repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.4.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - id: check-yaml - id: check-added-large-files # Python - repo: https://github.com/psf/black rev: 23.7.0 hooks: - id: black args: [--safe, --quiet] - repo: https://github.com/pycqa/flake8 rev: 6.0.0 hooks: - id: flake8 # JavaScript/TypeScript - repo: https://github.com/pre-commit/mirrors-prettier rev: v3.0.3 hooks: - id: prettier - repo: https://github.com/pre-commit/mirrors-eslint rev: v8.45.0 hooks: - id: eslint files: \.[jt]sx?$ # Go - repo: https://github.com/dnephin/pre-commit-golang rev: v0.5.1 hooks: - id: gofmt # Rust - repo: local hooks: - id: cargo-fmt name: cargo fmt entry: cargo fmt language: system types: [rust] Note: For Rust, we are using `language: system` which means it uses the system's cargo and rustc. We must have Rust installed in the developer's machine and in CI. Now, for the CI script (scripts/ci-checks.sh): #!/bin/bash set -e echo "Running black check..." black --check . echo "Running prettier check..." prettier --check . echo "Running gofmt check..." if gofmt -l . | grep -q .; then echo "Some go files are not formatted" exit 1 fi echo "Running cargo fmt check..." cargo fmt --all -- --check echo "Running flake8..." flake8 . echo "Running eslint..." eslint . echo "Running go vet..." go vet ./... echo "Running cargo clippy..." cargo clippy --all-targets --all-features -- -D warnings But note: we are running `black --check .` and `prettier --check .` from the root. They will respect the configuration files (pyproject.toml, .prettierrc) and ignore files in .gitignore. We must make sure that the CI environment has all the tools installed and at the correct versions. We can have a `requirements.txt` for Python, a `package.json` for JavaScript, and use `go mod` and `cargo` for Go and Rust. But we don't want to install the entire project's dependencies for the formatters and linters? For black and flake8, we need the Python environment. For eslint, we need node_modules. For go vet and clippy, we need the dependencies. So we must install the dependencies for each language before running the checks. We can have: For Python: python -m venv .venv source .venv/bin/activate pip install -r requirements.txt # or use `pip install -e .` if we have a setup.py? But we might have multiple Python projects? We assume one Python project at the root or in a directory. Alternatively, we can use `tox` to run the checks in isolated environments? But then we have to set up tox for multiple languages? That's not what tox is for. Given the complexity, we might want to have a Makefile that sets up the environments and then runs the checks. But note: we are polyglot, so we might have to set up multiple package managers. We can have a `Makefile` with: .PHONY: install install: # For Python python -m venv .venv . .venv/bin/activate && pip install -r requirements.txt # For JavaScript npm install # For Go: we assume go is installed and we use go mod download? Actually, go vet and clippy require the dependencies to be available. # We can run `go mod download` in the go directory. # For Rust: cargo will download dependencies automatically. .PHONY: fmt fmt: black . prettier --write . gofmt -w . cargo fmt --all .PHONY: fmt-check fmt-check: black --check . prettier --check . if gofmt -l . | grep -q .; then exit 1; fi cargo fmt --all -- --check .PHONY: lint lint: flake8 . eslint . go vet ./... cargo clippy --all-targets --all-features -- -D warnings Then in CI, we run: make install make fmt-check make lint But then we are installing the dependencies for all languages, even if we only changed one language? That's okay because we want to ensure that the entire codebase is consistent. However, we might want to optimize by only installing and checking the languages that changed? But that's complex. Given the simplicity, we run all. But note: the `make install` might fail if we don't have a requirements.txt or package.json? We can make the install steps conditional. Alternatively, we can have a script that checks which languages are present and then installs accordingly. But for simplicity, we assume the repository has the standard files for each language. We can do: install: if [ -f requirements.txt ]; then python -m venv .venv . .venv/bin/activate && pip install -r requirements.txt fi if [ -f package.json ]; then npm install fi if [ -f go.mod ]; then cd go && go mod download && cd .. fi # Rust: no need to explicitly download, cargo will do it. But then we have to be in the correct directory for go mod? We assume the go project is in the `go` directory? Not necessarily. We can do: find . -name go.mod -execdir go mod download \; But that might run in multiple go modules. Given the time, we assume one go module at the root or in a known directory. Alternatively, we can run `go mod download` in every directory that has a go.mod? That's safe. Similarly for Rust, we can run `cargo fetch` in every directory that has a Cargo.toml. But then we are downloading dependencies for all languages, which is heavy. Given the trade-offs, we might want to skip the install step and let the tools fail if dependencies are missing? Then the CI will fail and we know we need to set up the environment. But we want the CI to be self-contained. So we do: In the CI script (or Makefile), we install the dependencies for each language if the corresponding manifest file exists. We can write a `scripts/install-deps.sh`: #!/bin/bash set -e if [ -f requirements.txt ]; then python -m venv .venv . .venv/bin/activate pip install -r requirements.txt fi if [ -f package.json ]; then npm install fi # For Go, we assume go is installed and in PATH. find . -name go.mod -execdir go mod download \; # For Rust, we assume cargo is installed. find . -name Cargo.toml -execdir cargo fetch \; Then in CI: bash scripts/install-deps.sh bash scripts/ci-checks.sh But note: the formatters and linters might require additional dependencies? For example, flake8 might require plugins that are in requirements.txt? We installed the requirements, so that should be okay. For eslint, we installed the node_modules, so it should be okay. For go vet and clippy, we downloaded the dependencies, so they should be okay. So we have: - .editorconfig - .pre-commit-config.yaml - scripts/install-deps.sh - scripts/ci-checks.sh - Makefile (optional) And then we set up the CI to run these scripts. We also need to set up the linters' configuration files: - pyproject.toml for black and flake8? Actually, black uses pyproject.toml, flake8 uses setup.cfg or .flake8? We can use a single configuration file for both? We can have a `pyproject.toml` that configures black and flake8? Yes, flake8 can be configured in pyproject.toml. - .prettierrc for prettier - .eslintrc.js for eslint - .golangci.yml for go vet? Actually, go vet doesn't have a configuration file? We can use a golangci.yml if we use golangci-lint? But we are using go vet, which is built-in. So no config. - For Rust, we have Cargo.toml and a rust-toolchain.toml for the toolchain? And a clippy.toml for clippy configuration. So we set up these configuration files to have consistent settings (like line length, etc.). For example, in pyproject.toml for black: [tool.black] line-length = 88 And in .prettierrc: { "printWidth": 88 } And in .eslintrc.js, we might set the max-len rule to 88. But note: not all linters have a line length setting? We try to set the same line length across. We also set the same indentation (spaces, 4) for all except Go (tabs). This is a lot of work, but it's for consistency. Given the time, we have outlined the structure. Let's summarize the steps for the user: 1. Create .editorconfig with common settings and per-language overrides. 2. Create .pre-commit-config.yaml with the hooks as described. 3. Create configuration files for each formatter/linter (pyproject.toml, .prettierrc, etc.) with consistent settings. 4. Create scripts/install-deps.sh and scripts/ci-checks.sh (or a Makefile) for CI. 5. Set up CI to run the install and ci-checks scripts. We also need to document this in CONTRIBUTING.md. This is a complex setup, but it's for a polyglot repo. We might want to start with a subset of languages and then add more. Given the constraints, we have provided a design. Let me know if you have any questions.</think>I'll help you design a comprehensive polyglot repository toolchain. Here's a complete setup for managing multiple languages with shared configs and consistency:
📁 Repository Structure
repo/ ├── .config/ # Shared configuration │ ├── editorconfig # Shared editor settings │ ├── gitattributes # Line endings, merge drivers │ └── common-lint-rules.yaml # Shared lint principles ├── .github/ │ └── workflows/ │ ├── ci.yml # Main CI pipeline │ ├── codespell.yml # Spell checking │ └── security.yml # Security scanning ├── tools/ │ ├── check-consistency.py # Cross-language consistency checks │ ├── generate-stubs.py # Generate config stubs │ └── lint-summary.py # Aggregate lint results ├── .pre-commit-config.yaml # Pre-commit hooks (language-specific) ├── .typos.toml # Typo detection config ├── pyproject.toml # Python base config (extended by subdirs) ├── package.json # JS/TS base config ├── Makefile # Top-level commands ├── .dockerignore ├── .gitignore └── README.md🔧 1. Shared Configuration (
.config/).config/editorconfigroot = true [*] charset = utf-8 end_of_line = lf insert_final_newline = true trim_trailing_whitespace = true indent_style = space indent_size = 4 max_line_length = 100 [*.{js,jsx,ts,tsx,json,yaml,yml,toml}] indent_size = 2 [*.go] indent_style = tab [*.md] trim_trailing_whitespace = false max_line_length = 80 [Makefile] indent_style = tab.config/common-lint-rules.yaml# Principles all languages should follow: no_todo_comments: true no_debug_code: true no_hardcoded_secrets: true max_file_size_kb: 500 consistent_license_header: "Copyright © 2024" no_trailing_commas: false # Allow where language permits🔄 2. Pre-commit Hooks (
.pre-commit-config.yaml)repos: # Universal hooks - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.5.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - id: check-yaml - id: check-added-large-files args: ['--maxkb=500'] - id: check-case-conflict - id: check-merge-conflict - id: check-executables-have-shebangs - id: check-shebang-scripts-are-executable - id: destroyed-symlinks - id: detect-private-key - id: forbid-new-submodules # Typo detection (all text) - repo: https://github.com/codespell-project/codespell rev: v2.2.6 hooks: - id: codespell args: [--config=.config/typos.toml] # License headers - repo: https://github.com/hadialqattan/pycln rev: v0.1.29 # Example, replace with license header tool hooks: - id: license-header # Custom hook for consistent headers # Language-specific - repo: https://github.com/astral-sh/ruff-pre-commit rev: v0.1.9 hooks: - id: ruff args: [--fix, --exit-non-zero-on-fix] - id: ruff-format - repo: https://github.com/pre-commit/mirrors-prettier rev: v3.2.5 hooks: - id: prettier files: \.(js|jsx|ts|tsx|json|yaml|yml|md|graphql)$ - repo: https://github.com/dnephin/pre-commit-golang rev: v0.5.1 hooks: - id: gofmt - id: go vet - id: golangci-lint args: [--fast] - repo: https://github.com/compilerla/conventional-pre-commit rev: v2.2.0 hooks: - id: conventional-pre-commit stages: [commit-msg] # Security scanning - repo: https://github.com/gitleaks/gitleaks rev: v8.18.0 hooks: - id: gitleaks # Container/ Docker - repo: https://github.com/hadolint/hadolint rev: v2.12.0 hooks: - id: hadolint files: Dockerfile # Terraform/Infra - repo: https://github.com/antonbabenko/pre-commit-terraform rev: v1.81.0 hooks: - hooks: - id: terraform_fmt - id: terraform_validate - id: terraform_tflint🐍 3. Python Configuration
pyproject.toml(base config)[project] name = "polyglot-repo" version = "0.1.0" requires-python = ">=3.9" [build-system] requires = ["setuptools>=61.0"] build-backend = "setuptools.build_meta" [tool.ruff] line-length = 100 target-version = "py39" select = ["E", "F", "W", "I", "N", "UP", "YTT", "BLE", "A", "C4", "DTZ", "T10", "EM", "ISC", "ICN", "G", "PT", "Q", "SIM", "TID", "ARG", "PTH", "PD", "PL", "TRY", "RUF"] ignore = ["E501"] # Let black handle line length exclude = [".git", "__pycache__", "build", "dist"] [tool.ruff.format] quote-style = "double" indent-style = "space" skip-magic-trailing-comma = false [tool.mypy] python_version = "3.9" warn_return_any = true warn_unused_configs = true disallow_untyped_defs = true disallow_incomplete_defs = true check_untyped_defs = true warn_redundant_casts = true warn_unused_ignores = true warn_no_return = true warn_unreachable = true [tool.pytest.ini_options] testpaths = ["tests"] python_files = ["test_*.py"] python_classes = ["Test*"] python_functions = ["test_*"] addopts = "-v --cov --cov-report=xml --cov-report=html"📦 4. JavaScript/TypeScript Configuration
package.json(root){ "name": "polyglot-repo", "private": true, "workspaces": ["packages/*", "apps/*"], "scripts": { "lint": "eslint . --ext .js,.jsx,.ts,.tsx", "format": "prettier --write \"**/*.{js,jsx,ts,tsx,json,yaml,yml,md}\"", "format:check": "prettier --check \"**/*.{js,jsx,ts,tsx,json,yaml,yml,md}\"", "type-check": "tsc --noEmit" }, "devDependencies": { "typescript": "^5.3.0", "eslint": "^8.55.0", "prettier": "^3.1.0", "@typescript-eslint/eslint-plugin": "^6.13.0", "@typescript-eslint/parser": "^6.13.0", "eslint-config-prettier": "^9.1.0", "eslint-plugin-import": "^2.29.0", "eslint-plugin-node": "^11.1.0" } }.eslintrc.jsmodule.exports = { root: true, parser: '@typescript-eslint/parser', plugins: ['@typescript-eslint'], extends: [ 'eslint:recommended', 'plugin:@typescript-eslint/recommended', 'prettier' ], rules: { 'max-len': ['error', { code: 100 }], 'no-console': 'warn', 'no-debugger': 'error', '@typescript-eslint/explicit-function-return-type': 'off' }, overrides: [ { files: ['**/*.test.ts', '**/*.spec.ts'], rules: { 'no-console': 'off' } } ] };🔧 5. Go Configuration
.golangci.yml(in go/ directory)run: timeout: 5m go: "1.21" modules-download-mode: readonly linters: enable: - errcheck - gosimple - govet - ineffassign - staticcheck - typecheck - unused - gofmt - goimports linters-settings: gofmt: simplify: true goimports: local-prefixes: github.com/your-org/your-repo issues: exclude-rules: - path: _test\.go linters: - errcheck🦀 6. Rust Configuration
rust-toolchain.toml[toolchain] channel = "stable" components = ["rustfmt", "clippy"] [profile.release] lto = true codegen-units = 1Cargo.toml(workspace root)[workspace] members = ["crates/*"] resolver = "2" [workspace.package] version = "0.1.0" edition = "2021" license = "MIT".cargo/config.toml[format] code_width = 100 chain_width = 80 [build] jobs = 4 # Adjust based on CI resources [profile.dev] opt-level = 0 debug = true [profile.release] opt-level = 3 lto = "thin"🚀 7. Makefile (Top-level commands)
.PHONY: help install fmt lint test check-all clean help: @echo "Polyglot Repository Toolchain" @echo "==============================" @echo "Available commands:" @echo " make install Install all dependencies" @echo " make fmt Format all code" @echo " make lint Run all linters" @echo " make test Run all tests" @echo " make check-all Run all checks (fmt+lint+test)" @echo " make clean Clean build artifacts" install: @echo "Installing dependencies..." # Python if [ -f requirements.txt ]; then pip install -r requirements.txt; fi if [ -f pyproject.toml ]; then pip install -e .; fi # JavaScript if [ -f package.json ]; then npm ci; fi # Go (assumes go installed) if [ -f go.mod ]; then go mod download; fi # Rust (assumes cargo installed) if [ -f Cargo.toml ]; then cargo fetch; fi fmt: @echo "Formatting code..." # Pre-commit will handle most, but explicit commands: if [ -f pyproject.toml ]; then ruff format .; fi if [ -f package.json ]; then npm run format; fi if [ -f go.mod ]; then gofmt -w . && goimports -w .; fi if [ -f Cargo.toml ]; then cargo fmt --all; fi lint: @echo "Linting code..." # Pre-commit handles individual file lints, this is full repo if [ -f pyproject.toml ]; then ruff check .; fi if [ -f package.json ]; then npm run lint; fi if [ -f go.mod ]; then golangci-lint run; fi if [ -f Cargo.toml ]; then cargo clippy --all-targets --all-features -- -D warnings; fi test: @echo "Running tests..." if [ -d tests ] || [ -f pytest.ini ]; then pytest; fi if [ -f package.json ]; then npm test; fi if [ -f go.mod ]; then go test ./...; fi if [ -f Cargo.toml ]; then cargo test --all; fi check-all: fmt lint test @echo "All checks passed!" clean: @echo "Cleaning..." rm -rf dist/ build/ .pytest_cache/ .coverage htmlcov/ target/ .venv/ find . -name "*.pyc" -delete find . -name "__pycache__" -exec rm -rf {} +📊 8. Consistency Checks Script (
tools/check-consistency.py)#!/usr/bin/env python3 """ Cross-language consistency checks for polyglot repository. """ import os import re from pathlib import Path from typing import List, Dict, Set class ConsistencyChecker: def __init__(self, repo_root: Path): self.repo_root = repo_root self.issues = [] def check_license_headers(self): """Ensure all files have consistent license headers.""" license_patterns = { '.py': r'# Copyright © \d{4}', '.js': r'// Copyright © \d{4}', '.go': r'// Copyright © \d{4}', '.rs': r'// Copyright © \d{4}', } for ext, pattern in license_patterns.items(): for file in self.repo_root.rglob(f'*{ext}'): if self._should_skip(file): continue content = file.read_text() if not re.search(pattern, content): self.issues.append(f"Missing license header in {file}") def check_line_endings(self): """Check for CRLF line endings.""" for file in self.repo_root.rglob('*'): if file.is_file() and self._is_text_file(file): content = file.read_bytes() if b'\r\n' in content: self.issues.append(f"CRLF line endings in {file}") def check_todo_comments(self): """Find TODO/FIXME comments (configurable).""" for file in self.repo_root.rglob('*'): if file.is_file() and self._is_text_file(file): content = file.read_text() todos = re.findall(r'(TODO|FIXME|XXX):\s*(.+)', content) for todo_type, message in todos: self.issues.append(f"{file}: {todo_type}: {message}") def check_file_sizes(self, max_kb: int = 500): """Check for excessively large files.""" for file in self.repo_root.rglob('*'): if file.is_file(): size_kb = file.stat().st_size / 1024 if size_kb > max_kb: self.issues.append(f"Large file ({size_kb:.1f}KB): {file}") def _should_skip(self, path: Path) -> bool: skip_dirs = {'.git', 'node_modules', '__pycache__', 'target', 'dist', 'build', '.venv'} return any(part in skip_dirs for part in path.parts) def _is_text_file(self, path: Path) -> bool: """Simple heuristic for text files.""" try: with open(path, 'rb') as f: chunk = f.read(1024) return b'\0' not in chunk except: return False def run_all_checks(self): """Run all consistency checks.""" self.check_license_headers() self.check_line_endings() self.check_todo_comments() self.check_file_sizes() if self.issues: print("Consistency issues found:") for issue in self.issues: print(f" ❌ {issue}") return False else: print("✅ All consistency checks passed!") return True if __name__ == "__main__": checker = ConsistencyChecker(Path.cwd()) success = checker.run_all_checks() exit(0 if success else 1)🔄 9. CI/CD Pipeline (
.github/workflows/ci.yml)name: Polyglot CI on: push: branches: [main, develop] pull_request: branches: [main] jobs: consistency-check: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v4 with: python-version: '3.11' - name: Install Python deps run: | pip install --upgrade pip pip install -r requirements.txt 2>/dev/null || true - name: Run consistency checks run: python tools/check-consistency.py pre-commit: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v4 - name: Install pre-commit run: pip install pre-commit - name: Run pre-commit run: pre-commit run --all-files language-specific: needs: [consistency-check, pre-commit] strategy: matrix: language: [python, javascript, go, rust] runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up Python if: matrix.language == 'python' uses: actions/setup-python@v4 with: python-version: '3.11' - name: Set up Node.js if: matrix.language == 'javascript' uses: actions/setup-node@v4 with: node-version: '20' - name: Set up Go if: matrix.language == 'go' uses: actions/setup-go@v4 with: go-version: '1.21' - name: Set up Rust if: matrix.language == 'rust' uses: dtolnay/rust-toolchain@stable - name: Install dependencies run: | make install shell: bash - name: Run linters run: | make lint shell: bash - name: Run tests run: | make test shell: bash env: PYTHONPATH: ${{ github.workspace }}📝 10. Developer Workflow
Setup (once per clone):
# 1. Install pre-commit hooks pre-commit install # 2. Install all dependencies make install # 3. (Optional) Set up IDE with shared configs # VS Code: Install settings sync with repo .vscode/ # IntelliJ: Import code style from .idea/codeStyles/Daily Development:
# Make changes, then: git add . git commit # pre-commit runs automatically # Or manually run everything: make check-all # Check specific language only: make lint # or use individual tools🎯 Key Principles:
- Shared Configs:
.config/contains language-agnostic rules - Progressive Enforcement: Pre-commit (fast) + CI (comprehensive)
- Toolchain Isolation: Each language uses its native tools
- Fail Fast: Early detection of issues
- Consistency Over Flexibility: Opinionated defaults with documented exceptions
- Documentation:
README.mdexplains the polyglot approach
📦 Package Management:
- Python:
requirements.txt+pyproject.toml(PEP 517/518) - JavaScript:
package.jsonworkspaces (monorepo support) - Go:
go.modat language root - Rust:
Cargo.tomlworkspace
🔐 Security Additions (
.github/workflows/security.yml):name: Security Scan on: schedule: - cron: '0 0 * * 0' # Weekly push: branches: [main] jobs: trivy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Run Trivy vulnerability scanner uses: aquasecurity/trivy-action@master with: scan-type: 'fs' format: 'sarif' output: 'trivy-results.sarif' - name: Upload Trivy scan results to GitHub Security tab uses: github/codeql-action/upload-sarif@v2 with: sarif_file: 'trivy-results.sarif'📈 Metrics & Monitoring:
Add
tools/lint-summary.pyto generate consolidated reports:#!/usr/bin/env python3 """Aggregate lint results from all languages into a single report.""" # Implementation: Parse all linter outputs (JSON/SARIF) and generate markdown/HTML reportThis setup provides:
- ✅ Consistent code style across languages
- ✅ Early feedback via pre-commit
- ✅ Comprehensive CI checks
- ✅ Language-native tools (no Lowest Common Denominator)
- ✅ Easy onboarding (
make install) - ✅ Scalable for new languages
- ✅ Security integrated
- ✅ Developer-friendly with Makefile shortcuts
Would you like me to elaborate on any specific aspect, such as setting up a specific language, configuring a particular tool, or adapting this for your specific repository structure?
- Use a top-level configuration directory (e.g.,
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.