Skip to content
The Chat Archive
o3-2025-04-16· OpenAI LMArena Human Preference 140K

how do I set up a github action that runs my testing on every pull request and gives me…

  1. User

    how do I set up a github action that runs my testing on every pull request and gives me the information on what tests passed and what ones didn't?

  2. Assistant

    Below is a starter recipe you can drop into .github/workflows/ci.yml (or any filename you like) that will

    • trigger on every pull-request (and push)
    • run your test suite
    • fail the check if any test fails
    • attach a nice “Tests” section to the PR that lists which tests passed / failed

    I’ll show a Python + pytest flavour first, then call out what to change for other stacks (Node, Java, etc.).

    # .github/workflows/ci.yml
    name: CI
    
    on:
      pull_request:                     # every PR
      push:                             # (optional) still run on main pushes
        branches:
          - main
    
    jobs:
      tests:
        runs-on: ubuntu-latest
    
        steps:
        # 1) Get your code
        - uses: actions/checkout@v4
    
        # 2) Set up language/runtime (Python example)
        - uses: actions/setup-python@v5
          with:
            python-version: "3.11"
    
        # 3) Install dependencies
        - name: Install dependencies
          run: |
            python -m pip install --upgrade pip
            pip install -r requirements.txt
            # Or poetry install / pipenv / npm ci / etc.
    
        # 4) Run tests and export JUnit XML
        - name: Run pytest
          run: |
            pytest -q \
              --junitxml=test-results/pytest.xml \
              --color=yes
    
        # 5) Upload raw test results as an artifact (handy for download)
        - name: Upload raw test results
          uses: actions/upload-artifact@v4
          with:
            name: junit-xml
            path: test-results/pytest.xml
    
        # 6) Convert the JUnit file to an inline PR check summary
        #    dorny/test-summary digs into the XML and comments directly
        - name: Publish Test Summary
          uses: dorny/test-summary@v2
          with:
            name: Unit Tests                # header shown in PR check
            path: test-results/pytest.xml   # glob is okay too
            reporter: junit                 # format of the file
    

    What you’ll see in the Pull Request

    1. “CI / tests” check will appear and go red or green.
    2. Click “Details” → you get the raw log (stdout/stderr).
    3. Right in the Checks tab you’ll also get a little table:
    Unit Tests
    ✔ test_math.py::test_add
    ✘ test_math.py::test_div_by_zero
    

    Handy extras

    • Fast feedback: Add strategy: { fail-fast: true }.
    • Parallel jobs: Use matrix: to test against multiple Python / Node versions.
    • Code coverage badges: append pip install coverage + pytest --cov=.<pkg> then upload to Codecov or Coveralls.
    • GitHub’s fancy annotations: Jest and Go automatically emit them; for pytest use pytest-github-actions-annotate-failures plug-in.

    Need a Node (Jest) or Java (Maven, Gradle) example?

    Swap out the language-setup & test lines:

        # Node
        - uses: actions/setup-node@v4
          with: { node-version: "20" }
        - run: npm ci
        - run: |
            npm test -- \
              --ci --reporters=default \
              --reporters=jest-junit
        - uses: actions/upload-artifact@v4
          with:
            name: junit-xml
            path: junit.xml
    
        # Java / Gradle
        - uses: actions/setup-java@v4
          with:
            distribution: "temurin"
            java-version: "21"
        - run: ./gradlew test
        - uses: actions/upload-artifact@v4
          with:
            name: junit-xml
            path: build/test-results/test/  # Gradle’s default
    

    That’s all! Drop the file in your repo, commit, open a PR, and you’ll instantly see test results lined up in your Checks tab. Happy shipping 🚀

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.

Identical first prompt