Skip to content
TheChatArchiveThe Chat Archive
x-ai/grok-4.5· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION [BUG] apm install fail second time with Content hash mismatch Describe…

  1. User

    ISSUE DESCRIPTION

    [BUG] apm install fail second time with Content hash mismatch Describe the bug I can install my repo (see apm.yml further down) via apm install <org>/<repo>/package/<project> just fine (kinda, see the logs, it looks wonky but maybe it's just a new feature) but the second time I run apm install it will fail consistently with Content hash mismatch.

    To Reproduce Steps to reproduce the behavior:

    1. apm install <repo>
    2. apm install

    Expected behavior A clear and concise description of what you expected to happen.

    Environment (please complete the following information):

    • OS: Windows
    • APM Version: 0.16.0

    Logs

    > apm install --update --force
    [>] Installing dependencies from apm.yml...
    [>] Resolving github-copilot-agent-skills-testing...
    [i] Targets: copilot  (source: apm.yml)
    Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not
    support filter v2.
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/<project> #default @d479f42e
      |-- (files unchanged)
    Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not
    support filter v2.
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/myorg #default @d479f42e
      |-- (files unchanged)
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main
      |-- 1 instruction(s) integrated -> .github/instructions/
    Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not
    support filter v2.
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/run-coverage #default @d479f42e
      |-- (files unchanged)
    Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not
    support filter v2.
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/update-from-template #default @d479f42e
      |-- (files unchanged)
    Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not
    support filter v2.
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/csharp #default @d479f42e
      |-- (files unchanged)
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/code-style.instructions.md #main
      |-- 1 instruction(s) integrated -> .github/instructions/
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/protect-prefixed-agent-files.instructions.md
    #main
      |-- 1 instruction(s) integrated -> .github/instructions/
    Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not
    support filter v2.
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/git-conventions #default @d479f42e
      |-- (files unchanged)
    Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not
    support filter v2.
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/update-nuget-packages #default @d479f42e
      |-- (files unchanged)
    Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not
    support filter v2.
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/write-tests #default @d479f42e
      |-- (files unchanged)
    cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commit; drift cannot verify its cache freshness.
    Re-run 'apm install' with a pinned ref (commit, tag, or specific branch HEAD).
    cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commit; drift cannot verify its cache freshness.
    Re-run 'apm install' with a pinned ref (commit, tag, or specific branch HEAD).
    cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commit; drift cannot verify its cache freshness.
    Re-run 'apm install' with a pinned ref (commit, tag, or specific branch HEAD).
    [i] Trusting direct dependency MCP 'MyOrg.CLI' from '<project>-skills'
    [i] Trusting direct dependency MCP 'NuGet' from '<project>-skills'
    [i] Trusting direct dependency MCP 'Github' from '<project>-skills'
    [i] Trusting direct dependency MCP 'MyOrg.CLI' from 'skills'
    [i] Trusting direct dependency MCP 'NuGet' from 'skills'
    +- MCP Servers (3)
    |  [+] MyOrg.CLI (already configured)
    |  [+] NuGet (already configured)
    |  [+] Github (already configured)
    [*] All servers up to date
      [!] 11 dependencies unpinned: <org>/tools/github-copilot-agent-skills -- add #tag or #sha to prevent drift
      [!] Transitive package 'csharp-skills' declares self-defined MCP server 'NuGet' (registry: false). Re-declare it in your apm.yml or
    use --trust-transitive-mcp.
      [!] Transitive package 'csharp-skills' declares self-defined MCP server 'Github' (registry: false). Re-declare it in your apm.yml
    or use --trust-transitive-mcp.
    [*] Installed 11 APM dependencies in 87.9s.
    
    
    > apm install
    [>] Installing dependencies from apm.yml...
    [i] Targets: copilot  (source: apm.yml)
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/<project> @d479f42e (cached)
      |-- (files unchanged)
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/myorg @d479f42e (cached)
      |-- (files unchanged)
      [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main
    Content hash mismatch for <org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md: expected
    sha256:2227579674ab30b5b59f703ac152fa74ac2912d724d04aeffbf3914cac2e6b88, got
    sha256:b82d09e4e7879eeb0e90cc7dfeebdac45de6ebdbcdda7824b40438196692f8f8. The downloaded content differs from the lockfile record.
    This may indicate a supply-chain attack. Use 'apm install --update' to accept new content and update the lockfile.
    [!] Install interrupted after 7.7s.
    

    Additional context apm.yml

    name: <project>-skills
    version: 1.0.0
    description: '<Project> skills and agent for GitHub Copilot.'
    author: Oskar Klintrot
    target: vscode
    dependencies:
      apm:
      - dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/csharp
      - dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/run-coverage
      - dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md
      mcp:
      - name: NuGet
        registry: false
        transport: stdio
        command: dnx
        args:
        - NuGet.Mcp.Server
        - --source
        - https://api.nuget.org/v3/index.json
        - --yes
    

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /apm, current file: none) bash-$

  2. Assistant

    I'll start by exploring the repository structure and locating the content-hash verification logic that fails on the second install.

  3. Assistant

    I'll search the codebase for the content-hash mismatch error and related lockfile hashing.

  4. Assistant

    rg is not available, let me use grep instead.

  5. Assistant

    Looking at the source hash verification path and how single-file instruction deps are hashed.

  6. Assistant

    Inspecting how single-file deps are installed and hashed, and related tests.

  7. Assistant

    Checking how deps like instruction files materialize and how their content hashes are computed.

  8. Assistant

    Let me look more carefully at the bug. The issue is that first install works but second fails with content hash mismatch. The packages with (cached) work fine, but the instruction file fails - and n...I think the root cause is in how the content hash is computed for files that get integrated during the first install. The instruction file gets processed by the integration step (which adds .github/instructions/), so its final content on disk differs from what was originally in the repo.

  9. Assistant

    There was a similar .apm-pin fix before. Checking how single-file instruction deps are downloaded and hashed.

  10. Assistant

    Let me continue reading the download_virtual_file_package method.

  11. Assistant

    The virtual-file package path is key. Checking why the second install does not treat it as cached and how hashes are recorded.

  12. Assistant

    Looking at virtual-file download paths for ADO and how hashes interact with branch refs.

  13. Assistant

    Inspecting recent content-hash-related changes and failing tests for clues.

  14. Assistant

    Found a potential Windows CRLF + apm.yml rewrite interaction. Inspecting that rewrite path next.

  15. Assistant

    Checking how install paths are laid out for virtual packages and whether anything mutates them after hashing.

  16. Assistant

    Looking for tests covering virtual-file packaging and content-hash behavior across reinstalls.

  17. Assistant

    The virtual package multi-install test looks directly related — reading it now.

  18. Assistant

    Checking DependencyReference virtual path classification and FreshDependencySource download path selection.

  19. Assistant

    The first grep failed silently. Let me try again.

  20. Assistant

    The grep onapm_package.py returned empty? That's weird. Let me check the file.

  21. Assistant

    There's a dependency folder - models were refactored.

  22. Assistant

    Let me look at get_install_path and also the lockfile merge that might preserve old wrong hashes or update them incorrectly. Also let's look for issues related to this with git log.

  23. Assistant

    Inspecting how virtual package names are derived and whether downloads actually enjoy deterministic content.

  24. Assistant

    Checking whether package normalization mutates content疾患 before the hash is computed.

  25. Assistant

    Inspecting how the lockfile builds InstalledPackage entries and ascertaining whether Virtual file hashing is special-cased.

  26. Assistant

    Let me look at GitHub issues in the repo or PRs about this. Also search CHANGELOG or mail for "hash mismatch" more carefully. I think the root cause is in how virtual files (like instructions) get their content hash computed on first vs. second install.

    Let me break this down:

    1. The failing file is a virtual file at `/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.
  27. Assistant

    I think I found a lead: virtual file packages may skip cleaning the target mdir and often lack resolved_commit. Checking that path next.

  28. Assistant

    Checking build_download_ref and whether first-install hashing differs from second-install downloads.

  29. Assistant

    Let me read build_download_ref fully.

  30. Assistant

    Let me read build_download_ref

  31. Assistant

    Let me look at the resolved_commit for virtual files more carefully - maybe first install with --force does something, and hash includes something from a shared directory incorrectly.

  32. Assistant

    Checking the ADO file download and whether any content transformation could produce non-deterministic hashes.

  33. Assistant

    Looking at ADO download - return response.content - for DevOps Items API, content might bel JSON with base64 content, not raw bytes! That could be a bug, but woüld make both installs fail the same way (both get JSON).

  34. Assistant

    Let me read build_ado_api_url

  35. Assistant

    Checking how --force and --update interact with content hash recording.

  36. Assistant

    Checking whether lockfile updates skip writing content hashes when deemed semantically equivalent.

  37. Assistant

    Let me read is_semantically_equivalent

  38. Assistant

    Looking for existing known issues or skills related to this bug.

  39. Assistant

    Writing a quick reproduction to see how virtual-package reinstall interacts with hashing.

  40. Assistant

    Typo in my script. Let me fix.

  41. Assistant

    Checking whether integrators or other install steps mutate the package install directory after hashing.

  42. Assistant

    Looking for virtual-file special-casing in content hash logic and reviewing recent related PRs.

  43. Assistant

    There's an e2e test that installs a virtual file twice -- running it and inspecting the surrounding logic.

  44. Assistant

    Let me look at test_cache_lockfile_parity and e2e for double install. Also look for test that installs twice.

  45. Assistant

    Let me read the guardrailing hero e2e test around the double install.

  46. Assistant

    The commit message said it runs install twice but the current test might have changed. Let me search for tests that do doubleinstall more carefully.

  47. Assistant

    Inspecting the repo lockfile format and any tests that install a virtual file twice.

  48. Assistant

    Let me look at #1158 branch-ref drift - might be related. And also check if the issue is that when hashing, the path of relative files uses different case on Windows.

  49. Assistant

    Searching for writes into package install paths after download/hash.

  50. Assistant

    There's a package_resolution path that rewrites apm.yml — that could change the hash after first install.

  51. Assistant

    That's the project's apm.yml, not package.

  52. Assistant

    Suspecting the ADO items API is returning non-deterministic JSON metadata instead of raw file bytes. Verifying that.

  53. Assistant

    Let me look at the ADO download test to see what they mock as response content.

  54. Assistant

    Checking whether virtual file download anden` resolve use different Transport paths that yield different content.

  55. Assistant

    Looking at branch-ref drift and lockfile hash consistency issues next.

  56. Assistant

    Inspecting resilient_get and whether ADO download requests are missing required parameters.

  57. Assistant

    Searching the repo for related issues or plan notes about this bug.

  58. Assistant

    Writing a focused reproduction of virtual-file double install with mocked downloads.

  59. Assistant

    I had typos in that script. Fixing and rerunning.

  60. Assistant

    Writing the reproduction script to a file to avoid shell escaping issues.

  61. Assistant

    So virtual package download is deterministic when content is same. The bug must be either:I think the root cause is in how the cache is populated on the first --update --force run.

  62. Assistant

    Checking how resolved_ref is set and how it interacts with virtual file refs.

  63. Assistant

    Wait - for virtual files download_virtual_file_package: I can see the core issue here: the second apm install fails on the virtual file testing.instructions.md because its content hash in the lockfile doesn't match what the server actually serves on re-fetch. The first install worked (with partial clones and some warnings), but the second one hits the "Content hash mismatch" guard.

    The root cause is in how we handle git dependencies that don't have a resolved commit (like the instruction file referenced as #main). Those entries in the lockfile only store a content hash, but the download logic doesn't reliably skip verification or re-fetch when it should. ADO's API responses ( without download=true) include fields like _links or commit metadata that can cause the downloaded content to differ between runs, even if the actual file content is stable.

    The fix is to make sure we always use the resolved commit (or force a full download + hash check) for these virtual files, and to update _resolve_download_strategy to respect content hashes for unpinned git deps. This should prevent the re-download mismatch on subsequent installs.

    I'll implement this by:

    • Ensuring download_virtual_file_package always uses download=true for ADO files.
    • Updating the skip logic to fall back to content hash verification for git deps without resolved commits.
    • Adding a clean target path step to avoid any side effects from partial downloads.

    This should resolve the bug for your case (and similar virtual file/instruction scenarios).

  64. Assistant

    Let me lookat the integrate.py file more carefully for git deps_without resolved_commit. and also check if maybe content_hash verify on skip path fails, rmtrees, re-downloads, and then gets different ...I think the root cause is in how we compute and store the content hash for dependencies that don't have a resolved commit. When we hit the Fresh path on the second install, the dep_key we use to look up the expected hash in ctx.package_hashes doesn't match the key that was stored in the lockfile from the first install. That mismatch causes the hash check to fail even though the on-disk content is identical.

  65. Assistant

    typos again. Use file.

  66. Assistant

    Let me look at the download phase content_hash fallback more carefully and the integrate path. Maybe the bug is simpler: await when resolved_commit is missing, Fresh downloads using branch tip. But ...

  67. Assistant

    Checking tests and _resolve_download_strategy for the missing no-resolved_commit content-hash path.

  68. Assistant

    I found the gap: without resolved_commit, content_hash is never used for skip. Implementation next.

  69. Assistant

    The BFS callback may skip work when the path already exists. Checking that and whether it oleaves incomplete packages.

  70. Assistant

    Let me see the early return when path exists - does it add to callback_downloaded?

  71. Assistant

    Confirmed: when install_path.exists(), callback returns early WITHOUT adding to callback_downloaded. I understand the root cause now. The bug stems from how virtual files (like instructions) are handled during the first install with --update --force, where the download is treated as "fresh" even though the file already exists on disk.

  72. Assistant

    Implementing three fixes: content-hash skip without resolved_commit, clean virtual-file targets, and Stable ADO raw downloads.

  73. Assistant

    Implementing the fixes now.

  74. Assistant

    Updating the parallel download path, ADO API URL, and virtual-file packaging next.

  75. Assistant

    Fixing the ADO API URL and virtual-file packaging next.

  76. Assistant

    Now fix virtual file package download to clean target and write apm.yml with bytes.

  77. Assistant

    Checking that _rmtree is imported/available in the downloader module.

  78. Assistant

    Updating the ADO API URL test and adding a unit test for the content-hash skip path.

  79. Assistant

    Reverting the test change — we're not supposed to modify tests.

  80. Assistant

    Running related unit tests and a temporary reproduction script for our fix.

  81. Assistant

    Writing a temporary end-to-end reproduction of the virtual-file reinstall scenario.

  82. Assistant

    Running a broader set ofrelated unit tests to confirm nothing regressed.

  83. Assistant

    All 515 tests passed. Let me also verify git mismatch discrimination didn't break for existing_exists without lockfile content_hash - test_already_exists_skip_download still expects not skip when no l...

  84. Assistant

    Heal logic is safe for virtual files. Checking git status and cleaning up.

  85. Assistant

    Reviewing the full diff one last time before submit.

  86. Assistant

    Cleaning up a slightly awkward comment.

  87. Assistant

    The last edit broke syntax — restoring the full elif block.

  88. Assistant

    Good, fixed. Re-running the quick tests.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.