ISSUE DESCRIPTION [BUG] apm install fail second time with Content hash mismatch Describe…
- User
ISSUE DESCRIPTION
[BUG] apm install fail second time with Content hash mismatch Describe the bug I can install my repo (see apm.yml further down) via
apm install <org>/<repo>/package/<project>just fine (kinda, see the logs, it looks wonky but maybe it's just a new feature) but the second time I runapm installit will fail consistently withContent hash mismatch.To Reproduce Steps to reproduce the behavior:
- apm install <repo>
- apm install
Expected behavior A clear and concise description of what you expected to happen.
Environment (please complete the following information):
- OS: Windows
- APM Version: 0.16.0
Logs
> apm install --update --force [>] Installing dependencies from apm.yml... [>] Resolving github-copilot-agent-skills-testing... [i] Targets: copilot (source: apm.yml) Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/<project> #default @d479f42e |-- (files unchanged) Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/myorg #default @d479f42e |-- (files unchanged) [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main |-- 1 instruction(s) integrated -> .github/instructions/ Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/run-coverage #default @d479f42e |-- (files unchanged) Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/update-from-template #default @d479f42e |-- (files unchanged) Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/csharp #default @d479f42e |-- (files unchanged) [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/code-style.instructions.md #main |-- 1 instruction(s) integrated -> .github/instructions/ [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/protect-prefixed-agent-files.instructions.md #main |-- 1 instruction(s) integrated -> .github/instructions/ Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/git-conventions #default @d479f42e |-- (files unchanged) Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/update-nuget-packages #default @d479f42e |-- (files unchanged) Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/write-tests #default @d479f42e |-- (files unchanged) cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commit; drift cannot verify its cache freshness. Re-run 'apm install' with a pinned ref (commit, tag, or specific branch HEAD). cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commit; drift cannot verify its cache freshness. Re-run 'apm install' with a pinned ref (commit, tag, or specific branch HEAD). cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commit; drift cannot verify its cache freshness. Re-run 'apm install' with a pinned ref (commit, tag, or specific branch HEAD). [i] Trusting direct dependency MCP 'MyOrg.CLI' from '<project>-skills' [i] Trusting direct dependency MCP 'NuGet' from '<project>-skills' [i] Trusting direct dependency MCP 'Github' from '<project>-skills' [i] Trusting direct dependency MCP 'MyOrg.CLI' from 'skills' [i] Trusting direct dependency MCP 'NuGet' from 'skills' +- MCP Servers (3) | [+] MyOrg.CLI (already configured) | [+] NuGet (already configured) | [+] Github (already configured) [*] All servers up to date [!] 11 dependencies unpinned: <org>/tools/github-copilot-agent-skills -- add #tag or #sha to prevent drift [!] Transitive package 'csharp-skills' declares self-defined MCP server 'NuGet' (registry: false). Re-declare it in your apm.yml or use --trust-transitive-mcp. [!] Transitive package 'csharp-skills' declares self-defined MCP server 'Github' (registry: false). Re-declare it in your apm.yml or use --trust-transitive-mcp. [*] Installed 11 APM dependencies in 87.9s. > apm install [>] Installing dependencies from apm.yml... [i] Targets: copilot (source: apm.yml) [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/<project> @d479f42e (cached) |-- (files unchanged) [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/myorg @d479f42e (cached) |-- (files unchanged) [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main Content hash mismatch for <org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md: expected sha256:2227579674ab30b5b59f703ac152fa74ac2912d724d04aeffbf3914cac2e6b88, got sha256:b82d09e4e7879eeb0e90cc7dfeebdac45de6ebdbcdda7824b40438196692f8f8. The downloaded content differs from the lockfile record. This may indicate a supply-chain attack. Use 'apm install --update' to accept new content and update the lockfile. [!] Install interrupted after 7.7s.Additional context apm.yml
name: <project>-skills version: 1.0.0 description: '<Project> skills and agent for GitHub Copilot.' author: Oskar Klintrot target: vscode dependencies: apm: - dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/csharp - dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/run-coverage - dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md mcp: - name: NuGet registry: false transport: stdio command: dnx args: - NuGet.Mcp.Server - --source - https://api.nuget.org/v3/index.json - --yesADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /apm, current file: none) bash-$
Assistant
Let me analyze the issue:
- The user runs
apm install --update --forcewhich works fine - The second time they run
apm install, it fails with "Content hash mismatch" - The error says: "Content hash mismatch for .../testing.instructions.md: expected sha256:..., got sha256:... The downloaded content differs from the lockfile record."
This suggests that there's a bug where the hash recorded in the lockfile during the first install doesn't match what's computed during the second install. The hash mismatch occurs on a
.instructions.mdfile.Let me explore the repository to understand the structure and find the relevant code that handles content hashing and lockfile operations.
Let me start by exploring the repository structure.
- The user runs
Assistant
Let me explore the source code to understand the structure of the project and find relevant files for content hash handling.
Assistant
Let me explore the apm_cli directory structure.
Assistant
Let me look for files related to content hash, lockfile, and caching.
Assistant
Let me look at both these files to understand the content hash logic
Assistant
The open command might need different syntax. Let me try opening it directly with the
openspecial command.Assistant
Let me find the "Content hash mismatch" area in this file.
Assistant
Let me look at the context around line 777.
Assistant
Now let me find the
_compute_hashfunction and understand how hashes areAssistant
Let me look at the
compute_package_hashfunction incontent_hash.py.Assistant
Let me open that file.
Assistant
Let me also look at the integrate.py file that has "Content hash mismatch".
Assistant
Let me find the "Content hash mismatch" line in integrate.py.
Assistant
Let me look at the context around line 197 in integrate.py.
Assistant
Let me look at the instruction integrator to understand how instructions files are handled.
Assistant
Let me understand how single-file dependencies like
.instructions.mdfiles are installed. Let me look at the download and install path logicAssistant
Let me look at how dependencies are resolved and where they're installed. Let me check the model for how deps are structured
Assistant
Let me look at the cache_pin module.
Assistant
Let me open cache_pin.py to understand how the marker file works
Assistant
Let me look at how the install path is determined for dependencies and the download mechanism
Assistant
Let me look at the
get_install_pathfunction in the dependency reference modelAssistant
Let me look at
is_virtual_subdirectory()andget_virtual_package_name()Assistant
Let me look at the relevant methods
Assistant
Let me look more carefully at the FreshDependencySource class and how the hash is computed and
Assistant
Let me look at lines 730-790 where the hash is computed and verified
Assistant
Let me look at how the lockfile is built and how content_hash is stored
Assistant
Let me look at how content_hash is set in the LockfileBuilder
Assistant
Let me look at the CachedDependencySource to see how it computes the hash
Assistant
Let me look at the CachedDependencySource.acquire() method
Assistant
Let me continue reading the CachedDependencySource.acquire()
Assistant
Let me look at how virtual files are downloaded
Assistant
Let me look at
download_virtual_file_packagein github_downloader.pyAssistant
Let me continue reading `download_virtual_file_package
Assistant
Let me look at the lockfile writing phase
Assistant
Let me look at the
_attach_content_hashesmethodAssistant
Let me look at how the integration template works, specifically whether it modifies the package install path
Assistant
Let me look at the full FreshDependencySource.acquire() from the beginning
Assistant
Let me look at the
download_raw_filefunctionAssistant
Let me look at the
download_raw_filemethodAssistant
Let me look at the download strategies for ADO
Assistant
Let me look at the ADO file download
Assistant
Let me look at the
yaml_to_strfunctionAssistant
Let me look at the yaml_to_str
Assistant
Let me look at the lockfile_match logic in integrate.py more carefully
Assistant
Let me look at
_should_skip_redownloadAssistant
Let me look at BaseIntegrator to see if it modifies anything in the install path.
Assistant
Let me look at
find_files_by_globAssistant
Let me look at `find_files_by_glob
Assistant
Let me check how
--forceaffects the install flowAssistant
Let me search for tests related to content hash and virtual file packages
Assistant
Let me look at the test for virtual file package download
Assistant
Let me check the VIRTUAL_FILE_EXTENSIONS constant
Assistant
Let me look at the VIRTUAL_FILE_EXTENSIONS constant
Assistant
Let me check how
--updateaffects the flowAssistant
Let me look at how instruction integration works more carefully, specifically what it does with the package path
Assistant
Let me now look at the full integration template flow to see if anything writes to the install_path after hash computation
Assistant
Let me look at the LockedDependency class
Assistant
Let me look at recent git changes related to content hash or virtual file packages
Assistant
Let me look at
sync_markers_for_lockfileAssistant
Let me look at
sync_markers_for_lockfileAssistant
Let me look at the ADO API URL building
Assistant
Let me look at the ADO API URL builder
Assistant
Let me look at
download_packagein github_downloader.py where it decides between virtual file and normal downloadAssistant
Let me look at
build_download_reffunctionAssistant
Let me look at
build_download_refAssistant
Let me look at
detect_ref_changeAssistant
Let me look at
detect_ref_changeAssistant
Let me create a minimal reproduction script to understand the hash computation
Assistant
Let me use the venv to run the script
Assistant
Let me look at
LockedDependency.from_dict()Assistant
Let me look at the existing lockfile to understand its format
Assistant
Let me look at the git log for any recent changes related to content hash or virtual file packages
Assistant
Let me search for any TODO or FIXME comments related to content hash in the codebase
Assistant
Let me look at
to_dependency_refin LockedDependencyAssistant
Let me look at `to_dependency_ref
Assistant
Let me verify that yaml.safe_dump is deterministic
Assistant
Let me look at `resolve_commit_sha_for_ref
Assistant
Let me look at `resolve_commit_sha_for
Assistant
Let me look at the InstallContext to understand how
--forceand--updateaffect thingsAssistant
Let me look at
_resilient_getto see if there's anything non-deterministicAssistant
Let me look at
_resilient_getAssistant
Let me look at
resilient_getin download_strategies.pyAssistant
Let me check the git log for changes to relevant
Assistant
Let me check if
--forcecleans the install path before downloadAssistant
Let me search for tests about content hash mismatch
Assistant
Let me look at the diff_aware_install tests to understand the expected behavior
Assistant
Let me write a script to simulate the exact flow and see if I can reproduce the hash mismatch
Assistant
Let me look at how DependencyReference is parsed for ADO URLs to check if host is included in repo_url
Assistant
Let me look at how
InstalledPackage→LockedDependencyconversion worksAssistant
Let me look at
from_installed_packagesAssistant
Let me look at `from_installed_packages
Assistant
Let me look at
LockedDependency.from_dependency_refAssistant
Let me look at
from_dependency_refAssistant
Let me see the rest of
from_dependency_refAssistant
Let me check if
forceaffects the download or hash computationAssistant
Let me look at the integrate loop to see how dep_key is determined
Assistant
Let me look at `_dataclass_replace
Assistant
Let me check if DependencyReference is a dataclass that supports `
Assistant
Let me search for any commits or PRs related to this issue
Assistant
Let me check if the resolve phase pre-downloads virtual file packages
Assistant
Let me check whether
--forcecauses the install path to be cleanedAssistant
Now I have a good understanding of the issue. Let me look at the
_resolve_download_strategycode path more carefully to understand the flow for virtual file packages from ADO whereresolved_commitis None.The key issue I see is in
src/apm_cli/install/phases/integrate.py. For Azure DevOps virtual file packages,_resolve_commit_sha_for_refreturnsNone(as seen insrc/apm_cli/deps/git_reference_resolver.pylines 239-240). This meansresolved_commitis alwaysNonefor ADO virtual file deps.In
_resolve_download_strategy, thelockfile_matchcheck at line 115 requireslocked_dep.resolved_committo be non-None. Since it's None for ADO virtual files, the entire cache-hit path is skipped. There's a fallback for registry deps (line 151-160) but not for git-source deps with missingresolved_commit.Let me look at the exact lines to craft a fix:
Assistant
Now I understand the issue. Let me look at the
_resolve_download_strategyfunction more carefully. For Azure DevOps virtual file packages:_resolve_commit_sha_for_refingit_reference_resolver.py(lines 239-240) returnsNonefor ADO deps- This means
resolved_commitis alwaysNonefor ADO virtual file packages - In the lockfile,
resolved_commitis thereforeNone - In
_resolve_download_strategy, the check at line 115 requireslocked_dep.resolved_committo be truthy, which fails - The
elifat line 151 only handlessource == "registry", not git-source deps - So
lockfile_matchstaysFalse,skip_downloadisFalse, and the dep is always re-downloaded
The fix should add a fallback for deps that have
content_hashbut noresolved_commit(regardless of source), mirroring the registry path. Additionally, there's a scenario where the_resolve_download_strategydoesn't check_should_skip_redownloadwhenresolved_commitis missing for non-registry deps. This means even if the content hasn't changed, the dep goes throughFreshDependencySource, where the hash verification can fail.Let me add the fallback:
Assistant
Now let me also look at
download_virtual_file_packageto see if it cleans the install directory before creating the virtual package. If old files remain, the hash could differAssistant
I see another potential issue:
download_virtual_file_packagedoesn't clean the target directory before creating the virtual package. If there are leftover files from a previous version, they would be included in the hash. Let me add cleanup:Assistant
Let me check the
safe_rmtreefunction to make sure it can handle the target_path properlyAssistant
Let me look at
safe_rmtreeAssistant
Now let me run the relevant tests to make sure my changes don't break anything
Assistant
Let me run more tests related to the integrate phase and content hash
Assistant
Let me run the heal chain tests and some more unit tests
Assistant
Let me run a broader set of tests that might be affected by my changes
Assistant
Let me verify my changes by looking at the git diff
Assistant
Let me run a broader test to make sure nothing is broken
Assistant
Let me run the wave4 pure logic tests
Assistant
Good, the tests pass. Let me now think about edge cases and verify the fix is complete.
Let me also check that I haven't introduced any issues with the
safe_rmtreecall by verifying the import is available at that point in the code:Let me verify the import will work correctly in that context by checking what's already imported at that point in github_downloaderAssistant
Let me run the download strategies tests to make sure the safe_rmtree change doesn't break
Assistant
Let me run the install marketplace tests to be
Assistant
Let me run the test without the filter issue
Assistant
Let me run the full install marketplace tests to verify
Assistant
All tests pass. Let me now verify my changes one more time and also check for any potential issues with the
safe_rmtreecall in thedownload_virtual_file_packagefunction.Let me think about whether the
safe_rmtreeindownload_virtual_file_packagecould cause issues in specific scenarios:- Normal flow: target_path exists, is a regular directory → delete it → recreate it → fine
- First install: target_path doesn't exist → skip cleanup → fine
- Symlink attack: target_path is a symlink →
exists()returns True →safe_rmtree→ensure_path_withinresolves and detects traversal → raises → caught as download error
Let me verify there's an exception handler:Let me verify the exception handler in FreshDependencySource.acquire() covers errors from download_virtual_file_package
Assistant
Let me do a final review of both changed
Assistant
Now let me verify the changes once more and then submit. Let me run one more quick test to make sure the whole test suite for the install phases works
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-lunaCompare side by side →
- gpt-5.6-lunaCompare side by side →