ISSUE DESCRIPTION [BUG] apm install fail second time with Content hash mismatch Describe…
- User
ISSUE DESCRIPTION
[BUG] apm install fail second time with Content hash mismatch Describe the bug I can install my repo (see apm.yml further down) via
apm install <org>/<repo>/package/<project>just fine (kinda, see the logs, it looks wonky but maybe it's just a new feature) but the second time I runapm installit will fail consistently withContent hash mismatch.To Reproduce Steps to reproduce the behavior:
- apm install <repo>
- apm install
Expected behavior A clear and concise description of what you expected to happen.
Environment (please complete the following information):
- OS: Windows
- APM Version: 0.16.0
Logs
> apm install --update --force [>] Installing dependencies from apm.yml... [>] Resolving github-copilot-agent-skills-testing... [i] Targets: copilot (source: apm.yml) Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/<project> #default @d479f42e |-- (files unchanged) Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/myorg #default @d479f42e |-- (files unchanged) [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main |-- 1 instruction(s) integrated -> .github/instructions/ Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/run-coverage #default @d479f42e |-- (files unchanged) Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/update-from-template #default @d479f42e |-- (files unchanged) Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/csharp #default @d479f42e |-- (files unchanged) [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/code-style.instructions.md #main |-- 1 instruction(s) integrated -> .github/instructions/ [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/protect-prefixed-agent-files.instructions.md #main |-- 1 instruction(s) integrated -> .github/instructions/ Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/git-conventions #default @d479f42e |-- (files unchanged) Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/update-nuget-packages #default @d479f42e |-- (files unchanged) Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/write-tests #default @d479f42e |-- (files unchanged) cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commit; drift cannot verify its cache freshness. Re-run 'apm install' with a pinned ref (commit, tag, or specific branch HEAD). cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commit; drift cannot verify its cache freshness. Re-run 'apm install' with a pinned ref (commit, tag, or specific branch HEAD). cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commit; drift cannot verify its cache freshness. Re-run 'apm install' with a pinned ref (commit, tag, or specific branch HEAD). [i] Trusting direct dependency MCP 'MyOrg.CLI' from '<project>-skills' [i] Trusting direct dependency MCP 'NuGet' from '<project>-skills' [i] Trusting direct dependency MCP 'Github' from '<project>-skills' [i] Trusting direct dependency MCP 'MyOrg.CLI' from 'skills' [i] Trusting direct dependency MCP 'NuGet' from 'skills' +- MCP Servers (3) | [+] MyOrg.CLI (already configured) | [+] NuGet (already configured) | [+] Github (already configured) [*] All servers up to date [!] 11 dependencies unpinned: <org>/tools/github-copilot-agent-skills -- add #tag or #sha to prevent drift [!] Transitive package 'csharp-skills' declares self-defined MCP server 'NuGet' (registry: false). Re-declare it in your apm.yml or use --trust-transitive-mcp. [!] Transitive package 'csharp-skills' declares self-defined MCP server 'Github' (registry: false). Re-declare it in your apm.yml or use --trust-transitive-mcp. [*] Installed 11 APM dependencies in 87.9s. > apm install [>] Installing dependencies from apm.yml... [i] Targets: copilot (source: apm.yml) [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/<project> @d479f42e (cached) |-- (files unchanged) [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/myorg @d479f42e (cached) |-- (files unchanged) [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main Content hash mismatch for <org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md: expected sha256:2227579674ab30b5b59f703ac152fa74ac2912d724d04aeffbf3914cac2e6b88, got sha256:b82d09e4e7879eeb0e90cc7dfeebdac45de6ebdbcdda7824b40438196692f8f8. The downloaded content differs from the lockfile record. This may indicate a supply-chain attack. Use 'apm install --update' to accept new content and update the lockfile. [!] Install interrupted after 7.7s.Additional context apm.yml
name: <project>-skills version: 1.0.0 description: '<Project> skills and agent for GitHub Copilot.' author: Oskar Klintrot target: vscode dependencies: apm: - dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/csharp - dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/run-coverage - dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md mcp: - name: NuGet registry: false transport: stdio command: dnx args: - NuGet.Mcp.Server - --source - https://api.nuget.org/v3/index.json - --yesADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /apm, current file: none) bash-$
Assistant
I'll start by exploring the repository structure to understand the codebase and then investigate the "Content hash mismatch" bug.
Assistant
This is a Python project. Let me explore the source code structure to understand how the APM (Agent Package Manager) works, particularly around the hash mismatch issue.
Assistant
Let me search for the "Content hash mismatch" error message to find the relevant code:
Assistant
Now let me look at the
CachedDependencySourceclass more closely, since the issue happens on the second install (when it's cached):Assistant
Let me see the rest of the
CachedDependencySource.acquire()method:Assistant
Now I understand the flow. Let me look at the hash check more carefully. The issue is that when a dependency has a file (like an instruction file) pointing to
#main(a branch), the lockfile records a hash. On the second install, if the dep is marked as "cached" (skip_download=True), it reuses the cached files, BUT the hash is still computed from the install path, and compared against the lockfile.The key issue is that for files/instructions that point to a branch reference like
#main, the content might not actually be downloaded fresh, but the lockfile records a specific hash. Let me look at how theskip_downloaddecision is made:Assistant
Let me look at the
_resolve_download_strategyfunction in detail:Assistant
Now I see the issue more clearly. The
skip_downloadlogic has a content hash verification at lines 193-202 inintegrate.py. If the hash matches, it uses the cached content. But there's also a hash check insources.pyin theFreshDependencySource.acquire()method. Let me understand the flow better:- On first install: Files are downloaded, hash is computed, lockfile is saved.
- On second install: The
_resolve_download_strategydecides whether to skip download. For virtual packages (like instructions), the install_path is NOT a git repo, so it falls back to content hash verification.
Let me look at the
_should_skip_redownloadfunction:Assistant
Now let me look at the
content_hashmodule to understand how hashes are computed:Assistant
Now I see the
_EXCLUDED_ROOT_FILESincludes.apm-pin(the cache pin marker). Let me look at the cache pin module:Assistant
I see the issue now. Let me trace through what happens:
-
On first install (
apm install --update --force):- Dependencies are downloaded fresh
- The content hash is computed and stored in the lockfile
- The
.apm-pinmarker file is written
-
On second install (
apm install):_resolve_download_strategychecks if the cached content matches- For virtual packages (like instruction files), the install_path is NOT a git repo, so it falls back to content hash verification via
_should_skip_redownload - The content hash check passes (using
verify_package_hashwhich excludes.apm-pinfrom root files) skip_download = True- The
CachedDependencySource.acquire()is used - In
CachedDependencySource.acquire(), the hash is computed and stored inctx.package_hashes[dep_key]
Wait, but where does the error happen? Let me look at the error message again:
Content hash mismatch for <org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md: expected sha256:2227579674ab30b5b59f703ac152fa74ac2912d724d04aeffbf3914cac2e6b88, got sha256:b82d09e4e7879eeb0e90cc7dfeebdac45de6ebdbcdda7824b40438196692f8f8.This is a virtual package (instruction file) and the error comes from the
FreshDependencySource(since it says "The downloaded content differs from the lockfile record").Wait, let me re-read. On the second install, the instruction file dependency has
#mainas its reference (a branch), so it's NOT cacheable (is_cacheable checks for TAG or COMMIT). So it goes through thelockfile_matchpath.Since this is a virtual package, its install_path isn't a git repo, so the git-based HEAD check fails and falls back to content hash verification. The lockfile hash matches the cached content, so
skip_downloadshould be True—but that's not what's happening. The error is coming fromFreshDependencySource, which means the package is being downloaded fresh instead of using the cache.The real issue is that branch references like
#mainaren't considered cacheable in the current logic, so even though the content hash matches, the system still tries to re-download. For pinned commits or tags, this wouldn't be a problem since they'd be marked as cacheable. But virtual packages with branch refs get caught in this mismatch—they should be usable from cache when their content hasn't changed, but the cacheability check prevents it. I'm tracing through the hash comparison logic in FreshDependencySource.acquire(). The code checks if the freshly computed hash matches the lockfile hash, and if they don't align, it removes the install path and exits with an error about content hash mismatch. This is the supply-chain protection mechanism that's triggering the failure.But the real puzzle is why skip_download is False for this dependency in the first place. Looking back at the bug report, the first install used --update --force and succeeded, then the second install failed. The dependency has a branch reference (#main) rather than a pinned commit or tag, which means it wouldn't be considered cacheable and would need to be downloaded fresh.
When the second install runs, the git HEAD check can't verify the cached files since they came from a remote branch. The code then falls back to comparing content hashes instead. The lockfile has a recorded hash from the first install, so it needs to verify the cached content still matches before deciding whether to skip the download. Looking at the
_should_skip_redownloadfunction, it returnsTruewhen the content hash matches, which setslockfile_match = True. Then at line 186,skip_downloadbecomesTruesince the install path exists andlockfile_matchis satisfied. This means the code takes theCachedDependencySourcebranch rather thanFreshDependencySource. But I need to check if there's a hash validation happening withinCachedDependencySource.acquire()itself that might be causing the failure.Checking the method, it computes and stores the hash in
ctx.package_hashes[dep_key]but doesn't actually verify it against anything. The hash verification I'm seeing is inFreshDependencySource, not the cached path. Looking back at the logs, the second install shows the package marked as(cached), which indicates it went throughCachedDependencySource. Yet the error message about hash mismatch suggests the validation is happening somewhere in the template or another phase after acquisition completes.The error occurs for
testing.instructions.mdspecifically, which has a#mainbranch reference. This meansis_cacheablewould beFalsefor it. I'm wondering if there's a scenario where this dependency gets resolved through theFreshDependencySourcepath instead of the cached one.Looking at the error more carefully—it mentions the downloaded content differs from the lockfile record. But if the dependency is cached, why would content be downloaded at all? This suggests the resolver might be downloading it fresh, and then
skip_downloadlogic isn't working correctly for cached dependencies.The resolver's BFS callback could be downloading the package, which would set
skip_download=True, but the hash verification inFreshDependencySourcewould still run. For branch references pointing to#main, the HEAD might have moved since the first install, causing a hash mismatch even though the cached version matches what's in the lockfile.The real issue appears to be that
install_path.exists()returns true on the second run, so the code assumes the package is already downloaded. But for instruction files—virtual dependencies extracted from a repo—the actual content lives inside a subdirectory, and the hash verification might be checking against stale cached content that no longer matches what the branch now points to. I need to understand how virtual dependencies are processed during install, particularly whether the hash validation happens before or after content extraction from the parent repository.The core issue seems to be a mismatch: the first install computes a hash from the extracted content, but on the second install, the system might be extracting fresh content from the remote repo and comparing against the old cached hash. If the remote content changed or was extracted differently, the hashes won't match. The hash verification logic in
integrate.pycompares the cached content against the lockfile hash, and if they match, it skips re-downloading. But if the cache was invalidated or the remote changed, fresh content gets downloaded and verified against the lockfile hash—which would fail if the remote has diverged from what was originally cached. If the download fails for some reason, the cached content persists but with mismatched hashes. Looking at the error message mentioning "downloaded content," it seems the issue might stem from the integration phase after the download completes. I need to examine theintegrate.pyfile more closely to see what happens in that post-download hash verification step.<tool_call> <function=bash_terminal> <parameter=command> sed -n '1,50p' /apm/src/apm_cli/install/phases/integrate.py </tool_call>
-
Assistant
Let me try again with the bash_terminal function call.
Assistant
Let me search for the "Content hash mismatch" message more carefully in integrate.py to see if there's another place where this error is raised. </tool_call>
Assistant
The user seems to have encountered an error in my response. Let me continue investigating the issue by searching for the "Content hash mismatch" message in integrate.py.
Assistant
So there are two places where "Content hash mismatch" appears:
/apm/src/apm_cli/install/phases/integrate.pyline 197 - this is in the_resolve_download_strategyfunction, and it says "-- re-downloading"/apm/src/apm_cli/install/sources.py- this is the one that causes the actual error withsys.exit(1)
The error message from the bug report doesn't have "-- re-downloading" in it, so it's coming from
sources.pyinFreshDependencySource.Let me look at line 197 in integrate.py more closely.
Assistant
Now I see the flow:
-
In
_resolve_download_strategy, ifskip_download=Trueand the hash doesn't match, it:- Logs a warning
- Removes the install_path (safe_rmtree)
- Sets
skip_download = False
-
Then in
FreshDependencySource.acquire(), the package is downloaded fresh -
After downloading, it computes the hash and compares with lockfile
-
If they don't match, it exits with error
The bug is: when the cached content hash doesn't match (line 193-202 in integrate.py), it removes the cached content and forces a re-download. But then in FreshDependencySource, after the fresh download, the NEW content is hashed and compared to the lockfile. If the remote content has changed (or if there's some other issue), this will fail.
But wait, for a branch reference like
#main, the remote HEAD could have moved. So the new content could legitimately differ from the lockfile hash.BUT the issue says the first install works, and the second fails. If nothing changed on the remote, the second download should produce the same hash...
Let me think about what could cause the hash to differ:
- The
.apm-pinfile is written AFTER the first install's hash is recorded. So on the second install, if the hash computation includes.apm-pin, it would differ!
Wait, but
_EXCLUDED_ROOT_FILES = {_APM_PIN_MARKER}which is.apm-pin. So the hash computation excludes it.Let me look at the first install log more carefully:
cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commit; drift cannot verify its cache freshness.This warning appears 3 times. This means the parent repo
github-copilot-agent-skillsdoesn't have a pinned ref (no commit, tag, etc).For virtual dependencies like
testing.instructions.md #main, what is the resolved_commit? Since it's a file within a repo pointing to themainbranch, the resolved_commit would be the HEAD of main at the time of resolution.But the warning says the parent repo has no resolved_commit. This is
github-copilot-agent-skillsitself, not the individual files.Hmm, let me think about this differently. The error says:
Content hash mismatch for <org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md: expected sha256:2227579674ab30b5b59f703ac152fa74ac2912d724d04aeffbf3914cac2e6b88, got sha256:b82d09e4e7879eeb0e90cc7dfeebdac45de6ebdbcdda7824b40438196692f8f8.The expected hash is
sha256:22275796...and the actual hash issha256:b82d09e4....Now, on the first install:
- The file is downloaded
- The hash is computed (without .apm-pin)
- The lockfile is written with this hash
- .apm-pin is written AFTER the hash is recorded
On the second install:
- The cached file exists
- In
_resolve_download_strategy, since it's a branch ref (#main),is_cacheable = False - Since it's not a git repo (virtual package), it falls back to content hash check
- If the hash matches,
skip_download = True - If
skip_download = Trueand hash is verified, it usesCachedDependencySource
But wait! There's another path. What if the download was NOT skipped? What if the file was actually re-downloaded fresh?
Looking at the first install log again:
Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main |-- 1 instruction(s) integrated -> .github/instructions/And on the second install:
[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main Content hash mismatch for ...Note that on the second install, there's no "(cached)" suffix! This means
skip_download = Falseand it went throughFreshDependencySource.So why is
skip_download = False?Let me trace through the logic:
is_cacheable = Falsebecause the ref is a branch (#main), not a tag or commitalready_resolved- I need to check if this could be Truelockfile_match- This is checked via_should_skip_redownloadwhich verifies the content hash
So for
skip_downloadto be False, one of these must be true:- The content hash in
_should_skip_redownloaddoesn't match - Some heal chain logic changed
lockfile_matchto False - Registry config is enforcing a re-download
Wait! The
_resolve_download_strategyfunction has this code at lines 193-202:if skip_download and _dep_locked_chk and _dep_locked_chk.content_hash: from apm_cli.utils.content_hash import verify_package_hash if not verify_package_hash(install_path, _dep_locked_chk.content_hash): _hash_msg = f"Content hash mismatch for {dep_ref.get_unique_key()} -- re-downloading" diagnostics.warn(_hash_msg, package=dep_ref.get_unique_key()) if logger: logger.progress(_hash_msg) safe_rmtree(install_path, ctx.apm_modules_dir) skip_download = FalseThis code:
- Checks if
skip_downloadis True - If the lockfile has a content_hash
- Verifies the hash
- If verification fails, it removes the cached content and sets
skip_download = False
But this would cause a warning to be printed, which we don't see in the logs. So this path is NOT being taken.
Then why is
skip_download = False?Let me look at the heal chain logic:
lockfile_match, ref_changed = run_heal_chain( ctx, dep_ref, resolved_ref=resolved_ref, existing_lockfile=existing_lockfile, lockfile_match=lockfile_match, lockfile_match_via_content_hash_only=lockfile_match_via_content_hash_only, update_refs=update_refs, ref_changed=ref_changed, )Maybe the heal chain is changing
lockfile_matchto False?Actually, wait. Let me re-read the
_resolve_download_strategyfunction more carefully. I think I missed something.Looking at the
skip_downloadlogic:skip_download = install_path.exists() and ( (is_cacheable and not update_refs) or (already_resolved and not update_refs) or lockfile_match )So
skip_downloadis True if:install_path.exists()AND- One of:
(is_cacheable and not update_refs)OR(already_resolved and not update_refs)ORlockfile_match
Now, the virtual instruction file has
#mainas its ref, sois_cacheable = False. Ifalready_resolved = False(not pre-downloaded by resolver callback) andlockfile_match = False(content hash doesn't match), thenskip_download = False.But why would
lockfile_match = False? The_should_skip_redownloadfunction verifies the content hash:def _should_skip_redownload(locked_dep: Any, install_path: Path) -> bool: if locked_dep is None or not locked_dep.content_hash: return False if not install_path.is_dir(): return False from apm_cli.utils.content_hash import verify_package_hash return verify_package_hash(install_path, locked_dep.content_hash)This should return True if the hash matches...
OH WAIT! I think I found the issue!
The
.apm-pinmarker file is written bysync_markers_for_lockfilewhich is called after the lockfile is written. So on the FIRST install:- Content is downloaded
- Hash is computed (excluding .apm-pin since it doesn't exist yet)
- Lockfile is written with the hash
.apm-pinis written to the install_path
On the SECOND install:
- The install_path exists with the
.apm-pinfile _should_skip_redownloadcomputes the hash of the install_pathverify_package_hashcomputes the hash (excluding .apm-pin)- The hash should match... unless...
Wait, the hash computation already excludes
.apm-pinvia_EXCLUDED_ROOT_FILES = {_APM_PIN_MARKER}. So the hash should be the same!Unless... the
.apm-pinfile was written to a DIFFERENT path than what's being hashed!Let me look at where
.apm-pinis written. Incache_pin.py:def sync_markers_for_lockfile(...) -> int: ... for dep_key, dep in lockfile.dependencies.items(): if not _is_markable(dep): continue ... install_path = _resolve_install_path(dep, project_root, apm_modules_dir) ... write_marker(install_path, dep.resolved_commit)And
_resolve_install_path:def _resolve_install_path(dep, project_root, apm_modules_dir): dep_ref = dep.to_dependency_ref() return dep_ref.get_install_path(apm_modules_dir)So
.apm-pinis written todep_ref.get_install_path(apm_modules_dir)for each dependency.Now, for a virtual dependency like
github-copilot-agent-skills/.apm/instructions/testing.instructions.md, what is the install path?The install path for virtual dependencies is typically a subdirectory under
apm_modules/<owner>/<repo>/, likeapm_modules/dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md/.The
.apm-pinfile would be written to this path if the dep is markable.But wait,
_is_markablechecks:def _is_markable(dep): if getattr(dep, "source", None) == "local": return False commit = getattr(dep, "resolved_commit", None) return isinstance(commit, str) and bool(commit)So the dep needs a
resolved_committo be markable. For a branch reference like#main, the resolved_commit would be the actual commit SHA that main points to. So it should be markable.But the warning message says:
cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commitThis is for the parent repo, not the individual instruction file. The parent repo
github-copilot-agent-skillsmight not have a resolved_commit in the lockfile, so its marker is not written.But the instruction file dependency might still have a resolved_commit...
Actually, I'm getting confused about the dependency model. Let me think about this differently.
The instruction file
testing.instructions.md #mainis a "virtual package" - it's a single file extracted from a repo. Its install path would be something likeapm_modules/dev_azure_com_<org>_tools_github-copilot-agent-skills/_apm_instructions_testing.instructions.md/.The content hash for this virtual package is computed over just the instruction file itself.
Now, the
.apm-pinmarker... where is it written? For the parent repo? For each virtual package?Looking at
sync_markers_for_lockfile, it iterates overlockfile.dependencies.items(). Each entry in the lockfile has adep_keywhich is the unique key for the dependency.For virtual packages, the dep_key would be something like
<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md.If this dep has a
resolved_commit, then.apm-pinwould be written to its install path.Now here's the key question: Is the
.apm-pinfile being written to the SAME directory that's being hashed?If the hash computation is over
apm_modules/.../testing.instructions.md/and.apm-pinis written toapm_modules/.../testing.instructions.md/.apm-pin, then the hash computation would exclude.apm-pin(since it's in_EXCLUDED_ROOT_FILES), and the hashes would match.But what if
.apm-pinis written to a DIFFERENT directory? Like the parent repo's directory instead of the virtual package's directory?Let me check where
.apm-pinis actually written. Looking at_resolve_install_path:def _resolve_install_path(dep, project_root, apm_modules_dir): dep_ref = dep.to_dependency_ref() return dep_ref.get_install_path(apm_modules_dir)This converts the lockfile dep to a DependencyRef and gets its install path. The install path depends on how
get_install_pathis implemented for virtual packages.Let me find this implementation.
Actually, wait. Let me reconsider the issue.
The bug report says the FIRST install works, and the SECOND install fails. The lockfile is created after the first install. The
.apm-pinfile is written after the lockfile is created.On the SECOND install, the
.apm-pinfile exists in the install_path. If the hash computation includes files that were added/changed between the first and second install, the hashes would differ.But
.apm-pinis excluded from the hash computation. So it shouldn't affect the hash.UNLESS... there's something else being written to the install_path!
Let me look at what files are in the virtual package's install directory. For an instruction file, the install path would contain:
- The instruction file itself (e.g.,
testing.instructions.md) - Possibly
.apm-pin(excluded from hash)
If nothing else is added/removed, the hash should be the same.
But wait! What if the
.apm-pinfile is being written to the PARENT repo's install directory, not the virtual package's install directory?The parent repo's install directory might be
apm_modules/dev_azure_com_<org>_tools_github-copilot-agent-skills/. And the virtual package's install directory might beapm_modules/dev_azure_com_<org>_tools_github-copilot-agent-skills/_apm_instructions_testing.instructions.md/or similar.If
.apm-pinis written to the parent repo's directory, it wouldn't affect the virtual package's hash.But the warning says the parent repo has no resolved_commit, so
.apm-pinwouldn't be written for it anyway.Hmm, let me think about this from a different angle. Let me look at the error message format:
Content hash mismatch for <org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.mdThis is the dep_key for the instruction file. And the hash mismatch happens after a fresh download. So
skip_download = Falseon the second install.Why would
skip_download = False?Let me trace through
_resolve_download_strategyagain:is_cacheable = Falsebecause the ref is#main(a branch)already_resolved- is this True? The resolver callback might have pre-downloaded it.lockfile_match- this depends on_should_skip_redownload
Wait, I just realized something. The
already_resolvedcheck:already_resolved = dep_ref.get_unique_key() in ctx.callback_downloadedIf the resolver callback downloaded this dependency during the resolve phase, then
already_resolved = True. And theskip_downloadlogic includes:skip_download = install_path.exists() and ( ... or (already_resolved and not update_refs) ... )If
already_resolved = Trueandupdate_refs = False(normal install without --update), thenskip_download = True.But in the bug report, the second install is just
apm installwithout--update, soupdate_refs = False. Ifalready_resolved = True, thenskip_downloadshould be True.But then the cached source would be used, not the fresh source. And the cached source doesn't have the hash check that causes the error.
UNLESS the download happens through the resolver callback, which downloads fresh content. Then
already_resolved = True,skip_download = True, but the content was just downloaded fresh!Let me look at the
make_dependency_sourcefunction again:def make_dependency_source(ctx, dep_ref, install_path, dep_key, *, resolved_ref=None, dep_locked_chk=None, ref_changed=False, skip_download=False, fetched_this_run=False, progress=None): ... if skip_download: return CachedDependencySource(...) return FreshDependencySource(...)So if
skip_download = True, it usesCachedDependencySource, which doesn't do the hash check. The hash check is inFreshDependencySource.But the error message comes from
FreshDependencySource. Soskip_download = False.How can
skip_download = Falseifalready_resolved = Trueandupdate_refs = False?Unless
already_resolved = False! The resolver callback didn't download this dependency.Why wouldn't the resolver callback download it? Maybe because the install_path already exists and the resolver skips it?
Actually, let me look at the resolver callback logic. The resolver runs in parallel and pre-downloads dependencies. If a dependency is already cached, the resolver might skip it.
OK I think I need to look at the actual download flow more carefully. Let me search for the resolver callback.
Actually, let me approach this differently. Let me look at the tests to understand the expected behavior.
Let me search for tests related to content hash.
Actually, I think I should focus on understanding WHY the hash changes between the first and second install.
The expected hash is
sha256:2227579674ab30b5b59f703ac152fa74ac2912d724d04aeffbf3914cac2e6b88The actual hash issha256:b82d09e4e7879eeb0e90cc7dfeebdac45de6ebdbcdda7824b40438196692f8f8These are completely different. So something in the install directory is different between the first and second installs.
What could cause the content to change?
- The remote file changed (unlikely if it's a branch ref and the branch hasn't moved)
- Some extra files are being added to the install directory on the second install
- Some files are being removed from the install directory on the second install
Let me think about what files could be added. The
.apm-pinfile is excluded from the hash. What else?Oh wait! Let me look at the
_EXCLUDED_DIRSin the hash computation:_EXCLUDED_DIRS = {".git", "__pycache__"}And
_EXCLUDED_ROOT_FILES:_EXCLUDED_ROOT_FILES = {_APM_PIN_MARKER}So
.gitdirectories are excluded. But what if the install directory for a virtual package doesn't have a.gitdirectory (since it's just a single file)?Actually, virtual packages are extracted from a repo. They're not cloned as separate repos. So they wouldn't have a
.gitdirectory.Wait, I just had a thought. What if the
.apm-pinfile is written to a DIFFERENT location than where the hash is computed?Let me trace the flow more carefully:
- First install: The virtual package is downloaded to
apm_modules/.../testing.instructions.md/ - The hash is computed (excluding
.git,__pycache__, and.apm-pinat root) - The lockfile is written with this hash
.apm-pinis written... where?
Let me look at
sync_markers_for_lockfileagain. It writes.apm-pinto_resolve_install_path(dep, project_root, apm_modules_dir).And
_resolve_install_pathcallsdep_ref.get_install_path(apm_modules_dir).Now, for a virtual package, what is the install path? Let me search for
get_install_pathimplementation.OK let me actually run the code and look at the test cases to understand this better.
Actually, I realize I've been overthinking this. Let me just look at the specific scenario described in the bug report:
- First install:
apm install --update --force- downloads everything, computes hashes, writes lockfile - Second install:
apm install- should use cache, but fails with hash mismatch
The key observation is that the FIRST install works fine, including the instruction file. And the SECOND install fails specifically on the instruction file.
Let me look at the logs more carefully:
First install:
[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main |-- 1 instruction(s) integrated -> .github/instructions/Second install:
[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main Content hash mismatch for ...Note that in the second install, there's no "(cached)" suffix and no "|--" line. This means it went through
FreshDependencySource, notCachedDependencySource.So
skip_download = False. Why?The dependency has
#mainas its ref. Sois_cacheable = False(it's a branch, not a tag or commit).For
skip_downloadto be True, we needlockfile_match = True. Andlockfile_matchis set via_should_skip_redownloadwhich verifies the content hash.Wait! Maybe the hash verification in
_should_skip_redownloadis FAILING, which meanslockfile_match = False, which meansskip_download = False!But then the code at lines 193-202 would also detect this mismatch and print a warning:
Content hash mismatch for ... -- re-downloadingBut the bug report doesn't show this warning! So either:
- The warning was printed but not shown in the logs
_should_skip_redownloadis not being called (returns early)- The lockfile doesn't have a content_hash for this dep
Let me check if the lockfile has a content_hash for the instruction file dependency. The lockfile structure would have entries for each dependency, each with fields like
resolved_commit,content_hash, etc.If the instruction file dependency doesn't have a
content_hashin the lockfile, then_should_skip_redownloadwould return False immediately:if locked_dep is None or not locked_dep.content_hash: return FalseThis would make
lockfile_match = False, which would makeskip_download = False.But then the hash check in
FreshDependencySource.acquire()would also check:if ( not ctx.update_refs and dep_key not in _expected_hash_deps and dep_locked_chk and dep_locked_chk.content_hash and dep_key in ctx.package_hashes ):If
dep_locked_chk.content_hashis None/empty, this check would be skipped, and the hash mismatch error wouldn't occur!So the instruction file dependency MUST have a
content_hashin the lockfile.OK, I'm going in circles. Let me take a step back and think about this problem differently.
The issue is that the hash changes between the first and second install. The hash is computed over the install directory's contents. So the contents must be different.
What could cause the contents to change?
Option 1: The
.apm-pinfile is being written to a location that's included in the hash.But
.apm-pinis in_EXCLUDED_ROOT_FILES, so it should be excluded. Unless... the file is not at the root of the install directory?Let me re-read the exclusion logic:
if len(rel.parts) == 1 and rel.name in _EXCLUDED_ROOT_FILES: continueThis only excludes
.apm-pinif it's at the root of the package (rel.parts has length 1, meaning it's a direct child). If.apm-pinis in a subdirectory, it would NOT be excluded!But the
write_markerfunction writes it toinstall_path / MARKER_FILENAME, which is the root of the install directory. So it should be excluded.Unless... the hash is computed on a DIFFERENT path than where
.apm-pinis written!Wait! What if the
.apm-pinfile is written to the PARENT repo's directory, not the virtual package's directory?Looking at
sync_markers_for_lockfile:for dep_key, dep in lockfile.dependencies.items(): ... install_path = _resolve_install_path(dep, project_root, apm_modules_dir) ... write_marker(install_path, dep.resolved_commit)It writes to each dep's install path. So for the virtual package, it should write to the virtual package's install path.
But what if the lockfile doesn't have an entry for the virtual package? What if it only has an entry for the parent repo?
Actually, looking at the first install log:
cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commitThis suggests that the parent repo
github-copilot-agent-skillsis in the lockfile, but it doesn't have a resolved_commit. So no marker is written for it.But the instruction file dependency might be a SEPARATE entry in the lockfile. It would have its own resolved_commit (the SHA of main at the time of resolution) and its own content_hash.
Hmm, but looking at the log, the packages that are shown as
@d479f42eare the ones with resolved commits:[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/<project> #default @d479f42eBut the instruction file is shown as:
[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #mainNo
@<sha>suffix! This means it doesn't have a resolved_commit recorded.If the instruction file dependency doesn't have a
resolved_commitin the lockfile, then:_is_markablereturns False, so no.apm-pinis written- The content_hash might still be recorded
But wait, if there's no resolved_commit, then the
_should_skip_redownloadcheck might behave differently. Let me look at the code again:if install_path.exists() and existing_lockfile: locked_dep = existing_lockfile.get_dependency(dep_ref.get_unique_key()) if locked_dep and locked_dep.resolved_commit and locked_dep.resolved_commit != "cached": ... if _should_skip_redownload(locked_dep, install_path): lockfile_match = TrueIf
locked_dep.resolved_commitis None or "cached", then this whole block is skipped, andlockfile_matchremains False.But there's also a check for registry deps:
elif ( locked_dep and getattr(locked_dep, "source", None) == "registry" and not ref_changed and not update_refs ): if _should_skip_redownload(locked_dep, install_path): lockfile_match = TrueThis only applies to registry deps, not git deps.
So for a git-sourced virtual package without a resolved_commit,
lockfile_matchwould be False!And if
is_cacheable = False(branch ref) andalready_resolved = Falseandlockfile_match = False, thenskip_download = False.This means the dependency would be re-downloaded on the second install!
But wait, the instruction file dependency DOES have
#mainas its ref. When the resolver runs, it would resolve#mainto a commit SHA. Soresolved_ref.resolved_commitwould be set.But the lockfile might not have
resolved_commitif the first install used--update --forceand something went wrong in recording the commit.Actually, looking at the first install log:
[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #mainNo
@<sha>suffix. This suggests that the resolved_commit is NOT being recorded for this dependency.Hmm, but there's also this:
[!] 11 dependencies unpinned: <org>/tools/github-copilot-agent-skills -- add #tag or #sha to prevent driftThis suggests that the PARENT repo is unpinned, not necessarily the instruction file.
Actually wait, let me re-read the first install log more carefully:
[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/<project> #default @d479f42e [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/myorg #default @d479f42e [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/run-coverage #default @d479f42e [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/update-from-template #default @d479f42e [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/code-style.instructions.md #main [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/protect-prefixed-agent-files.instructions.md #main [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/git-conventions #default @d479f42e [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/update-nuget-packages #default @d479f42e [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/skills/write-tests #default @d479f42eI see a pattern:
- Dependencies with
#defaulthave@d479f42e(resolved commit) - Dependencies with
#maindo NOT have@<sha>(no resolved commit?)
But wait,
#mainis a branch ref. When the resolver resolves it, it should get the HEAD commit of main. So it should have a resolved_commit...Unless the resolver doesn't resolve
#mainbranches? Let me look at the resolution logic.Actually, looking at the code in
_resolve_download_strategy:is_cacheable = resolved_ref and resolved_ref.ref_type in [ GitReferenceType.TAG, GitReferenceType.COMMIT, ]This only considers TAG and COMMIT as cacheable. BRANCH is not cacheable.
And the resolution is only done for certain conditions:
if dep_ref.reference or (update_refs and _has_lockfile_sha): try: resolved_ref = ctx.downloader.resolve_git_reference(dep_ref) except Exception: passOn the first install with
--update --force,update_refs = True. So the resolver would try to resolve the ref.On the second install (without --update),
update_refs = False. Anddep_ref.referenceis#main, which is truthy. So the resolver would still try to resolve it.OK so the resolver should be resolving
#mainto a commit SHA on both installs.But wait, on the second install, there's a check:
if is_git_source and dep_ref.get_unique_key() not in ctx.pre_downloaded_keys:If the dep is already in
pre_downloaded_keys, the resolution is skipped. Andresolved_refwould be None.Hmm, but the pre_downloaded_keys would only contain deps that were pre-downloaded by the resolver callback during the resolve phase. On the second install, the resolver might not run at all if everything is cached...
Actually, let me step back. I think the issue might be simpler than I thought.
Let me look at the first install vs second install more carefully:
First install:
apm install --update --force--updatesetsupdate_refs = True--forcemight force re-download
Second install:
apm install- Normal install,
update_refs = False
On the second install, the dependency with
#main:is_cacheable = Falsebecause it's a branchalready_resolveddepends on whether the resolver callback pre-downloaded itlockfile_matchdepends on the content hash verification
If the content hash verification fails,
lockfile_match = Falseandskip_download = False.But then the fresh download would happen, and the hash would be computed again. If the remote content hasn't changed, the hash should be the same.
UNLESS the remote content HAS changed between the first and second install! If someone pushed to main, the file content would be different, and the hash would differ.
But the bug report says the issue is "second time" install fails. It doesn't mention any changes to the remote. So the remote content should be the same.
Wait, let me re-read the error message:
expected sha256:2227579674ab30b5b59f703ac152fa74ac2912d724d04aeffbf3914cac2e6b88, got sha256:b82d09e4e7879eeb0e90cc7dfeebdac45de6ebdbcdda7824b40438196692f8f8The expected hash is from the lockfile. The actual hash is computed from the freshly downloaded content.
If the remote content hasn't changed, these should be the same. But they're different!
So either:
- The remote content changed between installs
- The first install recorded a wrong hash in the lockfile
- The second install is computing a wrong hash
For option 2: The first install computes the hash correctly. But then
.apm-pinis written to some location. On the second install, if the hash computation includes something extra (like the.apm-pinfile), the hash would differ.But wait, the first install computes the hash BEFORE
.apm-pinis written. So the lockfile has the correct hash (without.apm-pin). The second install should compute the same hash (excluding.apm-pin).Unless... the hash is computed AFTER
.apm-pinis written on the first install! And then on the second install, the hash is computed BEFORE.apm-pinis re-written, resulting in a different hash.Let me check the order of operations:
- Download content
- Compute hash ->
ctx.package_hashes[dep_key] - Write lockfile (includes the hash)
- Write
.apm-pinmarker
So the hash is computed before
.apm-pinis written. The lockfile has the hash without.apm-pin.On the second install:
- Check if content matches (excluding
.apm-pin) - If match, use cache
- If no match, re-download
- After re-download, compute hash
- Compare with lockfile
If step 1 passes (hash matches), then step 2 uses cache and there's no error. If step 1 fails, step 3 re-downloads.
But the error happens after the re-download, when the new hash doesn't match the lockfile hash.
So the sequence is:
- Second install starts
_resolve_download_strategychecks the hash -> FAILS ->skip_download = FalseFreshDependencySourcedownloads fresh content- Hash is computed
- Hash doesn't match lockfile -> ERROR
But why does step 2 fail AND step 5 fail? If the hash doesn't match in step 2, it means the on-disk content differs from the lockfile. Then step 3 re-downloads. Step 4 computes the hash of the freshly downloaded content. If the remote hasn't changed, this should match the lockfile.
UNLESS step 2 fails for a DIFFERENT reason than the hash itself. Like if the lockfile doesn't have a
resolved_commitfor this dep, then the_should_skip_redownloadcheck is skipped entirely, andlockfile_match = Falseeven if the content hash matches.Let me re-read the code:
if install_path.exists() and existing_lockfile: locked_dep = existing_lockfile.get_dependency(dep_ref.get_unique_key()) if locked_dep and locked_dep.resolved_commit and locked_dep.resolved_commit != "cached": if update_refs: ... elif not ref_changed: try: from git import Repo as GitRepo local_repo = GitRepo(install_path) if local_repo.head.commit.hexsha == locked_dep.resolved_commit: lockfile_match = True except Exception: if _should_skip_redownload(locked_dep, install_path): lockfile_match = TrueSo if
locked_dep.resolved_commitis falsy or "cached", the entire block is skipped, and_should_skip_redownloadis never called!For a dependency with
#mainref, the lockfile might haveresolved_commit = Noneorresolved_commit = "cached".If the lockfile has
resolved_commit = "cached", then the checklocked_dep.resolved_commit and locked_dep.resolved_commit != "cached"would be False, and the block is skipped.This means
lockfile_match = Falseeven if the content hash matches!And since
is_cacheable = False(branch ref) andlockfile_match = False, we getskip_download = False.Then the fresh download happens, and the hash is computed. And if the lockfile has a content_hash, the check in
FreshDependencySourcecompares the new hash with the lockfile hash.But wait, on the second install, the fresh download should get the SAME content as the first install (if the remote hasn't changed). So the hash should match!
UNLESS the first install recorded a WRONG hash in the lockfile.
Hmm, let me think about what could cause a wrong hash to be recorded...
Actually, wait. Let me re-read the first install log:
cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commit; drift cannot verify its cache freshness.This warning is for the PARENT repo, not the individual instruction file. The parent repo doesn't have a
resolved_commitin the lockfile.But the individual instruction file might be a separate dependency in the lockfile, with its own
resolved_commit.Actually, looking at the lockfile structure, each dependency has a
dep_key. The instruction file would have a dep_key like<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md.This dep might or might not have a
resolved_commit.If the dep has
resolved_commit = Noneor an empty string, then:is_markablereturns False, so no.apm-pinis written- The content_hash is still recorded
- On the second install,
locked_dep.resolved_commitis falsy, solockfile_match = False skip_download = False, fresh download happens- Hash check passes (content hasn't changed)
Wait, but the error says the hash DOESN'T match in step 5. So the content must have changed between installs.
But how? The remote file should be the same...
Unless the issue is that on the first install, the hash was computed from different content than what's downloaded on the second install!
Let me think about what content is included in the hash:
- First install: The instruction file is extracted to
apm_modules/.../testing.instructions.md/ - The hash is computed over the contents of this directory
- On the second install: The same file is downloaded again
- The hash is computed again
If the directory contains the same file with the same content, the hash should be the same.
UNLESS there are other files in the directory! Like
.apm-pinor some metadata file.Oh wait! Let me look at the first install log again. There's a warning:
cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commit; drift cannot verify its cache freshness.This is printed 3 times. This suggests that there are 3 dependencies without
resolved_commit.These 3 dependencies are probably the instruction files (which have
#mainas their ref):testing.instructions.md #maincode-style.instructions.md #mainprotect-prefixed-agent-files.instructions.md #main
Wait, but the warning says the PARENT repo
github-copilot-agent-skillshas no resolved_commit. It's not about the individual files.Hmm, let me look at
find_unpinned_remote_deps:def find_unpinned_remote_deps(lockfile): unpinned: list[str] = [] for _key, dep in lockfile.dependencies.items(): if getattr(dep, "source", None) == "local": continue commit = getattr(dep, "resolved_commit", None) if not isinstance(commit, str) or not commit: unpinned.append(getattr(dep, "repo_url", "") or _key) return unpinnedThis iterates over all deps in the lockfile and checks if they have a
resolved_commit. The parent repo might not have aresolved_commit, and the individual instruction files might also not have one.But the warning message is:
cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commitIt only mentions the parent repo. Not the instruction files.
Looking at the loop in
sync_markers_for_lockfile:for repo in unpinned: _rich_warning(f"cache-pin: remote dep {repo!r} has no resolved_commit; ...")So it prints one warning per unpinned dep. And the
repoisgetattr(dep, "repo_url", "") or _key.If the instruction file dep has
repo_url = '<org>/tools/github-copilot-agent-skills', then all three instruction files would print the same warning (since they all have the same repo_url).But the warning is only printed 3 times, not once. And each time it's for the same repo. This suggests that there are 3 different deps with the same
repo_urlbut differentdep_key.So the instruction files ARE separate deps in the lockfile, and they don't have
resolved_commit.Now, the question is: why doesn't the instruction file have a
resolved_commit?When the dependency is resolved, the resolver gets the HEAD commit of
#main. This commit should be recorded in the lockfile asresolved_commit.Unless... the lockfile recording is not done correctly for virtual packages.
Actually, looking at the
InstalledPackagecreation inCachedDependencySource.acquire():ctx.installed_packages.append( InstalledPackage( dep_ref=dep_ref, resolved_commit=cached_commit, ... ) )And
cached_commitis determined by_resolve_cached_commit:def _resolve_cached_commit(self): ... elif ctx.existing_lockfile: locked_dep = ctx.existing_lockfile.get_dependency(dep_key) if locked_dep and locked_dep.resolved_commit and locked_dep.resolved_commit != "cached": cached_commit = locked_dep.resolved_commit if not cached_commit: cached_commit = dep_ref.reference return cached_commitIf the lockfile doesn't have a
resolved_commit(or it's "cached"), thencached_commitfalls back todep_ref.reference, which is#mainfor the instruction file.Wait, but
dep_ref.referenceis the original ref string, like#main. This is NOT a commit SHA. So theresolved_commitrecorded in the lockfile would be#main, not a commit SHA!Hmm, that doesn't seem right. Let me look at the lockfile structure more carefully.
Actually, I think the issue might be in how the first install records the
resolved_commitfor virtual packages with branch refs.Let me look at
FreshDependencySource.acquire()to see howresolved_commitis set:resolved = getattr(package_info, "resolved_reference", None) if resolved: resolved_commit = package_info.resolved_reference.resolved_commit ... ctx.installed_packages.append( InstalledPackage( dep_ref=dep_ref, resolved_commit=resolved_commit, ... ) )So the
resolved_commitcomes frompackage_info.resolved_reference.resolved_commit. This should be the actual commit SHA resolved by the downloader.But if the downloader doesn't resolve the ref (e.g., if the resolution failed or was skipped), then
resolved_commitmight be None or the original ref string.Hmm, let me think about this differently. The first install with
--update --forceshould resolve all refs and record the actual commit SHAs. So the lockfile should have properresolved_commitvalues.But the warning says the repo has no
resolved_commit. And the instruction files show#mainwithout a commit SHA in the log.I think the issue might be that for virtual packages, the
resolved_commitis recorded at the PARENT repo level, not at the individual file level.Actually, wait. Let me re-read the first install log:
[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/<project> #default @d479f42eThis shows
#default @d479f42e. The#defaultis the ref, and@d479f42eis the resolved commit. So the lockfile hasresolved_commit = d479f42e....[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #mainThis shows
#mainwithout a commit. This suggests that for the instruction file, theresolved_commitis not recorded (or it's None/empty).If the
resolved_commitis not recorded for the instruction file, then:_is_markablereturns False- No
.apm-pinis written for this dep - On the second install,
locked_dep.resolved_commitis falsy lockfile_matchis not set via the normal pathskip_download = False
But then the hash check should pass on the fresh download, since the content hasn't changed!
Unless... there's a different issue. Let me look at the lockfile format to understand what's recorded.
Let me look at the actual lockfile in the repo.
Let me look at the apm.lock.yaml file in the repo to understand the structure.
-
Assistant
This lockfile only has local dependencies. Let me look for tests or examples that have remote dependencies with branch refs to understand the issue better. Let me also search for how the lockfile records
resolved_commitfor remote dependencies.Let me trace through the code to understand the flow better. I need to find where the lockfile entries are created for remote dependencies, specifically looking at
resolved_commitandcontent_hashfields.Let me also look for existing tests related to content hash mismatch.
Assistant
Let me look at how the lockfile is built and understand when
content_hashis recorded. I need to trace back to whereInstalledPackageis converted to aLockedDependencyentry in the lockfile.Assistant
Let me trace through where
content_hashis set in the lockfile. I need to find whereLockedDependencyis created and see howcontent_hashis populated.The key insight is that the hash is computed in
ctx.package_hashes[dep_key]during the install process. Then when the lockfile is built, it should use this hash.Let me search for where the lockfile is built and how
content_hashis populated fromctx.package_hashes.Assistant
Now I can see the flow:
- In
sources.py, the hash is computed and stored inctx.package_hashes[dep_key] - In
phases/lockfile.py, the hash is transferred tolocked_dep.content_hash
Let me look at the lockfile building phase:
- In
Assistant
Now I understand the flow:
- Dependencies are installed (via
CachedDependencySourceorFreshDependencySource) - During installation, the hash is computed and stored in
ctx.package_hashes[dep_key] - When the lockfile is built,
_attach_content_hashescopies the hash tolocked_dep.content_hash
Now, let me trace the issue. The problem is:
- First install: Hash is computed, lockfile is written with the hash
- Second install: The dependency should be served from cache, but the hash check fails
Let me re-examine the
_resolve_download_strategyfunction to understand whenskip_downloadis False.Looking at the code, for a branch ref (
#main),is_cacheable = False. So theskip_downloadlogic depends on:already_resolved(pre-downloaded by resolver callback)lockfile_match(content hash verification)
For
lockfile_matchto be True, the code needs:if locked_dep and locked_dep.resolved_commit and locked_dep.resolved_commit != "cached": ... if _should_skip_redownload(locked_dep, install_path): lockfile_match = TrueIf
locked_dep.resolved_commitis None or "cached", this check is skipped andlockfile_match = False.Now, for a dependency with
#mainbranch ref, what is theresolved_commitin the lockfile?On the first install with
--update --force, the resolver should resolve#mainto a commit SHA. This SHA should be recorded in the lockfile asresolved_commit.But the first install log shows:
[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #mainNo
@<sha>suffix. This suggests theresolved_commitis not being displayed, but it might still be recorded in the lockfile.Let me look at how the log line is generated to understand if the
resolved_commitis actually missing or just not displayed.- Dependencies are installed (via
Assistant
Now I can see the log format:
- If both
_r(ref_name) and_s(short SHA) are present:#{_r} @{_s} - If only
_r:#{_r} - If only
_s:@{_s}
For the instruction file with
#main, the log shows just#main, which means:_r = "main"(the ref_name)_s = ""(no SHA, or the SHA is empty/sentinel)
This means
resolved.resolved_commitis either None, empty, or a sentinel value thatformat_short_shaconverts to "".Let me check what
format_short_shadoes:- If both
Assistant
Ah, now I see it! The
format_short_shafunction returns "" for:- Non-string values
- Sentinels: "cached", "unknown"
- Strings shorter than 8 chars
- Non-hex strings
So if the log shows
#mainwithout an@<sha>, it meansresolved.resolved_commitis either:- None
- "cached" or "unknown"
- Something that's not a valid hex string (like "#main" itself!)
Wait, for a branch reference like
#main, the resolver should resolve it to an actual commit SHA. If the resolution failed or was skipped, theresolved_commitmight be the original ref string or None.Let me think about this more carefully. The bug says the first install works but the second fails. On the first install, the dependency is downloaded fresh (since there's no cache). On the second install, the cache exists but the hash check fails.
Now, looking at the
_resolve_download_strategyfunction in integrate.py:if locked_dep and locked_dep.resolved_commit and locked_dep.resolved_commit != "cached": ... if _should_skip_redownload(locked_dep, install_path): lockfile_match = TrueIf
locked_dep.resolved_commitis None or "cached", thenlockfile_matchremains False, and the dependency is re-downloaded.But wait, there's also a check for the content hash verification AFTER
skip_downloadis determined:if skip_download and _dep_locked_chk and _dep_locked_chk.content_hash: if not verify_package_hash(install_path, _dep_locked_chk.content_hash): ... safe_rmtree(install_path, ctx.apm_modules_dir) skip_download = FalseThis means if
skip_download = Trueand the hash doesn't match, the cache is invalidated andskip_downloadis set to False.So the flow could be:
skip_download = True(e.g., becausealready_resolved = Trueorlockfile_match = True)- Content hash check: FAILS
- Cache is removed,
skip_download = False - Fresh download happens
- Fresh download hash is computed
- Hash comparison with lockfile: FAILS
The key question is: why does the fresh download hash differ from the lockfile hash?
If the remote content hasn't changed, the fresh download should produce the same hash as the first install.
Unless... the hash computation is different between the first and second installs!
One possibility: The
.apm-pinfile is being written to a location that's included in the hash computation on the second install but not on the first.But
_EXCLUDED_ROOT_FILES = {_APM_PIN_MARKER}should exclude it...WAIT! I just realized something. The
_EXCLUDED_ROOT_FILESonly excludes.apm-pinif it's at the ROOT of the package (i.e.,len(rel.parts) == 1). But what if the.apm-pinfile is NOT at the root?Let me check where
.apm-pinis written for virtual packages. Thesync_markers_for_lockfilefunction writes it to_resolve_install_path(dep, project_root, apm_modules_dir).For a virtual package, the install path might be a subdirectory of the parent repo's install path. Or it might be a completely separate directory.
Let me think about this more carefully. The virtual package for
testing.instructions.md #mainwould have an install path like:apm_modules/dev_azure_com_<org>_tools_github-copilot-agent-skills/.apm/instructions/testing.instructions.md/Wait, that doesn't look right. Let me search for how the install path is determined for virtual packages.
<tool_call> <function=bash_terminal> <parameter=command> grep -rn "get_install_path" /apm/src/apm_cli/ --include="*.py" | grep -v "pycache" </tool_call>
Assistant
I need to properly call the bash_terminal function to search for
get_install_path.Assistant
Let me look at the
get_install_pathimplementation to understand how virtual packages are installed.Assistant
Let me continue reading the
get_install_pathmethod to understand how virtual packages get their install paths.Assistant
Let me look at the tests related to content hash mismatch to understand the expected behavior better.
Assistant
Let me look at these tests to understand the expected behavior around content hash mismatch.
Assistant
I can see the test confirms the behavior. The hash mismatch happens in
FreshDependencySource.acquire()when:- The package is downloaded
- The hash is computed and stored in
ctx.package_hashes[dep_key] - The hash is compared with
dep_locked_chk.content_hash - If they don't match, it calls
sys.exit(1)
Now I need to understand WHY the hash differs between the first and second install.
Let me think about this more carefully. The issue is specifically with instruction files (virtual packages) that reference
#main(a branch).Looking at the bug report again:
- First install: Works fine
- Second install: Fails with hash mismatch for instruction files
The key observation is that the instruction file shows
#mainwithout a@<sha>in the log. This means theresolved_commitmight not be properly set.Let me look at what happens when
resolved_commitis None or "cached" for a branch reference:-
In
_resolve_download_strategy:- If
locked_dep.resolved_commitis None or "cached", thelockfile_matchcheck is skipped lockfile_matchremains Falseis_cacheable = False(branch ref)- If
already_resolved = False, thenskip_download = False
- If
-
With
skip_download = False,FreshDependencySourceis used -
Fresh download happens
-
Hash is computed
-
If hash doesn't match lockfile, error!
The question is: why doesn't the hash match?
Let me think about what files are in the install directory...
For a virtual instruction file, the install directory would contain:
- The instruction file itself (e.g.,
testing.instructions.md) - Possibly other files if the downloader creates them
.apm-pinif the dep is markable
Wait, I think I finally see the issue!
Let me trace the flow for the FIRST install:
- Instruction file is downloaded to
apm_modules/.../testing.instructions.md/ - Hash is computed (excludes
.apm-pinwhich doesn't exist yet) - Lockfile is written with the hash
.apm-pinis written bysync_markers_for_lockfile
But wait, the warning says "cache-pin: remote dep '/tools/github-copilot-agent-skills' has no resolved_commit". This means the PARENT repo doesn't have a resolved_commit. But what about the individual instruction file dependency?
If the instruction file dependency also doesn't have a resolved_commit in the lockfile, then:
_is_markablereturns False for it- No
.apm-pinis written to its install directory - The hash computation remains the same between installs
But the error still occurs...
Wait, let me look at this from a different angle. The
.apm-pinmarker is written bysync_markers_for_lockfile, which iterates over ALL dependencies in the lockfile. If a dependency has aresolved_commit, it gets a marker.What if the instruction file dependency DOES have a
resolved_commitin the lockfile? Then:.apm-pinIS written to its install directory- On the second install,
.apm-pinexists in the directory - The hash computation excludes
.apm-pin(since it's in_EXCLUDED_ROOT_FILES) - Hash should match!
But it doesn't match. So either:
.apm-pinis NOT being excluded properly- Something else is different
Let me re-read the hash computation code:
for item in package_path.rglob("*"): ... rel = item.relative_to(package_path) if any(part in _EXCLUDED_DIRS for part in rel.parts): continue if item.is_file(): if len(rel.parts) == 1 and rel.name in _EXCLUDED_ROOT_FILES: continue regular_files.append(rel)The exclusion logic for root files is:
if len(rel.parts) == 1 and rel.name in _EXCLUDED_ROOT_FILES: continueThis only excludes
.apm-piniflen(rel.parts) == 1. Butrel.partsis a tuple of path components. For a file at the root of the package, likeinstall_path / ".apm-pin", therelwould bePath(".apm-pin")andrel.partswould be(".apm-pin",)which has length 1.So
.apm-pinat the root SHOULD be excluded. Unless...OH WAIT! I think I see the issue now!
The instruction file is a VIRTUAL package. The install path for a virtual package might be structured differently. Let me think about this...
For a virtual file like
.apm/instructions/testing.instructions.md, the install path would be something like:apm_modules/<org>/<project>/<package_name>/And inside this directory, there would be:
testing.instructions.md(the actual file).apm-pin(if written)
Both at the root of the install directory. So
.apm-pinshould be excluded from the hash.Unless the
.apm-pinis written to a DIFFERENT location!Let me check
sync_markers_for_lockfileagain:install_path = _resolve_install_path(dep, project_root, apm_modules_dir)And
_resolve_install_path:def _resolve_install_path(dep, project_root, apm_modules_dir): dep_ref = dep.to_dependency_ref() return dep_ref.get_install_path(apm_modules_dir)This converts the lockfile dep to a
DependencyRefand gets the install path. So it should be the SAME path as where the package was installed.Unless... the lockfile dep's
to_dependency_ref()returns a differentDependencyRefthan what was used during install!Hmm, let me think about this more. The lockfile stores the dependency info. When converting back to a
DependencyRef, if something is lost or changed, the install path could be different.Actually, I wonder if the issue is with the
.apm-pinfile being written to the PARENT REPO's directory instead of the virtual package's directory.Looking at
sync_markers_for_lockfile:for dep_key, dep in lockfile.dependencies.items(): if not _is_markable(dep): continue ... install_path = _resolve_install_path(dep, project_root, apm_modules_dir) ... write_marker(install_path, dep.resolved_commit)This iterates over each dependency in the lockfile. For the instruction file dependency,
_is_markablewould check if it has aresolved_commit.If the instruction file dependency has a
resolved_commit, then.apm-pinwould be written to ITS install path (not the parent repo's path).But wait, what if the lockfile doesn't have an entry for the instruction file at all? What if the instruction file is part of the parent repo's dependency entry?
Looking at the lockfile structure, each dependency has its own entry. The instruction file should have a separate entry with its own
dep_key,resolved_commit,content_hash, etc.Actually, let me reconsider the entire flow. Let me look at how virtual packages are handled during install.
For a virtual package (like an instruction file), the installer downloads the entire repo (or a subdirectory), then extracts just the file(s) of interest to the virtual package's install directory.
Wait no, that's not right either. Let me look at how the downloader handles virtual packages.
Actually, I think I need to understand how the virtual package's content is different between the first and second install.
Let me look at the downloader code for virtual packages...
Actually, I realize I should focus on the actual code path. Let me look at what happens during the hash check in
FreshDependencySource:if install_path.is_dir(): ctx.package_hashes[dep_key] = _compute_hash(install_path) # Supply-chain protection: verify content hash on fresh # downloads when the lockfile already records a hash. ... if ( not ctx.update_refs and dep_key not in _expected_hash_deps and dep_locked_chk and dep_locked_chk.content_hash and dep_key in ctx.package_hashes ): _fresh_hash = ctx.package_hashes[dep_key] if _fresh_hash != dep_locked_chk.content_hash: ... sys.exit(1)The hash is computed AFTER the download. So the hash is of the freshly downloaded content. If the download produces the same content as the first install, the hash should match.
Unless the download produces DIFFERENT content!
What could cause different content on the second download?
- The remote file changed (branch moved)
- The downloader extracts different files
- Some files are added/removed during download
For option 2 and 3, I need to look at how the downloader works for virtual packages.
Actually wait, I just realized something. Let me look at the error message format again:
Content hash mismatch for <org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md: expected sha256:2227579674ab30b5b59f703ac152fa74ac2912d724d04aeffbf3914cac2e6b88, got sha256:b82d09e4e7879eeb0e90cc7dfeebdac45de6ebdbcdda7824b40438196692f8f8.The dep_key includes the full path to the file:
<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md. This is the unique key for the virtual package.Now, the key question: what is the install path for this virtual package?
For a virtual package, the install path would be something like:
apm_modules/dev_azure_com_<org>/tools/<sanitized-package-name>/Inside this directory, there would be:
- The instruction file
- Possibly other metadata files
Let me look at how the downloader creates the install directory for virtual packages...
Actually, let me focus on finding where
.apm-pincould be written to affect the hash. The_EXCLUDED_ROOT_FILESonly excludes files at the ROOT of the install path. If.apm-pinis somewhere else, it wouldn't be excluded.Wait, actually, I just realized something! Let me re-read the exclusion logic:
_EXCLUDED_ROOT_FILES = {_APM_PIN_MARKER}And
_APM_PIN_MARKERis imported fromcache_pin:from apm_cli.install.cache_pin import MARKER_FILENAME as _APM_PIN_MARKERAnd in
cache_pin.py:MARKER_FILENAME = ".apm-pin"So
_EXCLUDED_ROOT_FILES = {".apm-pin"}. The hash computation excludes.apm-pinat the root.But what if the marker is NOT at the root? Let me check
write_marker:def write_marker(install_path: Path, resolved_commit: str) -> None: ... marker = install_path / MARKER_FILENAME ... marker.write_text(json.dumps(payload), encoding="utf-8")So it writes to
install_path / ".apm-pin", which IS at the root.Unless... the install_path for the marker is DIFFERENT from the install_path for the hash computation!
Let me check if there could be a mismatch between where
.apm-pinis written and where the hash is computed.In
sync_markers_for_lockfile:install_path = _resolve_install_path(dep, project_root, apm_modules_dir)And
_resolve_install_path:def _resolve_install_path(dep, project_root, apm_modules_dir): dep_ref = dep.to_dependency_ref() return dep_ref.get_install_path(apm_modules_dir)In the source code (e.g.,
FreshDependencySource.acquire()), the hash is computed at:if install_path.is_dir(): ctx.package_hashes[dep_key] = _compute_hash(install_path)Where
install_pathis the same asdep_ref.get_install_path(apm_modules_dir).So they should be the same path!
Hmm, but what if there's a difference in how the lockfile dep's
to_dependency_ref()resolves compared to the original dep_ref?Let me look at
to_dependency_ref():<tool_call>
Actually, let me take a step back and think about this more carefully.
The bug says the FIRST install works but the SECOND fails. The first install is
apm install --update --force. The second isapm install.On the first install:
- Dependencies are downloaded fresh (because of
--forceor because there's no cache) - Hashes are computed
- Lockfile is written
- Markers are written
On the second install:
- Dependencies should be served from cache
- For branch refs like
#main, they might be re-downloaded - If re-downloaded, the hash is compared with the lockfile
The issue is that the hash comparison fails. This means the content downloaded on the second install is different from the content on the first install.
But why would the content be different if the remote hasn't changed?
One possibility: The
.apm-pinfile is being written to the install directory DURING the second install, BEFORE the hash is computed.Wait, no. The
.apm-pinis written bysync_markers_for_lockfile, which is called AFTER the install completes. Let me verify this.<tool_call>
<tool_call> <function=bash_terminal> <parameter=command> grep -rn "sync_markers_for_lockfile" /apm/src/apm_cli/ --include="*.py" | grep -v "pycache" </tool_call>
Assistant
I need to search for where
sync_markers_for_lockfileis called to understand when the.apm-pinmarker is written.Assistant
Let me look at the lockfile.py to see when sync_markers_for_lockfile is called relative to when the hash is recorded.
Assistant
The
.apm-pinmarkers are written AFTER the lockfile is built and saved. Let me look at the order of operations in the lockfile building process to understand the flow.Let me trace through the lockfile building process to see when the hash is attached vs when the markers are written.
Assistant
Let me search for how
resolved_commitis set for branch references and virtual packages, as I think this might be the root cause of the issue.Assistant
Looking at the code structure, I need to understand how the lockfile dependency is converted back to a
DependencyRefand how that affects the install path. Let me examine the lockfile'sto_dependency_refmethod.Assistant
I need to examine the
to_dependency_refmethod in the lockfile.py to understand how the lockfile dependency is converted back to a DependencyRef, and how that might affect the install path.Assistant
Now I understand the code structure better. Let me trace through the issue more carefully.
The
to_dependency_refmethod reconstructs aDependencyReferencefrom the lockfile entry. It usesself.virtual_pathandself.is_virtualto determine if it's a virtual package.Now, let me think about the bug:
- First install: Hash is computed, lockfile is written,
.apm-pinis written - Second install: The cached content should match the hash, but it doesn't
The key question is: what causes the hash to differ between installs?
Wait, I should look more carefully at the
_resolve_download_strategyfunction. Let me trace through what happens for a virtual instruction file with#mainref.Actually, I think I need to look at what happens when
resolved_commitis NOT in the lockfile. Looking at the code inintegrate.py:if locked_dep and locked_dep.resolved_commit and locked_dep.resolved_commit != "cached": ... if _should_skip_redownload(locked_dep, install_path): lockfile_match = TrueIf
locked_dep.resolved_commitis None or empty or "cached", the_should_skip_redownloadis never called, andlockfile_matchremains False.This means for dependencies without a resolved_commit in the lockfile, the cache check is skipped entirely, and they are always re-downloaded.
But then, after re-downloading, the hash check in
FreshDependencySourcecompares the new hash with the lockfile hash. If they match, no error. If they don't match, error.The issue is that the hash doesn't match. But why?
Let me think about this differently. Maybe the issue is that the lockfile DOESN'T have a content_hash for this dependency, and somehow a stale hash from somewhere else is being used?
Actually wait, let me re-read the error message:
Content hash mismatch for <org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md: expected sha256:2227579674ab30b5b59f703ac152fa74ac2912d724d04aeffbf3914cac2e6b88, got sha256:b82d09e4e7879eeb0e90cc7dfeebdac45de6ebdbcdda7824b40438196692f8f8.The "expected" hash is from the lockfile. The "got" hash is computed from the freshly downloaded content.
If the remote file hasn't changed, these should be the same. But they're different!
One possibility: The lockfile recorded a hash that was computed from a different set of files than what's downloaded now.
Let me think about what files are in the virtual package's install directory:
- The instruction file (e.g.,
testing.instructions.md) - Possibly
.apm-pin(excluded from hash) - Possibly other metadata files
What if there's a difference in what files are included between installs?
Actually, I just realized something important. Let me look at the
get_install_pathmethod again for virtual packages.For a virtual file package (not a subdirectory), the code does:
package_name = self.get_virtual_package_name()Let me see what
get_virtual_package_namereturns.Actually, I think the real issue might be simpler. Let me look at the flow again:
- First install: Downloads repo, extracts instruction file, computes hash, writes lockfile
- Second install: For branch refs, re-downloads, computes hash, compares
But wait, there's a check in
_resolve_download_strategy:if skip_download and _dep_locked_chk and _dep_locked_chk.content_hash: if not verify_package_hash(install_path, _dep_locked_chk.content_hash): ... safe_rmtree(install_path, ctx.apm_modules_dir) skip_download = FalseThis checks if the CACHED content matches the lockfile hash. If it doesn't match, the cache is invalidated and
skip_download = False.But then the fresh download happens. If the fresh download produces the same content as the first install, the hash should match!
Unless the fresh download produces DIFFERENT content!
What if the issue is that the first install computed the hash from the wrong files? Like, what if some files were added to the install directory AFTER the hash was computed but BEFORE the lockfile was written?
Hmm, but the hash is computed in
FreshDependencySource.acquire()right after the download. And the lockfile is written after all installs complete. So there shouldn't be any files added in between...Wait, unless the
.apm-pinfile is written BEFORE the hash is computed! Let me check the order of operations more carefully.Actually, looking at the lockfile building code:
# Attach content hashes captured at download/verify time self._attach_content_hashes(lockfile) ... # Only write when the semantic content has actually changed self._write_if_changed(lockfile, lockfile_path, _LF) # Self-heal cache pin markers EVERY install self._sync_cache_pin_markers(lockfile)So the order is:
- Attach content hashes (from
ctx.package_hasheswhich was populated during download) - Write lockfile
- Write cache pin markers
This means the
.apm-pinis written AFTER the lockfile is written. So the hash computed during download doesn't include.apm-pin.On the second install:
- The cache exists with
.apm-pin - Hash is verified (excluding
.apm-pin) - should pass - If pass, skip download
But if the verification fails, the cache is invalidated and a fresh download happens.
Hmm, but the error message doesn't show the warning "Content hash mismatch for ... -- re-downloading" that would be printed if the cache verification failed. So either:
- The warning was suppressed
- The cache verification passed but something else caused the error
Let me look at the error message format again. The error from
FreshDependencySourceis:_rich_error( f"Content hash mismatch for " f"{dep_key}: " f"expected {dep_locked_chk.content_hash}, " f"got {_fresh_hash}. " "The downloaded content differs from the " "lockfile record. This may indicate a " "supply-chain attack. Use 'apm install " "--update' to accept new content and " "update the lockfile." )And the warning from
_resolve_download_strategyis:_hash_msg = f"Content hash mismatch for {dep_ref.get_unique_key()} -- re-downloading"The error message from the bug report says "The downloaded content differs from the lockfile record." which matches the
FreshDependencySourceerror format. So the error is coming from the fresh source, not the cache verification.This means
skip_download = FalseandFreshDependencySourceis used. The fresh download happens, and then the hash comparison fails.Now, the question is: why does the fresh download produce a different hash than the lockfile?
Wait, let me look at the check more carefully:
if ( not ctx.update_refs and dep_key not in _expected_hash_deps and dep_locked_chk and dep_locked_chk.content_hash and dep_key in ctx.package_hashes ):This check only runs when:
not ctx.update_refs- Normal install (not --update)dep_key not in _expected_hash_deps- Not expected to changedep_locked_chk- There's a lockfile entrydep_locked_chk.content_hash- The lockfile has a content hashdep_key in ctx.package_hashes- The hash was computed
So on the second install (normal install without --update), this check runs. And if the hash doesn't match, it errors.
Now, the question is why the hash doesn't match.
One possibility: The virtual package's install path structure changed between the first and second install. Maybe the first install used a different directory structure than the second install.
But that seems unlikely since the install path is deterministic based on the dependency ref.
Another possibility: The
.apm-pinfile from the first install is causing the hash to differ on the second install.Wait, but the hash computation excludes
.apm-pinat the root. So this shouldn't affect the hash.Unless... the
.apm-pinis NOT being excluded properly. Let me look at the hash computation again:for item in package_path.rglob("*"): if item.is_symlink(): continue rel = item.relative_to(package_path) if any(part in _EXCLUDED_DIRS for part in rel.parts): continue if item.is_file(): if len(rel.parts) == 1 and rel.name in _EXCLUDED_ROOT_FILES: continue regular_files.append(rel)Wait,
rglob("*")returns both files and directories. The checkitem.is_file()only processes files. But before that check, there'sif any(part in _EXCLUDED_DIRS for part in rel.parts): continuewhich skips directories.So the flow is:
- Iterate all items (files and dirs)
- Skip symlinks
- Skip excluded dirs and their contents
- If it's a file: a. Skip excluded root files b. Add to list
This looks correct.
.apm-pinat the root should be skipped becauselen(rel.parts) == 1andrel.name == ".apm-pin"which is in_EXCLUDED_ROOT_FILES.Hmm, let me think about edge cases...
What if the
.apm-pinfile is written to a DIFFERENT path than the install path?Actually, let me look at what happens for virtual packages more carefully. The install path for a virtual package is determined by
get_install_path. For a virtual FILE package (not subdirectory), it uses a sanitized package name.Let me look at
get_virtual_package_name:Actually, I wonder if the issue is that the
.apm-pinmarker is being written to the PARENT repo's directory, not the virtual package's directory. And the parent repo's directory might overlap with the virtual package's directory structure.For example, if the parent repo is installed at:
apm_modules/dev_azure_com_<org>/tools/github-copilot-agent-skills/And the virtual package is installed at:
apm_modules/dev_azure_com_<org>/tools/<sanitized-virtual-name>/These are different directories. So
.apm-pinin the parent repo's directory wouldn't affect the virtual package's hash.But wait, for virtual packages, the content is extracted from the parent repo. The parent repo might or might not be separately installed.
Hmm, this is getting complicated. Let me look at the actual issue more carefully.
The bug says the issue is with instruction files that have
#mainas their ref. And the error only happens on the SECOND install.What if the issue is that the FIRST install didn't properly record a
resolved_commitfor the instruction file? Then on the second install:locked_dep.resolved_commitis None or "cached"lockfile_matchis False (because the check requiresresolved_commit)skip_download = False- Fresh download happens
- Hash comparison fails
But why would the hash comparison fail if the content is the same?
Unless the lockfile doesn't have a
content_hashfor this dependency either. In that case, the checkdep_locked_chk.content_hashwould be False, and the error wouldn't occur.But the error IS occurring, so the lockfile DOES have a
content_hash.OK let me think about this differently. The first install computes a hash. The second install computes a different hash. The content should be the same (same file from the same commit). So what's different?
One possibility: The
.apm-pinfile IS affecting the hash on the second install.But I've established that
.apm-pinshould be excluded. Unless there's a bug in the exclusion logic.Wait, let me look at
_EXCLUDED_ROOT_FILESagain:_EXCLUDED_ROOT_FILES = {_APM_PIN_MARKER}And
_APM_PIN_MARKERis imported fromcache_pin:from apm_cli.install.cache_pin import MARKER_FILENAME as _APM_PIN_MARKERLet me check if
MARKER_FILENAMEis exactly.apm-pin:MARKER_FILENAME = ".apm-pin"Yes, it is. So
_EXCLUDED_ROOT_FILES = {".apm-pin"}.The exclusion check is:
if len(rel.parts) == 1 and rel.name in _EXCLUDED_ROOT_FILES: continueFor a file at
install_path / ".apm-pin",relwould bePath(".apm-pin"),rel.partswould be(".apm-pin",),len(rel.parts)would be 1, andrel.namewould be.apm-pinwhich is in_EXCLUDED_ROOT_FILES. So it should be excluded.Unless...
rel.namereturns something different? Let me check...Actually,
Path(".apm-pin").namereturns.apm-pin. So it should match.Hmm, I'm stuck. Let me look at the issue from a completely different angle.
What if the issue is NOT with
.apm-pinbut with some other file? What if the virtual package's install directory contains different files between the first and second install?Let me think about what files are in a virtual package's install directory:
- The instruction file itself
- Any
.apm-pinmarker (if written) - Any other metadata files created during install
What if there's a metadata file that's created differently between installs?
Actually, wait. I just realized something. The virtual package for an instruction file might have more than just the instruction file. It might have other files from the repo that are part of the "package".
No wait, for a virtual file package, only the specified file is extracted. The install directory should only contain that one file.
Unless the downloader creates additional files...
OK, let me look at this from yet another angle. What if the issue is with the CONTENT of the instruction file itself?
The instruction file is downloaded from
#main. If the main branch HEAD has moved between the first and second install, the file content could be different!But the bug report says the issue happens on the "second time" immediately after the first install. It's unlikely the main branch moved in between.
Unless the first install resolved
#mainto one commit and the second install resolves it to a different commit. This could happen if:- The first install cached a stale HEAD
- The second install fetched a fresh HEAD
But the first install used
--update --force, which should have fetched fresh.Hmm, I'm going in circles. Let me look at the actual test cases to understand the expected behavior.
Actually, let me re-read the error message one more time:
Content hash mismatch for <org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md: expected sha256:2227579674ab30b5b59f703ac152fa74ac2912d724d04aeffbf3914cac2e6b88, got sha256:b82d09e4e7879eeb0e90cc7dfeebdac45de6ebdbcdda7824b40438196692f8f8.The expected hash (from lockfile) is
sha256:2227579674ab30b5b59f703ac152fa74ac2912d724d04aeffbf3914cac2e6b88. The actual hash (computed now) issha256:b82d09e4e7879eeb0e90cc7dfeebdac45de6ebdbcdda7824b40438196692f8f8.These are completely different hashes. This means the content is significantly different, not just a minor change.
If the content is the same file, the hash should be the same (or very similar for a small change). Completely different hashes suggest the file is completely different.
But wait,
compute_package_hashhashes the ENTIRE directory content (all files and their paths). If the directory structure is different, the hash would be completely different.What if on the first install, the directory only contained the instruction file, but on the second install, it contains additional files?
Or what if the first install's hash was computed from the parent repo's directory instead of the virtual package's directory?
Hmm, that's an interesting theory. What if there's a bug where the lockfile recorded the hash of the parent repo instead of the virtual package?
Let me look at how
ctx.package_hashes[dep_key]is populated. The key is thedep_key, and the value is the hash of the install path.In
FreshDependencySource.acquire():if install_path.is_dir(): ctx.package_hashes[dep_key] = _compute_hash(install_path)And
dep_key = self.dep_keywhich is passed to the constructor. Thedep_keyshould be the unique key of the dependency.For the instruction file, the
dep_keywould be<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md.And the
install_pathwould be the virtual package's install directory.So the hash should be of the virtual package's content, not the parent repo's content.
Unless there's a bug where
install_pathpoints to the wrong directory...Actually, let me think about this more carefully. For virtual packages, the download process is:
- Clone the parent repo to a temporary or parent directory
- Extract the specified file(s) to the virtual package's install directory
- Compute the hash of the virtual package's install directory
The hash should only include the files in the virtual package's install directory, not the parent repo's directory.
Unless the extraction process copies additional files...
Actually wait, I just realized something. Let me look at the FIRST install log again:
Partial clone (--filter=blob:none) failed for https://dev.azure.com/<org>/tools; retrying with full bare clone. Server may not support filter v2. [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/<project> #default @d479f42e |-- (files unchanged) [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main |-- 1 instruction(s) integrated -> .github/instructions/Notice that the instruction file shows
#mainwithout@<sha>. And then it says "1 instruction(s) integrated".This means the virtual package is being integrated (the instruction file is being copied to
.github/instructions/).Now, what if the integration process creates files in the virtual package's install directory? Like metadata files or something?
No, that doesn't make sense. The integration copies files FROM the install directory TO the project. It shouldn't create files in the install directory.
OK I think I need to actually look at what files are in the virtual package's install directory to understand the issue. Let me look for how virtual packages are created and what files they contain.
Actually, let me try a different approach. Let me search for any code that might write files to the install directory DURING the install process, which could affect the hash.
Actually wait, I think I might have found the issue!
Looking at the
sync_markers_for_lockfilefunction:for dep_key, dep in lockfile.dependencies.items(): if not _is_markable(dep): continue try: install_path = _resolve_install_path(dep, project_root, apm_modules_dir) except Exception: continue if install_path is None or not install_path.exists(): continue write_marker(install_path, dep.resolved_commit)For each markable dependency, it writes
.apm-pintoinstall_path. But_is_markablechecks:def _is_markable(dep): if getattr(dep, "source", None) == "local": return False commit = getattr(dep, "resolved_commit", None) return isinstance(commit, str) and bool(commit)So it only writes
.apm-pinif the dep has a non-emptyresolved_commit.Now, for the virtual instruction file, what is
resolved_commitin the lockfile?If the first install resolved
#mainto a commit SHA, thenresolved_commitwould be that SHA, and.apm-pinwould be written.But the first install log shows
#mainwithout@<sha>. This suggests theresolved_commitmight not have been recorded.Hmm, but the log format shows
@<sha>only ifresolved.resolved_commitis a valid hex string (viaformat_short_sha). If theresolved_commitwas recorded but is "main" or some non-hex value, the log wouldn't show it, but.apm-pinwouldn't be written either.Actually, let me check: what happens when the downloader downloads a virtual package? Does it record the
resolved_commit?For a branch reference like
#main, the downloader resolves it to a commit SHA. This SHA is stored inresolved_reference.resolved_commit.In
FreshDependencySource.acquire():resolved = getattr(package_info, "resolved_reference", None) if resolved: resolved_commit = package_info.resolved_reference.resolved_commitSo
resolved_commitshould be the actual SHA of the commit that#mainpoints to.And then:
ctx.installed_packages.append( InstalledPackage( dep_ref=dep_ref, resolved_commit=resolved_commit, ... ) )So
resolved_commitis stored inctx.installed_packages.And later, when the lockfile is built,
resolved_commitis transferred to the lockfile entry.So the lockfile SHOULD have the
resolved_commitfor the instruction file.But wait, what if the lockfile building process loses the
resolved_commitfor virtual packages?Let me look at how the lockfile is built from
ctx.installed_packages:Actually, I see that
LockFile.from_installed_packagesis used to create the lockfile. Let me check if this properly handlesresolved_commitfor virtual packages.Actually, maybe the issue is simpler. Let me look at the warning message again:
cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commitThis is printed for the PARENT repo. The parent repo might not have a
resolved_commitbecause it's not directly installed (it's a virtual parent). The virtual packages are installed separately.But the warning is printed 3 times. And the instruction files are:
testing.instructions.md #maincode-style.instructions.md #mainprotect-prefixed-agent-files.instructions.md #main
That's exactly 3! So maybe the warning is printed for the instruction files, not the parent repo!
Let me look at the warning code again:
unpinned = find_unpinned_remote_deps(lockfile) if unpinned: for repo in unpinned: _rich_warning(f"cache-pin: remote dep {repo!r} has no resolved_commit; ...")And
find_unpinned_remote_deps:def find_unpinned_remote_deps(lockfile): unpinned: list[str] = [] for _key, dep in lockfile.dependencies.items(): if getattr(dep, "source", None) == "local": continue commit = getattr(dep, "resolved_commit", None) if not isinstance(commit, str) or not commit: unpinned.append(getattr(dep, "repo_url", "") or _key) return unpinnedSo for each dependency without a
resolved_commit, it appendsdep.repo_urlto the list.For the instruction files, the
repo_urlwould be<org>/tools/github-copilot-agent-skills(the parent repo). So even though there are 3 instruction files, they all have the samerepo_url, so the same warning is printed 3 times.But wait, that's 3 instruction files with the same
repo_url. The list would have 3 entries of the same URL. And the loop prints each one, so the warning appears 3 times.OK so this confirms that the instruction files DON'T have a
resolved_commitin the lockfile.So why don't they have a
resolved_commit?On the first install, the downloader resolves
#mainto a commit SHA. This should be stored inresolved_commit. But it's not being recorded in the lockfile.Let me trace through the code to see where
resolved_commitis lost...Actually, looking at the first install log:
[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #mainNo
@<sha>. This meansresolved.resolved_commitis either None, "cached", or some non-hex value.But the instruction file was freshly downloaded (first install), so
resolved_commitshould be a valid SHA.Unless the downloader doesn't properly resolve the ref for virtual packages...
Actually, I think I need to look at how the downloader resolves refs for virtual packages. Maybe virtual packages with branch refs don't get their refs resolved?
Wait, actually I think the issue might be that the virtual package inherits the resolved_commit from the PARENT repo, not from its own resolution.
Let me think about this. When you have:
- Parent repo:
github-copilot-agent-skills - Virtual packages:
/.apm/instructions/testing.instructions.md #main
The parent repo might be resolved first, and then the virtual packages use the parent's resolved commit.
But the warning says the parent repo has no
resolved_commit. So neither does the virtual package.Hmm, but if the virtual package has
#mainas its ref, it should be resolved independently of the parent repo.Unless the resolution logic skips virtual packages...
Actually, let me look at the resolution logic again. In
_resolve_download_strategy:if is_git_source and dep_ref.get_unique_key() not in ctx.pre_downloaded_keys: _has_lockfile_sha = False if update_refs and existing_lockfile: _lck = existing_lockfile.get_dependency(dep_ref.get_unique_key()) _has_lockfile_sha = bool(_lck and _lck.resolved_commit and _lck.resolved_commit != "cached") if dep_ref.reference or (update_refs and _has_lockfile_sha): try: resolved_ref = ctx.downloader.resolve_git_reference(dep_ref) except Exception: passOn the first install with
--update --force:update_refs = Trueexisting_lockfile = None(first install, no lockfile yet)dep_ref.reference = "main"(truthy)
So the condition
dep_ref.reference or (update_refs and _has_lockfile_sha)is True (becausedep_ref.referenceis truthy). The resolver should try to resolve the ref.And
resolve_git_referenceshould resolve#mainto a commit SHA.So the
resolved_refshould have a validresolved_commit.Then in
FreshDependencySource.acquire(), theresolved_commitis extracted frompackage_info.resolved_reference.resolved_commit.Wait, but where does
package_info.resolved_referencecome from? It comes from the downloader'sdownload_packagemethod.Let me check what the downloader returns for virtual packages...
Actually, I think the key insight is: the
resolved_commitin the log might not be displayed because offormat_short_sha. If the SHA is very short (less than 8 chars) or is a sentinel value, it wouldn't be displayed.But a real Git SHA should be 40 chars (SHA-1) or 64 chars (SHA-256). Both are hex strings. So
format_short_shashould return the first 8 chars.Unless the resolved commit is NOT a valid hex string. Like maybe it's the branch name "main" instead of a SHA?
Hmm, let me check what happens when
resolve_git_referencefails or returns a partial result...Actually, I just realized something. Maybe the issue is NOT about
resolved_commitat all. Let me re-read the bug report more carefully.The first install works. The second fails with hash mismatch. The lockfile has a content_hash (otherwise the error wouldn't occur). The fresh download produces a different hash.
The only way the fresh download produces a different hash is if:
- The remote content changed
- The download process produces different output
For option 1, it's unlikely the main branch moved between two consecutive installs.
For option 2, what if the download process includes different files between installs?
What if on the first install, the virtual package's install directory contains ONLY the instruction file. But on the second install, it contains the instruction file PLUS something else (like a
.apm-pinfile that was written from a previous sync)?Wait, but
.apm-pinis excluded from the hash. So that shouldn't matter.Unless the
.apm-pinfile is NOT being excluded properly!Let me look at the exclusion logic one more time:
_EXCLUDED_ROOT_FILES = {_APM_PIN_MARKER} ... if len(rel.parts) == 1 and rel.name in _EXCLUDED_ROOT_FILES: continue_APM_PIN_MARKERis.apm-pin. Andrel.namefor a file at the root should be.apm-pin.Wait, but what if the file is named something slightly different? Like
\.apm-pinwith an escape character? Or.apm-pin\nwith a newline?That seems unlikely. Let me think about other possibilities...
Actually, wait! I just realized something important. The
.apm-pinfile might be written to a DIFFERENT directory than the virtual package's install directory!Let me trace through
sync_markers_for_lockfileagain:for dep_key, dep in lockfile.dependencies.items(): if not _is_markable(dep): continue ... install_path = _resolve_install_path(dep, project_root, apm_modules_dir) ... write_marker(install_path, dep.resolved_commit)_resolve_install_pathcallsdep.to_dependency_ref().get_install_path(apm_modules_dir).And
to_dependency_refreconstructs theDependencyReferencefrom the lockfile entry.Now, what if the lockfile entry for the virtual package has different fields than the original dependency? Like, what if
virtual_pathis not correctly recorded in the lockfile?Let me look at how
virtual_pathis stored in the lockfile:<tool_call>
<tool_call> <function=bash_terminal> <parameter=command> grep -n "virtual_path|is_virtual" /apm/src/apm_cli/deps/lockfile.py | head -30 </tool_call>
- First install: Hash is computed, lockfile is written,
Assistant
Let me search for how
virtual_pathandis_virtualare handled in the lockfile.Assistant
Let me search for tests that involve both
.apm-pinand content hash to understand the expected behavior.Assistant
This is interesting. Let me look at the content hash test that specifically tests
.apm-pinexclusion:Assistant
This test confirms
.apm-pinat the root is excluded from hashing, but NESTED.apm-pinfiles are NOT excluded. Let me look at the skill install test about.apm-pinleaking:Assistant
This is very helpful! I can see there was a similar issue (#1150) where
.apm-pinwas leaking into deploy targets. Let me now focus on understanding the actual bug by looking at howresolved_commitis recorded for virtual packages in the lockfile.Assistant
I see that
LockedDependency.from_dependency_reftakesresolved_commitas a parameter and stores it. Theresolved_commitcomes from theInstalledPackage.resolved_commitwhich is set during the download process.Now let me understand the issue better. Looking at the bug report:
- First install works
- Second install fails with content hash mismatch
The key is that on the second install, the hash computed from the freshly downloaded content doesn't match the hash in the lockfile.
But wait - I need to understand WHY the content would be different between installs. If the same file is downloaded from the same commit, the hash should be the same.
Let me look at this from a different angle. What if the issue is that the
.apm-pinfile IS being included in the hash computation on the second install, even though it should be excluded?Actually, I already verified that
.apm-pinat the root IS excluded. So that's not the issue.Let me think about what else could cause the hash to differ...
Actually, wait. Let me re-read the issue description more carefully:
The first install shows
#mainWITHOUT a commit SHA. And the warning says the repo has noresolved_commit.If the lockfile doesn't have a
resolved_commitfor this dependency, then:_is_markablereturns False- No
.apm-pinis written to the install directory - On the second install,
locked_dep.resolved_commitis falsy
And in
_resolve_download_strategy:if locked_dep and locked_dep.resolved_commit and locked_dep.resolved_commit != "cached": ...If
resolved_commitis None/falsy, this entire block is skipped, andlockfile_matchremains False.Then:
skip_download = install_path.exists() and ( (is_cacheable and not update_refs) or (already_resolved and not update_refs) or lockfile_match )For a branch ref:
is_cacheable = False(branch is not TAG or COMMIT)already_resolved = ?(depends on whether resolver callback pre-downloaded)lockfile_match = False(becauseresolved_commitis falsy)
If
already_resolved = Falseandlockfile_match = False, thenskip_download = False.Then
FreshDependencySourceis used, which downloads fresh and computes a hash.Now, the hash check in
FreshDependencySource:if ( not ctx.update_refs and dep_key not in _expected_hash_deps and dep_locked_chk and dep_locked_chk.content_hash and dep_key in ctx.package_hashes ):This checks if there's a
content_hashin the lockfile. If there is, it compares with the freshly computed hash.But wait - if the first install recorded a
content_hashin the lockfile, and the second install computes a different hash, there must be something different about the content.Let me think about what could be different...
Actually, I just realized something. What if the issue is with how the content_hash is calculated vs how the virtual package content is extracted?
Let me think about what happens for virtual packages:
- The downloader clones the parent repo (or uses a shared clone)
- The specific file is extracted to the virtual package's install directory
- The hash is computed over the install directory
But what if the virtual package's install directory contains MORE than just the specified file? Like, what if there are other files in the same directory in the repo?
Actually, for virtual FILE packages (not subdirectory packages), only the specified file should be extracted. Let me verify this.
Actually wait - let me reconsider. The virtual package path is
.apm/instructions/testing.instructions.md. This is a file in a directory structure. When extracted, what exactly is in the install directory?Looking at
get_install_pathfor virtual files:# Virtual file/collection: use sanitized package name (flattened) package_name = self.get_virtual_package_name()So the install path uses a sanitized package name. The file is extracted to this directory.
Now, what if the DOWNLOADER creates additional files in this directory? Like metadata files?
Actually, I wonder if the issue is related to the
.apm-pinfile being written to the SAME directory that's being hashed, but the exclusion isn't working properly for some reason.Let me look at the exclusion logic more carefully:
_EXCLUDED_ROOT_FILES = {_APM_PIN_MARKER} ... if len(rel.parts) == 1 and rel.name in _EXCLUDED_ROOT_FILES: continue_APM_PIN_MARKERis imported fromcache_pin.MARKER_FILENAMEwhich is.apm-pin.But wait - what if
_APM_PIN_MARKERis evaluated at import time, and the value changes later? No, that's not possible in Python.Actually, let me just focus on the actual issue. The error happens because:
- Fresh download computes hash X
- Lockfile has hash Y
- X != Y
For X to differ from Y, the downloaded content must be different.
What if the issue is that the FIRST install didn't properly record the hash? Like, what if the hash was computed BEFORE all files were extracted?
Or what if the SECOND install downloads different content than the first install?
Actually, I think I should look at this from a completely different angle. What if the issue is NOT about the content being different, but about the hash being computed from the wrong path?
Let me trace through the code path more carefully.
On the first install:
FreshDependencySource.acquire()downloads the package toinstall_path- Hash is computed:
ctx.package_hashes[dep_key] = _compute_hash(install_path) - Lockfile is built with this hash
On the second install:
FreshDependencySource.acquire()downloads the package toinstall_path(same path)- Hash is computed:
ctx.package_hashes[dep_key] = _compute_hash(install_path) - Hash is compared with lockfile
If both installs use the same
install_pathand download the same content, the hash should be the same.UNLESS the second install has
.apm-pinin the directory, and the first install didn't!Wait, but
.apm-pinis excluded. So that shouldn't matter.Hmm, but what if the
.apm-pinfile is NOT being written to the virtual package's install directory? What if it's being written to the PARENT repo's directory?Let me trace through
sync_markers_for_lockfileagain:for dep_key, dep in lockfile.dependencies.items(): if not _is_markable(dep): continue install_path = _resolve_install_path(dep, project_root, apm_modules_dir) write_marker(install_path, dep.resolved_commit)For the instruction file dependency:
_is_markable(dep)checks ifdep.resolved_commitis truthy- If the lockfile doesn't have a
resolved_commitfor this dep,_is_markablereturns False - No
.apm-pinis written
But the warning says the PARENT repo has no
resolved_commit, not the individual file. So the individual file might have aresolved_commit.Wait, the warning is:
cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commitThis is
dep.repo_url, which is the parent repo URL. Multiple dependencies (the 3 instruction files) could have the samerepo_url. So the warning is printed once for each dependency that doesn't have aresolved_commit.This means all 3 instruction files DON'T have a
resolved_commitin the lockfile.So
_is_markablereturns False for all 3 instruction files, and no.apm-pinis written to their install directories.So the
.apm-pintheory is ruled out for the instruction files.But then what causes the hash mismatch?
Let me think about this more carefully. The first install:
- Downloads fresh
- Computes hash
- Lockfile is written with hash
The second install:
skip_download = False(becauselockfile_match = False)- Downloads fresh again
- Computes hash
- Hash doesn't match lockfile
If the content is the same, the hash should be the same. So the content must be different.
What if the content is different because of how the virtual package is extracted?
Let me look at how virtual packages are downloaded. The downloader clones the parent repo and then extracts the specific file(s).
What if the extraction process creates different files between installs?
Actually, let me look at the downloader code for virtual packages...
Actually, I think I should just focus on finding the root cause by looking at what could cause different content between installs.
One possibility: The downloader creates a
.apm.ymlor other metadata file in the virtual package's install directory. If this file is created differently between installs, the hash would differ.Let me search for where metadata files might be created in the install directory:
Actually wait, I just had another thought. What if the issue is that the lockfile was written with the hash from a DIFFERENT dependency?
Like, what if the first install computed the hash for the parent repo but stored it under the instruction file's dep_key?
That seems unlikely but let me check...
Actually, let me look at the test case that tests this scenario. There should be tests for virtual packages with content hashes.
Let me search for tests related to virtual packages and content hash:
Actually, I think I should look at the actual tests for the content hash mismatch scenario to understand the expected behavior.
Let me look at the test that was mentioned:
/apm/tests/unit/install/test_sources_phase3.py:766: """Content hash mismatch → sys.exit(1)."""I already looked at this test. It confirms that when the hash doesn't match,
sys.exit(1)is called.But I need to understand WHY the hash doesn't match in the bug scenario.
Let me think about this more systematically. The hash is computed from:
- All file paths in the directory
- All file contents
For the hash to differ, either the file paths or the file contents must differ.
On the first install:
- Directory contains: instruction file
- Hash is computed
On the second install:
- Directory contains: instruction file (and possibly .apm-pin)
- Hash is computed
If
.apm-pinis excluded, the hash should be the same.But what if there's ANOTHER file that's different between installs?
What if the downloader creates a
.gitdirectory or some other metadata on the second install?Actually,
.gitis excluded from the hash computation:_EXCLUDED_DIRS = {".git", "__pycache__"}So
.gitwouldn't affect the hash.Hmm, I'm running out of ideas. Let me look at the issue from a different perspective.
Actually, wait. Let me re-read the bug report more carefully:
> apm install --update --force [>] Installing dependencies from apm.yml...And then the second install:
> apm install [>] Installing dependencies from apm.yml...Notice that on the second install, the dependencies show
(cached)for some but not all:[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/<project> @d479f42e (cached) |-- (files unchanged) [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/myorg @d479f42e (cached) |-- (files unchanged) [+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main Content hash mismatch for ...The packages with
#default @d479f42eshow(cached)- they usedCachedDependencySource.But the instruction file with
#maindoes NOT show(cached)- it usedFreshDependencySource.This confirms my earlier analysis:
- Package with commit SHA (
@d479f42e):is_cacheable = True,skip_download = True, usesCachedDependencySource - Package with branch ref (
#main):is_cacheable = False,skip_download = False, usesFreshDependencySource
So the instruction file is being downloaded fresh on the second install.
Now, the question is: why does the fresh download produce a different hash than what's in the lockfile?
The lockfile hash was computed during the FIRST install. The fresh download hash is computed during the SECOND install.
If the content is the same, the hash should be the same.
So the content must be different.
What could cause the content to be different?
- The remote file changed between installs
- The download process creates different output
- Some files were added/removed between installs
For #1, it's unlikely the file changed between two consecutive installs.
For #2, what if the downloader creates additional metadata files on the second install that weren't created on the first install?
For #3, what if the
.apm-pinfile from a PREVIOUS sync is included in the hash?Wait, actually, let me think about #3 more carefully. The
.apm-pinfile is written AFTER the lockfile is written. So on the first install:- Download content
- Compute hash (no .apm-pin yet)
- Write lockfile with hash
- Write .apm-pin
On the second install:
- Check cache (cache exists with .apm-pin)
- Hash verification in
_resolve_download_strategy(excludes .apm-pin) - If verification fails, re-download
- After re-download, compute hash (excludes .apm-pin)
- Compare with lockfile hash
But wait, on the second install, the cache verification might pass (because .apm-pin is excluded), and the dependency might use
CachedDependencySourceinstead ofFreshDependencySource.But the logs show that the instruction file uses
FreshDependencySource(no(cached)suffix). So the cache verification must have failed, orskip_download = Falsefor some other reason.Actually, I think the issue is that
lockfile_match = Falsebecauseresolved_commitis falsy. Andalready_resolved = Falsebecause the resolver callback didn't pre-download it. Andis_cacheable = Falsebecause it's a branch ref.So
skip_download = False, andFreshDependencySourceis used.Now, the fresh download happens. The hash is computed. And the hash doesn't match the lockfile.
But why would the hash not match if the content is the same?
Unless... the first install recorded the hash of the WRONG directory!
What if the first install computed the hash of the PARENT repo's directory instead of the virtual package's directory?
Let me trace through the code path for the first install:
In
FreshDependencySource.acquire():if install_path.is_dir(): ctx.package_hashes[dep_key] = _compute_hash(install_path)install_pathis the virtual package's install directory. So the hash should be of the virtual package, not the parent repo.Unless
install_pathis wrong...Actually, let me look at what
install_pathis for virtual packages. Inintegrate.py:install_path = dep_ref.get_install_path(apm_modules_dir)This calls
get_install_pathon the dependency reference. For virtual packages, this returns a specific path based on the sanitized package name.So the hash should be computed from the correct directory.
Hmm, I'm really stuck. Let me try to create a reproduction script to understand the issue better.
Actually, wait. Let me re-read the bug description one more time:
The first install:
apm install --update --forceThe second install:apm installOn the first install,
--update --forceis used.--forcemight force re-download of everything, including things that would normally be cached.But on the second install,
--forceis NOT used. So cached things should be used.But the instruction file is NOT cached because it's a branch ref. So it's downloaded fresh.
And the hash doesn't match the lockfile.
OK so the question is: what's in the lockfile that's different from what's downloaded?
Wait, let me look at the lockfile more carefully. The lockfile has a
content_hashfield. This hash was computed during the FIRST install.On the FIRST install, the hash was computed after downloading. So it should be the hash of the downloaded content.
On the SECOND install, the hash is computed after downloading again. If the download produces the same content, the hash should be the same.
So either:
- The download produces different content
- The hash is computed differently
For #1, the download might include different files between installs. For #2, the hash computation might include/exclude different files between installs.
For #2, what if on the first install, the hash was computed BEFORE the
.apm-pinwas written (which is the case), and on the second install, the hash is computed AFTER the.apm-pinwas already present (from a previous sync)?Wait, but
.apm-pinis excluded. So it shouldn't matter.Unless the exclusion logic is different between installs!
Actually, I just realized something. The hash is computed in
FreshDependencySource.acquire()which is called during the integrate phase. The.apm-pinmarkers are written insync_markers_for_lockfilewhich is called in the lockfile phase.The integrate phase runs BEFORE the lockfile phase. So on the first install:
- Integrate phase: Download, compute hash (no .apm-pin)
- Lockfile phase: Write lockfile, write .apm-pin
On the second install:
- Integrate phase: Download, compute hash (.apm-pin exists but is excluded)
- Lockfile phase: Write lockfile, write .apm-pin
So on the second install, the hash should still exclude
.apm-pin(because it's in_EXCLUDED_ROOT_FILES).So the hash should be the same.
UNLESS there's a bug where
.apm-pinis NOT being excluded on the second install.Wait, what if the issue is that the
.apm-pinfile is written to a DIFFERENT directory than the virtual package's install directory?Let me trace through
sync_markers_for_lockfileone more time:for dep_key, dep in lockfile.dependencies.items(): if not _is_markable(dep): continue install_path = _resolve_install_path(dep, project_root, apm_modules_dir) write_marker(install_path, dep.resolved_commit)For the instruction file dep,
_is_markable(dep)checks ifdep.resolved_commitis truthy.If
dep.resolved_commitis None or empty,_is_markablereturns False, and no.apm-pinis written.But the warning says the instruction file deps don't have
resolved_commit. So no.apm-pinshould be written to their directories.But wait, the PARENT repo might have a
resolved_commit! Let me check...Actually, looking at the first install log:
[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/<project> #default @d479f42eThe packages with
#defaulthave@d479f42e. So they have aresolved_commit.But the instruction files with
#maindon't have@<sha>. So they might not have aresolved_commit.And the warning says the PARENT repo
github-copilot-agent-skillshas noresolved_commit. But the parent repo might be a SEPARATE dependency in the lockfile, not the same as the virtual file dependencies.Actually, I wonder if the parent repo IS in the lockfile. Let me check what dependencies are in the lockfile...
The lockfile entries are created from
ctx.installed_packages. Andctx.installed_packagescontains entries for all installed dependencies.For virtual packages, the install process downloads the parent repo and extracts the specific files. But is the parent repo also added to
ctx.installed_packages?Looking at the code, I don't think the parent repo is added separately. The virtual packages are added with their own
dep_key(which includes the virtual path).So the lockfile has entries for the virtual packages (instruction files), not for the parent repo.
But the warning says "remote dep '/tools/github-copilot-agent-skills' has no resolved_commit". This is the
repo_urlof the virtual packages. So the virtual packages themselves don't have aresolved_commit.OK so this confirms:
- Virtual packages (instruction files) don't have
resolved_commitin the lockfile _is_markablereturns False for them- No
.apm-pinis written to their directories
So the
.apm-pintheory is ruled out.But then what causes the hash mismatch?
Let me think about this differently. What if the issue is NOT about the files in the directory, but about the PATH NAMES in the hash computation?
The hash computation includes the relative path of each file. If the file is named differently between installs, the hash would differ.
But the file name is determined by the dependency reference. It should be the same between installs.
Unless there's a bug in how the virtual package name is sanitized...
Actually, let me look at
get_virtual_package_name:<tool_call>
<tool_call> <function=bash_terminal> <parameter=command> grep -n "get_virtual_package_name" /apm/src/apm_cli/models/dependency/reference.py </tool_call>
Assistant
I need to search for the
get_virtual_package_namemethod to understand how virtual packages get their names, which could affect the install path and hash computation.Assistant
Let me think carefully about the issue:
-
The bug is that the second
apm installfails with "Content hash mismatch" for virtual package files liketesting.instructions.md #main. -
The first install (
--update --force) succeeds, records a hash in the lockfile. -
The second install (
apm install) - the dependency has a branch ref#main, so:is_cacheable = False(branches aren't cacheable)lockfile_match = Falsebecauseresolved_commitis None/empty/falsy (the warning says the repo has noresolved_commit)- So
skip_download = FalseandFreshDependencySourceis used
-
The fresh download happens, hash is computed, and it doesn't match the lockfile hash.
The key question is: why does the hash differ between the first and second install?
Looking at the flow:
- First install: Hash is computed AFTER download, BEFORE
.apm-pinis written - Second install: Hash is computed AFTER fresh download
The
.apm-pinfile is excluded from hash computation. So that's not the issue.But wait - let me reconsider. The
.apm-pinfile is excluded from hash computation at the ROOT of the package. But what if.apm-pinis being written to a DIFFERENT path?Actually, I think I need to look more carefully at what happens when
resolved_commitis None.Looking at
_resolve_download_strategy:if locked_dep and locked_dep.resolved_commit and locked_dep.resolved_commit != "cached": ... if _should_skip_redownload(locked_dep, install_path): lockfile_match = TrueIf
resolved_commitis None, this block is SKIPPED entirely.lockfile_matchremains False.Then
skip_download = False(becauseis_cacheable = False,already_resolved = False,lockfile_match = False).So
FreshDependencySourceis used. Fresh download happens. Hash is computed.Now, the check in
FreshDependencySource:if ( not ctx.update_refs and dep_key not in _expected_hash_deps and dep_locked_chk and dep_locked_chk.content_hash and dep_key in ctx.package_hashes ): _fresh_hash = ctx.package_hashes[dep_key] if _fresh_hash != dep_locked_chk.content_hash: ... sys.exit(1)So the issue is that
_fresh_hash != dep_locked_chk.content_hash.The question is: why?
Let me think about what could cause the hash to differ:
- The remote content changed - unlikely between two consecutive installs
- The download process creates different output - possible if there's a bug
- The hash is computed differently - possible if there's a bug in exclusion logic
- Something was added to the install directory between the first and second install
For #4, the
.apm-pinfile is a candidate. But it's excluded. Unless it's NOT being excluded properly.Wait - let me look at the exclusion more carefully:
_EXCLUDED_ROOT_FILES = {_APM_PIN_MARKER}Where
_APM_PIN_MARKERis imported fromcache_pin.MARKER_FILENAMEwhich is.apm-pin.But wait - what if there's ANOTHER marker file? What if the PARENT repo has a
.apm-pinfile that's being copied to the virtual package directory?Actually no, that doesn't make sense. The virtual package directory is separate.
Hmm, let me think about this differently. What if the issue is that the lockfile doesn't have a
resolved_commitfor the virtual package, and this causes some other issue?Looking at the
CachedDependencySource._resolve_cached_commit():elif ctx.existing_lockfile: locked_dep = ctx.existing_lockfile.get_dependency(dep_key) if locked_dep and locked_dep.resolved_commit and locked_dep.resolved_commit != "cached": cached_commit = locked_dep.resolved_commit if not cached_commit: cached_commit = dep_ref.referenceIf
resolved_commitis not in the lockfile,cached_commitfalls back todep_ref.reference, which is the branch name (like "main" or "#main").But this is for
CachedDependencySource, notFreshDependencySource.Actually, wait. I think I finally see the issue!
Looking at
_resolve_download_strategy, there's this check:if skip_download and _dep_locked_chk and _dep_locked_chk.content_hash: from apm_cli.utils.content_hash import verify_package_hash if not verify_package_hash(install_path, _dep_locked_chk.content_hash): _hash_msg = f"Content hash mismatch for {dep_ref.get_unique_key()} -- re-downloading" diagnostics.warn(_hash_msg, package=dep_ref.get_unique_key()) if logger: logger.progress(_hash_msg) safe_rmtree(install_path, ctx.apm_modules_dir) skip_download = FalseThis checks if the CACHED content matches the lockfile hash. If not, it removes the cache and forces a re-download.
But this check is ONLY done when
skip_download = True. Andskip_downloadrequireslockfile_match = True(among other things).But
lockfile_matchis only set to True via_should_skip_redownload, which requireslocked_dep.resolved_committo be truthy!So for virtual packages without
resolved_commit:lockfile_match = Falseskip_download = False- The cache verification check is NEVER reached
FreshDependencySourceis used- Fresh download happens
- Hash is computed
- Hash is compared with lockfile
So the issue is in step 7: the hash doesn't match.
But why would the hash not match if the same content is downloaded?
Unless... the FIRST install recorded a DIFFERENT hash than what should have been recorded.
Wait, let me look at the flow more carefully. On the first install with
--update --force:update_refs = True- Dependencies are downloaded fresh
- Hashes are computed
- Lockfile is written
So the lockfile has the correct hash from the first install.
On the second install (without
--update):update_refs = False- For virtual packages with
#main:is_cacheable = Falselocked_dep.resolved_commitis Nonelockfile_match = Falseskip_download = False
- Fresh download happens
- Hash is computed
- Hash comparison: FAILS
So the question is: why does the hash in step 4 differ from the hash in the lockfile (from step 4 of the first install)?
The only way this can happen is if:
- The downloaded content is different
- The hash computation is different
For the content to be different, the remote file must have changed. But the bug says it happens immediately on the second run.
For the hash computation to be different, there must be a bug in the computation.
Actually, wait! I just realized something. What if the lockfile doesn't have the hash for the VIRTUAL PACKAGE dependency? What if the hash was recorded for a different dependency?
Let me check the lockfile building code:
def _attach_content_hashes(self, lockfile: LockFile) -> None: for dep_key, locked_dep in lockfile.dependencies.items(): if dep_key in self.ctx.package_hashes: locked_dep.content_hash = self.ctx.package_hashes[dep_key]This attaches the hash if the
dep_keyis inctx.package_hashes. Thedep_keymust match exactly.And
ctx.package_hashesis populated inFreshDependencySource.acquire():if install_path.is_dir(): ctx.package_hashes[dep_key] = _compute_hash(install_path)Where
dep_key = self.dep_keywhich is passed to the constructor.So the hash is stored under the correct
dep_key.Unless there's a mismatch in how
dep_keyis computed between the first and second install...Actually,
dep_key = dep_ref.get_unique_key(). Let me check if this could be different between installs.Hmm, let me look at
get_unique_key():Actually wait - I think I need to look at this from a completely different angle. Let me check what happens when
resolved_commitis None in the lockfile and the dependency is re-downloaded.Actually, I think the issue might be simpler. Let me check if the lockfile has a
content_hashfor the virtual package dependency.The lockfile is built from
ctx.installed_packages. Andctx.installed_packageshas entries for each installed dependency. Thecontent_hashis attached separately via_attach_content_hashes.If the virtual package was installed, it should be in
ctx.installed_packages, and the hash should be inctx.package_hashes.But what if the virtual package wasn't installed on the first install? What if it was skipped for some reason?
Looking at the first install log:
[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #main |-- 1 instruction(s) integrated -> .github/instructions/So it WAS installed. And "1 instruction(s) integrated" confirms the integration happened.
OK so the virtual package was installed, and the hash should have been recorded.
Let me think about this more carefully. On the first install, the hash is computed and recorded. On the second install, a fresh download happens, and the hash is computed again.
For the hashes to differ:
- Different content is downloaded
- Different hash computation
For #1, what if the downloader extracts different files? What if on the second install, it extracts the file AND some metadata files?
Actually, let me look at how virtual packages are downloaded. The downloader might:
- Clone the parent repo to a temp location
- Copy the specific file to the virtual package's install directory
- Clean up
If step 2 copies additional files (like
.apm-pinfrom the parent repo), the hash would differ.But
.apm-pinis excluded from the hash computation...Wait, what if
.apm-pinis NOT at the root of the virtual package directory? What if it's in a subdirectory?The exclusion logic is:
if len(rel.parts) == 1 and rel.name in _EXCLUDED_ROOT_FILES: continueThis only excludes
.apm-pinat the ROOT (depth 1). If.apm-pinis in a subdirectory (depth > 1), it would NOT be excluded!So if the downloader copies
.apm-pinto a subdirectory of the virtual package, it would be included in the hash!But wait, the downloader shouldn't copy
.apm-pin. It should only copy the specified file.Unless the downloader copies the ENTIRE parent directory instead of just the file...
Actually, for virtual file packages, the downloader should only copy the specified file. Let me verify this.
Actually, I think I need to understand the virtual package extraction logic better. Let me search for how virtual packages are extracted.
Actually, I think the real issue is about what happens when there's no
resolved_commitin the lockfile. Let me look at this scenario more carefully.When
resolved_commitis None in the lockfile:_is_markablereturns False - no.apm-pinmarker written_resolve_download_strategyskips thelockfile_matchcheckskip_download = False- fresh download happens- Hash is computed and compared with lockfile
The issue is in step 4. The hash doesn't match.
But why? If the same content is downloaded, the hash should be the same.
Unless... the FIRST install recorded a different hash!
What if the first install's hash was computed BEFORE the download was complete? Or what if the hash was computed from a different directory?
Actually, let me think about this more carefully. On the first install with
--update --force:update_refs = Truedep_ref.reference = "main"(truthy)resolved_ref = ctx.downloader.resolve_git_reference(dep_ref)- resolves to a commit SHAFreshDependencySourcedownloads the package- Hash is computed:
ctx.package_hashes[dep_key] = _compute_hash(install_path) - Lockfile is written with this hash
So the hash should be correct.
On the second install:
update_refs = Falsedep_ref.reference = "main"(truthy)resolved_ref = ctx.downloader.resolve_git_reference(dep_ref)- resolves to a commit SHAis_cacheable = False(branch ref)locked_dep.resolved_commit- what is this?
Here's the key question: What is
locked_dep.resolved_commitfor the instruction file dependency?On the first install, the
resolved_commitwas set to the SHA of the commit that#mainresolved to. This should have been recorded in the lockfile.But the warning says the repo has no
resolved_commit. This suggests that theresolved_commitwas NOT recorded in the lockfile.Why would
resolved_commitnot be recorded?Looking at
FreshDependencySource.acquire():resolved = getattr(package_info, "resolved_reference", None) if resolved: resolved_commit = package_info.resolved_reference.resolved_commit ... ctx.installed_packages.append( InstalledPackage( dep_ref=dep_ref, resolved_commit=resolved_commit, ... ) )So
resolved_commitcomes frompackage_info.resolved_reference.resolved_commit. This should be set by the downloader.If the downloader doesn't set
resolved_reference.resolved_commit, thenresolved_commitwould be None.And then the lockfile would have
resolved_commit = None.But wait, the log shows:
[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md #mainNo
@<sha>suffix. This suggestsresolved.resolved_commitis not a valid hex string.But if the downloader resolved
#mainto a commit SHA, it should be a valid hex string!Unless the downloader doesn't resolve the ref for virtual packages...
Hmm, let me look at how the downloader resolves refs for virtual packages.
Actually, I think the issue might be that for virtual packages, the download happens differently. Instead of cloning the repo and checking out the ref, the downloader might:
- Use a shared clone of the parent repo
- Extract the specific file from the shared clone
In this case, the
resolved_referencemight not be set for the virtual package itself. It might inherit from the parent.If the parent repo doesn't have a
resolved_commit, then the virtual package wouldn't either.But the log shows that some packages DO have a
resolved_commit:[+] dev.azure.com/<org>/tools/github-copilot-agent-skills/packages/<project> #default @d479f42eThese are packages, not virtual files. They have
#defaultas their ref and@d479f42eas their resolved commit.But the virtual files have
#mainand no resolved commit.This suggests that packages and virtual files are handled differently. Packages get their refs resolved and recorded. Virtual files don't.
Actually, let me look at this more carefully. The packages with
#defaultmight use a different download path than the virtual files with#main.For packages (
/packages/<project>), the download might clone the subdirectory. For virtual files (/.apm/instructions/testing.instructions.md), the download might just extract the file from a shared clone.In the second case, the
resolved_referencemight not be set because the file is just copied from the shared clone.So
resolved_commitwould be None for virtual files, and the lockfile wouldn't have it.And this causes the issue:
locked_dep.resolved_commit = Nonelockfile_match = False(check skipped)skip_download = False- Fresh download
- Hash mismatch
But why hash mismatch? If the same file is extracted from the same shared clone, the hash should be the same.
Unless the shared clone is different between installs!
On the first install with
--update --force, the shared clone is fetched fresh. The file is extracted. Hash is recorded.On the second install, the shared clone might be STALE. But since
update_refs = False, the shared clone is not updated. The file is extracted from the stale clone.But wait, if the shared clone is stale (same as before), the extracted file should be the same. So the hash should be the same.
Unless... the shared clone was UPDATED between the first and second install (by some other process), and the extracted file is different.
But the bug says the issue happens immediately on the second run. So there shouldn't be time for an external update.
Hmm, I'm really stuck. Let me try a different approach.
Let me look at the issue from the perspective of the
_resolve_download_strategyfunction. The key is understanding whenskip_download = Truevsskip_download = False.For the virtual instruction file:
is_cacheable = False(branch ref)already_resolved- is the dep inctx.callback_downloaded?lockfile_match- depends onresolved_commitbeing truthy
For
skip_downloadto be True, one of these must be True:(is_cacheable and not update_refs)=(False and True)= False(already_resolved and not update_refs)=(? and True)= dependslockfile_match= False (becauseresolved_commitis falsy)
So
skip_downloadis True only ifalready_resolved = True.Is
already_resolved = True? Let me check:already_resolved = dep_ref.get_unique_key() in ctx.callback_downloadedctx.callback_downloadedis populated by the resolver's BFS download callback. If the resolver downloaded this dependency, it would be in the set.For a branch ref like
#main, the resolver might or might not pre-download the dependency. Let me check the resolver logic.Actually, I think the issue might be in the resolver logic. For branch refs, the resolver might skip pre-downloading if the cache exists.
But even if
already_resolved = True, the content should be the same as the first install (if the remote hasn't changed).Wait, unless the resolver downloaded a DIFFERENT version of the file!
On the first install with
--update --force, the resolver fetched the latest#main. On the second install, the resolver might fetch a stale version from the cache.But if the resolver fetched the same version, the hash should be the same.
Unless the resolver fetched a different version...
Actually, I think I need to understand the resolver's cache logic for branch refs.
Hmm, this is getting too complex. Let me take a step back and think about what the fix should be.
The issue is that for virtual packages with branch refs:
resolved_commitis not recorded in the lockfilelockfile_matchcannot be computed (requiresresolved_commit)- Fresh download happens on every install
- Hash check fails because... something is different
The fix should ensure that either:
- The hash check passes (content is the same)
- The hash check is skipped (no need to verify)
- The cache is used (no fresh download)
Option 3 would require
resolved_committo be recorded. But the warning says it's not.Option 2 would require
dep_locked_chk.content_hashto be None. But the error shows it's not.Option 1 would require the content to be the same. But the error shows it's not.
So the issue must be in the content being different. But why?
Actually, wait. Let me re-read the error message one more time:
Content hash mismatch for <org>/tools/github-copilot-agent-skills/.apm/instructions/testing.instructions.md: expected sha256:2227579674ab30b5b59f703ac152fa74ac2912d724d04aeffbf3914cac2e6b88, got sha256:b82d09e4e7879eeb0e90cc7dfeebdac45de6ebdbcdda7824b40438196692f8f8.These are TWO COMPLETELY DIFFERENT hashes. If it was just a minor content change (like a newline difference), the hashes would be different but not completely unrelated.
But these hashes are completely different. This suggests that the CONTENT is completely different.
What if the second install downloaded the file from a DIFFERENT branch or commit?
Or what if the second install downloaded the file from a DIFFERENT directory structure?
Actually, I just had a thought. What if the issue is that the SECOND install downloads a DIFFERENT set of files than the FIRST install?
On the first install with
--update --force, maybe the downloader:- Clones the entire repo
- Extracts the file to the virtual package directory
- The virtual package directory contains ONLY the instruction file
On the second install, maybe the downloader:
- Uses a cached/bare clone
- Extracts the file to the virtual package directory
- The virtual package directory contains the instruction file AND something else
What could the "something else" be?
Actually, wait. What if the issue is that the SECOND install uses
CachedDependencySourcefor the PARENT repo, but the virtual package is extracted from the cached parent repo which has.apm-pinin it?No, that doesn't make sense. The virtual package directory is separate from the parent repo directory.
Hmm, let me look at how the virtual package directory is created and what files it contains.
Actually, I wonder if the issue is related to how the bare clone cache works. On the first install, the bare clone might be created fresh. On the second install, the bare clone is reused.
If the bare clone has
.apm-pinor other metadata, and this metadata is somehow included in the virtual package extraction, the hash would differ.But the
.apm-pinfile is excluded from the hash. So this shouldn't matter.Unless the exclusion isn't working for some reason...
OK let me try a completely different approach. Let me look at the tests for this specific scenario and see if there's a clue.
Actually, I just realized something. The bug report mentions that the first install is
apm install --update --force. The--forceflag might force re-download of everything.But on the second install, there's no
--force. So the resolver might use cached content.What if the cached content for the parent repo has
.apm-pinin it (written during the first install)? And the virtual package extraction copies the file FROM this cached content?If the extraction copies the file directly, the virtual package directory should only contain the instruction file. The
.apm-pinfrom the parent repo shouldn't be copied.Unless the extraction copies the ENTIRE directory instead of just the file...
Actually, I think I should look at how virtual packages are extracted from the shared clone. Let me search for the extraction logic.
Let me search for "is_virtual" in the downloader to understand how virtual packages are handled.
Let me trace through how virtual packages are downloaded. I need to find the downloader code that handles virtual packages.
<tool_call> <function=bash_terminal> <parameter=command> grep -rn "is_virtual|virtual_path" /apm/src/apm_cli/install/phases/download.py | head -30 </tool_call>
-
Assistant
I need to look at how virtual packages are downloaded to understand why the hash might differ between installs.
Assistant
Let me look at the downloader implementation to understand how virtual packages are downloaded.
Assistant
Let me search more broadly for how virtual packages are handled in the download process.
Assistant
Let me look at the github_downloader.py to understand how virtual file packages are downloaded, particularly the
_download_virtual_filemethod.Assistant
Now I can see the key issue! Let me continue reading to understand what files are created in the virtual package directory.
Assistant
Let me look at the
yaml_to_strfunction to check if there's any non-determinism that could cause differentapm.ymlcontent between installs:Assistant
...
Assistant
Looking at this, I can see the issue! The
_download_virtual_filemethod creates anapm.ymlfile with YAML serialization. Let me check if there's any non-determinism in the YAML output.Assistant
Let me look at how the commit SHA is resolved for virtual files, since this might explain why the content differs between installs.
Assistant
...
Assistant
Now I'm starting to understand the flow better. For virtual file packages (like instruction files), the downloader:
- Resolves the ref (e.g.,
#main) to a commit SHA - Downloads the file content
- Creates a directory structure with
.apm/instructions/<filename>andapm.yml - Returns a
PackageInfowith the resolved commit
The key insight is that the
resolved_commitIS resolved during the download. Let me check if this resolved commit is properly propagated to the lockfile.Let me look at the issue more carefully. The bug says:
- First install (
--update --force) succeeds - Second install fails with "Content hash mismatch"
The error happens because the hash of the freshly downloaded content doesn't match the lockfile.
Looking at the
_download_virtual_filemethod, I can see that:- It creates an
apm.ymlfile withyaml_to_str - It creates a
.apm/instructions/<filename>file
The
apm.ymlcontent is generated dynamically. If the YAML serialization is non-deterministic (e.g., key ordering), the content could differ between installs.Let me check the
_DUMP_DEFAULTS:_DUMP_DEFAULTS: dict[str, Any] = dict( default_flow_style=False, sort_keys=False, # <-- NOT sorting keys! allow_unicode=True, )sort_keys=Falsemeans the YAML keys are NOT sorted. And Python dicts preserve insertion order. So the order of keys should be deterministic...But wait, the dict is created in this order:
apm_yml_data = { "name": package_name, "version": "1.0.0", "description": description, "author": dep_ref.repo_url.split("/")[0], }This should be deterministic. So the YAML content should be the same between installs.
Hmm, but wait! Let me check the
descriptionfield. It's extracted from the file's frontmatter:description = f"Virtual package containing {filename}" try: content_str = file_content.decode("utf-8") # Simple frontmatter parsing (YAML between --- markers) if content_str.startswith("---\n"): ... for line in frontmatter.split("\n"): if line.startswith("description:"): description = line.split(":", 1)[1].strip().strip("\"'") break except Exception: passIf the frontmatter parsing succeeds, the
descriptionwould be the value from the file. If it fails, it would be"Virtual package containing {filename}".This should be deterministic for the same file content.
So the YAML content should be the same between installs...
Actually, let me think about this differently. What if the issue is with the
.apm-pinfile being included in the hash?Wait, I already established that
.apm-pinat the root is excluded. But let me check if the virtual package directory has a.apmsubdirectory:Looking at the code:
apm_dir = target_path / ".apm" / subdir apm_dir.mkdir(parents=True, exist_ok=True)So the virtual package directory structure is:
<target_path>/ apm.yml .apm/ instructions/ testing.instructions.mdNow, if
.apm-pinis written to<target_path>/.apm-pin, it would be at the root and excluded.But what if
.apm-pinis written to<target_path>/.apm/? That would NOT be at the root, so it would NOT be excluded!Let me check where
sync_markers_for_lockfilewrites the marker:install_path = _resolve_install_path(dep, project_root, apm_modules_dir) write_marker(install_path, dep.resolved_commit)And
write_marker:marker = install_path / MARKER_FILENAME marker.write_text(json.dumps(payload), encoding="utf-8")So
.apm-pinis written toinstall_path / ".apm-pin", which is at the root of the package.But wait, what if
_resolve_install_pathreturns a DIFFERENT path than expected?Let me check
_resolve_install_path:def _resolve_install_path(dep, project_root, apm_modules_dir): dep_ref = dep.to_dependency_ref() return dep_ref.get_install_path(apm_modules_dir)And
to_dependency_ref:return DependencyReference( repo_url=self.repo_url, ... virtual_path=self.virtual_path, is_virtual=self.is_virtual, ... )So it reconstructs the
DependencyReferencefrom the lockfile entry, using the samevirtual_pathandis_virtual.And
get_install_pathshould return the same path as during the download.So
.apm-pinshould be written to the correct location (at the root), and it should be excluded from the hash.Hmm, but wait! I just realized something. The
_EXCLUDED_DIRSin the hash computation includes.gitbut NOT.apm:_EXCLUDED_DIRS = {".git", "__pycache__"}So the
.apmdirectory is NOT excluded! The files inside.apm/instructions/are included in the hash.This is correct - the
.apm/instructions/<file>IS part of the package content and should be hashed.OK so the
.apmdirectory is included in the hash. This is expected.Now, what if
.apm-pinis written to<target_path>/.apm/.apm-pin(inside the.apmsubdirectory)?Looking at
write_marker:marker = install_path / MARKER_FILENAMEThis is
install_path / ".apm-pin", notinstall_path / ".apm" / ".apm-pin".So
.apm-pinis at the root, not inside.apm/.OK so the
.apm-pintheory is still not explaining the issue.Let me think about this more carefully. The hash is computed over:
apm.ymlat the root.apm/instructions/<filename>
Both of these should be the same between installs (same content).
Unless... the
descriptioninapm.ymlis different!The
descriptionis extracted from the file's frontmatter. If the frontmatter parsing is non-deterministic (e.g., due to different encoding or line endings), the description could differ.But this seems unlikely.
Actually, let me look at the issue from yet another angle. The bug report shows the first install log:
cache-pin: remote dep '<org>/tools/github-copilot-agent-skills' has no resolved_commit; drift cannot verify its cache freshness.This warning is printed 3 times. And there are exactly 3 instruction files with
#main.So the 3 instruction files don't have
resolved_commitin the lockfile.But wait, the
_download_virtual_filemethod DOES resolve the commit SHA:resolved_commit = self._resolve_commit_sha_for_ref(dep_ref, ref)And it creates a
ResolvedReferencewith this commit:resolved_ref = ResolvedReference( original_ref=str(dep_ref.reference) if dep_ref.reference else ref, ref_name=ref, ref_type=ref_type, resolved_commit=resolved_commit, )So the
resolved_commitSHOULD be set in thePackageInfo.resolved_reference.And in
FreshDependencySource.acquire():resolved = getattr(package_info, "resolved_reference", None) if resolved: resolved_commit = package_info.resolved_reference.resolved_commitSo
resolved_commitshould be extracted from thePackageInfo.And then:
ctx.installed_packages.append( InstalledPackage( dep_ref=dep_ref, resolved_commit=resolved_commit, ... ) )So
resolved_commitshould be in theInstalledPackage.And then the lockfile is built from
InstalledPackage:lockfile = _LF.from_installed_packages(self.ctx.installed_packages, ...)And in
from_installed_packages, theresolved_commitis transferred toLockedDependency:LockedDependency.from_dependency_ref( dep_ref, resolved_commit, depth, resolved_by, ... )So the
resolved_commitSHOULD be in the lockfile!But the warning says it's not. Why?
Oh wait! The warning is about
find_unpinned_remote_deps:def find_unpinned_remote_deps(lockfile): unpinned: list[str] = [] for _key, dep in lockfile.dependencies.items(): if getattr(dep, "source", None) == "local": continue commit = getattr(dep, "resolved_commit", None) if not isinstance(commit, str) or not commit: unpinned.append(getattr(dep, "repo_url", "") or _key) return unpinnedSo if
resolved_commitisNoneor empty, the dep is considered unpinned.But the
_download_virtual_filemethod resolves the commit SHA. Soresolved_commitshould NOT beNone!Unless
_resolve_commit_sha_for_refreturnsNone!Looking at the method:
def _resolve_commit_sha_for_ref(self, dep_ref, ref): return self._refs.resolve_commit_sha_for_ref(dep_ref, ref)And there's a comment in
_download_virtual_file:# Resolve the commit SHA cheaply BEFORE the file download. This is one # short HTTP call (Accept: application/vnd.github.sha returns just the # 40-char SHA in the body) and the result is propagated into PackageInfo # so the lockfile and per-dep header can render the SHA suffix instead # of just the ref name. On non-GitHub hosts or any failure this returns # None and we fall back to ref-name only -- the install never fails on # SHA resolution.So for non-GitHub hosts (like Azure DevOps), the commit SHA resolution might fail and return
None!The bug report shows
dev.azure.com/<org>/toolsas the host. This is Azure DevOps!So for Azure DevOps hosts,
_resolve_commit_sha_for_refmight returnNone, andresolved_commitwould beNonein the lockfile!This explains why the warning says "no resolved_commit" - the SHA resolution fails for Azure DevOps hosts!
Now, let me trace through what happens when
resolved_commit = None:- First install:
resolved_commit = None(SHA resolution failed for ADO) - Lockfile is written with
resolved_commit = None sync_markers_for_lockfileskips this dep because_is_markablereturns False (noresolved_commit)
On the second install:
locked_dep.resolved_commit = None- In
_resolve_download_strategy:locked_dep and locked_dep.resolved_commit and locked_dep.resolved_commit != "cached"is False- The
lockfile_matchcheck is SKIPPED lockfile_match = False
skip_download = False(becauseis_cacheable = False,already_resolved = False,lockfile_match = False)FreshDependencySourceis used- Fresh download happens
- Hash is computed
- Hash comparison fails!
But WHY does the hash comparison fail? If the same content is downloaded, the hash should be the same!
Unless... the content is different between installs!
Wait, I think I finally found it!
Let me look at the
_download_virtual_filemethod again. It creates:apm.yml- generated withyaml_to_str.apm/instructions/<filename>- the downloaded file content
The
apm.ymlincludes adescriptionfield. Let me check if the description is deterministic.Actually, I see that the description is extracted from the file's frontmatter. If the file content is the same, the description should be the same.
But wait, what if the YAML serialization is NOT deterministic?
Let me check
yaml_to_str:def yaml_to_str(data, *, sort_keys=False): return yaml.safe_dump(data, **{**_DUMP_DEFAULTS, "sort_keys": sort_keys})And
_DUMP_DEFAULTS:_DUMP_DEFAULTS: dict[str, Any] = dict( default_flow_style=False, sort_keys=False, allow_unicode=True, )So
sort_keys=Falseby default. This means the YAML keys are in insertion order.The dict is created as:
apm_yml_data = { "name": package_name, "version": "1.0.0", "description": description, "author": dep_ref.repo_url.split("/")[0], }Python dicts preserve insertion order. So the YAML output should be deterministic.
Hmm, but wait! What if there's something non-deterministic in the
descriptionfield?Let me look at the description extraction again:
description = f"Virtual package containing {filename}" try: content_str = file_content.decode("utf-8") if content_str.startswith("---\n"): end_idx = content_str.find("\n---\n", 4) if end_idx > 0: frontmatter = content_str[4:end_idx] for line in frontmatter.split("\n"): if line.startswith("description:"): description = line.split(":", 1)[1].strip().strip("\"'") break except Exception: passWhat if the file has different line endings between downloads (e.g.,
\r\nvs\n)?If the file has
\r\nline endings,content_str.startswith("---\n")would be False, and the description would remain as"Virtual package containing {filename}".But if the file has
\nline endings, the frontmatter parsing might succeed, and the description would be extracted from the file.If the file content is the same between downloads (same commit), the line endings should be the same. So the description should be the same.
Unless... the download method normalizes line endings differently!
Let me check
download_raw_file. For Azure DevOps, it calls_download_ado_file:- Resolves the ref (e.g.,
Assistant
Let me continue investigating the ADO file download method to understand if there could be any non-determinism in how files are downloaded.
Assistant
Looking at the code flow, I now understand the issue. Let me examine the
_resolve_download_strategyfunction more closely to identify the exact fix needed.Assistant
I now understand the issue! Let me trace through the problem:
- For virtual packages (instruction files) on Azure DevOps, the commit SHA resolution fails (ADO doesn't support it), so
resolved_commit = Nonein the lockfile - In
_resolve_download_strategy, the content-hash check (_should_skip_redownload) is ONLY done whenresolved_commitis truthy - for deps withoutresolved_commit, this check is skipped entirely - There's a special case for registry deps, but NOT for git-source deps without
resolved_commit - This causes
lockfile_match = False, leading toskip_download = False, which triggers a fresh download - The fresh download's hash then fails to match the lockfile (for various reasons - possibly non-deterministic YAML generation, line endings, etc.)
The fix should extend the content-hash verification to ALL deps without
resolved_commit, not just registry deps.Let me implement the fix:
- For virtual packages (instruction files) on Azure DevOps, the commit SHA resolution fails (ADO doesn't support it), so
Assistant
Now I can see the issue clearly. The code has a special case for registry deps that fall back to content-hash verification when there's no
resolved_commit. But there's no equivalent fallback for git-source deps (like virtual packages on Azure DevOps) that also don't haveresolved_commit.The fix is to extend the content-hash fallback to ALL non-local deps without
resolved_commit, not just registry deps.Assistant
The fix has been applied. Let me verify it was applied correctly and then run the tests to make sure I didn't break anything.
Assistant
Good, the fix has been applied. Now let me verify it looks correct and run the relevant tests to make sure the fix doesn't break anything.
Assistant
Let me find the correct test files related to the integrate phase:
Assistant
...
Assistant
I need to install the required dependencies first.
Assistant
The tests pass. Now let me run more tests related to the redownload logic and content hash verification to make sure the fix is correct.
Assistant
All tests pass. Let me also run the source-related tests to make sure the fix doesn't break the source acquisition logic.
Assistant
All 84 tests pass! Let me also run the content hash tests to make sure the fix doesn't break anything related to hash verification:
Assistant
All content hash tests pass. Let me also run the heal-related tests to make sure the fix integrates well with the self-heal pipeline:
Assistant
All tests pass. Let me submit the changes now.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-lunaCompare side by side →
- gpt-5.6-lunaCompare side by side →