A colleague opened a small PR to harden dependency pinning on a Lambda. It was a real fix for a real bug. I read it, thought "good, this is more rigorous than what we had," and approved it.
It broke every infrastructure apply in the workspace for two days.
Nobody noticed, because the one thing that would have exercised the change was the one thing our pipeline never ran.
The bug it was fixing was real
The Lambda gets packaged and deployed by Terraform. Its rebuild trigger was a hash of the requirements file:
triggers_replace = {
requirements = filemd5("${path.module}/src/requirements.txt")
}That looks reasonable until you notice what the requirements file actually pinned. One direct dependency, pinned exactly. Everything underneath it floated.
So the transitive tree could move without the file changing a single byte. A new release of a downstream crypto library would land, the packaged artifact would happily pick it up on the next rebuild, and the trigger would see the same hash it saw last week. No rebuild, no change to source_code_hash, no signal that anything had moved. The trigger was watching a file that could not see the thing it was supposed to be watching.
That is a legitimate problem and worth fixing.
The fix that arrived
The PR did the textbook thing.
It introduced a human-edited source of truth with one line in it, the direct dependency. It compiled that into a fully hash-locked requirements file with uv:
uv pip compile requirements.in \
--generate-hashes \
--python-version 3.12 \
--python-platform x86_64-manylinux2014
Out came a dozen pinned transitive dependencies, each with its hashes. Then it added --require-hashes to the install step in the build, so nothing could sneak in unpinned.
The PR body made the case cleanly: the trigger now fingerprints the entire dependency tree, so any transitive bump becomes a visible, reviewed diff that forces a rebuild.
Every part of that is correct reasoning. It is also the exact reasoning I would have written myself.
The failure
The build installs with pip, not uv:
pip3 install \
--platform manylinux2014_x86_64 \
--only-binary=:all: \
--python-version 3.12 \
--require-hashes \
-r requirements.txt \
-t ./build
Here is what it does now, every time:
ERROR: In --require-hashes mode, all requirements must have their
versions pinned with ==. These do not:
typing-extensions>=4.13.2 (from cryptography==49.0.0)
The lockfile pins cryptography. cryptography requires typing-extensions. typing-extensions is not in the lockfile. It never was. It does not appear anywhere in the file, and git blame confirms the pin that requires it was there in the original lockfile commit, so this was not a later dependency bump knocking things loose. The lockfile was incomplete from birth.
--require-hashes refuses to fetch anything that was not pre-pinned and pre-hashed. So the install dies, the zip never gets built, source_code_hash never moves, and the apply exits non-zero. Not just for that Lambda. The whole workspace apply fails.
Read that against the goal one more time. The PR set out to stop transitive drift from silently skipping rebuilds. What it shipped was a build that loudly fails and still never rebuilds. Same stale artifact, worse blast radius.
Root cause: one lockfile, two resolvers
The lockfile was generated by uv and installed by pip.
For the same target, Python 3.12 on manylinux2014, uv's computed transitive closure excluded typing-extensions. pip's install-time resolution for that same target includes it. Two tools, same inputs, different answers about what the dependency tree contains. Why they diverge at that level, I didn't reproduce, and it doesn't change what broke next.
Under normal conditions that disagreement is invisible and harmless. pip would notice typing-extensions was missing, download a compatible version, and move on. Slightly unpinned, completely working, nobody ever finds out.
--require-hashes is what turned that into a wall.
This is the part I keep chewing on. The strictness was the whole point. It is what guarantees you get the bytes you reviewed and nothing else. And it is precisely what converted a benign resolver disagreement into a total, unrecoverable build failure. The hardening did not fail to protect us. The hardening is the mechanism by which we got hurt.
A lockfile is only complete relative to the tool that produced it. "Fingerprints the entire dependency tree" quietly assumed uv's tree and pip's install-time tree were the same tree. They were not, and --require-hashes made that assumption load-bearing.
Why it survived review
Three misses, and I own the middle one.
CI never ran it. The pipeline builds the application, tests it, and deploys it. It does not run terraform apply, and it does not run the Lambda's packaging step either. Infrastructure is applied by hand. So a lockfile that cannot install is completely invisible to CI. The one artifact that would have caught this in thirty seconds is not in the pipeline at all. If the only thing that exercises a build step is a manual command someone runs locally, that build step has no test. It has a habit.
I approved it. The honest version is not that I skimmed it. I read it. The problem is that the diff looked more rigorous than what it replaced. Hashes. Exact pins. A dedicated source-of-truth file. An added strictness flag. Every visible property of that change signaled care, and I graded the change on the properties I could see instead of the one that mattered: does the resulting file install?
That is the trap. Hardening changes come pre-loaded with the appearance of correctness. A diff that adds --require-hashes reads as unambiguously safer than one that removes it, and the reviewer's instinct is to nod. I never asked whether the artifact had been run.
The AI reviewer caught the cosmetics. Copilot did several passes on this PR. It flagged a comment mentioning the wrong library and got it corrected. It suggested adding the source file to the trigger list, which we declined for reasonable cause. It noticed that uv spells the platform x86_64-manylinux2014 while pip spells it manylinux2014_x86_64, called that easy to mix up, and its autofix committed a note about the spelling.
On the lockfile itself: reviewed three of three files, generated no new comments.
So it observed, in writing, that uv and pip are two different tools with two different spellings for the same target. Then it fixed the spelling and shipped. It flagged the surface symptom of the exact bug class that was sitting three lines below in the same diff. I do not think that is a dunk on the tool so much as a good illustration of what it is currently good at: it reads diffs, not artifacts. Neither did I.
What actually fixes it
Immediate. Regenerate the lockfile and then prove it installs before merging. The gap here was never "forgot to regenerate." It was "never verified the generated file installs." Those are different failures and only one of them is caught by regenerating more carefully.
Structural. Do not generate with one resolver and install with another under --require-hashes. Either install with uv as well, or compile with pip-tools so that generation and installation share pip's resolver. One resolver, end to end. The moment you have two, the lockfile is a claim about a tree that the installer may not agree exists.
CI. Run the packaging step in the pipeline. Not the apply, just the install into a throwaway directory:
pip install --require-hashes -r requirements.txt -t /tmp/verify
Ten seconds. It converts "a colleague's apply fails two days from now" into "this PR is red." More broadly, if apply is manual, the parts of the infrastructure that build things should still be exercised automatically.
The thing I took away
A lockfile that does not install is worse than a floating dependency. A floating dependency is a known, bounded risk that you can reason about. A broken lockfile is a landmine dressed as rigor, and it gets waved through review precisely because it is wearing the costume.
--require-hashes is only as safe as the lockfile is complete. And "it compiled" is not "it installs."
I still think the PR was the right idea. I would approve the same intent again. What I would do differently is ask one question I did not ask, the same question that turns out to matter for most build tooling: has anyone actually run this, or does it just look correct?
//Comments