One Month After the Hackathon: Why My 4-Service App Stopped Working (and What @mansio Caught in the Verifiers) - DEV Community

Posted on Nov 13 • Edited on Nov 13 • AI-assisted
Details subtle auth bugs and community PR fixes
What breaks a month after a hackathon, and what a fellow DEV writer caught in the verifiers.
TL;DR: A month after shipping README Clew 2, I opened it up and every scan failed. Nothing was wrong with the deterministic verifiers or with Jev on Glasser. Instead, three small operational traps lined up at once: an expired 30-day GitHub token on a tool that only reads public repos, a rotated Claude Haiku key scoped only to Development in Vercel, and a substring bug that fellow DEV writer Mikhail (@mansio) found and fixed in PR #2.
In October I built README Clew 2 for the AdaL "What Would Jev Do?" hackathon. It checks whether a public JavaScript or TypeScript repo's README matches its code, sorting every checkable line into four buckets: Contradicted, Missing, Unverifiable, and Verified.
The core rule of the build is that models never decide what is true:
When you wire that many moving parts together (AdaL, Jev, Glasser, Haiku, GitHub, and Vercel) and come back a month later to a red SCAN FAILED banner, it is hard to tell at first glance which piece broke.
Here is what was actually wrong, and the two fixes that came out of it.
Before I even got to the deployment keys, Mikhail (@mansio) had opened and landed PR #2, the project's first external code contribution.
Mikhail was testing README Clew 2 against real repositories and noticed something strange on sindresorhus/yocto-queue. The repo uses ava as a test runner, and the README never mentions ava. The coverage verifier should have flagged ava in the Missing bucket. Instead, it stayed silent.
Why? Because the README contained a link to Mozilla's documentation:
And inside the word javascript, in lowercase, sits the three-letter substring ava.
Our coverage check was using a plain substring match:
Even worse, the same substring check lived in the dependency verifier's prose guard (inCodeContext), which checks whether a candidate package name appears on an install line. A short name like can inside npm i canvas-parser or lodash inside npm i lodash-es could bypass the prose guard and reach Contradicted, the exact false positive the guard was built to prevent.
Mikhail replaced both substring checks with a single whole-token rule, mentionsWord(), bounded by npm's own package name characters (A-Za-z0-9_-) plus @ and /:
Mikhail shipped the fix with a 27-case test matrix and verifier controls in tests/mentions-word.test.js, matching the repo's file-size cap and offline test rules on the first try. If you have not read his engineering field notes on DEV (like The Dataset Was Lying: 4 of 6 "False" Facts Were True), go read him.
Once Mikhail's PR was merged, I still had a live site that refused to scan.
Every click on SCAN returned HTTP 502:
The ironic part: README Clew 2 only scans public repositories. It makes 4 metadata calls to api.github.com per scan and fetches the files from raw.githubusercontent.com. You do not even need a GITHUB_TOKEN for low-volume public reads (GitHub allows 60 unauthenticated REST calls per hour per IP, or 15 full scans per hour).
During the October hackathon, I had added a 30-day GitHub Personal Access Token so repeated test scans wouldn't hit the unauthenticated rate limit. Thirty days passed, the token expired, and lib/github.js kept sending Authorization: Bearer <expired_token> on every request.
GitHub allows requests with no Authorization header, but immediately rejects requests carrying an expired Authorization header with 401 Bad credentials. Because step 1 threw a GithubError, the pipeline stopped before Jev or Haiku ever ran.
First, I removed the expired GITHUB_TOKEN from Vercel. Second, we hardened lib/github.js so that if an expired token is ever present in .env or server environment variables and GitHub answers 401, the client drops the authorization header from the shared headers object and retries unauthenticated:
Because headers is shared across the 4 metadata calls in fetchRepo(), dropping headers.authorization on the first 401 means the remaining 3 calls go straight out without the bad token.
The last piece was Claude Haiku. By design, Haiku is fail-open in README Clew 2: if the summary call fails or times out, the report still renders in full and the clew mascot falls back to a fixed one-line summary per mood.
My October Anthropic key had also been rotated. When I added the new ANTHROPIC_API_KEY in Vercel, two classic traps bit me at the same time:
Running vercel env ls made the split visible right away:
Overriding ANTHROPIC_API_KEY across all three environments (Production, Preview, Development) and running vercel --prod brought the Haiku summary and bucket notes back on the first scan. Meanwhile, Glasser and Jev (jev-1.13.0) never missed a beat: all 4 batched calls completed in 1.37 seconds for about half a cent ($0.0055), with $10.80 of the original hackathon balance still untouched.
Two posts from the community hit the exact operational edges from today's debugging session:
Thank you again to Mikhail (@mansio) for finding the substring bug on a real repo and shipping a clean, well-tested fix.
Quick context if you are new here. I work in the California courts, running court operations for the county. I started building with AI in July 2025 and I have been learning in public ever since. I do not write the code. I direct, the agents generate, I validate and decide. I build the Clew Suite, a set of civic tech tools for making complex systems easier to inspect. That is the lens I am writing from.
AI Assisted. Human Approved. Powered by NLP.
There’s one edge case I’d pay extra attention to in the GitHub fallback: concurrency.
If fetchRepo() eventually runs those four metadata requests through Promise.all, mutating a shared headers object after the first 401 creates a race. Some requests may already have left with the expired token. I’d keep request headers immutable and make the retry decision at the individual reques