Testing JSON Schema's format Keyword Against the RFCs

JSON Schema's format keyword asserts that a string looks like an email address, an IPv6 address, a date, a URI, and so on. This year, Google Summer of Code funded a project at JSON Schema to check whether that assertion is actually true - not against what implementations happen to do, but against the RFC each format value points at.
This is a report on what that testing found.
The specification does not define what an email address or a URI looks like. It delegates to an RFC per format: email to RFC 5321, ipv6 to RFC 4291, duration to the ABNF in RFC 3339 Appendix A, and so on across 18 formats. So format is really eighteen separate grammars sitting behind one keyword, each with its own edge cases, and none of them defined by JSON Schema itself.
The JSON Schema Test Suite is the shared conformance baseline every implementation runs against. Before this project, its format coverage was thin relative to the other keywords - a handful of representative cases per format rather than a systematic pass over each grammar's boundaries. That gap matters beyond the suite itself: sourcemeta/blaze defers format assertion by design, in part because there was no test corpus solid enough to implement against with confidence.
Step 4 is the one that matters. Two libraries disagreeing is easy to find. A test case only earns a place in the suite once you can point at the specific line in the RFC that settles the disagreement, and the specific line of code that gets it wrong.
Across the 13 formats with published evidence, this method produced 270 documented findings against 29 JSON Schema validators - cases where a real, in-use implementation gives a different verdict than its governing RFC. Most of these validators pass the entire existing published suite.
A note on what those numbers are: they count documented findings in the evidence repository, not failing suite tests. Several findings collapse into a single test case, and inputs where the RFC is ambiguous - leading zeros in ipv4, the leap-second month-end - are excluded entirely rather than counted against anyone. sourcemeta/core does not appear because across all 13 formats it was the faithful reference, with its only two non-conforming readings both being open specification questions rather than defects.
A separate group sits underneath this table: another 28 libraries that are not JSON Schema validators but that validators delegate to - C's inet_aton, Java's Guava, the WHATWG URL parser, ICU4J, GNU libidn2, Python's email.utils.parseaddr, Go's net/mail, the Rust and Python idna crates. Several of the rows above are inherited from these rather than written by the validator authors.
The findings are not spread evenly, and the reason is more interesting than the count. python-jsonschema accounts for the largest single share, because its format checkers delegate to third-party libraries - isoduration, rfc3339-validator, rfc3987, jsonpointer - each of which implements a grammar slightly different from the RFC the JSON Schema specification actually points at. One delegation choice then produces divergences across several formats at once.
A second recurring mechanism is anchoring. A regex that is correct in shape but ends in $, or omits the end anchor entirely, will accept a trailing newline or trailing junk: jsonschemafriend (Java $ matching before a final line terminator), opis/json-schema (missing end anchor) and tdegrunt/jsonschema (no anchors at all, so any string containing a valid value passes) all fail this way, in three different languages, for the same underlying reason.
What the table shows overall is that this is not a story about one bad library. Twenty-nine validators, in ten languages, each got something wrong - and the two mechanisms above account for a large share of it. email and uri-reference are the worst affected, which makes sense: they have the largest grammars and the most places to go wrong.
Most findings belong in the test suite rather than in a bug tracker: a shared conformance test is the right way to communicate "this input has this verdict" to every implementation at once, and many cases are places where the RFC is ambiguous rather than where a library is broken. Nine were different - reproducible defects with an identifiable cause, worth reporting directly:
All nine were merged or accepted by their respective maintainers.
Kelvin.example.com is a hostname. Its first character is not the letter K - it's U+212A, the KELVIN SIGN, the unit symbol for temperature, which Unicode encodes separately from the letter it resembles.
RFC 1123 hostnames are ASCII, so this string is invalid. ajv and sourcemeta/core both reject it correctly. python-jsonschema accepts it, because its hostname checker calls .lower() on the string before running the hostname regex, and Python's .lower() maps U+212A to a plain ASCII k. The regex is correct; by the time it runs, the non-ASCII character is already gone.
That single mechanism - case-folding before validation instead of after - turned out to open the door for 643 distinct non-ASCII codepoints through this one checker, via three separate paths. The Kelvin sign is now a published test case: suite#1024.
The clearest single measure is the published test suite itself. Comparing the format test files at the start of the coding period against their state now:
Published format tests across the 19 formats tracked grew from 618 to 843 over the project - a net addition of 225 tests, essentially all of it landing through this project's pull requests. duration, idn-email, and the regex dialect tests had close to no dedicated coverage before this and now have a real baseline. email alone more than doubled, from 27 to 71. One format, json-pointer, needed no additions at all - its existing coverage held up against the same testing everything else went through.
The work also landed in a second repository. The project was mentored by Juan Cruz Viotti, who maintains sourcemeta/core - the C++ library behind Blaze - so a