Microformats – building blocks for data-rich web pages

A (very) belated follow up to Getting Started with Microformats 2, covering the basics of consuming and using microformats 2 data. Originally posted on waterpigs.co.uk.
More and more people are using microformats 2 to mark up profiles, posts, events and other data on their personal sites, enabling developers to build applications which use this data in useful and interesting ways. Whether you want to add basic support for webmention comments to your personal site, or have ambitious plans for a structured-data-aware-social-graph-search-engine-super-feed-reader, you’re going to need a solid grasp of how to parse and handle microformats 2 data.
To turn a web page containing data marked up with microformats 2 (or classic microformats, if supported) into a canonical MF2 JSON data structure, you’ll need a parser.
At the time of writing, there are actively supported microformats 2 parsers available for the following programming languages:
Parsers for various other languages exist, but might not be actively supported or support recent changes to the parsing specification.
There are also various websites which you can use to experiment with microformats markup without having to download a library and write any code:
If there’s not currently a parser available for your language of choice, you have a few options:
Most real-world microformats data is fetched from a URL, which could potentially redirect to a different URL one or more times. The final URL in the redirect chain is called the “effective URL”. HTML often contains relative URLs, which need to be resolved against a base URL in order to be useful out of context.
If your parser has a function for “parsing microformats from a URL”, it should deal with all of this for you. If you’re making the request yourself (e.g. to use custom caching or network settings) and then passing the response HTML and base URL to the parser, make sure to use the effective URL, not the starting URL! The parser will handle relative URL resolution, but it needs to know the correct base URL.
When parsing microformats, an HTTP request which returns a non-200 value doesn’t necessarily mean that there’s nothing to parse! For example, a 410 Gone response might contain a h-entry with a message explaining the deletion of whatever was there before.
When consuming microformats 2 data, you’ll most often be fetching raw HTML from a URL, parsing it to canonical JSON, then finally processing it into a simpler, cleaned and sanitised format ready for use in your website or application. That’s three different representations of the same data — you’ll most likely end up storing the derived data somewhere for quick access, but what about the other two?
Experience shows that, over time:
Therefore, if it makes sense for your use case, I recommend archiving a copy of the original HTML as well as your derived data, leaving out the intermediate canonical JSON. That way, you can easily create scripts or background jobs to update all the derived data based on the original HTML, taking advantage of both parser improvements and improvements to your own code at the same time, without having to re-fetch potentially hundreds of potentially broken links.
As mentioned in the previous section, if you archive original HTML for re-parsing, you’ll need to additionally store the effective URL for correct relative URL resolution.
For some languages, there are already libraries (such as XRay for PHP) which will perform common cleaning and sanitisation for you. If the assumptions with which these libraries are built suit your applications, you may be able to avoid a lot of the hard work of handling raw microformats 2 data structures!
A parsed page may contain a number of microformat data structures (mf structs), in various different places.
Take a look at the parsed canonical microformats JSON for the article you’re reading right now, for example.
items is a list of top-level mf structs, each of which may contain nested mf structs either under their properties or children keys.
Each individual mf struct is guaranteed to have at least two keys, type and properties. type is the primary way of identifying what sort of thing that struct represents (e.g. a person, a post, an event). Structs can have more than one type if they represent multiple things at once without wanting to nest them — for example, a post detailing an event might be both a h-entry and a h-event at the same time. Structs can also have additional top-level keys such as id and lang.
Generally speaking, type information is most useful when dealing with top-level mf structs, and mf structs nested under a children key. Nested mf structs found in properties will also have type information, but their usage is usually implied by the property name they’re found under.
For many common use cases (e.g. a homepage feed and profile) there are several different ways people might nest mf structs to achieve the same goals, so it’s important that your code is capable of searching the entire tree, rather than just looking at the top-level mf structs. Never assume that the microformat struct you’re looking for will be in the top-level of the items list! You need to search the whole tree.
I recommend writing some functions which can traverse a mf tree and return all structs which match a filtering callback. This can then be used as a basis for writing more specific convenience functions for common tasks such as finding all microformats on a page of a particular type, or where a certain property matches a certain value.
See my microformats2 PHP functions for some working examples.
Each key in a mf struct’s properties dict maps to a list of values for that property. Every property may map to multiple values, and those values may be a mixture of any of the following:
A plain string value, containing no HTML, and leaving HTML entities unescaped (e.g. <)
(In future examples I will leave out the encapsulating {"items": [{"type": [•••], •••}]} for brevity, focusing on t