AI usage

The goal of this project is to improve the connectivity and discoverability of real, human authors of web content. So it’s probably important to talk about AI and its use on the Outer Web.

AI summaries

The most obvious use of AI on this project got its own post. When we come across a webpage that has no description, we add it to a queue for a locally-run LLM to generate one. Having dogfooded our site we know firsthand how important it is to know what a post is about before clicking through.

To the extent that a given post is more than a single sentence, AI summaries are really more of an exercise in classification - the generative part is just because classification output isn’t easy to read. Since we algorithmically exclude posts that are not contentful, there’s no danger of the AI summary stealing a click from the destination post.

AI tagging

There are some standardized ways to tag webpages but, like with descriptions, adoption is far from universal. Our LLM summary pass also suggests tags that are used in the similarity computation that defines the structure of the Outer Web. This is notionally equivalent to a traditional classifier. Our similarity heuristic could survive without tags, but its much better with them.

Embeddings

In addition to tags and raw text content, page similarity is determined by cosine similarity between text embeddings (whose values are a product of machine learning).

Pixel art

The retro pixel art used throughout this site is a combination of gen-ai and hand editing. We’d rather write code than create a dozen gas giants pixel by pixel. We could probably create a neat algorithm to do this but this was the path of least resistance. Not that hand editing pixel art is a cakewalk.