Building the Social Graph with Hubble and Jetstream

Recently Northsky Social launched it's own Social App to much fanfare and we got a lot of great feedback on it!

Of course, one thing that many users noticed was that their follow counts weren't quite accurate and compared it to what they would expect to see on Bluesky which was significantly different by hundreds and thousands. If it was off by a few I doubt it would have been noticed but at such a magnitude, we needed to take a look.

We're using the Blacksky Appview while we build ours and something I was aware of is that Blackskys follow numbers at the start weren't entirely accurate, and this not exactly a surprise if you know how that number is calculated. Every Appview in operation today indexes all content in order to serve hydrated feeds and that includes the coveted Following feed. That feed is quite simply everyone you are following, there's also mutuals feeds and others that are all based off your social graph.

This social graph is everyone you are following and is following you. These follows are recording in the repos as app.bsky.graph.follow and in order for a feed to hydrate this the appview also needs to have indexed all of this from the firehose (minus deletes, deactived/deleted accounts, and whatever was dropped that week). Considering that Blacksky launched their Appview end of last year it's no surprise that their social graph is considered "incomplete".

In theory another method is to enumerate the PLC then crawl all users repos across the PDS network to build this graph. I'm still carrying the scars of running an indexer with relay mirror at home and so wasn't keen to set that up again; PLC crawling also brings it's own challenges. The options are there but each with their own nuance, nuance that resulted in my building two services to figure out which one was ideal.

Enter Jetstream v2

Jetstream has always been the ideal way to consume firehose: it eats the relay's CAR-file stream and hands you clean JSON records. Just about everyone uses it. But v1 made no promises as it was still a "a lossy convenience relay" with roughly a 36 hour replay window. Anything beyond was "your own problem) so you were left to enumerate every PDS to do a backfill or microcosm sheningans if you needed it.

v2 takes this to the next level with archival replay, read the announcement if you haven't yet. It serves the entire network in compressed segments, and every event gets a sequence number. A client pages through a snapshot plan, downloads segments with striped range requests, folds them into its index, then connects to the live websocket with the same cursor. In short per the docs, jetstream is your buffer. Deletes are event kept as tombstones forever, so a replay is delete-correct, and the stated delivery posture is "we crash rather than corrupt," which is exactly what I want from something feeding my index.

For the first time, "replay everything" is a thing Jetstream can just do. So the obvious experiment was: can I build a correct follow counter with Jetstream alone?

Icarus: replaying history

Icarus is the first experiment, and a very simple one. It reads follow creates and deletes, keeps only what a count needs in a compressed store, and serves a single XRPC query (de.atverkackt.icarus.getProfile) with the graph count:

A dead simple query.

The backfill was less so. The limitations showed up in two phases. Bluesky's own Jetstream services rate limited me like it was going out of style and the connection was capped at around 2Mbps for my token. This is likely because the social graph collection I was requesting is massive. The archive replay itself is metered by API key, and the meter does not mess around: my projection for the full backfill came out around 2.6 TiB of transfer across ten days. With an archive around 23.5 billion events, you can understand how painful it would be.

Then zzstoatzz.io came to my rescue and suggested I use his jetstream server which has been solid and I was able to backfill much faster which then surfaced other issues in Icarus, such as when I assumed his server was misbehaving when in fact I'd simply set GC a little too aggressively which careted the download rate. I rate limited myself.

But the real problem I "learned" from Icarus, is that because jetstream is tracking every single event, that scoped social graph collection was 23.5 billion events. These are every single follow create and delete when what I was after was the follow count today. Replaying the history meant that I was tracking history that was effectively meaningless to me. Consuming Repos are simply the fastest way to learn the present state of the network.

This then raises the question - why am I replaying an event archive when I could be consuming repos?

Picon: skipping to the end

hubble is microcosm.blue's repo mirror — in its own words, "a whole-Atmosphere public data mirror, synchronizing every atproto repository in real-time." It follows the firehose so you don't have to, keeps every repo current, and serves them back over plain com.atproto.sync.getRepo. No API key, no quota (yet). And it has one twist that makes it perfect for this: content negotiation for a format called STAR-lite.

If you follow bad-example.com you might have seem them posting about this format and I've been very hyped to use it somewhere.

A normal CAR export interleaves record blocks with MST node blocks, and nothing guarantees the order, so a parser ends up buffering the whole repo in memory. STAR-lite is too cool to need all that. The spec describes it as "a flat list of every key/record pair in a repository, in lexicographic key order, with commit details in its header." No MST blocks, just sorted records, built for single-pass streaming. Add in compression that comes out about 2x smaller than a CAR, you can see why I wanted to use it.

The sorted order is the entire trick. Picon keeps its own copy of a repo's follows in that exact same order, so updating one is just reading the two lists side by side and stepping through them together. Where both lists have the same key, nothing changed. Where the download has a key the store doesn't, that's a new follow. Where the store has one the download doesn't, that follow is gone. One pass, both lists, done.
Because picon never has to hold more than the record it's currently looking at memory stays flat whether the repo has ten follows or ten thousand.

Picon finished its first pass over all 41.3 million repos while icarus was still under halfway through the event archive.

Tracking changes beyond hubble

A snapshot only solves the question of "What is the graph at this point in time"; the network keeps moving. So picon runs two readers at once: the sweep, and a Jetstream live tail — the plain websocket, which (thankfully) needs no key.

The interesting part is the handoff. Jetstream v2's replay has one global sequence number where the archive ends and the live stream begins. Picon instead has 41 million, one per repo. While a repo is downloading, its live events are held; on commit the revision decides which held events are newer and get applied, and which are already inside the snapshot and get dropped. There are a few moving parts beyond the one cursor Jetstream provides, but it's all gated by the repos own revision, which is the one piece that can't lie about ordering.

Would tap have been simpler?

Maybe, it does the full MST and sig verification where picon simply trusts the mirror and the way it buffers live events during backfill is similar to how picon merges hubble with jetstream.

The difference is really their purpose. With tap you say what repos you want and it gets them from whatever PDS the account lives on, it does the dirty work of crawling across multiple PDS where some may not be online (or running who knows where). I also am not a big fan of tap as for PDS operators they now get significant traffic straight from tap and many simply rate limit that agent.

But I'm not interested in seeding it with a list and waiting to fetch across each PDS, I'm greedy and want all of them. I want them from one place that has already gone and done that dirty work for me, in a format I can read as fast as my disk will process it. That's where hubble wins.

So looking back: v1 Jetstream couldn't even pretend to do this - 36 hour window is for service disruption, not backfill. v2 can, and it's global cursor is simpler but the current API usage restriction and long backfill times for billions of events ruled it out for a cold start system. Tap simply solves a different smaller-scale version of the problem very well. Backfilling from hubble and tailing events from Jetstream takes the best part of each and gives me a system that I can setup and run efficiently.

So what now

Picon and Icarus are running side by side on a VPS somewhere in Germany, Icarus burned a bit too brightly and is still backfilling while Picon is in what I'd call beta mode. I've got a little dashboard running to track the gap and it should be closed by the end of this weekend and we'll see how they do.

I encourage you to try picon and let me know what you think, the code is on my tangled. One small caveat I want to raise: I don't expect these counts to be 100% accurate because blocks aren't taken into consideration (maybe a future iteration?) so there will be a little drift but then again, we've been arguing over the true user count on the Atmosphere since the start so I don't mind it.