How to Understand and Query Farcaster Data (Casts, Users, Channels, and Frames)
How to Understand and Query Farcaster Data (Casts, Users, Channels, and Frames)
Farcaster is the top new web3 decentralized social platform. I'll walk through how the data structures work (onchain, hubs, clients) and how to query it on Dune and Neynar.
Farcaster is a social app - like Twitter - that’s been decentralized and uses Ethereum addresses as the basis for accounts. If you’re curious about why that matters or what use cases that enables, read Packy’s post. I’m going to cover the key data structures behind the protocol, and teach you how to query the protocol using Dune and Neynar.
In this guide I’ll walk through:
The difference between onchain contracts, hubs, and clients
Querying main interactions (casts, reactions, links, profiles, fids, verifications)
A bit about Frames
Connecting onchain data to Farcaster data
Dune gets its Farcaster tables data from Neynar every 12 hours. If you’re looking to join onchain activity to Farcaster tables, then querying in Dune is the way to go. If you’re looking to build an app/bot on top of Farcaster, you should be using Neynar (think of them like Alchemy).
The easiest way to sign up to Farcaster is to just download Warpcast from the app store and purchase in app. You can use my invite link.
📊 Follow me on Farcaster and join the /data channel
🎧 Do you prefer audio? I recorded a 20-minute voice overview of this guide:
What data exists where?
Onchain Contracts
First, let’s cover what is onchain, on hubs, and on clients. This will teach you where you need to go to access what type of data.
Let’s start with the onchain contracts. Farcaster has three contracts on Optimism, which act as “registries” for ids, storage, and keys respectively (shown below). I believe the main reason for having this onchain component is to have payment for both ids and storage - as it would be very expensive to decentralize a social app if any bot could costlessly sign up and spam.
When a user registers, they are given an fid and custody address, as well as some storage units. Here is an example of a registration transaction on Optimism. You can see that transaction broken down in Dune, with the tables to query for this data:
You’ll most likely need to query these tables if you want some raw economic data on protocol revenue and costs. There are great example queries in this dashboard you can reference (near the bottom).
Once a user is signed up, they can also rent more storage within an app like Warpcast. Here’s the purchase screen below in my account:
Hubs
Once a user is registered, hubs take care of the rest. the bulk of actual Farcaster data is stored on hubs. Anyone can run a hub using Hubble, and they hash messages that are then streamed to each other and validated.
There are currently 560 hubs being run (see here). You can sync a hub in 2 hours, and it takes 8GB ram with 20GB storage to run the hub (requirements here).
Hubs store all the actual interaction data - profiles, casts, channels, reactions, links (follows), and verifications. You can see an overview with the data documentation on Dune. We’ll go through all these data tables soon.
This is where Neynar gets their data (they actively run a hub), and Dune gets their hub data from Neynar which syncs once a data.
Clients
Clients are unrelated to hubs, so don’t make the “oh geth is a client to run ethereum” mental connection here.
Clients mean apps in the Farcaster context. The biggest client is Warpcast, a mobile/web app built on top of Farcaster. Warpcast stores data from custom interactions that don’t get published to hubs. There are three examples here:
private direct and group messages. Note that these are not to be confused with the “hashed messages” sent between hubs.
frame actions (clicking on a frame). Note that frames as a concept can be implemented on any client, but the data will always only be passed between the client and the frame server.
warps (in-app currency native to Warpcast)
channel follows. You can get channels using parent_url (explained later below) but the best proxy for “following” is looking at if a user has casted, recasted, or liked a cast in a given channel.
There is also sometimes a delay between when a cast appears on a client like Warpcast and when it gets published. Warpcast publishes in small batches, so sometimes you may cast on Warpcast and not see the cast available on Neynar for a couple seconds/minutes.
You can’t query data off of clients unless they open up an API for it. An example here could be using Warpcast to “login with Farcaster” on another application. Warpcast is considered a “wallet app” within the category of clients.
Explaining Interaction Data (from Hubs)
Whew - that’s a lot of background knowledge, we’re finally ready to get into the real data.
Profiles, FIDs, Addresses, and Followers
Each fid has a custody address (the one that was added in the registration process). This is found in the dune.neynar.dataset_farcaster_fids table. If you wanted to get their profile data, you would JOIN the dune.neynar.dataset_farcaster_user_data on “fid” and do a WHERE on “type” to get their fname (username), profile, display name, avatar URL, or bio.
A user can choose to link other addresses if they want through offchain signatures in a wallet app (i.e. Warpcast → settings → connected addresses). These go in the dune.neynar.dataset_farcaster_verifications table (JOIN again on “fid”).
To make it simple, you can just join the fids table to the aggregated table dune.neynar.dataset_farcaster_profile_with_addresses to get fid, profile data, custody address, and verified addresses. To query followers, you’ll want to use the dune.neynar.dataset_farcaster_links table, where “fid” is the follower and “target_fid” is the user being followed.
Here’s a query putting this all together:
On Neynar, you can use the users endpoints to get this data.
Casts, Replies, Channels, and Reactions
There are only two tables for you to work with here.
Keep in mind that many tables have a “hash” column, which represents the id of an action on that table. Only the “hash” on the casts table represents a cast hash.
The reactions table is easy to work with, just JOIN on “fid” or “target_fid” as needed and JOIN “target_hash” to the “hash” column on the casts table. Let’s walk through some of the key columns of the casts table:
hash: the hash of the actual cast. This is truncated to just the first 4 bytes (8 characters) in the Warpcast URLs.
parent_hash: the hash of the cast being replied to (if any)
root_parent_hash: the hash of the cast at the top of the reply thread (if any)
parent_url: the channel id of the cast (if any, none means “home”)
As for the cast text, it’s broken up into three parts:
text: any raw text from the cast, with links and mentions removed
mentions: the fids of any users @ mentioned in the cast. The position of the mention is in the “mention_positions” column. The arrays are 1:1 in terms of index.
embeds: any attachments to the cast, like an image
Here’s an example query that puts some of these cast concepts together:
For casts analysis, you’ll need to just get used to doing self joins (joining the casts table on itself to get replies). This can get pretty complex, if you want to see some more difficult queries then check out how I’ve broken down Bountycaster in this Dune dashboard. Bountycaster relies on a user to cast with certain words like “complete” or “cancel” or “add” while tagging the bountybot.
You’ll see that in the “bounties” CTE of this query, I look for any cast that’s a top level cast (no parent_hash that it’s tied to) and then look for any replies with keywords that indicate the status of the bounty. I then use some regex to pull out information like bounty value.
On Neynar, you can get casts from a feed and then unroll a cast using retrieve all casts from thread. Check out my code here for calling and parsing a cast, and also validating some frame data.
About Frames
Frames are really simple right now, involving an image and a few buttons. The interaction in the app then asks a server (API backend of the frame developer) what to show next. It uses the OpenGraph (OG) standard, which is what is used to power those link previews you see on any site when you paste a link. Here’s an example:
Frames are the newest feature on the Warpcast client. Because it’s a client concept, you can’t query frame actions data off of Neynar or Dune. But if you build a frame, you will get the “frame action” passed to your frame server:
Note that frames as a concept can be implemented on any client (i.e. Supercast supports them), but the data will always be only passed between the client and the frame server.
There are some workarounds to getting frame action data. I’ve built a frame system that pushes actions to Ethereum Attestation Service (EAS) to act as an onchain logs framework. I tweeted an example query - we’ll go more in depth in the next section.
🚨 Frames are still rapidly changing, and this guide will likely fall out of date quickly. I’ll update this when I can, but will likely do a full frames guide in another article in the future.
Connecting Farcaster data to other onchain data
The real unlock of having a web3 native social graph is that you can now join a Farcaster user’s onchain activity to their hub activity. I took the voters from my EAS frame poll, and joined the user custody_address and verified wallets together onchain summary data in this dashboard. Here’s a nice little summary query:
Again, “verified wallets” are pulled from the verifications table. They are added by the user through offchain signatures done in a wallet app (i.e. Warpcast → settings → connected addresses).
The complexity of the onchain data here has been abstracted in a materialized view named “dune.dune.result_wallet_all_chain_activity_summary” that is defined in this query. In the matview query, you will find how I calculate statistics like age, transactions, volumes, and contract deployments.
Here’s another example, of me pulling in Farcaster fids and usernames onto a list of token holders (bytepass NFTs). These are all top data experts in the ecosystem, so you should follow everyone here 😁
All Dune queries can be accessed from the API as well. Here’s some code in javascript that I used to calculate poll totals from EAS data. Dune also has a python SDK.
Dune API documentation can be found here.
Continue Learning Data with the Bytexplorers
That’s all for this Farcaster data guide! If you enjoyed it, be sure to like and subscribe. If you have questions or work you want to share, then DM me on Twitter or Farcaster.
I’m also building an onchain crypto data community called the Bytexplorers. We’re building out some Farcaster specific initiatives like the ByteLight token, and would love to have more people participate with us.