I’ll go over my workflow for creating this hack analysis and monitoring report in a GitHub Gist that’s updated every 5 minutes.

As soon as I saw the initial $50m mint, I opened Claude Code (which already had Herd MCP installed) and told it to analyze USR and tell me what caused the hack.

The hack was pretty simple (unfortunately, a team's private key was compromised), but it was still nice to understand it and find the related hack transactions within a minute. I immediately knew I wanted to monitor the attacker’s wallet for any further activity and to check the team’s multisig to see their responses.

The agent was able to track transfers to and from the attacker wallet to keep expanding the wallets to monitor, and it was also able to check across the USR contracts to find the team multisig and read the queued transactions.

I immediately set this loop:

The initial report was now being uploaded to GitHub in a gist for me to easily read. Now, I could keep up with the activity and add things to the report as I did my own sleuthing alongside the agent.

You can read the gist yourself here; whenever I told Claude to add a new section, I would tell it to add and keep track of the methodology too. So for prices, I told it to check the spot price after each block of swaps, by reading traces data and querying onchain read functions.

This would have taken hours to do by hand, and I’m finding myself using AI to help with research and analysis in crypto more and more now.

Subscribe now

If you’ve spent any time building with Cursor, Claude Code, Codex, or Openclaw, you’ll realize the hardest part of coding is not the writing and running the code, but the planning and validation that comes at each step of development.

Building in crypto makes this problem 10x more apparent due to the unbounded, adversarial nature of blockchains. Most of the time, agents are working in a bounded box - your codebase is all yours, and only updated by you. Blockchains are shared codebases that everyone constantly adds to and commits to. What’s true about the “code” and states now might not be in the next minute. This has been coined as the “dark forest” nature of crypto.

The good thing is that there are already thousands of examples of whatever you want to do onchain, for your agent to enhance your plan and validate it against real transaction data. But these are not laid out in clean end-to-end user journeys. They’re broken up across millions of transactions, contracts, and wallets - and you just have to know how to search and piece them together.

However, your agent can’t just “web search” for addresses either, because they can easily be misled into the wrong contracts, outdated documentation, or missing edge cases. This leads to brute-forcing with many assumptions about the blockchain, and there is no way to create a learning test loop to help with planning or validation. In crypto, that could mean your or your customers’ money is at risk. Searching and learning about protocols is typically done by very technical researchers/devs opening 50+ tabs of different tools like Etherscan, Dune, Blocksec, docs, and Github pages - but not anymore!

Agents are missing a crypto “web search” tool

Web search tools help agents search and extract relevant information from docs, GitHub repos, and more. Given 50 relevant sources, it can probably triangulate on the right answer. Smart contracts are different - if the agent finds 50 addresses, then it’s likely to go down a very wrong rabbithole. Herd’s tools help agents navigate precise relationships when starting from any contract, transaction, or wallet:

You can think of it as building a relationship graph on the fly - the difference being that we aren’t hardcoding what a “node” or “relationship” is because you can’t really categorize all onchain objects that cleanly (many of us have tried). Instead, we let the agent build a knowledge graph tailored to the task at hand.

Herd has built the most searchable and human-readable Ethereum block explorer over the last year, and we’re now releasing an MCP that includes all the functionality of our explorer (and more). It’s a set of tools focused on navigating the “unknown unknowns” of onchain relationships, with lookup functions for every transaction type to enhance planning and QA validation. It also includes bookmarking tools that let you easily reference the same contracts, wallets, and transactions across chats.

On top of relationship lookups, our tools contain some of the most in-depth contract and transaction data in the industry, combining our own data engineering with 10+ top SDKs and APIs in the space. We also support analysis for all types of wallets: EOA, 4337, 7702, multisig - you name it.

How to Leverage the Herd MCP as a Builder

Let’s walk through developing something in crypto (a data pipeline/query, transaction script, miniapp, etc) using an agent. You must guide the agent through four phases:

(1) 🗺️ Research: Think of this as going wide to find the right contracts to call/analyze. There are many “unknown unknowns” that may not be documented. What’s the user journey through the contract? Who can participate during the journey, when, and through what contracts? Here are some example questions:

• Tell me about USDC on Ethereum. Who is handling minting? Can you check the last few mints?

• Find the basenames registration contract on Base. Can you figure out what contracts are called during registration? Which contracts can be called to mint basenames, and how is this managed?

• Look up the steakhouse morpho vault on ethereum. What are the key functions called by users versus other roles in the contract? What’s the market structure under the vault - where are those tokens located? Who can move those tokens?

(2) ✍️ Planning: Once you have a complete set of contracts to build on, it’s time to go “deep”. Here, you want to focus on studying wallets to understand different user patterns and to look for edge cases in transactions that help break any assumptions the agent makes.

• Given the steakhouse multisig, analyze how they interact with their steakUSDC vault. How does this compare to how the Gauntlet multisig interacts with the vault? At each state, what read functions or queries would you write to understand the allocation queue?

• Looking at the Nouns contract, analyze how the last five auction winners interact with the token after receiving the token. Are there any interesting patterns in how/where they transfer the token to (i.e. either trading or cold storage)?

(3) 🤖 Implementation: This is when you let the agent do its thing with code. You should enable more skills and MCPs at this point to help with its development cycles. Here are some good ones to add:

• Dune Analytics or Allium for SQL queries (MCP)

• Alchemy’s RPC (MCP)

• Trail of Bits Security (Skills)

• Coingecko Token (MCP)

• Privy Wallets (MCP)

• Context7 (docs retrieval MCP)

(4) ✅ Validation: Finally, you want to catch hallucinations or mistakes. You can do this by comparing historical onchain data to the outputs produced by your code. You can ask “Use Herd to look up relevant transactions based on our code, and check that the historical transaction results data matches the outputs from what we’ve built.” Some of these transactions likely already surfaced during the research/planning process, but it helps to explicitly tell the agent to check again.

Having the agent do this and document it also provides you with a reference you can share to help others build confidence in your/the agent's work, too.

🤫 Herd is experimenting with tools that let you create modular actions and queries for simulating, transacting, and reading from the blockchain. DM/email me if you are interested in trying this out.

Now you’re ready to go and safely build in crypto with your favorite agents/LLMs! You can always reach out to me at andrew@herd.eco or andrewhong5297 on Twitter and Telegram for any questions or feature requests.

Subscribe now

But if you don’t really understand crypto (underneath the tokens), it probably just looks like copycat stablecoins and some mystery yield are driving the newest wave of adoption. It becomes very hard to discern the underlying patterns and strategies at play, even though it’s all technically transparent and open on the blockchain. It’s also very easy to think that wild token incentives are still driving the growth. But the 2020-2023 era of pure incentives farming and rotations is mostly out the door.

Decentralized finance (defi) has evolved with each crypto cycle, each time condensing hundreds of years of learning from traditional finance and economics into onchain protocols that are more efficient, trustless, and permissionless than their offchain counterparts. First came wrapping assets into tokens, then wrapping tokens together into pools that facilitate lending and trading, and now wrapping those pools with portfolio management and complex structured product protocols called vaults.

The spread math for where yield comes from is conceptually simple: you can stack yields through multiple tokens by placing them in pools and vaults, where you earn fees from anyone using those tokens for financial transactions:

The spreads already somewhat exist in traditional finance, but are only accessible to large intermediaries (banks/funds), who then give you that 1-cent yield in your Chase account. In crypto, it goes directly to you as the user/liquidity provider.

Both the opportunities and complexity in defi have gone up exponentially over the last five years - I want to take the time to unwrap it all for you. You must know these fundamental concepts and how they’re connected if you want to navigate defi:

• Tokens: How billion-dollar token flywheels get built

• Pools: How tokens plug into markets for trading/lending, and capital efficiency

• Vaults: How liquidity is managed across pools for safer and higher yields

This article will cover more of the business “what” and “why” behind these concepts, rather than the technical “how” at the code level. All three of the above are operated using “smart contracts” that are deployed in sets that work together as a “protocol”.

@herd_eco will be hosting a workshop (and happy hour) in NYC this Thursday, October 30th, covering everything here and demoing our new product for creating/executing defi strategies and more.

Subscribe now

Understanding the Token Flywheel

A token is a smart contract that usually follows a specific standard (ERC20, ERC721, ERC1155). For stablecoin and aspiring blue-chip tokens, there are two fundamental goals:

• Sustainably manage the issuance and redemption of some asset, which gives it value

• Create liquidity loops on top of token wrappers that expand the yield of the token and its prominence in the ecosystem

Any token that can satisfy that dual mandate ends up with a strong growth flywheel:

It’s hard to follow a token across many constantly changing protocols (contracts), leading most people to think of a token as a singular asset and engage only in surface-level trading discussions. That’s the wrong way to think about a token.

Let’s start by understanding how token issuance and redemption are managed. Tokens with actual “collateral” underlying them usually have code modules that delegate minting and burning functionality to another contract or allowlist.

Below are some primary categories of stablecoins (with their owner/deploy date):

Fiat Backed: USDC (circle, 2018), USDT (tether, 2017), PyUSD (paypal, 2023)

• 1:1 pegged redemptions, with treasury bills (T-bills) and other high-quality liquid assets held offchain as backing. Most of the time it’s offchain, but some do onchain redemptions (Bold, Liquity). Anyone can now launch something similar by pairing Bridge (acq by Stripe) for offchain issuance/reserve management and m0 for onchain liquidity/cross-chain token management (mUSD, Metamask)

Over-Collateralized: Dai (Maker/Sky, 2019)

• Users “borrow” these tokens against collateral (originally ETH) they deposit, paying a borrow rate (stability fee) on the position. This pattern is called a collateralized debt position (CDP). Maker also offers a savings rate (akin to the Fed funds rate) on the token. The protocol can adjust both of these to maintain supply and demand and the soft $1 peg.

“Algorithmic” Stablecoins: Fei/Tribe (Fei Protocol, 2021), UST/luna (Terra, 2021)

• Tries to use fancy dual-token tokenomics to uphold value and prevent a run on collateral. None of them worked, but they’re worth reading about.

Delta-Neutral: USDe (Ethena, 2023), Neutrl (Neutrl, 2025)

• Users can deposit collateral (a specified set of tokens, such as ETH, USDT, etc.) to mint USDe, and Ethena takes a hedge against the collateral in open markets. This hedge is taking a short position on perps to earn money from the funding rate (which pays shorts most of the time) and protect against collateral value drops. Ethena has started offering a stablecoin launching service to manage both the issuance (partnered with Anchorage) and the yield onchain.

You can read more about these different types of tokens in this article. Generally, they have evolved to require more diverse sets of collateral with less over-collateralization (the delta-neutral strategies like Ethena are at 101% collateralization, versus Dai at around 270%). The algo stablecoin era is over due to the highly unpredictable nature of the fancy tokenomics model.

Next, let’s talk about wrapper tokens and their role in the flywheel. The canonical example of a wrapper token is WETH (Wrapped ETH), which turns the native ETH chain asset into an ERC20 token. This is necessary because most contracts are built around the ERC20 token standard, so converting to WETH simplifies the logic. You can also unwrap the WETH into ETH at any time, redeeming the underlying ETH. The amount of ETH is just tied to the amount of WETH, and has no ties to the wallet that originally wrapped it.

Wrapper tokens can be used by protocols to represent:

• Interest/reward accrual, where the wrapper balance is automatically increased over time (like lending in Aave aTokens). Sometimes these are referred to as “rebasing” tokens.

• Breaking down the underlying token into more modular pieces, such as splitting a yield token from Aave into principal and yield tokens (pendle). Or turning an NFT into a set of fungible tokens for fractional ownership (fractional.art).

• Wrapping many underlying tokens at once, such as in a liquidity pool (Uniswap) when you need to represent both tokens in the pair or in a vault where the wrapper represents a share of all tokens in the vault contract (MetaMorpho).

I gave only one example protocol for the cases here, but these are all common patterns used by hundreds of protocols on Ethereum.

Because wrapper tokens follow ERC standards, they can often be reused in other protocols. Let’s examine USDe (the best new example of a strong token flywheel, with a $12bn market cap as of 2025/10/22). You could stake it for revenue share from Ethena’s protocol revenue to get sUSDe, split that yield (YT-sUSDe) and principal (PT-sUSDe) using Pendle, supply the principal token into a Morpho Market to earn on PT-sUSDe/USDC, or deposit into a Morpho vault that holds a more diversified set of positions (where the PT market position may make up 40% of the vault) to get MC_PTS. Don’t worry if this jargon doesn’t make sense to you - we’ll review the types of pools and actions behind these wrappers next. The flow looks like this:

USDe -(stake)-> sUSDe -(strip)-> PT-sUSDe and YT-sUSDe -(supply)-> PT-sUSDe/USDC -(vault)-> MC_PTS

You could consider all of the following wrappers as “extensions” of the base USDe token. They have some set of extended value and capabilities that the base token does not. When analyzing a token, look for all the wrappers around it to understand its full value and growth (easier said than done, I know).

How Pools Work and Make Money

In the examples above, the wrapper token returned represents a share of a “pool,” which is the contract into which the underlying token is deposited. Pools have become a blanket term for any contract that holds the token deposits of many different users at once, to be easily utilized by other users. It’s a peer-to-peer (P2P) system with a protocol that manages all ownership, math, and order-matching logic. The point of pools has always been to maximize capital efficiency for a set of financial orders/actions and to earn usage fees paid to users.

Here are the main types of pools you need to know, in the order that they were developed and became popular in the ecosystem:

Lending Pools (Money Markets):

Actions: Lend, Borrow, Loop

Wrapper: Rebasing token on either the lent or borrowed amount, based on the interest rate (floating).

History: CDPs (like over-collateralized DAI borrowing) kicked off basic onchain lending, with Aave and Compound launching pooled money markets in 2019 where users could lend and borrow from shared liquidity. The pools all have some interest rate curve that has some target “utilization rate” that defines the supply and borrow rates. The slope increases slowly until the target is reached, then becomes very steep until 100%. Each token also has a custom set risk profile which determines the Loan-to-Value and Liquidation-Loan-to-Value parameters - Risk profiles and LTV/LLTV are set on the collateral, defining how much you can borrow against it. From 2020 to 2023, we saw many attempts to make collateral management and liquidation auctions more efficient (see Rari isolated pools, Alchemix self-paying loans, and B-protocol pooled liquidations). This is also where “flash loans” came from, where you can borrow and repay a large sum of money within one block with zero collateral.

Key Trend: Tokens have to be manually added to these markets through governance proposals (with some exceptions), leading to some heavily contentious discussion about riskier assets like CRV and other bad actors attempting to game the pool systems. As the crypto space onboards more institutions and adds RWAs as collateral, the requirements and counterparty risks of pools will become even more sensitive. On the institutional side, you don’t want to mix KYC and non-KYC funds so you end up with allowlisted pools like Aave Horizon RWA. For accessing a wider range of institutional borrowers and enabling undercollateralized lending, we see products taking normal deposits into a pool but then lending out to borrowers through privately managed and credit scoring/slashing methods (Maple and 3Jane respectively). This will be interesting to watch as we see thousands of stablecoins and RWAs get launched, because you will have this tension between issuers wanting to access the yield benefits of all these pools by being added in, but also the protocols managing the pools have to satisfy the institutional requirements which means being strict about the asset classes and participants allowed.

Trading Pools (Liquidity Pools)

Actions: Add/Remove Liquidity, Swap

Wrapper: LP tokens, which redeem for an equal share of the underlying pool tokens plus trading fees earned (usually configured between 0.05%-1% of volume). This pool token balances may have changed since the deposit, leading to impermanent loss (it’s permanent).

History: What started as a two-token pair with Uniswap’s xy=k curve to create an automed market maker (AMM) has evolved mostly in math and the granularity of the price curves (exchange rates) to improve price impact and idle liquidity in pools. There have been stablecoin-specific curves, weighted 8 token curves, concentration curves to manage liquidity in specific tick ranges so that you can control impermanent loss, and most recently pool hooks to allow more composability with other protocols before/during/after each swap. Unlike lending pools, liquidity pools have always been permissionless, so anyone can add any tokens to them (because pairs don’t affect each other in a “collateral” sense). A lot of focus in the last year has also been on token launchpads, which are heavily influenced by the trading curve tied to the initial liquidity. There have been many new protocols like pump.fun, clanker, doppler (used by zora) with unique liquidity bootstrapping or bonding curves that give a token more initial momentum, price discovery, and/or anti-sniper mechanisms.

Key Trend: Both lending and liquidity pools aim to minimize idle capital, so new protocol versions have introduced extensions that enable more seamless capital transfer between the two. Some examples are Balancer boosted pools which keep idle liquidity in a yield-bearing pool (like Aave), or Uniswap hooks that allow protocols like Eulerswap to push liquidity from their lending pool into Uniswap for JIT swaps. This kind of rebalancing is becoming more popular within each pool, too, with protocols like Arrakis managing concentrated liquidity price targets and Morpho’s PublicAllocator for balancing the target rate across lending pool markets.

There’s a lot more to be said on the constant debate about how orders are matched for trading pairs, be it AMMs, CLOBs, intent market makers, perp DEXs, etc. But those differences aren’t really relevant to the point of this article. For another time!

Staking Pools (Insurance Funds)

Actions: Stake, Lock

Wrapper: Some token with “s” in front of it, usually earning some split of protocol revenue.

History: With staking, you are usually putting up your tokens to be potentially slashed (losing stake) in a black swan/negative event. This can range from staking ETH on a validator and getting slashed if you fail a block proposal, to absorbing bad debt from lending pools like Aave. Sometimes it’s just meant to be a lockup to prevent immediate selling, so supply is more stable - Ethena does this with USDe -> sUSDe. In all these cases, the protocol will often share some revenue directly with stakers for taking on the extra risk/illiquidity.

Key Trend: Staking has always been and will continue to be a core component of maintaining protocol security and providing initial yield to users. It’s the simplest of the pools here, but core to the token flywheel. You might hear people say “restaking” which just means using an existing blue-chip token to stake in a different protocol (instead of issuing a protocol native token) - it’s the same staking mechanics, don’t overthink it.

Incentivized Pools (Gauges)

Actions: Vote

Wrapper: was common to add “ve” to the token being staked, starting with Curve (veCRV). You earn the protocol token as yield, on top of the usage fees from liquidity/lending pools.

History: Let’s say you’re a token issuer and you have all these lending/liquidity pools with somewhat similar yields. How can you push users to choose your tokens’ pools over another? By adding token incentives that can be earned, of course! Many protocols (regardless of if it’s a token, pool, vault, etc) will launch a protocol token for governance/ownership purposes. These can then be used to pay out liquidity providers over time (called “emissions”). So a pool protocol (like Aave) can emit tokens for deposits into their pools, and then Ethena could add ENA incentives on top of specifically the aEthUSDe Aave pool for more rewards. Emission pools emerged as a way to direct token incentives to specific liquidity or lending pools, helping protocols compete for user liquidity. Curve pioneered this model with vote-escrow (ve) mechanics, allowing tokenholders to lock CRV and vote on where emissions should flow (to boost their own rewards) in “gauges”. There were then many pooling layers built on top to fight for voting majority in the underlying protocol pools - such as Yearn, Convex, Tokemak, Butterfly, and Olympus. This happened in tandem with the airdrop and vampire attack era, where teams like Sushiswap would fork other protocols and do heavy incentives to drain usage from competitors. This is where the 1670% yields came from.

Key Trend: It was an ugly but notable part of defi pool history, incentivized pools still exist but are much tamer than before. See aerodrome and pendle for some modern examples of incentive pool voting. The good news is that a lot of the technology they built transitioned well into “vaults” for more serious defi strategy management. Yearn and Tokemak (Auto Finance) are two that have stuck around the post-farming era.

Strategy Pools (Vaults)

Actions: Deposit, Redeem

Wrapper: shares of the vault token, as a percentage. The percentage ensures you can automatically redeem any earned interest on any tokens in the vault.

History: Vaults largely emerged from the emissions- and incentive-based farming era mentioned above. They’re now standardized with ERC4626, and are fully focused on efficient allocation and rebalancing across pools. I’ll talk a lot more about vault structures in the next section instead of yapping here.

Structured Pools (Structured Finance)

Actions: Strip, Split

Wrapper: Really can be anything that represents a component of the underlying token (and what it owns/earns). The goal is to create a more liquid wrapper token. Commonly “L” prefixed, such as LSTs, LYTs, LRTs, Liquid lockers etc.

History: These are pools that take in an asset and restructure it into smaller components. Breaking tokens down creates more opportunities for wrapper reuse. Pendle has been the most successful example of this with principal (PT) and yield (YT) tokens stripped from yield-bearing tokens being implemented all across defi. There have been many attempts at “fractionalization” as well during the NFT cycle in 2022, such as turning NFTs into fungible shares for fractional ownership.

Key Trend: As defi becomes more mature, we can expect to see many more protocols that convert traditional structured finance products into their crypto native counterparts. This kind of pool is the most technically complex to implement and usually tough to communicate and coordinate with users.

Subscribe now

Vault Management and Yield Strategies

Many of these pools have yields that are isolated from one another at a protocol level but dependent through wrappers. How can a user easily manage all these wrappers and pools to optimize yield? The answer lies in “strategies”.

From 2019-2022, users had to discover and manage strategies across tokens and pools individually. Some popular strategies included “levering up” on positions, for example in a lending pool if you deposited $100 of ETH you could borrow $80 of USDC, swap that into ETH, and deposit it again to earn extra yield. You’re taking on the risk of getting liquidated if ETH price falls below the LTV, but otherwise earning yield (assuming the USDC borrow rate < ETH supply rate). Getting into the various farming vaults often required going through a set of swap/pool/stake steps across multiple transactions. To get around this, solidity developers would aggregate the function calls in a separate contract so that a user could sign just one transaction to enter some position - this was called a “zap” (first seen in defi zap). Products like Zapper, Instadapp, and Defisaver then built out numerous contracts for zapping into each new protocol that popped up, like a game of whack-a-mole. These are not simple to deploy either, given the security risks - you can read about Defisaver’s architecture for a glimpse of the complexity. As one would expect, it became impossible for any one team to keep up with every new defi protocol/version/vault. The retail products for stuff like this today do have 10x better UX than in 2020 - everything from leverage management with the products mentioned above or portfolio/ETF management with products like Reserve and Glider.fi.

So, naturally, it makes sense to have users deposit into a vault that manages different zaps themselves - and just call them “strategies” instead. Most vaults today follow the ERC4626 standard, which consists of:

• A user who deposits some token into the vault to get a vault wrapper token representing a percentage share of all tokens in the vault, where that share includes any yield earned in the vault that can be redeemed anytime.

• Functions in the vault for managing “strategies” that move around tokens and positions (sometimes these functions are called “adapters” since they have custom logic that needs to be adapted to specific protocols and pools).

• A curator who defines and executes the strategies over time and adds/removes tokens.

As with any ERC standard, there end up being many extension standards on top of the base standard (ERC4626), such as ERC7540. These extensions aren’t always made into a standard, and may be protocol specific such as the adapters in Morpho vaults v2.

The vault strategies right now are pretty similar: a vault supplies tokens to a set of lending/liquidity pools. Most of them don’t take on much/any leverage. For the more manually managed vaults, the best example is how Morpho curators work by managing caps and flows across different markets. Yearn and Lagoon are other great examples. Then there are more protocol automated vaults like Sommelier and Beefy. Vaults works for trading strategies too, see Hyperliquid Vaults. We’re in a period right now where teams are figuring out how to build the most flexible vaults (access the most pools/tokens) and execute and manage strategies within them.

The yield structure for the user (depositor) looks something like this (Morpho vault):

The “native fee” comes from underlying lending pool utilization across allocations, the MORPHO and SEAM tokens are emissions on top, and the vault curator takes a hefty 15% performance fee.

Now, vaults do have a transparency problem, in that users don’t always understand what strategies are actually underneath them. That’s something we’re hoping to help with using Herd, soon. Be careful when depositing into high-yield vaults.

Now tying everything we’ve learn back together in the token flywheel using the Ethena stablecoin USDe as the example, we get a flow diagram like this:

Every pool, wrapper, and vault adds to the value of the base USDe token, and as Ethena grows, the ecosystem will push for more complex pools and vaults to be built on top. All this creates more demand for minting the USDe token.

Naturally, the token issuer wants to own as much of this flywheel yield as possible. So it’s becoming more popular for the token issuer to offer vaults of their own, see Perena and Midas (tbd on how safe these are, I have just seen them mentioned here and there).

Concluding Thoughts

If you made it this far - first off, props for actually reading in the age of AI - you should now understand the history and growth of defi much better. Next time someone talks about a token or pool, you’ll know how to traverse the web of related tokens, pools, and vaults to have a much more nuanced discussion.

There is a growing problem with executing transactions across all these different strategies. Herd is working on a new standard and AI for easily composing these transactions, stay tuned for the follow-up article on our new product (or come to our event in NYC this Thursday, Oct 30th to see the live demo).

I’ll also note that I’ve left out many lower-level concepts and nuances related to token, pool, and vault operation risks. Many of these risks revolve around how to price a token across protocols, getting into the oracle problem (aggregating off-chain price feeds/calculations onchain as a source of truth for token prices) - I recommend searching up the oracles listed here. As well as the dark forest of MEV/sniper bots that run behind the scenes. If people enjoy this article, I can do another one on the operational “how” side of defi that covers these deeper and more difficult topics.

I’m always open to chat about any of this if you dm me - be it questions, critiques, or ideas!

Subscribe now

That is, until today. I created a trail on Herd that combines a Dune balances API with the 0x swap contracts, to make building this app trivial for anyone.

ICYMI: We built a next generation block explorer called Herd (herd.eco), available to everyone! We built it to make exploring onchain and building apps 100x easier. You can go to our docs for a quick overview.

Our main feature is called a “trail”, which is a series of onchain functions optimized for human readability and LLM tools (like v0 and cursor).

We have a Discord you can join to meet other trailblazers and ask questions.

I’ll walk through how I built this trail and then vibe coded the app, breaking down key concepts and components along the way. There is a full video guide covering everything in this article in much more depth too:

👉 You don’t need to be technical, we’ll be using a simple no-code interface and code agents for this whole guide.

Key reference links:

• The swap trail

• The app

• The github repo

• The v0 chat

Subscribe now

Building the Trail

First, we start by adding a “primary step” to the trail, which means a function that will be executed to create a transaction onchain. We click “add step”, and then toggle to the “swap” tab to add the swap component to our trail. This uses the 0x dex aggregator to route swaps between one token to another, with a specified “sell” amount.

So in the “edit” panel, we can see there is a buy_token_address, sell_token_address, and sell_token_amount. We’ll creator_hardcode the buy token to USDC, and let the user choose which sell token they want to use and how much of it to sell.

You’ll notice it says “user_input” before the form field, this is a derive method. We support many derive methods for inputs, such as using read functions and events, typescript code, and more - read about them here.

You could publish the trail now to test it out in the overlook (a simple sandbox that comes with each trail), where inputting 0.1 ETH (0x000…) gives us this swap:

Next, we need to know the balances of the user’s wallet to be able to know what tokens and amounts can be swapped. Dune’s SIM balances API is a perfect tool for this, giving us the tokens with their balances, USD values, and metadata in a single call. We add this to a code node like so, and enable the read API:

We can easily test the code node, which give us this response:

And that’s all we need in the trail! Now it’s time to build the app.

Subscribe now

Building the App

To start, we create an app in Herd and copy the guidebook prompt from the trail. All you need to do is click the “guidebook” button and then copy the prompt link

The prompt is just a long llms text file that covers how to integrate the trails API and each of the steps. You can read the one for this trail here, and our docs cover the guidebook in more detail.

We start by prompting v0 to create a simple mobile app that loads the tokens and lets users select any number of tokens as well as some percentage to sell (10%, 50%, 100%). One v0 starts to struggle on some harder requests, I moved to Cursor by creating a GitHub repo, so that they can share the same code and I can use the more powerful agent that Cursor provides. This part is easier to follow in the youtube video, so I won’t do screenshots here. With some UI tweaking, we end up with this:

For batching all swaps in one transaction, I just fed it the sendCalls and waitForCallsStatus from the viem docs to basically call the trail once for each selected token to get the swap calldata, and then batch all of them at once. This uses EIP7702 by default with a fallback to EIP5792 if wallet doesn’t support it. EIP7702 is a standard where you can treat your wallet as a proxy and set an “implementation” contract, meaning that you can now call your wallet from your wallet to call a function on the contract (i.e. your address is both the “from” and “to” on the transaction). This requires setting up this delegation, which some wallets might not have done yet. So, EIP5792 is just a wallet standard that allows you to just sign many transactions in one signature but each one is still sent onchain separately (effectively still providing you the same UX improvements).

And that’s it! In less than half an hour, you have a fully functional and sleek swap app. Please try it out yourself and share your app with me when you’re done! We have a community Discord where you can reach out if you have any questions or problems.

Subscribe now

I know that it’s been a long time since I’ve posted here, and I’m excited to get back into creating guides for you guys again 🤗

Herd has a Discord server too, come join and learn about new protocols and apps with the community.

From Launchpad Protocol to Launcher App in Minutes

Clanker is a token launchpad that has created over 300k+ tokens and done $3bn+ of trading volume. It’s a crowd favorite that started off on Farcaster in 2024, and has gone through multiple protocol versions (new contract deployments) since.

I built this simple token launcher app on top of Clanker in just a few minutes (no code at all) using v0 and cursor, by leveraging a one-click copy/paste prompt from Herd.

And all I had to do was copy and paste the “guidebook” from a Herd trail, which creates a specialized LLM prompt for any AI to easily build on top of these contracts:

What is a trail? How does building the app without coding work? Where does the Herd explorer fit into this? I know, I know, it’s a lot. Let’s starting by using Herd to find and understand a few key contracts.

Subscribe now

Exploring and Understanding Clanker Contracts

Our goal is to understand the full user journey of Clanker, meaning “what are all the functions a user is expected to call, with what variables and in what order?”

Here’s a video that goes in depth on the contracts and building up the trail - we’ll cover some of the main exploration steps in the rest of this guide.

One of the first features you’ll notice on Herd is that we have a more contextual search bar. When you enter a query, you’ll be searching across the protocol names, contract names, and functions/functionalities all in one go. Let’s start by searching for “clanker”.

The results are shown in a google style format with a short description, and some stats on recent transactions/wallets and deployment age. We’ll actually click the “protocol” results and open up the Clanker protocol page first, which gives us this:

The protocol page gives us the key links to their socials and documentation, to make it easier to start sorting through the noise. There is also a full chronological ordering of all contracts they’ve deployed, so you can easily see what is being used the most and also what they are most recently developing (looks like Clanker has some new auction and fee contracts in the last month).

By looking through the results (page 3), I can see there are two main “Clanker” factory contracts that deploy tokens - these match up with versions 3.1 and 4.0 in the documentation. We’ll focus on version 3.1 for this guide.

When I paste the 3.1 token factory address (0x2A78…7382) into the explorer, I’m taken to the contract page.

We have a custom solidity parser and LLM processing pipeline to summarize how a contract works - shown in the summary tab above. I can see there are a few ways of deploying a token, a few admin functions, and a way to collect rewards (presumably fees).

The user journey must begin with token deployment, so let’s start by finding that function. We can switch to the “functions” tab on the contract page and get this view:

Functions are ordered by actual onchain call stats, so you can see the most popular ones. They also include a short summary for quickly understanding what they do. Looks like there were 60 tokens deployed in the last day from this contract, let’s look at the function details page now.

This is where the meat of the information is - we have a summary of the function and all inputs/outputs, as well as showing what events are emitted by the function. The table below also shows up the most recent calls of this function, so we can very quickly find an example transaction to dig into and understand (you can filter for direct or indirect calls from an EOA too if you’d like).

Now let’s click on the first transaction, to better understand what happens when a token is deployed and if there are any other contracts we should be aware of. We can easily see that there were two contracts deployed in the transaction, one token and one pool:

From here, if I scroll down through the call traces I can see that there is another contract called at the end called “LpLockerv2” which seems to facilitate all token rewards. I can tell at a glance because of the event summary and decoded arguments are side-by-side in the trace card:

And that makes sense, because if we quickly examine the code tab in the contract page we can see that the “claimRewards” function we saw earlier on the functions tab actually just calls the LpLockerv2 contract with a “collectRewards” call.

Now, if you looked through the docs on the protocol page earlier, you might have found that there is also mention of vesting tokens. This can be seen in the transaction as well, where there is a vault deposit that emits an event showing the amount and end time of vesting:

Another way you could have found these two other contracts and information would be to use the “related contracts” tab on the contract page, where we can see what other contracts are called by Clanker contract (or vice versa too).

And with that, we have a great understanding now of how a token deployment works and what other functions the user likely needs to call!

1. deployToken on Clanker to launch the token

2. claimRewards or collectRewards (on Clanker or LpLockerv2) to claim fees

3. withdraw on ClankerVault for vested tokens

Creating the Trail

For actually creating the trail, the hard work is already all done! We can directly just paste in these transaction hashes to pull in the function and data into each step of the trail to reuse some of the past function inputs.

In the YouTube video from earlier, I cover this much more in depth, showing off a splits contract as well (which handles splitting funds between any number of wallets).

We hope you find the new explorer useful in sleuthing protocols, and you can always reach the team and the community in our Discord if you have any questions, feedback, or feature requests!

Subscribe now

Thanks to Storm, Jeff, and Mats for their review and input into this year’s guide.

I’ve left Dune to work on my own data product now, and have been setting up our data engineering and backend for the past few months. As such, this year’s guide will focus on Ethereum (EVM) data engineering instead of the entire data tooling stack. Most of the interesting product changes in the last year have been in data engineering anyways.

In this guide I’ll teach you how to navigate efficiently acquiring any onchain datasets you need for your app in a continuous ETL pipeline. Some topics we will cover:

• Backfills, and the increasing cost of raw data as chains scale

• Solutions to the “backfill and transform” problem, such as post-execution versus re-execution transforms

Here are some definitions you need to know to understand this guide:

• Client: Repositories containing software implementations of the Ethereum Virtual Machine (EVM) that execute transactions and process blockchain state, such as Geth, Reth, and Erigon. Often referred to as “nodes” when run.

• RPCs: Standard API interfaces on clients, which are then run by providers like Alchemy, so developers can query blockchain data without running their own nodes. Alchemy has a great sandbox for learning RPCs.

• Events/Functions/Transactions (Execution Lifecycle): Transactions trigger contract functions, emitting events as logs. Transactions are bundled into blocks. Transactions can be replayed for a full memory trace.

• Data Platforms (like Dune): Browser/cloud platforms that aggregate, decode, and visualize on-chain data, allowing users to create and share custom queries.

• Utility API Libraries: Developer libraries that abstract complex RPC calls into intuitive programming interfaces:

  • Typescript: viem, ethers

  • Python: ctc, ape, web3

  • Rust: ethers

• ETL, specifically transformations (dbt models, data lineage): The process of applying some processing logic to raw data to create abstracted tables called “models” that you can then use for app/analytics purposes.

As always, feel free to reach out to me with any thoughts or questions.

Subscribe now

Backfills, and the Increasing Cost of Raw Data

EVM data has three core raw datasets:

• blocks/transactions (top level transaction from/to/gas/data)

• logs/receipts (events emitted from contracts, tied to transactions)

• traces (full call traces of each transaction)

Shovel has a guide covering these basic datasets fairly well, and Cryo also provides many more almost 40 raw datasets in their cli tool:

Downloading all the data from past blocks is called a “backfill”, with continuous syncs called a “frontfill.” Backfilling blocks + receipts on Base (EVM) using Quicknode RPC costs almost $10k:

After discounts, most platforms quote around $4k for a Base backfill of traces and logs, and monthly frontfill cost of about $2k. You can optimize some costs here by running your own reth node on a service like Blockjoy for $650/mo (though keeping a node running and client up to date has its own labor costs).

Many teams try to backfill directly into their own database and then run transforms on top after (ELT), as it my take many transformation model attempts to get to the final dataset you want. This was much cheaper to do in the past with small chains like Ethereum mainnet (I remember when backfills only cost $200).

Blockchains have scaled wildly in the last year, with Solana and Base (EVM) leading the way. To give you a sense of scale, core Solana data takes up roughly 500TB and Base data takes up 50TB. With this current storage scale, you can add another $4-10k a month for storing, transforming, and querying all this data on GCP/AWS - then multiply this all by the number of chains you’re trying to support. These costs start to add up quickly if you aren’t careful!

Keep in mind that all chains are starting to hyper scale, with Base growing it’s gas limit exponentially (read as number of transactions/state) and Solana aiming for 1 million TPS. The devs are only working to minimize size of state that nodes need, not the poor souls who have to index and work with the data. So if you plan to store all raw data from a chain as part of your data strategy, you’re in for a lot of $$$ pain.

We’re also quickly moving away from a block-by-block processing model (goodbye 2 second block times) to a pure event streaming model (100-200ms per block, and even faster with preconfs). Again, this means that some of the slower data pipelines and delayed transforms you’ve been relying on will start to fall apart - make an error, and who knows when your pipeline will catch up to the latest block/event again.

So what can you do? It’s time to start studying different methods of backfilling and transforming crypto data!

A Mental Model for Crypto Data Transforms

Usually, the first iteration of the data pipeline involves tracking a few events from a set of contracts and then performing basic transforms on them before inserting them into a database.

Subgraphs on theGraph created the best developer experience (DX) around this pattern, letting you track events and factories and also saving necessary related parameters such as lending pool reserves and user params at time of liquidation on Aave. The process is as follows:

1. List a set of contracts and ABIs in a yaml/schema

2. Define a set of mapping (transform logic) typescript files that process each event into columns on the schema

3. Run these mappings over the whole history of the chain and persist these schemas to a database block-by-block

4. theGraph handles the raw data so you don’t have to deal with the RPC and storage issues of a full backfill

However, over the years these initial subgraphs ran into several limitations:

• Backfills can run into performance issues (so every time you edit a subgraph), and can also lag behind on faster chains.

• If you require some enhanced decoding, read functions, or RPC calls then you can’t use viem or other helper packages.

• Only supports graphQL, so aggregation queries and further transformations on top are painful.

Realizing these limitations, many of the data products from 2022/2023 contained some sort of “subgraph migration” feature, such as Satsuma (acq. Alchemy), Goldsky, Ponder, Subquery, Subsquid, and StreamingFast. All of these take different approaches at keeping the simple DX of subgraphs while solving some of the problems mentioned earlier. theGraph is of course also trying to tackle some of these, choosing StreamingFast as their core partner.

We’ll get into all the core data engineering problems in the following sections, but at this stage we can define our transform needs as follows:

Hook onto an event/function (like a Swap) and then gather “periphery” information at the time of the hook using a combination of packages and read functionality. This is also a good time to gather a filtered version of a larger dataset, such as full traces only when there was a Swap in the transaction.

Subscribe now

Post-Execution Transforms

So we know we can combine the extract and transform steps with some subgraph-esque hooks. Let’s call that a “post-execution transform,” defined as transforming the default raw data from an RPC after a transaction/block is committed. Below are some tools that I’ve found most useful for a specific use case.

Remember that it’s very unlikely a single platform will fulfill all your needs, so don’t get too attached to always using the same tool.

Let’s start with the tools that let you directly build from the RPC:

Easiest Self-Hosted Transform Solution: Ponder (Typescript Library)

The best way to do this without relying on a cloud vendor is by using ponder.sh. You can hook onto any ABI event, and then use viem to do any onchain calls/reads you need to enhance the data before inserting it into your database. Here’s a simple example of combining a hook and a read function together. You can imagine hooking onto a Uniswap V3 Swap event, then calling their TWAP oracle and combining that with reads on the tick movements at the start and end of the block.

I personally use ponder to power the Bytexplorer mint page to track ERC6551, ERC20, ERC721, ERC1155, and some factory contracts.

If you’re tracking a few specific contracts, need to make various API calls during your transformation, and want to store this data in your own database; then Ponder will be the simplest and lowest cost option when combined with an RPC provider like Quicknode or Alchemy.

You can check out envio for a similar solution.

Lightweight Cloud Solution: Quicknode (Typescript/Python)

If you want a more lightweight solution than Ponder with a simple key value store (like a mongoDB), then check out Quicknode functions (they also provide an RPC). I would pick this if you don’t need to join the data heavily to some internal database table, but still want something more flexible than a subgraph.

They have a great guide here with Aave. You can think of functions like a lambda on AWS, with three key steps:

1. Set up a “stream” of backfill data to run through the function

2. Write some arbitrary typescript/python scripts as a “function” that can be chained together to transform data

3. Store outputs in a key-value store that can be queried via API

4. If you want to store the stream directly in your DB with some logic on a stream, use “filters” in the stream settings instead.

I believe indexing.co is the only other platform to support functions as well, you can reach out to their team to test it out.

This is the most flexible and lightweight option out there, useful if you are working with object specific data (sets of wallets, tokens, and contracts that don’t require many aggregations or joins).

Quickest Self-Hosted Bulk Analytics Solution: Cryo (Rust, Python)

I already mentioned them earlier, but Cryo supports almost 40 raw datasets that are basic transforms on top of raw RPC data. Storm (the maintainer) gave a great high level talk on self-hosted data and where Cryo fits in:

You can find a more recent talk here too that goes more in depth

If you just want generic raw datasets for one-time analytics purposes, you should check to see if there are any supported datasets on cryo that you can easily pull with their CLI (run node or get an RPC → pull with cryo → saves in parquet files → query with duckdb). This is NOT for building a continuous pipeline, or for adding transforms during extraction though.

Now onto the SQL tools that let you build on top of basic tables built out of RPC data…

Transform/Query Without Your Own Data Warehouse: Dune (SQL)

My recommendation is to always use a cloud SQL data provider like Dune first to test out your transformation logic without cost, before jumping straight into the raw data with some of the tools I will talk about later. You should think of your contract data as a small subset of all the available data, and using a platform like Dune will help you build familiarity with transforms across the post-execution space. Everything from creation_traces (deployed contracts), ERC20 transfers, prices, DEX trades, ENS names, and more will be tables you likely create or use down the line. You can create your own query views and matviews for a pseudo DAG like this:

I often utilize the Dune API filters to test these transforms in my backend before ever loading large tables into my database - just remember to query filters on results and not re-execute the query each call (which will be really expensive). You can also upload small sets of your data to Dune for testing against some of your internal schemas too. It’s also great for daily or weekly jobs you may need to run, like Cowswap does for distributing solver rewards.

Remember that Dune data is not real-time, with large variations of freshness across tables (between 1 minute to 1 hour to 1 day), and the execution engine can run into queues/variance during busy times. With Dune, do not expect to be able to pull raw datasets into your own database without paying a lot of money. The ideal setup is needing to pull already deeply abstracted tables like dex.trades, where you can pull up the last 100 million trades (100m rows, 25 columns at 25,000 datapoints per credit with 100,000 credits) using the premium plan of $1k/mo. 100 million trades is roughly 2 months of trades on Base (as of december 2024). You can use functions like array_agg to pull as many values as you can into one cell so save on credits.

Allium has a SQL offering with a more powerful API and data extraction, that is real-time within 5 seconds and has a dedicated engine - albeit with fewer community example queries and tables.

Choose these options if you want a data solution where you don’t need an data warehouse of your own for onchain data. Your complexity here will be in finding and creating the SQL tables you need within the platform. You can expect to pay at least $1k a month at scale for immediate access to the most powerful data pipelines in crypto.

Easiest SQL-to-your-Warehouse Transforms: Goldsky (SQL)

Unfortunately, Dune does not have good data engineering DX. It’s not easy to manage pipelines of query views to sink into an external database. I’ve found that Goldsky has the best self-serve experience for this, with a DAG table builder:

They support many sinks (pushing data from queries into your database) for a large number of data destinations with one click, coming close to the ease of use level of a tool like fivetran. I believe they have some of the most startup friendly pricing for first-time backfills or academic usage as well, if you reach out nicely to the team.

You can create a DAG of queries in Goldsky the same way that you can in Dune, however Goldsky actually supports stateful transforms through streaming SQL with Flink (instead of batch like dbt). You also can interweave data from external handlers using http requests or webhooks, for you to externally manage some custom functions before pushing and joining with other Goldsky datasets.

Subscribe now

Re-Execution Transforms

If you’re a beginner data engineer, I highly recommend using one of the earlier solutions for whatever you are trying to do. When you get into more advanced and data heavy pipelines, you may need to study re-execution transforms.

Forking a chain with custom execution has been a common practice in Solidity testing workflows for years. The basic idea is that you can take a chain and it’s state, and copy, edit, and run it locally so that you can do anything you want to it:

• Submitting transactions at any block from any wallet (without the key)

• Editing contract code to add events or even new variables to enhance the data

• Adding opcodes or other low level functionality for more visibility (cheatcodes)

Once you’ve made your edits, you can replay a previous/future block and then use the same RPC endpoints to extract your enhanced data.

There has been an ongoing trend to build data tooling closer to the execution level, since any edits you make here will save you time doing expensive joins and queries post-execution.

The only downside here is that you have to learn Solidity and some lower-level VM lifecycle concepts to leverage these tools.

Easiest Events Editor: Shadow (Cloud or Self-Hosted)

Sometimes you will want to enhance the data around a key emitted event (like a Swap). Shadow makes it easy to add new events that let you capture more data around the exact time of execution. Note that getting data at transaction level granularity requires a re-execution environment, all the tools from post-execution can really only get you data at a block level granularity. This is very important for data like prices, liquidity, balances, health factors, oracle math, and more.

Here’s an example of their added events to Uniswap V3, namely making it easier to grab liquidity and price data during the swap. There is a registry of examples they’ve built to easily learn and run alongside their shadow reth node which takes advantage of some fancy reth client exex features. Don’t worry if that’s all word soup to you, basically just means you can edit events and run it yourself somewhat simply.

Emily Hsia, CTO of Shadow, gave a great presentation talking about using Shadow in your data pipeline below (2:39:18):

The RPC standard support doesn’t change at all when you edit events, so you can then use Ponder with a Shadow RPC to create a subgraph/hook data engineering experience on top.

Other options include ghostlogs and sim patch, but these are not open-source and self hostable (if that matters to you).

Most Powerful Re-Execution Hooks: Sim (smlXL, acq. Dune)

For more complex transforms, you may need data from different points in time of execution and not just a single event. You can create hooks inside of a smart contract using Sim, such as hooking on to every time a storage variable like a balance mapping has changed, every transfer event, or every call trace in a transaction/block:

You can easily get real-time balances at the latest block using something like Dune Echo API, but my use case required being able to get a balance at any historical block - even within a specific transaction. So one fun Sim canvas I created is getting balance changes of all addresses and tokens involved in a transaction (pre and post) using a combination of storage hooks and block hooks.

The logic in my lambda (set of hooks in solidity) is as follows:

1. For a given transaction, before the balance mapping is changed store the owner, token address, and current balance to a data mapping called “balances” with a struct of “BalanceInfo”

2. Each time the balance is changed, update the final balance value in “BalanceInfo”.

3. We keep only the last update, and at the end of the block we return the whole “balances” mapping with the complete before and after “BalanceInfo” data (shown in the test logs).

4. Nothing is saved to a database, I get to process this per API call using an “external trigger” component.

This kind of data extraction requires RPC tracing, storage layouts, and low level memory diff tracking to produce - the kind of expertise very few people in crypto have. Meanwhile I can create that pipeline using Sim in just 100 lines. Getting dynamic data like balances at a transaction level granularity was previously impossible, but now it can be done.

Sim is the data tool that I use the most for my new data engineering stack, covering 90% of my use cases. The other 10% is covered by Dune and Quicknode.

After these transformations above, you can then write SQL queries on top and execute those as API endpoints or directly sink it to your database like Goldsky.

Sim also supports stateful solidity transforms, meaning you can add a storage variable whose state carries on through all blocks. With a simple mapping variable, you essentially can create a key value store in your lambda!

Lightweight Custom Read Functions: TEVM (Typescript Library)

A common data gathering utility in crypto is the “multicall”, where you essentially create a new contract with a function that calls a set of other read functions when called. This way you can gather a bunch of data across contracts while only making one RPC call (saving you time and money). Viem actually has a powered up version of this in their simulate call that lets you override state too.

We can now take it a step further, and rewrite read functions on any existing contract completely. This is like editing events, but it is written directly in typescript making it even more lightweight as it doesn’t have the full compile/deployment process that Shadow and Sim have.

To be honest I have not built many examples with TEVM yet, but I have some fun public experiments coming up soon. I’m very excited about it so I’m including this tool in here now for others to check out, too.

Subscribe now

Concluding Thoughts: Tradeoffs on Transforms

All in all, your data pipeline will depend on when and where you handle your transforms (thanks captain obvious). Below is a little chart of the tools and concepts I’ve mentioned today and when they can be used:

Remember these are only a subset of all the crypto data tools, there are many others but they largely fall under the same usage archetypes that are shown above.

As a rule of thumb, I usually try and get a basic pipeline working on a cloud platform provider first and then decide later if I want to pull it out into an external pipeline/database later. Doing every dataset yourself from scratch each time will slow you down immensely and cost you both in infra and engineers.

In the end, you will most likely need a combination of tools here to meet both your app and analytics data requirements. The open source and open data nature of crypto means we’re given a lot of powerful tools and platforms, instead of just closed APIs.

Many analysts live in the world of Dune, and only know SQL. Learning the rest of the data stack with some of the tools here will be a way to take your skills and career to the next level, because as the space evolves, so will the number and quality of apps. Those apps will need good data. And good data requires great data engineers and engineering - I hope that will be driven by you.

Best of luck to all of you in 2025!

Subscribe now

This article was written by the amazing Pat Doyle - go follow him on Twitter!

If you want to be a guest author on Crypto Data Bytes, you can dm me.

In this article we will go through analyzing Solana data using only RPC methods. We will also have the corresponding analysis for Ethereum using only RPC methods. The goal here is to give data engineers and analysts coming from the Ethereum world a better understanding of how Solana data is structured and can be used in analysis.  

The audience should have some technical experience and familiarity with analyzing Ethereum logs and traces. 

Subscribe now

Requirements:

• Ability to run Python/Jupyter Notebooks

• Ethereum & Solana Nodes Endpoints (I am using Alchemy, the free credits should be plenty to run the examples)

We will walk through 3 examples: 

1. Getting the Latest Block 

2. Analyzing Compute Units / Gas Fees

3. Analyzing Token Transfers

I have included additional resources at the bottom of the article if you are brand new to Solana and want to dig deeper.

Get Current Block - Starting Simple

code for this section

Why do we want this?

Let’s start with a really simple example.  Getting the current block in a blockchain is always helpful. Why do I need this? Well, imagine I have a pipeline of data that is running on a schedule every hour.  To avoid over processing data we would only want to process our latest state plus 1 block and the most recent block.  This will allow our data consumption to move faster and our rpc request to stay lower.

Ethereum eth_blockNumber:

Solana getSlot:

Cool. That was easy.  Not much to it. Let's keep doing more analysis at the block level. 

Get Block Gas Fees & Compute Units

code for this section

Why do we want this?

I want to understand how network demand is changing over time.  Understanding the change in gas fees is a good way to get a view into this trend.  

Ethereum uses the concept of gas while Solana uses compute units.  They are slightly different but can be used to measure similar trends.

Ethereum

The ETH Block header data just gives us the value we need so parsing the data is actually pretty straight forward. We will use the eth_getBlockByNumber method and pass in the block number of interest in the params.  For each block we will return the gas_used / gas_limit and plot those values, with the current gas limit is a value of 30,000,000

Solana

In the case of Solana - The compute units need to be aggregated for each transaction within the block.  In the request, we set transactionDetails to True.  This returns all the transaction data within the block itself. The transaction data contains all the compute units of interest.  To do this we will use the method getBlock.  This method has a parameter that allows us to request all the transaction data within the block. As mentioned - The compute units used are within the transaction data so we need to set this to true.  Now we just iterate over each block, parse the computeUnitsUsed and group the values by block. To calculate the % of compute units used we divide our total used CUs by the max CUs per block of 48,000,000.

Subscribe now

Get Tokens Transferred in a Block

code for this section

Why do we want this?

Token transfers are at the core of every blockchain researcher’s tool belt.  They give us insight into where assets are moving, how balances change over time, token issuance and much more.

• eth_getLogs

• solana getBlock

• solana getAccountInfo

Ethereum

Ethereum provides nice and straightforward methods to retrieve logs.  Thankfully the ERC process has given us a set of standard contract events that allow us to simply and easily fetch this data.  For this example we will get USDC transfers that occur within a 10 block range and plot them.  First we begin by creating a function that will get the logs emitted by a contract. We use the eth_getLogs method and include the parameters in our payload:  The transfer event signature, the beginning block, the end block and the contract address.  Then we iterate through a block range, aggregate our data and plot it.

Solana

There are two different approaches you can take to pull the amount transferred per block from Solana. Each approach is slightly different and will yield different results for a few reasons. It is important to understand what you are analyzing and how these different approaches can be used.  One important thing to note that is a key difference from the EVM is that all Solana tokens share the same token program (SPL Token Program).  So in each instruction you will be able to see that a token transfer instruction will come from the SPL Token Program.  Compare this to the EVM all the event data will come from the specific token contract itself.

Approach #1: Parsing Transfer with Instructions

Lets walk through getting the transfers of USDC per block first.  Buckle up, this isn’t as straightforward as ETH but still doable.  First, what we will do is request all transactions in a range of blocks. The function below will return all the transactions and transaction details in a block.  

Once we get all the transactions we have to do quite a bit of data filtering and cleaning.  A single transaction can have many instructions. Each instruction can then have an inner instruction.  Transfer instructions can exist in both the Outer and Inner instruction.  There are also different types of instructions that indicate tokens moving - these would be transfer, transferChecked, mintTo, mintToChecked, and Burn.  We need to make sure we parse all of these instruction types. 

For all the transactions we will need to filter for transfers from the SPL Program where one of the transfer types above is present. 

Ok, great - progress. But there is one big problem.  For standard `transfer` & `mintTo` types we actually do not get any information about the token. We need the token mint address, the decimal precision.  

Below is an example of a transfer of `wSOL`

It tells us the sender, recipient and the amount transferred but we can see the token address is missing.  Without this information our analysis will be incomplete.  The reason for this is basically a detail of SPL Token Program/PDAs works. Remember, when you create a transfer of an asset, the program creates a Program Derived Account which actually holds the balance and maps back to your EOA wallet.  If this is a bit confusing it might be worth revisiting how tokens work on Solana.  Andrew Hong has another great write up on tokens here.  

To get the information about the token what we actually can do is make an additional RPC call for each Source / Destination address in the transfer.  We use the getAccountInfo method and call this function for each source/destination address in a transfer.

This function will return the Owner (or the EOA), the mint address (token address) and token decimal and amount they hold (we don’t need this but I thought it nice to show). I think of it similar to the balanceOf function on an ERC20 address.  Below is an example response from the getAccountInfo method. 

Finally, we arrive at a data frame with all the necessary information we need to start analysis.  We can filter our dataframe for just USDC Transfers, sum the amount, group by block number and plot. 

With this data we can start to do all sorts of token analysis.  Getting to this point is not a linear path and requires multiple RPC calls. This is the best approach to getting transfers. 

Approach #2: Parsing Transfers with Pre & Post Token Balance Changes

Each transaction includes details about the balances of accounts before and after a transaction.  So a simple approach to calculate the total value that was transferred within a block is to calculate the balance change for each address and token involved in the transaction.  We can then sum these up and group them.  This approach will tell us the balance changes and thus we can back out the transferred amount.  However, these balance changes are different from transfers since a wallet can send/receive the same amount of an asset in a transaction and will have a 0 balance change but there will be transfers in the instructions. 

After we parse the balance changes we get an output data frame that looks like this:
We see the Owner, the amount of the token before and after the transaction, the change and the token address.

From here we can start to filter for our token of interest. In this case it will be USDC and then we will plot it. 

Remember, this is the path of least resistance - However, what we are actually tracking is the total amount of an asset that changed wallets in a transaction.  This transaction is a good example of why this technique is not 100% accurate for tracking transfers. 

How do our 2 approaches compare?

Let’s put our two approaches side by side and you can see that the charts follow a similar trend. 

We can see that in block 294291897, our transfer values do not line up. While the general trend remains consistent it is not a 1 to 1 approach.  I wanted to show both techniques because I think it is important to explain that transfers & balance changes are not the same across the board. 

What’s next?

I hope this was helpful as a baseline for getting some of the fundamental analysis going on Solana.  In part 2, we will dive deeper into analyzing a specific protocol using just RPC methods and the program IDLs.  If you have feedback, comments, questions please feel free to DM me on Twitter/X.

I am always open to feedback and ideas. 

Additional Resources

• A Guide to Solana for Ethereum Analysts by Andrew Hong is a great resource for learning terminology and concepts for Solana. 

• A Guide to Solana Token2022 by Andrew Hong

• MEV Monitoring on Solana by Ghost

• Solana Docs have a ton of great resources as well. 

Resources mentioned in the video:

• Slides

• Solana Dex Dashboard

• Example ETL Query

• Solana Table Docs

• Program Quickstart Dashboard

Subscribe now

Other Beginner Resources:

• Crypto Data Analyst Roadmap

• Beginners SQL and Ethereum Guide

• Basics of reading a block explorer

It’s easier and more fun to build with a community, come join the Bytexplorers to learn and earn during your data journey.

Oftentimes you start looking into a protocol, and quickly find an example transaction that calls 5+ contracts and evoke 20+ logs. You go to their docs (if they have any) and maybe see something like this:

How can you navigate these solidity contracts to find the answers and data you need for your analysis? I’ll cover a full example with my tips and tricks in this guide. We’ll be studying the new Zora x Uniswap contracts, deployed in early August 2024.

You should try searching on Dune for existing queries/dashboards covering a protocol first, since it is usually easier to read/fork existing SQL than to go straight into reading Solidity.

Subscribe now

Starting From Transactions

Whenever you start analyzing a protocol, there are two example transactions you want to find:

• The main contract deployment transaction (deploying a new nft or uniswap pool)

• The main action transaction (swap, mint, etc)

I say “main” because oftentimes there are many contracts and actions, but they usually all stem from one starter contract/action. Most of the time your actual start comes from a tweet somewhere, and you have to work your way to these two transactions.

Let’s take the example of Zora’s new uniswap extension on their protocol - essentially allows you to create an ERC1155 that can be traded on uniswap as secondary as an ERC20 instead of opensea/blur. They tweeted out this short overview guide and also their first mint on this new protocol. In true crypto fashion, the real documentation doesn’t come till much later (or sometimes never). You should still try searching for documentation first, but this example will rely on purely onchain data for sleuthing.

I normally find the easiest way to get the main contracts and transactions is by just interacting with the product myself. So I minted some limitless zorbs when it first went live.

This transaction gives me the main action I can refer back to later for volume calculations, but more importantly it gives me two of the main contracts to refer to:

1. ZoraTimedSaleStrategyImpl: I know this must be central to this new protocol upgrade since its the main “to” contract, and likely is where configurations for mints/secondary are set

2. Limitless Zorb: I know this is the ERC1155 token that somehow gets converted to an ERC20 for uniswap trading.

Contract pages on explorers always have a “creation tx hash” link (usually you are first taken the token page, then you have to click on the contract again to get to the contract page).

So if we quickly aggregate what we have, we get:

• ZoraTimedSaleStrategy (ZTSS)

  • created txn - a quick look tells me this is NOT a factory contract, because it calls a “Deterministic Caller” contract which is used to deploy with CREATE2 instead of CREATE. This deploys it at an exact address, usually so the same contract can have the same address across multiple chains.

• The ERC1155/ERC20 Collection (Limitless Zorb)

  • creation txn - I see a “factory” is called to deploy this, so likely this is a factory created contract where each collection gets its own deploy.

  • a mint txn - the one I sent from the product interface

  • we don’t have a secondary/uniswap transaction yet

With this base in mind, we can define a question and dive into the code.

Subscribe now

Diving Into the Contracts

Alright, once you have a general lay of the land with the main contracts and transactions, you will be ready to start sleuthing around. You should lay out your questions and try and navigate the code to answer each one, for this guide I want to answer the question:

OUESTIONS: “How is the Uniswap pool initiated after the mint period ends, who controls that first LP position, and who gets the swap fees from it?”

If you think you’re good at this, try and answer this question using only the links I’ve given thus far.

We can guess that permissions/functions on sales are tied to the ZoraTimedSaleStrategy (ZTSS) contract, and not to the collection contract itself. Funny enough, I don’t see any events from this contract emitted in the collection creation transaction - that’s a signal for me that you can create an ERC1155 and likely attach on the ERC20/timed sale component afterwards.

I go the ZTSS contract code page on the explorer, and first thing I see is that it’s a proxy. This means that the “proxy” handles storage of variables but there is an “implementation” contract with the actual contract logic. You usually have to go to the “read proxy” or “write proxy” tabs to then get the implementation address:

⚠ WARNING: Proxies can be upgraded to point at different implementations to change the logic. The proxy in this example was upgraded on 8/23/2024 to this implementation instead, but the rest of the guide references the old implementation. This is to introduce a new launchMarket() condition.

Alright, now we’re getting somewhere…. kind of. Looking at the code explorer, I can see there are actually dozens of inherited/referenced contracts here.

A lot of this you can straight up ignore - however the most important ones to keep an eye on are the “interfaces”. Interfaces are used to let one contract call another contract function. I know that Uniswap is actually a different set of contracts, so I want to look for where the uniswap interface for creating/adding liquidity to a pool is.

This is as simple as ctrl+f “IUniswapV3Pool” which I see in the imports at the top of the file. I get 5 hits, 2 of which are the import line and 3 of which sit on lines 285-290.

Scrolling up a bit, I can see that it’s this “launchMarket()” function being called. The notes here say that anyone can call it after a primary sale ends to create the uniswap pool.

Now if I scroll back down, I will notice that I don’t see the uniswap pool being created or liquidity added actually - just a price being set. Instead, the end of the function has an “IERC20Z” interface being called with the “activate” function.

Subscribe now

Alright - so now we know we have to find an ERC20Z contract somehow and check the “activate” function definition. Your mind should hopefully go to finding the ERC20Z related to the limitless zorb collection. There are a few ways to go about this:

• Expect a read function on the collection to get the erc20 address

• Look for some event from the collection creation txn for the erc20 address

• Look for some event/function on the ZTSS that ties the erc20 contract to the collection contract

The first two are actually dead ends in this case, because the erc20 is an optional component of the collection. If I go to the ZTSS proxy page and the “logs” column, I can see a bunch of SaleSet() events that seem to contain a “erc20zAddress” and “poolAddress” fields. This is not always the most foolproof method, I recommend you plug in the contract into my EVM quickstart dashboard and check examples of the most emitted events/functions.

I’ll click into one of the recent SetSale() transactions to see what is going on. By following the logs, I can see that the ERC20z and uniswap pool are actually already deployed (CREATE2) when the sale extension is activated! It just doesn’t have any liquidity yet, so no trades can be made.

The ERC20z is also deployed from the ZTSS contract, so I can be 100% sure there is some read function to search up a collection and gets its ERC20z (and uniswap pool) details. Sure enough, we have a “sale()” read function:

I plug in the Limitless Zorb collection address and token id (from the URL) and get our erc20zAddress and poolAddress! You could have also found this by ctrl + f “IERC20Z” in the ZTSS implementation code if you had noticed it earlier 😉.

Alright now we the erc20z address - so now let’s get back to looking for the “activate()” function. It’s a proxy, so I again go “read contract” → “implementation” to get to the actual code. Then I just ctrl + f for “function activate(“ since that’s how functions are defined in solidity.

Within that function, I see this key line that calls the Uniswap position manager through an interface to create the first liquidity position:

It actually goes to a “royalties” address! Alright, one more ctrl + f to find where royalties is defined.

It’s a public variable, which means it automatically gets a read function. I go read contract “royalties()” to get the royalty contract address:

We’re so close now - let’s go to the royalties contract code next. This one is simple in terms of there are just two relevant public functions: claim() and claimFor() which passes the fees from the liquidity pool position to the creator of the token. The complexity here lies in reading “internal” functions, which are basically functions that can’t be called outside the contract and are called within the external functions.

I usually paste the code into chatgpt and ask it “Here is a solidity contract. Can you draw a node graph that shows all function call invocation flows? Show public/external functions in orange, internal functions in light blue, and interface functions in grey. Display this as a tree diagram from left to right, where the start of a function call is on the left. Increase figure size to fit all nodes.”

So the the function invocation flow we’re interested in looks like this:

• claim() → _claim() → _collect() / _transfer()

You will only see “claim()” called in the transaction traces since in-contract calls don’t have traces (besides memory traces but we won’t get into that). _collect() gets the fees from the pool by calling the position manager uniswap interface, and _transfer() passes those fees to the creator.

Notably, the _transfer() function has a fee that goes to Zora team.

We can find a public variable feeBps in getFee() function, so we read it from the contract “read” page:

Its in “bps” which means basis points, where 100 basis points goes into 1 percentage point. 2500/100 is 25, so they take a 25% fee off of the initial LP rewards! If you go to the uniswap pool you’ll see it takes a 1% fee on all swaps. So effectively, the creator takes a 0.75% fee and Zora takes a 0.25% fee on all swaps.

Notably, there is no withdraw LP position function anywhere in the contract - so that LP position can NEVER be withdrawn from the pool. The creator (and zora) can never rug the pool, albeit those ERC20/ETH LP tokens are unclaimable now too (unless you buy out the whole curve).

All in all, we have our answers now:

• How is the Uniswap pool initiated after the mint period ends? The launchMarket() function, and anyone can call it. The pool is already deployed with the erc20z on SetSale() event.

• Who controls that first LP position? No one, it goes to the royalty contract and is burned. The royalty contract can still collect fees from the position though.

• Who gets the swap fees from it? It’s split between Zora and the Creator, effectively 0.25% and 0.75% respectively.

Subscribe now

Contract Mastery

What a journey! If you’ve follow everything here, then you truly have a grasp of how to navigate across contracts, interfaces, proxies, and transactions. Knowing how to do this by only relying on block explorers and tools like Dune will make you a super analyst.

Again, all of that was to answer “How is the Uniswap pool initiated after the mint period ends, who controls that first LP position, and who gets the swap fees from it?”

Our list of found resources over this journey:

• ZoraTimedSaleStrategy (ZTSS), (impl)

  • created txn - a quick look tells me this is NOT a factory contract. So it must be shared across all collections.

  • SaleSet ERC20/Pool creation txn

• The ERC1155/ERC20 Collection (Limitless Zorb)

  • creation txn - I see a “factory” is called to deploy this, so likely this is a factory created contract where each collection gets its own deploy.

  • a mint txn - the one I sent from the product interface

  • erc20z address (impl)

• Royalties Contract (not a proxy)

You’ll pick up patterns on how functions are structured and stuff like system parameters and deploys are managed the more you dig into the data. Keeping a running list of notes like I have above when doing your research will make your life a lot easier too.

Happy learning, and good luck with your queries!

Learn about OpenRank and the eigentrust algorithm, and check out the example python notebook (with input graphs from dune queries)

🏆 We have a competition to create the best graph datasets ending next Sunday (August 18th). Join the discord if you have questions.

Eigentrust: Hubs and Authorities (HNA)

Make sure to read the original eigentrust overview to understand the algorithm. To review, there are two main inputs:

• localtrust: this is your measurement of interactions between two nodes, as node “i” signals some value to node “j”. This could be a token transfer, attestation, vote reply/like, etc.

• pretrust: this is your "seed" selection of nodes who you think should be more trusted in the network.

Now, this works if all of your nodes are homogenous (i.e. all the same type). But many interaction graphs in crypto are not just wallet to wallet, they often include a mix of different contract interactions instead.

To capture this nuance, we create an interaction graph of node type 1 to node type 2, i.e. a wallet interaction with an NFT (mint/trade). We then take the matrix dot product of this graph and it’s transpose, essentially creating a relationship between all minters of the same NFT(s) - think of it like a Wallet to NFT to Wallet matrix. This end matrix is our new localtrust! We refer to it as a “round trip” in our examples. It’s still the same eigentrust algorithm used from here on out.

Pretrust methodologies stay the same, the notebook shows off many different “variants” for selecting a pretrust seed.

You can find the step by step calculation and model creation in this jupyter notebook. This contains a toy example not covered in the video.

A video walkthrough of the zora NFT recommendation feed can be found below:

Subscribe now

📹 Watch this OpenRank overview video (with hands-on code walkthrough)

Mission quests are partnerships with foundations across the crypto ecosystem to openly discuss and answer high-value research questions. In the last mission, we spent six weeks helping the Optimism NumbaNerds research all aspects of their 4th airdrop.

For mission #2, we’ll be creating reputation graphs that power feeds using OpenRank. The mission will run until August 18th, 2024.

Explorers can discuss their ideas and questions in the Openrank discord or mission central channel of our telegram. We’ll also host weekly office hours in the discord on Tuesday evenings (6pm EST) and Saturday afternoons (1pm EST).

Building with Graph Algorithms in Crypto

Learn about OpenRank and the eigentrust algorithm, and check out the example python notebook (with input graphs from dune queries)

Here’s an example using “hubs and authorities” in a slightly more advanced approach.

Openrank is a platform that computes different kinds of graph algorithms for you, given a set of input graphs. It specializes in peer to peer graphs, and are a perfect fit for crypto due to the nature of addresses in unifying identities and interactions onchain/offchain.

The eigentrust algorithm takes in an interaction graph and a set of trusted nodes inside that graph, and outputs a ranked score of all nodes. These two inputs are called “localtrust” and “pretrust” respectively:

• localtrust: this is your measurement of interactions between two nodes, as node “i” signals some value to node “j”. This could be a token transfer, attestation, vote reply/like, etc.

• pretrust: this is your "seed" selection of nodes who you think should be more trusted in the network.

In most interaction graphs, the action “value” is exponentially distributed. Think of voting power, transfer USD values, cast likes/recasts. To normalize trust across the graph, nodes that are close to the “pretrust” seed set will inherit half their trust value from the seeds.

These inputs go into an eigentrust model, which gives us a ranked value of all connected nodes. You can then weight the activity of those nodes to curate a feed, such as weighting interactions with contracts in the last seven days to create a “trending contracts” feed:

The easiest way to create these two inputs is by using Dune (if you’re using onchain data). In my example notebook that generated the feed above, I created a transfer token graph of all casters in the /base channel on farcaster for my localtrust graph, and then created a pretrust seed by ranking all casters by engagement on their casts.

There is a lot of creativity and flexibility in curating these two inputs, which we will aim to explore in the competition.

Subscribe now

Competing to Create the Best Graph Inputs

Knowing python is not strictly necessary, you could give us your input graphs and we’ll help run the eigentrust algorithm for you (and upload the results back into Dune so you can test against feed queries).

Below are our three feeds that we want to power with openrank. We’ve listed a few ideas for generating pretrust and localtrust graphs that you can start building from. These are just suggestions, you can be as creative as you would like.

• (submission) Optimism Token House: Recommend governance proposals using Agora feeds.

  • Pretrust Ideas: Top delegated to (OP only or many tokens), most active voters, top token holders, active token house or grant members in forums.

  • Localtrust Ideas: Delegate to, voted on proposal creator. Check out submissions to mission #1 to see example of delegation data. You can also use github repo data, put together by Open Source Observer.

  • Feed Query: Use the governance_optimism.agora_proposals table in Dune.

• (submission) Trending Mints: Recommend top NFT mints by chain.

  • Pretrust Ideas: Top NFT traders, top minters, age of wallet, top creators of NFTs, first minters

  • Localtrust Ideas: You could use nft transfers, token transfers, interactions on zora, interactions on farcaster

  • Feed Query: Use the nft.transfers table in Dune (example query). See Zora mints example here too.

• (submission) Degen Cast Tipping: Recommend top users and casts to tip.

  • Pretrust Ideas: Top DEGEN holders, top tippers, top tipped, number of unique tipper (or tipped) accounts, most influence on farcaster, power badge users. Some example user rankings in this dashboard.

  • Localtrust Ideas: tipping interactions, reply/recast/like interactions (filter for quality of engagement). Reference queries in this dashboard for DEGEN data.

  • Feed Query: Use the dune.neynar.dataset_farcaster_casts table to show the feed of recommended casts (group by fid if you want to recommend users).

Remember that the point of these graph generated feeds is to such that we can curate different types of feeds that fulfill different purposes. Creating a Zora focused feed based on first-minters seed will give us different recommendations than a generic NFT minting feed using a farcaster frame minter seed or a top NFT trader seed.

Think about the end feed you want to generate before you tackle the mission. You’ll likely also want to scope down on your feed at the start, for example I started with just /base casters in my example so that I only needed to compute 50k addresses of graph data.

There are four steps to create your own feed:

1. Create your localtrust and pretrust queries

2. Input your queries into the eigentrust algorithm using the sdk

3. Fetch your eigentrust results and plug it into a feed query, using the score for some weighted ranking of objects.

Again, you should reference the example python notebook to see these steps in depth. These are reference links you should be aware of:

• OpenRank Docs

• Python SDK repo

• Python Notebook

  • Another Python Notebook

• Dune feed dashboard

Mission Participation, Rewards, and Guidelines

We’ve made the process and rewards simpler than the last mission.

• Feed Races: We’ll be using jokerace, there will be one race to submit into per feed.

  • Governance feed submission

  • NFT mint feed submission

  • Degen cast feed submission

• Submissions: Explorers will get a maximum of two submissions per feed race. Only one reward can be won per race. Your submission should be in the form of a jupyter notebook (github, google colab, hex, etc), and should include your input datasets, output graph, and an output feed example.

• Rewards: For each feed category, the top submission by votes will earn $300 USDC. Second will get $200 USDC, and third will get $100 USDC.

• Voting: Only the OpenRank team members will get votes, they each get 5000 votes per race. Criteria for voting are expanded on below.

You will need a Bytexplorer pass to earn rewards, you can still submit your work without a pass though. Work must be submitted by August 18th 2024.

Winning Criteria:

• Novelty of localtrust and pretrust queries: There are lots of combinations of onchain and offchain interactions between addresses. We’re looking for the most creative ways you can combine these into an interaction graph.

• Size/coverage of the output graph: It’s important to have well connected graphs, so that our feed is not missing large portions of activity. For example, if there are 10,000 casters in a farcaster channel then having at least 3,000 scores in the output of eigentrust will be ideal.

• Subjective quality of the feed results: Different graph inputs will give us different kinds of feeds. You should provide rationale on why this feed is useful and different than just a simple ranked feed based on value/count.

• Well explained rationale: Clearly written logic explaining your choice of graph inputs, ranking method in the feed, and use case(s) for the feed. This is the most important part of your submission.

Explorers can discuss their ideas and questions in the Openrank discord or mission central channel of our telegram. We’ll also host weekly office hours in the discord on Tuesday evenings (6pm EST) and Saturday afternoons (1pm EST).

Subscribe now

Solana Learning Resources:

• 4 part Starter Guide to Solana data

• Ethereum <> Solana Translation

• 1 hour video guide to token transfers and balances

Subscribe now

The Solana Token program (spl_token) has existed for years, and I’ve covered before how the token/account structure works with fungible tokens and NFTs. It will be helpful background context, but is not required knowledge to follow this guide.

Token2022 is the newest iteration of that program and enables much more flexible token mechanics through “extensions”. It’s often referred to now as the “Token Extension Program”. The program is already being used by companies like PayPal for their PyUSD stablecoin.

I’ll walk through the following concepts:

• Differences between spl_token and token2022

• Understanding the instruction and enum patterns

• The new transfer lifecycle

• Tables to query on Dune

• ZK and Compressed Tokens

Below are some important links that I will refer to:

• Token2022 on Solana.fm (Block Explorer)

• Token2022 Dune Dashboard Overview

• Solana Token2022 Docs

• Token2022 Rust Crate Docs

  • spl_token Rust Crate Docs

• Token2022 Rust Code Github

  • spl_token Rust Github

Big shoutout to Jon Wong for answering all my questions, be sure to follow him to keep up with all Solana developments.

token2022 versus spl_token

The overall account structure has not changed with token2022, a single token mint account is still created per token and an associated account is created using the same associated token (atoken) program. Remember that wallets can be assigned as the own of an associated account, and each associated account can hold SOL (for rent) and one token mint balance.

Token2022 also retains the same transferChecked() instruction and arguments, as to be easily backwards compatible (the “transfer()” instruction is gone though). You can also still create metadata and NFT edition accounts using the same old Metaplex token metadata and master edition programs.

However, a lot of new functionality has been added. Let’s take a look at all token extensions (official docs):

✅ = live, ⏲ = in development

Mint token account extensions (these affect ALL created associated accounts):

• ✅ transfer fees: set a percentage fee to take on all transfers. You can set a maximum fee to take as well.

• ✅ closing mint: set an owner who can delete the token mint forever.

• ✅ interest-bearing tokens: set an interest rate for ALL tokens to accrue at. Can change the rate at any time.

• ✅ transfer hook: assign a program id, where after every transfer a CPI to that program will call an arbitrary “execute()” instruction on the assigned program.

• ✅ non-transferable tokens: tokens cannot be transferred after mint. Can still be burned.

• ✅ permanent delegate: approve a delegate who can control transfers of any amount forever

• ✅ default account state: sets default account data

• ✅ immutable ownership: account owner cannot be changed (this is different from the program account owner)

• ✅ metadata pointer: pointer to metadata program id

• ✅ metadata: token metadata (basic program contains name, symbol, uri)

• ⏲ group pointer: like collection mint id from metaplex standard, which groups tokens together (useful for NFTs).

• ⏲ group metadata: metadata of the collection

• ⏲ member pointer: id for a set of tokens within a group

• ⏲ member metadata: metadata/permissions for the set of tokens

⚠ IMPORTANT NOTE: All of these extensions must be initialized before the mint account is created. You can’t add or remove extensions after creating the token.

You can also combine most of these extensions, with obvious caveats around non-transferable tokens.

Associated token account extensions (set per account):

• ✅ approve delegate (same as in spl_token): set a delegate who can control transfers up to a certain amount

• ✅ freeze (same as in spl_token): set an authority who can freeze and thaw a token at will (make non-transferrable).

• ✅ memo: call the memo program with some text string, after a transfer. requirement is set on a per-account level.

• ⏲ confidential transfers: keep transfer values private but source and destination accounts are public, account owner must opt-in and create an encryption key. Can deposit and withdraw from the private account.

  • ⏲ confidential transfer fees: take fees on private transfers, this is set on ALL accounts (no opt out).

• ⏲ CPI guard: Disable certain cross program invocations (CPIs) to the token2022 program. This intends to allow for wallet to wallet transfers only.

If we look at all the token2022 mints created so far, fee tokens have been the main extension utilized in these new tokens.

Subscribe now

Breaking Down Instructions and the Enum Pattern

If you’ve looked into Solana data before, you should be familiar with “discriminators” which are the signatures of specific functions on a program. If not, I’ve created this dashboard for quickly breaking down any decoded program in Dune and giving you some data intuition. One of the queries I constantly use for identifying discriminators is this sleuther query (this one works even if the program isn’t decoded).

Read this guide if you want to learn more about discriminators and program types

I’ve hardcoded the token2022 program into that sleuther query, showing the instructions and call stats over the last 365 days:

As it is a native program, the discriminators are 1 byte and follow a nice and orderly number system. There are 41 different instructions - it’s worth looking at each instruction’s example tx and then also its respective instruction code and logic in the processor file.

Most extensions use an extra enum on top of the discriminator to decide which extension instruction to invoke. Let’s take a look at the transfer extension, which is enum number 26 (0x1a). It contains the following nested enum instructions:

You can see there are actually 6 instructions nested within this extension, numbering 0-5 respectively. So if I’m filtering for the InitializeTransferFeeConfig instruction, I would check that the first 2 bytes of the call data are equal to 0x1a00. Unfortunately, this means you’ll need to do manual decoding of any call data variables since Dune does not handle this nested function format. You can look at my authority tracker query below for some examples.

On Authorities

Authorities for who can freeze accounts, change fees and interest rates, add hooks, and more can be changed after initialization. This honestly does not matter much today, but as people build more programmatic ways of changing these variables it will matter more. This can be changed with setAuthority() instruction, where an enum in the call data decides which authority is being changed:

I’ve created this query that reconstructs all authorities for a token mint, you could adjust any subquery here to get historical authorities at any given block slot for a given authority type.

Note that you could set authorities to 0x0000… (32 bytes of 0x00) to immutably “turn off changes” to an extension. The becomes 11111111111111111111111111111111 in base58, which is the system program id.

Transfer Lifecycle

In the original spl_token, the transfer instruction is about 100 lines of code. In the token2022 program, the transfer instruction is about 250 lines of code. The main reason for this is all of the new checks that occur (on both mint and account level), you can ctrl + F for “get_extension” to see when each extension is referenced.

Let’s go over how transfer fees, interest accrual, and transfer hooks work in this lifecycle.

If a memo extension is active on the token account, then the memo program will be called in an instruction BEFORE the transfer happens.

Transfer fees are set as a percentage of the amount transferred, and there is a max fee that can be taken (both are set by the transfer fee config authority). What’s confusing here is that users can call transferChecked() or transferCheckedWithFee() as 0x1a01. The latter clearly defines the fee amount, the former does not. You’ll soon notice that in a transfer transaction of a fee token, there is no transfer of that fee amount to some treasury (in an instruction or account activity/balances). That’s because the fee amount is held on the destination account, which can then be withdrawn from at some future date using HarvestWithheldTokensToMint() and then taken out to the treasury with WithdrawWithheldTokensFromMint(). I’ve captured the correct fee amount in the tokens_solana.transfer table in the “fee” column by calculating the set fee percentage at the time of the transfer.

Interest bearing tokens are a bit harder to intuitively understand. No matter when the tokens were minted, interest accrual on them is counted starting from the time when the token mint was first created. Logically, you can see this in the amount_to_ui_amount function that just takes the cumulative compounding interest of token amount scaled by total time passed. So if there was a token with a 5% interest rate created 1 year ago, and I minted you 100 tokens today, then in actuality you would have received roughly 105 tokens instead. Onchain in both the transaction and the balances you would only see 100 tokens though. It’s not intuitive, and the data work to get the right balance is also difficult. The Solana team is working on a few helper functions to correctly give you the interest accrued balance in an RPC call, but that is not ready yet.

Transfer hooks are called at the end of the transfer function, and will look for a generic “execute()” function on the assigned hook program id. The hook authority can assign any one program id to be hooked onto the token. If no execute function is found, it will continue normally with the transfer. You can revert a transfer by throwing an error in the hook, and you could have a generic hook chaining program if you wanted to string together multiple hooks to a single token. Some interesting applications include identity hooks by Civic, but it is largely an unexplored feature.

Subscribe now

Finding the Data Tables in Dune

If you’ve clicked through the linked queries and dashboard, then you have likely already spotted the logic and tables I use for token2022. I’ll list them below:

• tokens_solana.fungible: get all token ids, symbols, and deciamls. I added a token_version column, model logic

• tokens_solana.transfers: get all token mint, burn, and transfers. I added a token_version column and a fee column, model logic

• solana_utils.daily_balances (or latest_balances): you can still use “address” as the associated account address and filter for token2022 mint ids, model logic

• tokens_solana.fees_history: reconstructed fee history for tokens, model logic

• all authorities: see this query for latest authorities and how to construct historical authorities

• interest bearing accruals: I’ve calculated current, average, 1 year forward looking return rate, and total effective historical return rate in this query.

Note that I have not included token2022 in tokens_solana.nft yet, because NFT standards on Solana are a hot mess that I’m not motivated to touch right now.

Overall, you’ll see that token2022 has not really hit developer adoption yet besides a small blip in Feb 2024. Only 70,000 fungible tokens have been created using this standard as of July 2024.

This is partially because it’s not yet supported by most DEXs or defi protocols yet, besides some limited usage in Fluxbeam, Printdex, and Orca. You can see some of the top protocols using token2022 below:

ZK and Compressed Tokens

If you’ve been following along for a while, you have likely realized that Solana has many competing token standards - even I don’t understand all of them. I want to make a quick note to differentiate between confidential tokens, compressed tokens, and ZK tokens; as those are going to potentially be very important but also very confusing.

• Confidential transfers with private transfer amounts is an extension on token2022, where you can deposit into and withdraw with a hidden amount. The source and destination of funds are still public. A user can have confidential and non-confidential balances of the same token, at will.

• Compressed token accounts are an old concept already seen with encoding accounts into leaves of a merkle tree (because call data is basically free and not stored on the node). You typically have at least three accounts (the mint, the associated, and the metadata accounts) so avoiding creating them saves a lot of SOL. This was created for compressed NFTs, which saw great adoption last year. Metaplex built a program on top called “Bubblegum” which allowed you to transfer/mint/burn these leaves of the tree. The NFT version does not have an option to transition between normal accounts and compressed accounts.

• ZK compression tokens goes further to include proofs for token actions like transfers directly in the data, versus using an onchain component like bubblegum. You’re essentially updating your leaf in the merkle tree each time you transact. All transaction data is still written to the mainnet chain, just in call data. Nothing is saved in the state of the blockchain (i.e. in other accounts or balances), so this is considered stateless transactions. Think of the proof verification into a merkle tree as a Verkle tree, and the fact you can update individual leaves as a Reckle tree (recursive proof). What’s cool here is you can actually compress and decompress this token into a normal token account at will. Note that these tokens are NOT privacy preserving.

Stateless transactions, compression of data, and privacy preservation are three key parts to the end state of blockchains, so while these are more technically complex they are worth reading about to keep up with. I’ll come out with more in depth guides for confidential transfers and ZK compression tokens when they are fully live on mainnet.

In Conclusion

Tokens on Solana are fully in experimentation mode, with different teams opting for varying amounts of data and programmatic flexibility. While the many standards can be confusing at times, it’s exciting to see most of these theoretical constructs finally live. While usage of the new token2022 extensions is still low right now, I’m excited to revisit it in half a year to see what exciting applications have been built.

Subscribe now

Join our research mission the month of July to build the best graph datasets to power feeds for governance, nft mints, and social discovery.

Crypto frontends today mostly contain simple leaderboards - top tokens by volume, liquidity, mints, points, votes, etc. If we want to move into consumer crypto experiences that beat the behemoths of web2 today, we’ll need more than leaderboards in our app feeds.

OpenRank is one of the building blocks that will help get us there, and is already used by Metamask Snaps, Degen tips, and Supercast. OpenRank is a compute layer where many reputation graph algorithms can be run. The first of which is the eigentrust algorithm.

In this article, I’m going to introduce you to the OpenRank eigentrust algorithm and cover:

• The importance of community built graphs, and why you’ll need them

• Key concepts of the algorithm and how it works

• Creating your own graph, referencing one I made myself in a python notebook

Let’s get into it!

I’ll be running a Bytexplorers mission within the next month where we’ll compete to make the best graphs for feeds on top apps in crypto - be sure to subscribe to be notified of when that kicks off!

Subscribe now

Why build recommendation graphs with the community instead of only your own ML team?

When building algorithms and feeds in crypto, you’ll quickly face a few data problems:

• Transactions contain many inter-layered actions

• The relationships between addresses can be infinitely complex, built up over many transactions.

• An address itself contains partial identities, each of which are relevant in different contexts.

• All three of the above are ever-evolving at an exponential rate, let’s call those growing elements our “context”.

Your small ML team isn’t going to be able to keep up with this never-ending context.

You aren’t going to want to task your backend or data engineering team with this either, they have a product to build. The days of an application owning the user and user data structures is over. You don’t get a simple link, user id, like/reply/share, and post id anymore. You can have claims, splits, drops, swaps, stakes, delegated, voted, mints… there’s a new “action” that drops almost every day. And a new chain, new type of wallet, new type of identity… good luck with all that!

I believe we’ll see a graph data science community evolve in crypto in the next year, built around the OpenRank protocol and product.

I’ve been part of the wizard community at Dune for years now, and have seen the power that the community has over a small team. And I’ve seen almost every small team in crypto go from “yeah we can do this ourselves with a node and RDS database” to “we need to tap into community built data tools like the Graph and Dune”. To me, creating the combinations of queries and graphs tuned for specific kinds of feeds and communities is a similar problem. We’ll need to start collecting and testing graphs that can power feeds on every app from Farcaster clients to block explorers.

The idea of “one feed” is skeuomorphic and will be left behind. The user is the curator.

In crypto, users not only want to take their social graphs with them across apps, but the contexts hidden within them too. If I’m actively following the /degen community on farcaster, I want to see the actions of that community as the base of recommendations if I’m on Zora, Roam.xyz, or OnceUpon too. And I want to be able to toggle that feed to the context of another community I’m part of, say the artblocks collectors. The future will be one where users discover and pick their own feeds, out of contexts that are naturally occurring and created - not boxed within some group or channel feature on a single platform.

How does OpenRank Eigentrust work?

The eigentrust algorithm is similar to pagerank in that it ranks nodes in a graph network. The difference is that it focuses on capturing complex peer-to-peer relationships, as a distribution of trust. It was first built for assigning trust scores in a file sharing network. In crypto, you could imagine using this to proxy high-quality governance delegates or identify trustworthy smart contracts.

Below is the eigentrust formula you’ll soon become familiar with:

There are two key inputs above: pretrust nodes and localtrust graphs. "P" is your pretrust and "S" is your localtrust.

• localtrust: this is your measurement of interactions between two nodes, as node “i” signals some value to node “j”. This could be a token transfer, attestation, vote reply/like, etc.

• pretrust: this is your "seed" selection of nodes who you think should be more trusted in the network.

• “c”: this constant (between 0 to 1) is a weighting of the trust value between the overall localtrust graph and the pretrust seed. Interaction graphs typically have power law distributions so a higher pretrust weight helps normalize the distribution of final ranked values.

If the math doesn’t make sense, think of a social graph like Twitter. Influence (followers, likes, replies) is typically concentrated on just a few individuals, leading to a power law dynamic. By setting a diverse set of influential individuals and selecting a constant “c” of 0.5 or higher, you’re essentially saying that the people those trusted individuals interact with shall inherit half the value of that influence. This is what equalizes and distributes the trust score across the network more evenly.

Inputs and outputs are all normalized, so it doesn’t matter the scale of the values you use for localtrust/pretrust. Outputs are also always between 0 and 1 from the eigentrust model.

How is this related to choosing any context and creating any feed?

Let’s say you wanted to create an NFT mint recommendation feed. You could use a localtrust of NFT transfers/sales, and create a pretrust of the top 100 wallets by USD value of transfers/sales. This would be a global feed, and one that likely closely mimics a generic leaderboard. Our goal here is personalization, so it’s important we actually select a more contextual set of pretrust nodes. So instead, you could create a list of 100 wallets ranked by number of artblocks curated projects minted. This will now affect the output graph in a way that artblocks community wallets (and those wallets within 2-3 degrees of the community) get the highest values of “trust” in the network. Now, your feed will be more personalized to the artblocks community. You could also plug in just your own wallet(s) into the pretrust, so that you get an output of what your closest friend wallets (and their friends) are minting onchain.

From this example, it should hopefully be clear that having a focus on the kind of feed you want to build will determine what input graphs you want to create. I know this may still be too abstract, so I’ll walk through a concrete code example in the next section. Remember, OpenRank handles the compute and storage of these eigentrust graphs, and recommends feeds you can use outputs in. All you have to do is decide on the pretrust and localtrust inputs.

Subscribe now

How do you build a eigentrust graph with OpenRank?

The end goal

In this example, I want to provide a feed of recommended contracts based on the wallets of casters in the /base channel on Farcaster (it’s an app like Twitter). The output of eigentrust is just a list of ids and values, in my graph each id is tied to a farcaster user id (fid).

Remember that usually there is a power law dynamic in graph networks, so if I just took the top casters in the /base channel by number of casts or followers or engagement and used that to weight onchain actions, then it will be heavily dependent on just the top 10-20 casters. Using eigentrust, I’m able to more evenly disperse that trust across 12,000 casters instead which will give me a much more diverse weighting of onchain interactions.

After creating that eigentrust output, I weight their top contract interactions over the last week give us this recommendation feed:

You can check out this dashboard to see other feeds created from that graph, such as NFT minting, DEX token trading, and Farcaster channel activity.

Into the code of the model

Now that you’ve seen the end goal, let’s talk about how I created the ranked graph.

All code for this example can be found in this hex.tech notebook, and the jupyter notebook if you prefer to run locally.

To start our journey, I created two queries for our pretrust and localtrust respectively:

The first query is for our “pretrust nodes”. This query outputs the top casters in the /base channel based on engagement received (likes, recasts, replies), I take the (likes + 3*recasts + 10*replies) as my formula. We’ll be taking the top 100 ids from this query to use as our trusted nodes.

The second query is for tracking onchain interactions between nodes, using the linked addresses of casters in the /base channel. Because the feed is going to be recommending onchain actions, I want to make sure to choose an interaction graph heavily based off onchain interaction volume. Taking the USD value transferred between nodes is a good general proxy - I tracked stablecoin and ETH transfers on optimism, base, and ethereum mainnet.

Analyzing our input graphs, and testing our output eigentrust graphs

Now that we have our pretrust nodes and localtrust graph, let’s check some summary statistics. There are 65,755 users in the /base channel who have transferred tokens to someone else in that channel, and from our pretrust nodes we are able to traverse 19% of the graph (i.e. connected nodes). This percentage may be much lower or higher depending on how sybiled the localtrust data of the graph is. Token transfers can be high signal, but can also be wash traded so it’s not surprising that much of the graph is disconnected.

After we have checked the input data is sensible in size and connection, we can run and save our eigentrust graph. I’ve saved mine to the id “base_transfer_50” - you can see below it only takes 10 lines to train the graph. Think of openrank sdk as the scikit-learn of crypto graph models.

Remember the constant “c” from the formula earlier? Let’s do a grid search with different c values (I call it alpha) and different pretrust seed sizes and see which gives us the most log-normal trust scores and highest coverage ratios:

There are many tradeoffs here, and no optimal value to aim for. High normalization and coverage is a nice behavior if you want a strong diversity of recommendations, but for something high stakes like governance voting you may actually want a higher concentration of trust. Use your own intuition here.

From here, we can plug in our values into the feed query from the dashboard linked at the start on Dune to get a contracts interaction feed from trusted users in the /base channel. This subjective feed output helps us better connect the generic metrics from earlier to our intuition on the expected quality of recommendation outputs.

And we’re done! You could use this Dune API to power any of your apps immediately.

Learn to build your own OpenRank eigentrust graphs

Are you excited and ready to build on your own? You can fork my notebook and try it yourself, all the links you need are here:

• OpenRank Docs

• Python SDK repo

• Python Notebook

• Dune feed dashboard

Join our research mission the month of July to build the best graph datasets to power feeds for governance, nft mints, and social discovery.

Subscribe now

Other Beginner Resources:

• Crypto Data Analyst Roadmap

• Beginners SQL and Ethereum Guide

• Learn to navigate Solidity Contract Code

It’s easier and more fun to build with a community, come join the Bytexplorers to learn and earn during your data journey.

Subscribe now

What’s in a transaction?

If you've ever made a transaction on Ethereum (or any smart contract enabled blockchain), then you've probably looked it up on a block explorer like etherscan.io and seen this heap of information:

And if you tried looking at logs or traces (internal txs), you might have seen these confusing pages:

Learning to read the details of a transaction on block explorers will be the foundation for all your Ethereum data analysis and knowledge, so let's cover all the pieces and how to work with them in SQL.

I’m only going over how to understand these concepts at a high level; if you want to learn to decode these by hand, then you’ll need to get familiar with how data is encoded (it’s the same for transactions/traces/logs) and how to use Dune’s bytearray/hex functions to go between different types.

By the end of this guide, you’ll be able to understand and navigate the data tables for any contract using this transaction table finder query:

After you’ve learned the concepts in this guide, you should also learn to use my EVM quickstart dashboard to get started on any contract analysis.

Transactions

Transactions are only the tip of data iceberg, all traces and logs are invoked AFTER the initial input data kicks off the top level function. Let’s first label all the fields you’ll see in the transaction page of the block explorer:

These are the same fields you’ll see when you query “ethereum.transactions” on Dune. The key item to learn to identify here is if the “to” is a contract or not. Normally, contracts will be clearly labelled. If it’s a contract, there should be “input data” which contain a function call.

Out of all these concepts, the first one to learn well is an EOA versus a contract address. Contracts are deployed by EOAs, and can be called in the “to” field of a transaction. If you click on an address, the explorers will show on the top left if it’s a contract or an account. On dune you can join on the ethereum.creation_traces table to check if it’s a contract. Note that only EOAs can be the tx “from” signer.

It’s important to learn what data comes from directly onchain versus what data the explorer/frontends have added on top. Everything in the blockchain is represented as hex (sometimes called binary or bytes), so a 1inch swap call will have this input data string:

The first 4 bytes (8 characters) is the “function signature”, which is the keccak hash of the function name and input types. Etherscan has a nice “decode” button for some contracts, giving you this readable form:

As you can see, there are many variables packed together into that one long hex string from earlier. The way they are encoded follows the application binary interface (ABI) specification of smart contracts.

ABI’s are like API documentation for smart contracts (like OpenAPI specs), you can read more on the technical details here. Most developers will verify their ABI matches the contract and upload the ABI for everyone else to reference in decoding. Many contracts may be MEV/trading related, where the developer wants to keep things closed source and private - so we don’t get any decoding from them.

In Dune, we have decoded tables based on contract ABIs submitted to a contracts table (i.e. ethereum.contracts), functions and events are converted to bytes signatures (ethereum.signatures) which are then matched against traces and logs to give you decoded tables such as uniswap_v2_ethereum.Pair_evt_Swap which stores all swaps for all pair contracts created by the Uniswap v2 pair factory. You can filter for swaps on a specific pair by looking at the contract_address table for events.

On Dune, you would want to query this table for this function call oneinch_ethereum.AggregationRouterV6_call_swap. You’ll see this table name is at the top of the query results in the table finder at the start of the guide.

For the following sections on traces and logs, we’ll be using the same 1inch aggregator swap transaction. This is a good example because a router will swap tokens across numerous DEX contracts, so we’ll get a good diversity of traces and logs to investigate.

Subscribe now

Logs

Let’s talk about event logs next. Logs can be emitted at any point in a function call. Devs typically will emit a log at the end of a function, after all transfers/logic are completed without errors. Let’s look at the uniswap v3 swap event emitted from the transaction earlier:

You’ll see there is a topic0, topic1, topic2, and data field. topic0 is akin to the function signature, except it’s 32 bytes instead of just 4 bytes (still hashed the same way). Events can have “indexed” fields for faster data filtering, which can appear in topic1, topic2, or topic3. All other fields are encoded together in the “data” object. Again, they follow the same encoding rules as transactions and traces. The “28” is the index of the event in the whole block. It can sometimes be useful to join on when you want the first swap or transfer in a tx.

To find the logic behind where and how this event was emitted, I’ll need to dive into the solidity code. I’ll click the linked address of the event, go to the contract tab, and search “emit swap” because I know that all events have “emit” right before they are invoked in the code.

I can see that this is emitted in line 786 of the contract, as part of the “swap” function.

Being able to navigate functions and events lineage across contracts will be a key skill you’ll need to pick up to accurately understand the lineage of the data you’re querying. You don’t need to learn solidity in depth to navigate these files, just know how to understand contract interfaces and when functions/events are called (function and emit are your keywords).

For in depth example of sleuthing the code for functions and events, check out this breakdown of Sudoswap contracts and data.

Using the table finder query from earlier, I can see that the table I should query for this swap is uniswap_v3_ethereum.Pair_evt_Swap and that it’s emitted after the swap() function is called.

Traces (ethereum.traces)

Traces can quickly become very difficult to navigate, because of how nested calls between different contracts get. Let’s first understand the types of traces:

• CREATE: this is a trace emitted when a new contract is deployed. You can deploy a contract directly at the top of a transaction, this will mean there is no “to” address in the transaction data. You can also deploy a contract within a function call, hence the existence of contract factories. Check out the ethereum.creation_traces table for a simpler view of these.

• DELEGATECALL: this goes on your mental “ignore” list when looking at a transaction. Think of this as forwarding a request from one server to a next without changing any logic. This is related to proxies and storage, you can check out more details here.

• CALL: this is the most common and generic trace. A call can be just a transfer of ETH value without any contracts involved. It can also be any function call on any contract.

• STATICCALL: this is a function call that does NOT modify any state, and is purely used for calculations. Stuff like oracle price feeds, AMM price calculations, liquidation ratio checks, balance checks, etc all happen in staticcalls. Commonly seen in solidity as “view” or “pure” function types.

You’ll also need to understand the trace_address column/index. This is the [0,1,1,1,1] pattern you often see. Imagine it’s like bullet points, where the number of numbers in the array indicates the depth and order of the function calls.

A (null) --the transaction first input has a trace_address of []
  CALLs B (0)
    CALLs C (0,0)
  CALLs D (1)
    CALLs E (1,0)
      CALLs F (1,0,0)
    CALLs G (1,1)
  CALLs H (2)

As you can tell from our earlier internal transactions (traces) screenshot, etherscan is not a friendly place for viewing traces. I prefer to use phalcon blocksec instead, which unwraps the transaction like so:

This might look overwhelming, but it is actually a super easy way to explore all the functions, events, and arguments in the flow of a transaction. Once you’re able to understand this, then you can safely say you understand all the data in a transaction. Notice that my table finder query is an almost exact copy of this layout, I was largely inspired by them!

Note that on Dune, we automatically decode both transaction calls and traces of the same function name to the same table. You may be wondering if you can easily join events and traces/transactions in the nice ordering shown in phalcon. On Dune, you can join on transaction hash to generally tie data together, but you can’t join on any index to recreate the exact ordering of interactions. It’s an unfortunate limitation at the moment that requires a custom indexer.

Subscribe now

Onwards, deeper into the dark forest of crypto

If you understand the concepts I’ve laid out in this guide, then you’re ready to dig deeper and write more complex queries. Navigating data across transactions using multiple different tools will be one of the most key skills you’ll need to excel in this space.

There are probably 10 different explorers I find myself using every week, and the number of the tools is 10 times that amount. I write an annual guide covering how the data tooling stack continues to evolve, and what you should use each tool for:

The more data tools you learn to use, the more flexibly you’ll be able to build and communicate in this giant ecosystem! As always, my DMs are open on Twitter if you have feedback or questions.

None of this is financial advice, please do your own research and be careful before buying any tokens - especially with memecoins.

Subscribe now

Every market cycle comes with memecoins. If you’re not familiar with the concept, think about the craze of r/wallstreetbets driving the rally of AMC in 2021. A bunch of people rally around some meme and bid the price of some asset up for a short while (anywhere from one day to a couple months). It’s become a popular go to market strategy in crypto for everything from the blockchain layer to the app layer - as you drive up not just prices but attention around an ecosystem. Some like Avalanche go as far as creating an actual official fund for memecoins.

The line between memecoin and any other token can sometimes be blurred, but the general assumption here is that there is nothing backing the token besides the meme. For example, DOGE as a token only really has the shiba inu doge picture/idea backing it. Unlike a token like ETH that’s secured by the ethereum blockchain, UNI which has the full Uniswap protocol behind it, or a token like MKR that has a full collateralized stablecoin service behind it.

Farcaster memecoins (like DEGEN) are the newest flavor, with the primary benefit being that the social is openly accessible. So with that, I wanted to test the waters with some basic analysis combining social and financial data.

I ended up with this 📊 dashboard, which I will walk through in this article.

Measuring the Meme

I started with the hypothesis that we could compare all memecoins with a simple chart that graphs social and financial health:

I’ve categorized five main areas in this chart:

• Extreme Risk: Memecoins with low liquidity and volume are very risky and subject to rugs (as likely there are only a few liquidity providers)

• Bot Arena: Most memecoins will never make it out of the trenches, where thousands of coins (many with the same token name/variation) are competing for social and financial attention

• Volatile Growth: Memecoins that do break out of their initial circles now have to sustain their momentum and growth. You’re likely to see many days of 100-500% swings in price in both directions, and also a bunch of influencers start to latch onto the meme.

• Well Established: The leaders will sit at the top socially and financially for a while, and be very clearly separated from the pack. It’s likely that the week to week numbers are less volatile, and attention is sustained as there is an equilibrium of inflows and outflows of people.

• Sleeping Giant: Memecoins that grew a lot socially and financially - and didn’t get rugged - may die down at sit in this corner for a while. Likely a DAO has formed, they’ve started shipping random mints or products, and are dealing with the mess of a community. All while hoping to rotating into the spotlight again.

The bulk of memecoins should be stuck in the “bot arena”, with a few catchy ones in the “volatile growth” segment, and maybe one or two making it “well established”. Along the way, some may lose social power and become “sleeping giants” and others may lose their financial backing (liquidity) to become “extreme risks”.

I believe that the average successful memecoin goes on a journey like this:

And you probably see many memecoins that get botted/influencer pushed to have a high social score, but has low financial score (liquidity) and leads to rugs.

After much data engineering and cleaning, I was able to produce this chart in Dune focused on Farcaster memecoins:

This actually fit my expectations pretty well, you have DEGEN far out in the top right corner and then a few others like ENJOY, HIGHER, TN100X, and EVERY out in the middle. Everyone else is stuck in the left competing for attention and liquidity.

It’s worth noting that I have not filtered out sybil/bots here, so social scores for some memecoins may be skewed. An improvement for the future!

Now, let’s walk backwards from this end chart to explain how those two scores were created. I’ll also pose further research questions and detail my query lineage for those who want to dig deeper.

The dashboard can be found here, there are a few charts that I have not included in this article.

Subscribe now

Social Score

Each score is made up of a “base” component and a “growth” component. For the social score, we start by measuring the casts and engagement of ticker mentions. So “$DEGEN” will count but “DEGEN” won’t in this methodology. Related symbols to the token (like the degen hat, higher arrow, tn100x ham) are also included in counts.

That gives us five main columns:

• Casters: Number of people who have casted a given ticker

• % Recipients: Percent of those casters that have recieved the token before

• Casts: Number of casts with a given ticker

• Channels: Number of channels that had a cast with a given ticker

• Activity Level: The engagement (likes + replies) plus the casts times cube root of casters times cube root of channels.

The overall social score is calculated with the “base” being the activity level, and “growth” multipliers based on week over week change in unique casters and recipient casters. The idea here is that if you’re seeing large growth in people casting a ticker and those people buying/acquiring the token, that’s an extremely healthy sign.

Overall, it looks like this in a table:

Financial Score

These are your main financial metric columns:

• Fully Diluted Value (FDV): Total supply multiplied by price

• Price: Most recent price based on DEX trades

• % DoD, WoW, MoM Price Change: Percent changes in price on a daily, weekly, and monthly basis

• Liquidity: This is non-token liquidity, meaning for a DEGEN-WETH pool we only count the WETH portion of it. It gives us a more stable sign of how much good liquidity there is for a given token.

• Trades: Number of DEX trades in last 30 days

• Transfers: Number of erc20 transfers in the last 30 days

• Total Volume: USD volume of DEX trades in last 7 days

The “base” for the financial score is its non-token liquidity and DEX volume, with a “growth” component based on week over week change in liquidity.

Overall, it looks like this in a table:

Subscribe now

Follow Up Questions

Below are some top of mind questions that I’d love to see people dig into from here:

• How can you do sybil/bot scoring of casters? Weight it by the dollar value of tokens in the wallet?

• Can you tier casters based on relationship to known clusters and/or communities of people?

• Categorize the waves of social and financial growth - how are they correlated? Is there a time delay or dependency one way or another?

• Are the the same circles/types of people driving the initial growth waves? What about the people who are early, on-time, and late to the meme? Are there relationships with the user tier?

• What happens when a meme “rotates” from one to another? What or who drives this?

• How “interested” is a person based off of their social or financial stake? Do people cast more/less after they buy/sell a given token?

• What’s the average lifecycle of a meme? Socially and financially?

DM me if you want help ideating or figuring out approaches to tackling any of these.

Want to join an onchain data community and learn/build answers to these questions with 200+ others? 🔍 Learn about the Bytexplorers, then mint your passport and join in

You can also further explore Farcaster trends in users and channels here:

Query Lineage for the Data Nerds

While all the output looks like it all came from a single query here, there is actually a bit of data complexity in the background. Below you’ll find a query lineage map that shows what data came from which data sources:

I’ll quickly walk through these:

• The “dune.neynar…” datasets are uploaded by the Neynar team, who indexes all farcaster data. If you’re unfamiliar with Farcaster data concepts, then start with my beginners guide. In general, if you see “dataset” in the tablename that means it was uploaded by whichever team name shows up after the “dune.” prefix.

• Any time you see a “result” in the table name, it’s a matview. That means there is another Dune query being read from here. I created the “erc20 token summary” table to get me all trade, price, liquidity, and transfer data on a given token. This matview is further broken down in child queries such that the final query can run in only 3 minutes. Combined runtime for all the queries here is around 60-70 minutes.

• Any time you see a table name that doesn’t follow the above prefixes (and also is not a raw or decoded table), it’s likely from spellbook (abstracted tables). So tokens.erc20 gives me all tokens with symbols/decimals, dex.trades gives me all DEX swaps across all blockchains, etc. You can find the table logic in the github repo by doing a code search for the table namespace or table name (i.e. “dex” or “trades”).

If you have questions on any of these queries, then please reach out to me. Check out the memecoin query for links to each matview, they are in the comments next to whenever I utilize a child table.

Subscribe now

👉 Interactive roadmap can be found here

I’ve created an interactive roadmap with all the tips and guides you’ll need to go from a beginner to an expert in crypto data. This is not meant to be completed all at once, you should work your way through the levels and only come back to the next step when you feel like you’re ready.

Each section has multiple levels of modules (L1, L2, and L3). Each module has a “you should be able to…” that suggests what you should be capable of doing.

Sections:

• Intro to Crypto: if you know nothing, then these links will be helpful for getting up to speed.

• Blockchains: the basics of blockchains come down to transaction structure, common standards, and ecosystem coverage.

• Exploration Tools: without learning how to spot trends and explore addresses and transactions, you won’t be able to go much deeper.

• Querying Tools: diving deeper than surface level trends requires writing queries.

  • Technical Languages: you can use SQL, Python, or Javacript. You don’t need to learn them all at once.

  • Navigating Protocols: the more protocols you dive into, the easier this gets. Learn to build up from functions to feeds to labels and metrics.

• Building in Public: the steps to follow if you want to get hired full time in the space.

• Knowledge Domains: seven of the key domains you’ll become familiar with the longer you spend time in crypto.

I’ll add to each section over time, if you have questions or contributions then please DM me.

Subscribe now

🔉 I explained the mission in this Twitter space recording.

Now that the Bytexplorer community is close to 200 members, we’re kicking off our first mission quest!! Mission quests are partnerships with foundations across the crypto ecosystem to openly discuss and answer high-value research questions.

The first mission starts today, and it will last for six weeks. We will choose a partner and mission each quarter, and are hoping to collectively tackle some of the biggest questions in the crypto data space.

Our first mission will be with the Optimism NumbaNERDs!

You will need a Bytexplorer pass to participate, if you’ve completed an Optimism NumbaNERD mission before then you can get one dropped to you for free by filling out this form. Otherwise, you can mint a pass here.

NOTE: This mission has concluded, we had 18 explorers submit 30+ discoveries. Thank you to everyone who participated! Distribution of votes and rewards are as follows:

Subscribe now

Optimism Airdrop Research

Optimism announced Airdrop #4 recently to allocate over 10M OP to over 22k unique addresses and reward Superchain creators (docs).

The focus of this mission will be understanding onchain behaviors of recipients before and after the drop, relating to usage of protocols and governance activity. Questions will be split into three levels:

• Large (1500 OP): Open-ended questions where we value novelty and creativity of approach the most

• Medium (400 OP): Questions that have definite answers, but have a large surface area (i.e. identifying different sybil groups).

• Small (120 OP): Questions that are bite-sized and have a definite answer

There will be a novelty reward (400 OP) split across most novel submissions too. You will also earn Optimism Guild Tokens for your participation and quality of work.

Below are the list of questions we are interested in answering. The section after this will detail how to participate.

Large (👉 Submit Here, Closed April 28th):

🔊 Twitter Space discussing questions

• LTV analysis on airdropped users i.e. how much did the airdrop contribute to long term usage expected versus actual? 

  • i.e. let's say a user transacts twice a month on an nft marketpace, around 200 USDC. That's annualized to be 12 txs a year or 2400 USDC. Maybe after an airdrop, the annualized goes up is 20 txs a year with higher avg value size so maybe 5000 USDC (subtract your airdrop value amount to reduce noise).

  • LTV can include growth rate as well. It’s possible that a user’s pre versus post airdrop growth rate of activity (value) changed - so that’s important to take into consideration. You could also attribute value from collectors brought in as well, so it’s not just the creator’s individual transaction value.

• Compare optimism airdrop to other protocol airdrops metrics/parameters?

  • Comparison of tiers and retention - could even do across just optimism airdrops, and even grant funding. Note that OP airdrop 1 and 4 were claims, 2 and 3 were pushed to wallets.

  • What are good metrics to use to compare against airdrops?

• What other factors make up a quality creator? Are there better onchain metrics?

  • quality of collectors (fees/value spent)? farcaster activity of creator/collectors? secondary volumes?

• Which delegates have the most “active” delegator base? Is it correlated with the most actively voting delegates?

• In depth analysis of delegate clusters on specific kinds/types of proposals, using proposal metadata. How many delegates do we have per “topic”, sometimes can be seen in titles?

  • https://community.optimism.io/docs/governance/token-house-history/

  • https://github.com/voteagora/governor (proposal rules)

Medium (👉 Submit Here, Closed April 7th):

🔊 Twitter Space discussing questions

• Segmentation of claimers based on protocols.

• Did this airdrop help to grow the creator and NFT community on OP Chains?

• Any noticable sybil aggregations of airdrop claims?

• Any overall protocol usage changes (i.e. did any specific protocol see changes in volumes/users)?

Small (👉 Submit Here, Closed March 24th): 

🔊 Twitter Space discussing questions

• How many claimers of the airdrop?

• What are the overall claim rates? 

• What is the claim distribution by amount?

• Change in delegation totals and percentage of totals by delegate? 

• Did delegatees end up delegating their airdrop to the same delegates they already supported?

• How many claimers that were already delegated stayed delegated (in addresses terms & OP terms)?

• How many new delegations remained delegated?

  • Note: delegation was part of the claim process

If you have questions that are related to the ones here that you wish to tackle (or you want clarification), please reach out in the Bytexplorer telegram group and we can discuss adding them.

Subscribe now

Mission Participation and Guidelines

Every two weeks, participants will submit their answers to the next level of questions -I’ll call them “checkpoints”. Anyone can submit, and you can choose to submit your work to only one checkpoint level or all of them if you want.

The checkpoint timeline looks like this:

1. “Small” questions are due after the first two week period (March 24th)

2. “Medium” questions are due after the second two week period (April 7th)

3. “Large” questions are due after the third two week period (April 28st)

Voting for each checkpoint lasts two weeks. Novelty rewards will be decided at the very end. The NumbaNERD committee will get 50% of voting power, and the rest will go to trailblazers in the Bytexplorer community. It is a requirement to get 100 votes from the NumbaNERD committee for your submission to count towards a given level.

We will use jokerace for this. The split of reward is based on sum of points of submissions that qualified in category, where points are defined using the quality of submission (valid, great, superb, exceptional). This is the same as the weekly quest rubric. So if 15 people all score exceptional (64 points) on the large checkpoint, then that’s a total of 960 points. Each person would earn 100 OP tokens (64/960)*1500.

Again, you will need a Bytexplorer pass to participate. If you’ve completed an Optimism NumbaNERD mission before, you’ll get a pass dropped to your wallet by filling out this form. Otherwise, you can mint a pass here.

Learning and Data Resources

We will host weekly Twitter spaces to dive into questions and topics, and will also be active in the Bytexplorer telegram group to help with ideation and queries.

Here are a list of relevant data resources you can use to jumpstart research from:

• Claim Contract: 0xfb4d5a94b516df77fbdbcf3cfeb262baaf7d4db7

• Address list: https://github.com/ethereum-optimism/op-analytics/tree/main/reference_data/address_lists

  • dune.oplabspbc.dataset_op_airdrop_4_addresses_detailed_list

  • dune.oplabspbc.dataset_op_airdrop_4_simple_list

• Logic that you can use to map creators to contracts: https://dune.com/queries/3452893

  • Note that this is not perfect as contract creation could be done by a relayed / sponsored transaction or first minter as the deployer. If you have better ways to attribute, please let us know!

• Onchain contract <> project name mappings: `contracts.contract_mapping`

• Governance delegates and proposals

  • “op_governance” and “governance” spell namespaces

    

  • check “governance_optimism.agora_proposals” namespaces for more up to date proposals. op_governance has older ones.

  • OP token house delegate and delegatees analysis (dash)

Best of luck on your mission, explorers 🚀

🎧 If you prefer a live walkthrough, check out the video above

What Open Social Data Enables

This is the first time in history that three datasets have come together in the open:

• High quality user transaction data (blockchain data)

• High quality social graph data (profiles, followers, and actions)

• Verified linkage of addresses (user X owns wallets A,B,C)

And on top of that, these datasets have come together in a vertically integrated data stack (Neynar for Farcaster data, and Dune for onchain data) that is open for everyone to use.

Doing the equivalent with reddit, twitter, plaid, and stripe data would take hundreds of thousands of dollars to scrape and build data infra for. This leads to a concentration of data wealth and product in the hands of a select few companies (those overflowing with money and connections).

I’ve built a dashboard that combines the three datasets above and showcases some basic clustering and trends across words, users, and channels.

📊 Farcaster Trends Explorer Dashboard

⚙Contribute to the methodology in the dashboard GitHub repo

🔔Use our Farcaster query templates and set some Slack/Discord alerts

Let’s walk through the kinds of trends and analysis you can kickoff from here.

Words and Topics

Most people have seen screenshots from google trends before. You get to put in any word and see the interest/performance over time:

Here’s a simple example of week over week “word” trends on Farcaster, week over week:

Some column definitions:

• Word: each word is isolated from every cast in the last 14 days

• Casters: number of users who included a given word in their cast

• TF: term frequency, or number of times this word appeared in casts.

• IDF: inverse document frequency - basically the higher this is, the more unique the word is. I used each caster as a “document”, so in this context a higher IDF means it was cast by less people.

• TF-IDF: the multiplication between frequency and uniqueness measures

You can immediately get a high quality view of what’s trending in the Farcaster ecosystem, with a single SQL query. This query is easily extensible to filter for trending words on a subset of users or channels, and for you to apply your own measurements on top.

Now, you’ll also notice that I filtered on casts only from “active badge” users. Let’s get into how I labeled users next.

Subscribe now

User Labels and Trends

Okay, let’s take it to the next level now. I want to be able to identify the quality of a user and then rank users based on age, Farcaster activity, and onchain activity. For this dashboard, I came up with the following five tiers:

• 🤖 npc: Less than 400 followers

• 🥉 active: 400+ followers, 1+ casts, 100+ engagement score

• 🥈 star: 1k+ followers, 5+ casts, 5k+ engagement score

• 🥇 influencer: 10k+ followers, 10+ casts, 25k+ engagement score

• 💎 vip: 50k+ followers, 10+ casts, 50k+ engagement score

Where engagement score is [likes + recasts*3 + replies*10] and only “received (R)” actions are counted to this score. It’s likely you could use actions “given” as part of user tiering too, but just measuring casts felt good enough to me for a basic model.

Out of 83k users who sent a cast in the last 14 days, about 3k of them are higher quality users. With this in mind, we can create a leaderboard of these users:

I can not only see that are Dan (dwr) and Vitalik the top followed users on the platform, but also that Vitalik is active in the “geopolitics” channel and mainly shares either his own blogs (vitalik.eth.limo) or github links, and that Dan has been very active in deploying contracts (198 contracts deployed).

Let me cover some of the more complex column definitions:

• Channels: This is the number of channels that a user casted in (over the last 14 days)

• Top Channels: These are the top three channel-ids that a user casted in, ranked by number of user casts in them.

• Top Domains: These are the top three domains found in links that a user casted, ranked by number of times the domain appears

• Txs, Volume, Contract Deploys: These are the all-time totals across all verified addresses, across all EVM chains that Dune currently covers.

Now, what if I want to add filters to see trending users based on some criteria? For example “show me the trending users based on engagement that @jessepollak has followed, who are less than 7 days old.” This will then give me a list of users that I might consider as the “potential new builders on Base”.

You’ll see there are also “user channel filter” and “held token address” filters there too. That means that I could filter for trending users in only a given channel, or trending users who have ever held the Nouns token. The possible combinations here are endless, I’ve only added six parameters that I thought were useful - but you can fork the query and add your own!

Subscribe now

Channel Labels and Trends

Channels are like “subreddits”, where they have followers who cast inside them and hosts who manage the channel. I’ve created five tiers of channels:

• 💤 quiet: less than 5 casts and less than 50 engagement score

• 🍻 friends: 5+ casts, 50+ engagement score

• 🔍 niche: 25+ casts, 5,000+ engagement score, 100+ casters

• 🎭 subculture: 100+ casts, 25,000+ engagement score, 50+ rising stars and 2+ influencers/vips

• 👑 stadium: 250+ casts, 100,000+ engagement score, 10+ influencers/vips

The idea here is that by combining user tiers and channel tiers, you can get a much better sense of which channels are high signal. Let’s start off with an all active channels ranking:

I’ve sorted by “engagement” here, and you’ll notice that not all high engagement channels are “stadiums”. This is because there are plenty of channels with a large concentration of “npc” or “active” users, but not many “star”, “influencer”, or “vip” users. I still consider these to be pretty niche.

Covering some column definitions again:

• Top Influential Casters: These are the top three influencer/vip tier users, ordered by number of casts in channel in the last 14 days

• 🤖,🥉,🥈 ,🥇 ,💎: The emojis in the table match up with the emojis I used for user tiers. This measures the number of users in a tier who sent a cast in the last week in that channel

• Avg Txs, Volume, Contract Deploys: I took the average of these onchain metrics across all users with at least one transaction (columns are on the far right). I think there is still more work to be done here to make this a strong channel signal.

So for the trending view, I might want to see which channels are under 7 days old but have the highest number of “influencers” actively casting in them:

I quickly find that there are a bunch of niche channels like “superbowl”, “donothing”, and “consumercrypto” that seem to be popular with influencers but not have that much engagement or other users in there yet. I could use this information to try and join channels early, to grow my own account and relationships.

You can also filter by trending channels given a “username filter”, i.e. what are the trending channels Vitalik is a part of. Or filter based on “channel held token address”, which if you put in Nouns would show you channels trending amongst Nouns holders.

Subscribe now

📊 Farcaster Trends Explorer Dashboard

⚙Contribute to the methodology in the dashboard GitHub repo

🔔Use our Farcaster query templates and set some Slack/Discord alerts

The tip of the social data iceberg

This dashboard is fundamentally three different queries that aim to build up a much stronger base for everyone to build social analysis with. Again, you can fork and build off of any of these queries yourself, or use them as endpoints in the Dune API to build your own external applications.

I’m on Warpcast if you want to follow me for more insights, and my DMs are always open to bounce ideas and answer questions.

I’m also building an onchain crypto data community called the Bytexplorers. We’re building out some Farcaster specific initiatives like the ByteLight token, and would love to have more people participate with us.

In this guide I’ll walk through:

• The difference between onchain contracts, hubs, and clients

• Querying main interactions (casts, reactions, links, profiles, fids, verifications)

• A bit about Frames

• Connecting onchain data to Farcaster data

Dune gets its Farcaster tables data from Neynar every 12 hours. If you’re looking to join onchain activity to Farcaster tables, then querying in Dune is the way to go. If you’re looking to build an app/bot on top of Farcaster, you should be using Neynar (think of them like Alchemy).

The easiest way to sign up to Farcaster is to just download Warpcast from the app store and purchase in app. You can use my invite link.

📊 Follow me on Farcaster and join the /data channel

Subscribe now

🎧 Do you prefer audio? I recorded a 20-minute voice overview of this guide:

What data exists where?

Onchain Contracts

First, let’s cover what is onchain, on hubs, and on clients. This will teach you where you need to go to access what type of data.

Let’s start with the onchain contracts. Farcaster has three contracts on Optimism, which act as “registries” for ids, storage, and keys respectively (shown below). I believe the main reason for having this onchain component is to have payment for both ids and storage - as it would be very expensive to decentralize a social app if any bot could costlessly sign up and spam.

When a user registers, they are given an fid and custody address, as well as some storage units. Here is an example of a registration transaction on Optimism. You can see that transaction broken down in Dune, with the tables to query for this data:

You’ll most likely need to query these tables if you want some raw economic data on protocol revenue and costs. There are great example queries in this dashboard you can reference (near the bottom).

Once a user is signed up, they can also rent more storage within an app like Warpcast. Here’s the purchase screen below in my account:

Hubs

Once a user is registered, hubs take care of the rest. the bulk of actual Farcaster data is stored on hubs. Anyone can run a hub using Hubble, and they hash messages that are then streamed to each other and validated.

There are currently 560 hubs being run (see here). You can sync a hub in 2 hours, and it takes 8GB ram with 20GB storage to run the hub (requirements here).

Hubs store all the actual interaction data - profiles, casts, channels, reactions, links (follows), and verifications. You can see an overview with the data documentation on Dune. We’ll go through all these data tables soon.

This is where Neynar gets their data (they actively run a hub), and Dune gets their hub data from Neynar which syncs once a data.

Clients

Clients are unrelated to hubs, so don’t make the “oh geth is a client to run ethereum” mental connection here.

Clients mean apps in the Farcaster context. The biggest client is Warpcast, a mobile/web app built on top of Farcaster. Warpcast stores data from custom interactions that don’t get published to hubs. There are three examples here:

• private direct and group messages. Note that these are not to be confused with the “hashed messages” sent between hubs.

• frame actions (clicking on a frame). Note that frames as a concept can be implemented on any client, but the data will always only be passed between the client and the frame server.

• warps (in-app currency native to Warpcast)

• channel follows. You can get channels using parent_url (explained later below) but the best proxy for “following” is looking at if a user has casted, recasted, or liked a cast in a given channel.

There is also sometimes a delay between when a cast appears on a client like Warpcast and when it gets published. Warpcast publishes in small batches, so sometimes you may cast on Warpcast and not see the cast available on Neynar for a couple seconds/minutes.

You can’t query data off of clients unless they open up an API for it. An example here could be using Warpcast to “login with Farcaster” on another application. Warpcast is considered a “wallet app” within the category of clients.

Subscribe now

Explaining Interaction Data (from Hubs)

Whew - that’s a lot of background knowledge, we’re finally ready to get into the real data.

Profiles, FIDs, Addresses, and Followers

Each fid has a custody address (the one that was added in the registration process). This is found in the dune.neynar.dataset_farcaster_fids table. If you wanted to get their profile data, you would JOIN the dune.neynar.dataset_farcaster_user_data on “fid” and do a WHERE on “type” to get their fname (username), profile, display name, avatar URL, or bio.

A user can choose to link other addresses if they want through offchain signatures in a wallet app (i.e. Warpcast → settings → connected addresses). These go in the dune.neynar.dataset_farcaster_verifications table (JOIN again on “fid”).

To make it simple, you can just join the fids table to the aggregated table dune.neynar.dataset_farcaster_profile_with_addresses to get fid, profile data, custody address, and verified addresses. To query followers, you’ll want to use the dune.neynar.dataset_farcaster_links table, where “fid” is the follower and “target_fid” is the user being followed.

Here’s a query putting this all together:

On Neynar, you can use the users endpoints to get this data.

Casts, Replies, Channels, and Reactions

There are only two tables for you to work with here.

• dune.neynar.dataset_farcaster_casts

• dune.neynar.dataset_farcaster_reactions

Keep in mind that many tables have a “hash” column, which represents the id of an action on that table. Only the “hash” on the casts table represents a cast hash.

The reactions table is easy to work with, just JOIN on “fid” or “target_fid” as needed and JOIN “target_hash” to the “hash” column on the casts table. Let’s walk through some of the key columns of the casts table:

• hash: the hash of the actual cast. This is truncated to just the first 4 bytes (8 characters) in the Warpcast URLs.

• parent_hash: the hash of the cast being replied to (if any)

• root_parent_hash: the hash of the cast at the top of the reply thread (if any)

• parent_url: the channel id of the cast (if any, none means “home”)

As for the cast text, it’s broken up into three parts:

• text: any raw text from the cast, with links and mentions removed

• mentions: the fids of any users @ mentioned in the cast. The position of the mention is in the “mention_positions” column. The arrays are 1:1 in terms of index.

• embeds: any attachments to the cast, like an image

Here’s an example query that puts some of these cast concepts together:

For casts analysis, you’ll need to just get used to doing self joins (joining the casts table on itself to get replies). This can get pretty complex, if you want to see some more difficult queries then check out how I’ve broken down Bountycaster in this Dune dashboard. Bountycaster relies on a user to cast with certain words like “complete” or “cancel” or “add” while tagging the bountybot.

You’ll see that in the “bounties” CTE of this query, I look for any cast that’s a top level cast (no parent_hash that it’s tied to) and then look for any replies with keywords that indicate the status of the bounty. I then use some regex to pull out information like bounty value.

On Neynar, you can get casts from a feed and then unroll a cast using retrieve all casts from thread. Check out my code here for calling and parsing a cast, and also validating some frame data.

About Frames

Frames are really simple right now, involving an image and a few buttons. The interaction in the app then asks a server (API backend of the frame developer) what to show next. It uses the OpenGraph (OG) standard, which is what is used to power those link previews you see on any site when you paste a link. Here’s an example:

Frames are the newest feature on the Warpcast client. Because it’s a client concept, you can’t query frame actions data off of Neynar or Dune. But if you build a frame, you will get the “frame action” passed to your frame server:

Note that frames as a concept can be implemented on any client (i.e. Supercast supports them), but the data will always be only passed between the client and the frame server.

There are some workarounds to getting frame action data. I’ve built a frame system that pushes actions to Ethereum Attestation Service (EAS) to act as an onchain logs framework. I tweeted an example query - we’ll go more in depth in the next section.

🚨 Frames are still rapidly changing, and this guide will likely fall out of date quickly. I’ll update this when I can, but will likely do a full frames guide in another article in the future.

Connecting Farcaster data to other onchain data

The real unlock of having a web3 native social graph is that you can now join a Farcaster user’s onchain activity to their hub activity. I took the voters from my EAS frame poll, and joined the user custody_address and verified wallets together onchain summary data in this dashboard. Here’s a nice little summary query:

Again, “verified wallets” are pulled from the verifications table. They are added by the user through offchain signatures done in a wallet app (i.e. Warpcast → settings → connected addresses).

The complexity of the onchain data here has been abstracted in a materialized view named “dune.dune.result_wallet_all_chain_activity_summary” that is defined in this query. In the matview query, you will find how I calculate statistics like age, transactions, volumes, and contract deployments.

Here’s another example, of me pulling in Farcaster fids and usernames onto a list of token holders (bytepass NFTs). These are all top data experts in the ecosystem, so you should follow everyone here 😁

All Dune queries can be accessed from the API as well. Here’s some code in javascript that I used to calculate poll totals from EAS data. Dune also has a python SDK.

Dune API documentation can be found here.

Continue Learning Data with the Bytexplorers

That’s all for this Farcaster data guide! If you enjoyed it, be sure to like and subscribe. If you have questions or work you want to share, then DM me on Twitter or Farcaster.

I’m also building an onchain crypto data community called the Bytexplorers. We’re building out some Farcaster specific initiatives like the ByteLight token, and would love to have more people participate with us.

Subscribe now

^🚨 this quest has been paused for now

Read on to learn more about the token usage and mechanics.

ByteLight is an ERC1155 token where any holder can ask a crypto question in the Farcaster /data channel, and then the post will be spotlighted into the Bytexplorer community (100+ data analysts, engineers, and scientists).

Collectively we have knowledge on everything from user behaviors, dozens of chain ecosystems, protocol patterns, domain trends, and complex data structures. The breadth and depth of topics we can help discuss or answer is massive.

This is meant to bring the crypto community together with fun new incentives that don’t add much friction.

Thanks to Vishesh for ideating on the token mechanics with me.

Subscribe now

What kind of questions can I ask?

Data community question/answering already happens every day on Twitter, but there are still tons of good questions that get missed. I answer questions like this one from Binji almost daily:

This kind of question is too simple for a paid bounty process, but still very valuable for making key decisions. The ByteLight token is for spotlighting such questions from those who are either non-technical or don’t have time to dig into the data.

Some other example questions you might ask:

• Trend Diving: I don’t understand how Solana DEX volumes have increased this significantly, can someone help me figure out what tokens are driving this?

• Ecosystem Context: I’m trying to understand the trend of why Arbitrum TVL has been increasing the last few weeks, does anyone have an idea?

• Tweaking Charts: This chart shows active users of Uniswap, can someone help me show retention instead?

• Protocol Releases: We’re releasing our new protocol contracts soon (or a new EIP/standard), and would love to have some data nerds let us know what events we should add or edit

• Data Tooling Questions: We’re considering using this data product to build ABC and would love to get some honest thoughts on the pros and cons of XYZ features

🚨 This is NOT for requesting for full dashboards or data models - that kind of work should go through Bountycaster. That said, asking smaller data questions in public will help you find the right data person to hire.

How ByteLight tokens work, and how to get them

The lifecycle of a question:

1. Send a cast with your question into the /data channel on Farcaster. It must be a completely new cast, not a reply to another cast. It must also contain at least one question mark “?”.

2. The cast will get pushed to the private Bytexplorer telegram group, and ranked in a feed on this Dune dashboard.

3. The question gets answered through replies to the original cast. It stays open for 14 days.

4. If an answer is satisfactory, reply with “@bytecast answered”

5. You keep your ByteLight token(s) until you decide to sell them. You can ask as many questions as you want while you hold them - within reason, if you ask ten times a day then you won’t get a response to all of them.

There is no promise that you’ll get the question answered perfectly, but if it is a reasonable ask I am sure the Bytexplorer community will try and help you out.

Buy ByteLight tokens on Sudoswap

👉💵 Buy or Sell from this SudoSwap pool, with a starting price of 0.03 ETH that goes up 0.001 ETH with each purchase.

It’s non-transferrable, and I will increase the supply of tokens over time, based on demand. I will also set the price lower or higher manually over time. It can be sold back at roughly half the current purchase price.

Legal Disclaimer: The ByteLight token is a utility token for use within the Bytexplorer ecosystem and does not represent a financial instrument, security, or commodity. It is not intended for speculation or investment. Ownership of ByteLight tokens does not guarantee any rights to services, profits, or dividends. The value and utility of ByteLight tokens may fluctuate and are not assured by Crypto Data Bytes. The token is designed to facilitate engagement and is not a promise or obligation to deliver any product, service, or performance. We encourage all users to approach ByteLight token ownership with caution and to seek independent advice. Crypto Data Bytes disclaims any responsibility for the token's value or the fulfillment of any services or benefits.

Rewards and Participation for Bytexplorers:

🔍 Learn about the Bytexplorers and join us

This is our second reputation token! It will be the counter in the top left of your Bytepass, with the red/white face icon. This is meant to signal to people how generally “helpful” you have been both inside and outside the data community. I personally think this is a great way to better understand what’s on other peoples minds, and to also build relationships that turn into bounty work or jobs.

Explorers who answer questions from ByteLight holders will earn ByteInsight tokens, for helping casters get insights. If you respond with a valid answer to a cast, you will earn 1 ByteInsight token for each ByteLight token held by the caster.

You can use this query to see open questions and top three answers:

Point System: For an answer to count as valid, it must either get marked as “@bytecast answered” in a reply by question author or get a like from someone of trailblazer tier or higher. If a like comes from a bytexplorer, it’s weighted based on the tier. If the question author liked the answer cast, that like is worth 10.

Keep in mind that:

• You must own a Bytepass to get rewards.

• The top three answers after 14 days will be rewarded. Answers will be ranked based on likes from the Bytexplorer community, and likes will be weighted based on guild token balance.

• If a question is completely unanswerable, I will reply “unanswerable” and it will not be eligible for rewards.

• If you wrote queries for your answer, you can submit those to the weekly quest to earn guild tokens on them.

The plan is 75% of revenue from ByteLight mint sales and trading fees will be split amongst the community, based on explorer tokens earned. 25% will be taken by me for now to cover time/cost of admin and operations - this percentage will decrease over time as the community grows in size. It will likely be distributed through quarterly Splits or Sablier airstreams.

The method and weight of reward distribution is still to be finalized, More details will come over the next couple of months.

Legal Disclaimer: The information provided regarding the distribution of revenue from ByteLight sales and trading fees represents our current intentions and is shared to outline our preliminary vision. Please be aware that these plans are not final and are subject to change as we continue to refine our strategy and respond to regulatory, technical, and community feedback. No statement in this communication is intended to be, nor should it be construed as, a promise or commitment. The actual implementation of revenue distribution may differ, and we reserve the right to modify any of the mentioned mechanisms. We recommend our community members stay informed through our official channels for the latest updates and developments. Depending on jurisdiction, KYC (Know Your Customer) procedures may be required, and the specifics of revenue distribution may be adjusted to comply with legal and regulatory standards.

Subscribe now

Technical Notes:

• The ByteLight token will be an erc1155 token, distributed through airdrops or bought/sold through a single sudoswap pool. Transferability otherwise will be turned off. A 15% fee will apply to all trades.

• The ByteInsight token will be an erc1155 token that is nontransferable. The count of an explorer’s balance will show up under the top left of the Bytexplorer pass.