Warp Syncer

Warp-Syncer is an umbrella project for implementing various tasks related to downloading and uploading SmartWeave data to Arweave.

The Need for Syncer: Why It Matters

When SmartWeave was first conceived, the protocol only supported one type of transaction: the Arweave base layer upload, often referred to as Arweave L1 transaction. Given the high demand for the network's limited block size and long block finality window, the ANS-104 transaction bundling standard was introduced. In essence, this standard lets us bundle large amounts of data and store it on Arweave via an intermediary. This quickly became the go-to upload standard, steering Arweave towards a clear path of addressing distributed network scalability challenges.

While bundling has immensely benefited the Arweave Ecosystem, it also raised composability issues within the SmartWeave Protocol. The vast majority of SmartWeave activity now occurs on a higher layer through vertical scaling, utilizing the Warp Sequencer and settling data using the Bundlr Network onto the Arweave base layer commonly referred to as L2 transactions. This shift has posed challenges in harmonizing both L1 and L2 transaction types. It brought forward issues like potential double-spending attacks, inconsistencies in contract states, and difficulties in accessing and filtering bundled SmartWeave transaction data.

Solution

Warp Syncer addresses the SmartWeave composability issues between the base layer (Arweave L1) transactions and those generated by the Warp Sequencer, known as L2 transactions.

It's designed to maintain the efficiency of Warp Contracts while seamlessly integrating both transaction types within a single SmartWeave contract, ensuring security and deterministic results.

Methodology

For a seamless integration of L1 and L2 transactions, we need a sequencing method that bridges both dimensions. Relying solely on Arweave's block height is constraining since multiple SmartWeave activities can occur within a single 2-minute block. That's where Warp's SortKey comes into play. The measure assigns a unique value to each contract transaction, allowing the SmartWeave protocol to arrange transactions in a lexicographic sequence.

The generation algorithm can be reviewed here.

Crucially, the lastSortKey parameter aids in pinpointing the SortKey of the preceding transaction, ensuring a consistently ordered chain of SmartWeave transactions.

For some time, the lastSortKey has been an integral part of the Warp SDK. What's novel now is its role in orchestrating the order between L1 and L2 transactions.

To qualify for Syncer's inclusion, interactions must satisfy two conditions:

• Syncer waits for 10 blocks before downloading transactions and interactions.
• Syncer demands consensus from 10 out of 15 internally selected network nodes, based on sync timing and performance. More details on the schema, here.

High Level Syncer Workflow

Once these criteria are met, Syncer proceeds to its primary synchronization modules. On a high level, these actions consist of downloading transactions, parsing all of their interactions, and saving them into an internal Postgres database.

Developer Look: Syncer's Infrastructure: here.

Afterward, Syncer moves to assign the ID of the latest block that it has processed.

It's worth noting the deep integration between the Warp Syncer and Sequencer, both vital components of the SDK that closely interplay. At present, the Sequencer derives SortKey based on the latest block height served by Syncer.

If you're keen on delving deeper into the intricacies of Warp Sequencer, you can find more information here.

In the earlier version of Warp Syncer, the SortKey for L2 interactions was determined using the block_height metric, fetched directly from the arweave.net gateway. Now, the Sequencer sources the block height metric from the internal Postgres database. This metric represents the latest block height captured after the primary synchronization module, outlined here. This updated approach offers enhanced finality guarantees for the synced block. By eliminating the risk of including forked blocks, it ensures that the Sequencer's L2 transactions will always have a SortKey higher than the last fully synced Arweave block. This maintains the integrity of the transaction sequence and ensures determinism.

This method also provides a safety cushion against potential cascading issues, ensuring the Warp stack doesn't succumb to a singular point of failure. If the Sequencer were to generate a SortKey using the current block height and the Syncer went offline, transactions from the block at the time of the failure might not sync upon restart. Consequently, DRE execution nodes could miss these transactions, as they would jump to a higher SortKey set by the Warp Sequencer.

Commands

Warp-Syncer is a single binary that can be run with different commands:

Command	Description
sync	Download L1 interactions from Arweave
forward	Send L1 and L2 interactions from the database to Redis. Assign last_sort_key to L1 interactions
contract	Download L1 contracts
bundle	Send L2 interactions to bundlr.network
check	Monitor if bundles are finalized
gateway	Run a REST API server
help	Help about any command
env	Prints evnironment variables used for configuration