Operate an Oracle

Operator running Oracle

Oracle Operators are responsible for periodically reporting consensus layer data to the Liquid Collective protocol.

In particular, they are responsible for reporting network rewards earned by validators on the consensus layer so those network rewards can be accounted for, accruing the value of the LsETH token (cToken) by increasing the Conversion Rate.

The Oracle smart contract is responsible for synchronizing Oracle Operators.

Oracle Operators

Oracle operators are responsible for periodically reporting consensus layer data to the execution layer.

In particular, they are responsible for reporting network rewards earned by validators on the consensus layer so those network rewards can be accounted for, accruing the value of the LsETH token (cToken) by increasing the Conversion Rate.

The Oracle smart contract is responsible for synchronizing Oracle operators. In particular it:

• Lists the addresses of Oracle members that are expected to report

• Sets the next target epoch to be reported by Oracle members

• Expects a quorum of Oracle members to report the same data at the target epoch before forwarding it to the River contract

Oracle reports consist of:

• The count of validators on the consensus layer

• The total balance of those validators on the consensus layer

Oracle Operator Procedure

Oracle Operators interested in participating in the Oracle set should go through the following procedure:

1. One-Time Protocol Onboarding

   1. Oracle Operator generates a wallet to be used as Operator members

   2. Oracle member address gets approved on the Oracle smart contract

2. Ongoing Operations

   1. Oracle Operator configures and runs the Oracle daemon in its infrastructure connecting self-managed Ethereum EL and CL clients

   2. Oracle Operator makes sure that the Operator member account always has a sufficient ETH balance to pay for gas fees implied by report transactions

Oracle Daemon

Description

To facilitate operations, Oracle Operators are recommended to use the Oracle daemon which is an open-source application maintained by Liquid Collective. While the Oracle daemon is recommended, it is not mandatory, and Oracle Operators can opt-out of it and use their own preferred solution.

The Oracle daemon is a long-living application that:

• Continuously listens for event logs emitted by the Operators Registry contract

• Calls the Oracle contract to get the next target epoch to report

• Exposes various metrics about the activity of the Oracle

When the target epoch to report is reached, it:

• Collects consensus layer data for the target epoch

• Generates a report from the collected data (validators count and total balance on the consensus layer)

• Sends the report transaction to the Oracle contract

The Oracle daemon can optionally run in --dry-mode in which case it does not send report transactions to the chain.

Architecture

The Oracle daemon needs to be connected to:

• Ethereum Execution Layer RPC endpoint

  • to listen for event logs emitted by the Operator registry contract (in particular events when validator keys get funded)

  • to aggregate all withdrawals data since the Shapella fork

  • to call the Oracle contract for the next epoch to report

• Ethereum Consensus Layer HTTP endpoint

  • to query validator data

The Oracle Daemon also needs access to a private key for signing report transactions. This wallet corresponds to the oracle member address that has been approved on the oracle contract.

Since the Shapella fork, it has become necessary to retrieve withdrawals for all Liquid Collective validators. However, there is no available API for this task. One solution is to read all blocks, but this consumes a large number of requests. An alternative solution is to implement a stateful approach.

As a result, we have decided to introduce a new flag in the lceth CLI, --data-dir, which stores the state of withdrawals. The stored state encompasses all withdrawals and their corresponding blocks. (The estimated size of the storage is approximately 4 GB.)

Dependencies

• A synced Execution Layer client with JSON-RPC endpoint enabled. All implementations are supported (Geth, Erigon, Besu, etc.)

• A synced Consensus Layer client with API endpoint enabled. All implementations are supported (Prysm, Teku, Lighthouse, etc.)

Installation

The recommended installation is to use the public Docker image public.ecr.aws/alluvial/liquid-collective/lceth:latest

It is also possible to build the binary from sources and run it.

Estimated gas cost

Sending an Oracle report uses about 100,000 gas for a regular report and ~300,000 gas when reaching the quorum.

Let's estimate how much in gas fees an Oracle Operator should pay per year, considering:

• the protocol expects 1 report per day

• the average gas price is 30 gwei

• there are 5 oracle operators

• the quorum is 3

• all operators use an implementation which alternates submissions order

This means that each day there is a 2/5 probability of sending a regular report and a 1/5 probability of sending a report that reaches quorum:

(365 _ 2/5 _ 100000 + 365 _ 1/5 _ 300000) * 30 = 1 095 000 000 Gwei = 1.09 ETH / year

Guidelines

Protocol onboarding

Generate Oracle member wallet

To generate such a wallet you can use the following command:

CLI

env KEYSTORE_PASSWORD={password to encrypt the key file} lceth eth1keys generate

Docker

Once generated you should securely store the key file and password for later usage, as you will need it each time you will run the Oracle daemon.

Submit Oracle member

Submit the Oracle member address to the administrator to get it approved on the Oracle contract members list.

On-going operations

Oracle has become stateful; be sure to choose an appropriate data directory using the --data-dir flag.

Dry-Run Mode

In dry run the Oracle daemon collects data and generates reports but does not send transactions to the Oracle contract.

It is recommended to use dry run mode when:

• You are not an Oracle Operator and you are only interested in collecting Oracle metrics data

• You are an Oracle Operator and you want to first test in dry run before running in live mode

To run the oracle in dry-run mode, you can run:

CLI

env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
ETH_CL_ADDR={ethereum consensus layer endpoint} \
lceth oracle run --dry-run --data-dir ./data

Docker

Live Mode

In live mode, the daemon runs full capabilities and sends transactions to the chain for every epoch to be reported.

In live mode, you need the daemon to have access to the Oracle member wallet that will be used to sign report transactions.

Top up Oracle Operator account

Before starting the daemon you should make sure that the Oracle Operator account has been topped-up with ETH so it can pay for the gas fees for the report transactions.

After starting the daemon, an Oracle Operator should make sure that the Oracle member account always has a sufficient balance of ETH to send report transactions.

Oracle Operators are responsible for paying for the gas implied by the report transaction.

To run the daemon in live mode, you can run:

CLI

env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
ETH_CL_ADDR={ethereum consensus layer endpoint} \
KEYSTORE_PASSWORD={password used to encrypt the Oracle member key file} \
ORACLE_MEMBER_ADDR={address of the Oracle member wallet} \
ORACLE_DATA_DIR="/data" \
lceth oracle run

Docker

Oracle Daemon Command

Run Oracle reporter deamon

Usage:
  lceth oracle run [flags]

Flags:
      --eth-cl-addr string          Address of the Ethereum consensus layer node to connect to [env: ETH_CL_ADDR]
      --dry-run                     Run oracle reporter without attempting to submit on-chain reports [env: ORACLE_DRY_RUN]
      --gas-limit int               Set gas limit when submitting reports [env: ORACLE_GAS_LIMIT] (default 600000)
      --member-addr string          Ethereum address of the oracle operator used to sign and submit reports [env: ORACLE_MEMBER_ADDR]
      --report-distance int         Maximum count of epochs after frame start where oracle has to report [env: ORACLE_REPORT_DISTANCE]
      --withdrawal-batch-size int   Maximum count of withdrawals blocks to fetch in parallel [env: ORACLE_WITHDRAWAL_BATCH_SIZE] (default 20)
      --data-dir string             Local directory where to store withdrawal data [env: ORACLE_DATA_DIR] (default "./data")
      --loop-sleep-time duration    Duration in seconds between 2 fetch calls [env: ORACLE_LOOP_SLEEP_TIME] (default 15s)
  -h, --help                        help for run

Global Flags:
      --allowlist-addr string            Address of the Allowlist contract [env: ALLOWLIST_ADDR]
      --deployment-block uint            Deployment block of the contracts [env: DEPLOYMENT_BLOCK]
      --el-fee-recipient-addr string     Address of the Execution Layer fee recipient contract [env: EL_FEE_RECIPIENT_ADDR]
      --eth-el-addr string               JSON-RPC address of the Ethereum execution layer node to connect to [env: ETH_EL_ADDR]
      --keystore-password string         Password used to encrypt key files [env: KEYSTORE_PASSWORD]
      --keystore-path string             Directory where to store keys [env: KEYSTORE_PATH]
      --log-format string                Log output format (text or json) [env: LOG_FORMAT] (default "text")
      --log-level string                 Log output level [env: LOG_LEVEL] (default "info")
      --operators-registry-addr string   Address of the Operators Registry contract [env: OPERATORS_REGISTRY_ADDR]
      --oracle-addr string               Address of the Oracle contract [env: ORACLE_ADDR]
      --redeem-manager-addr string       Address of the RedeemManager contract [env: REDEEM_MANAGER_ADDR]
      --river-addr string                Address of the River contract [env: RIVER_ADDR]
      --tlc-addr string                  Address of the TLC contract [env: TLC_ADDR]
      --withdraw-addr string             Address of the Withdraw contract [env: WITHDRAW_ADDR]
      --wls-eth-addr string              Address of the WlsEth contract [env: WLSETH_ADDR]

Monitoring and Metrics

The Oracle service exposes a health endpoint that exposes routes for monitoring:

• /live Liveness endpoint

• /ready Readiness endpoint

• /metrics Prometheus metrics