Operate Validator Nodes

This guide is intended for Node Operators participating in the Liquid Collective protocol.

Operate Validator Nodes

Node Operators are validator infrastructure providers responsible for running validator nodes on the Ethereum consensus layer in the name of the protocol. Node Operators must have the capacity to scale to a large number of validator nodes, meet certain compliance requirements, and meet certain performance requirements, including Liquid Collective's Node Operator Performance SLAs.

Node Operators receive ETH delegation from the protocol and they are remunerated proportionally to the delegation they receive.

Operate Validator Nodes

Node Operator Procedure

Node Operators interested in receiving ETH delegation should go through the following flow.

1. One-time protocol onboarding

   1. Node operator generates Node Operator wallet

   2. Node Operator address gets approved on the Node Operator registry contracts

2. On-going operations

   1. Pre-registration of validator keys

      • Node Operator generates validator keys in its infrastructure with expected configuration (withdrawal credentials, execution layer fee recipient, etc.)

      • Node Operator submits validator keys to the Node Operators registry contract by sending a transaction using the Node Operator wallet

      • From this point in time, validator keys should be ready to get funded

   2. Protocol administrator reviews and confirms added keys are valid

      • Validates validator keys, ensuring withdrawal credentials, deposit data, and signatures are correct

      • Administrator increases operator limit making validator keys eligible for funding. From this point in time, validators keys can be funded at any time

   3. Validator keys are funded and activated

      • Protocol regularly picks eligible validator keys and deposits them to the official Ethereum deposit contract

      • Once deposited, validator keys enter the activation queue as per the standard Ethereum staking procedure

   4. Run the Exit Daemon

      • Node Operators are responsible for exiting their own validators

      • Node Operators have the possibility of running the lceth exit daemon in order to be notified when they should exit their validators

CLI

CLI provides various commands to facilitate Node Operators in the process of managing validator keys.

CLI is compatible with the Ethereum deposit CLI and can be used in conjunction with it.

Installation

The recommended installation is to use the public Docker image public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0

It is also possible to build the binary from sources.

Guidelines

Protocol Onboarding

Generate (or Import) Node Operator Wallet

The Node Operator needs a wallet to submit validator keys. The wallet must be approved on the Node Operator Registry Contract.

To generate such a wallet you can use the following command:

CLI

env KEYSTORE_PASSWORD={password to encrypt the key file} lceth eth1keys generate

Docker

Once generated or imported you should securely store the key file and password for later usage, as you will need it each time you need to register keys.

It is also possible to import an existing wallet:

CLI

env KEYSTORE_PASSWORD={password to encrypt the key file} lceth eth1keys import --priv-key {private key in hex format

Docker

Approve Node Operator Wallet

A Node Operator should provide administrators with:

• name: As the Node Operator will be publicly listed on the Node Operator contract

• address: The Node Operator wallet address previously generated

Those values can be updated at any time after first approval.

Node Operator Index

Once approved, a Node Operator will get its operator index on the Node Operators Registry. This index will never change over time.

To get your node operator index, run the below command and find your name in the returned list.

CLI

env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
 lceth operators list

Docker

Ongoing Operations: Pre-Registration of Validator Keys

Pre-registering validators consists of submitting new validators keys to the Node Operators registry contract so they can be picked for validation and funded.

As a Node Operator, you should typically pre-register new validators keys when:

• It is the first time you pre-register keys

• Most of the keys that you have pre-registered have been funded

It is the responsibility of the Node Operator to make sure validator keys are available for funding.

In the case that a Node Operator has no validator keys available, it will not receive an ETH delegation from the protocol.

Generate validator keys, deposit data and signatures

A Node Operator is responsible for generating validator keys in its own infrastructure.

For each generated key, a Node Operator is also expected to generate the corresponding DepositMessage and BLS12-381 signature as per the Ethereum 2.0 specs

Node Operators are required to set withdrawal credentials to the address of Withdrawal contract.

Per the Ethereum protocol, once a Type 1 (0x01) withdrawal address is set those withdrawal credentials cannot be changed.

To get withdrawal credentials you can run:

CLI

env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
 lceth withdrawal credentials

Docker

As a result of the validator key generation, a Node Operator should obtain a JSON file matching the following format:

validator_keys.json

[
  {
    "pubkey": "814dc0f55ac3fb02431668adf6f8fa1c37fb9baa5b87f5be519a373205933dfe742f3df566cba3a35b5be1940e1dffd5",
    "signature": "81b24791a89f3596abebb294e13502e398d96cbdd2a70d87b7fa89cc93019080a2905bb7447e0c408f2569a17b0ce628163e8caca4ea5b69af4db458dffcca100f8fcb96edeef30072ca4322721adc5591da8aadffac8b730d78f84c3b9f3f92"
  },
  {
    "pubkey": "88d1ac7f33780fd328bee60957b2325cfa41b3719614b662616d4525e5b478b3a81d490671526936e3ea412428c84451",
    "signature": "9985f2eb6f445f5e2bae226333f4796127c01efed7f4a4a35b62719dd5470879d1377e0035c00ccaa1cf892542bb427b072113a8e690e963d3e5b590ffd0051c0680e345cd1ee9318f2559796585445c490b8e9db417e99ac55e99c7e281aced"
  }
]

It is possible to use the Ethereum deposit CLI to generate keys, which produces a file in the above format.

Configure validator exec layer fee recipient address

Node Operators are required to set the fee recipient of the validators to the ELFeeRecipient contract that is responsible to flow the execution layer network rewards to the core River contract.

To get the ELFeeRecipient address you can run:

CLI

env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
 lceth el-fee-recipient address

Docker

Exec layer fee recipient address and withdrawal address are not the same.

Submit validator keys

Submit validator keys consists of sending a addValidatorKeys(...) transaction to the OperatorsRegistry contract. The transactions should be signed by the Node Operator address:

index.sol

/// @notice Adds new keys for an operator
/// @dev Only callable by the administrator or the operator address
/// @param _name The name identifying the operator
/// @param _keyCount The amount of keys provided
/// @param _publicKeys Public keys of the validator, concatenated
/// @param _signatures Signatures of the validator keys, concatenated
function addValidatorKeys(
    string calldata _name,
    uint256 _keyCount,
    bytes calldata _publicKeys,
    bytes calldata _signatures
)

To submit validator keys you can run the following command:

CLI

env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
KEYSTORE_PASSWORD={password to encrypt the key file} \
lceth validators add \
--operator-idx {operator index} \
--deposit-data validator_keys.json \
--from {operator address} \
--send

Docker

Once the transaction has been sent to the network and validated, the validator keys are listed on the Node Operators Registry contract and will be reviewed by the administrator.

Deposit and Node Activation

Node Operators should be careful that validator keys can be picked, funded, and deposited to the official Ethereum Deposit Contract at any time.

Once a validator key has been funded and deposited, it enters the activation queue. When this happens, the Node Operator should make sure that the corresponding validator key gets properly deployed to its validator infrastructure so the validator is ready when validator activation occurs.

We strongly recommend Node Operators effectively monitor the Deposit Contract, in particular watching DepositEvent to never miss an activation.

Watching the exit requests

• Node Operators are responsible for exiting their own validators

• Node Operators can use the provided Exit Daemon

Useful events:

• FundedValidatorKeys

• RequestedValidatorExits

Exit Daemon

Description

The Validator Exit Daemon is an off-chain application designed to help Liquid Collective Node Operators exit validator keys with ease.

Node Operators are responsible for exiting validator keys from the Consensus Layer, allowing withdrawn funds to be used for satisfying conversions of LsETH for ETH.

Flow

• LsETH holders convert LsETH for ETH through the protocol, creating a request for ETH.

• The protocol regularly rebalance's ETH positions. When it needs ETH funds to fulfill redeem demands, it withdraws funds from the Consensus Layer and signals Node Operators by emitting an event requesting validator keys to be exited. The protocol is responsible for selecting the Node Operators that need to exit keys, depending on the current validator key allocation.

• Upon receiving a validator exit request event, the Node Operator exits the requested number of validator keys, which results in broadcasting ValidatorExit messages to the Consensus Layer. This step is facilitated by the Validator Exit Daemon application.

• After validator keys pass through the exit queue, funds are withdrawn to the LC Withdraw contract, and the Liquid Collective protocol takes over the ETH to satisfy the LsETH conversion requests.

Technical considerations & assumptions:

• Validator exit is triggered by broadcasting a ValidatorExit message signed with the validator's private key on the Consensus Layer.

• We cannot rely on the Dual-Key exit design to also allow triggering the validator key exit from the withdrawal_credentials. Even if this option is preferred, it is not expected to be available on day 1 of the withdrawals.

• Validator keys may be slashed, resulting in the keys undergoing the Consensus Layer slashing process and eventually leading to the withdrawal of funds after incurring penalties (this takes at least 36 days).

• Existing Liquid Collective validator keys have all been set with a 0x1 prefix and point to the Withdraw contract.

• Liquid Collective Node Operators use diverse infrastructure base layers (clouds, bare metal, containers/orchestration), client software (Prysm, Lighthouse, etc.), and may use or not use signers (e.g., Web3Signer). This diversity is expected to grow with staking innovations (e.g., DVT).

• Liquid Collective Node Operators have (or are building) infrastructure & processes

Architecture

The Validator Exit Daemon is a long-lived application containerized in a Docker image designed to be run by Node Operators in their infrastructure.

It accesses data from both the Execution Layer and Consensus Layer connected to nodes in the Node Operator infrastructure.

It assumes that Node Operators have an existing system and process in place for proceeding with validator key exits.

The Daemon can interact with the existing exit system through one or both of the following methods:

• HTTP hook: Called by the Daemon each time a validator request exit event is emitted, and a validator key should be exited. Node Operators can configure the callback to match any endpoint.

• API polling: The Daemon exposes an API that Node Operators can poll to retrieve the key to exit.

The Validator Exit Daemon will provide the following information to the Node Operator:

• Number of validator keys to exit

• Recommended keys to exit

Diagram

Validator Exit Daemon

Sequence diagram

Validator Exit Daemon Sequence Diagram

Dependencies

• A synced Execution Layer client with a JSON-RPC endpoint enabled. All implementations are supported (Geth, Erigon, Besu, etc.)

• A synced Consensus Layer client with an API endpoint enabled. All implementations are supported (Prysm, Teku, Lighthouse, etc.)

Installation

The recommended installation is to use the public Docker image public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0

For those running kubernetes and helm you can use this image:public.ecr.aws/alluvial/helm-charts/lceth:1.3.0

Usage

API

By default, the Exit daemon exposes an API route that Node Operators can use, either in conjunction with or without the HTTP hook.

This allows an operator to retrieve the recommended keys to exit at the current time.

CLI

env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
env ETH_CL_ADDR={ethereum consensus layer rpc endpoint} \
lceth exit run \
--operator-idx {operator index}

Docker

Operator index could be retrieved with lceth operators list

The API meets the following requirements:

paths:
  '/':
    get:
      description: 'Get validator keys to exit'
      responses:
        200:
          description: 'Success.'
          schema:
            type: object
            properties:
              chain_id:
                type: 'string'
                example: '1'
              total_requested_exits:
                type: 'integer'
                description: 'Number of validators requested to exit'
                example: 2
              validators_to_exit:
                type: 'array'
                description: 'List of recommended validators to exit'
                items:
                  type: 'object'
                  properties:
                    pubkey:
                      type: 'string'
                      example: '0x8a9e1213459337631232447e360faf162781aee0a3df533306a02e2eae481b4db4d1313d1756da7a121da437d9964fb0'
                    index:
                      type: 'integer'
                      example: 473944
        500:
          description: 'Internal unexpected error'

Webhook

The Exit daemon can run an HTTP hook system that sends validator exit request callbacks to the operator's existing systems when it receives a Validator Exit request event from the Consensus Layer.

The Exit daemon is stateless. If the webhook is enabled, it will send a call at each start if there is a pending exit request.

Operators can customize the webhook call by providing a template for the body, endpoint, and headers, and use the following macros to pass data related to the exit event:

| Name                       | Type     | Description                               |
| -------------------------- | -------- | ----------------------------------------- |
| chain_id                   | string   | Network identifier                        |
| total_requested_exits      | int      | Number of validators requested to exit    |
| validators_to_exit_indexes | int[]    | List of validators pubkey to exit         |
| validators_to_exit_pubkeys | string[] | List of validators index to exit          |
| validators_to_exit         | object[] | List of validators pubkey + index to exit |

• total_requested_exits represents the total requested exits since the beginning of the contract. It’s a incremental-only counter emitted by the RequestedValidatorExits event.

• validators_to_exit (as well as validators_to_exit_pubkeys & validators_to_exit_indexes) is an array of the validators to exit at a given time. To get them, the exit daemon fetches all the keys of an operator (via the FundedValidatorKeys event), check their status on the consensus layer to know those already exited, and then make a diff with total_requested_exits to know how many keys are remaining to exit.

The webhook templating imports the Golang Sprig library. Which means additional functions are available to format the output.

For instance:

• using environment variables via {{env "ENV_VAR_NAME"}}

• printing array via {{validators_to_exit_pubkeys | toJson}}

• creating a quoted string from an array via {{.validators_to_exit_indexes | join "," | quote}}

• computing array length via {{.validators_to_exit_indexes | len}}

• etc.

Example of webhook template:

CLI

env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
env ETH_CL_ADDR={ethereum consensus layer rpc endpoint} \
lceth exit run \
--operator-idx {operator index} \
--webhook \
--webhook-method="POST" \
--webhook-endpoint="http://127.0.0.1:9080/call?chain_id={{.chain_id}}&count={{.total_requested_exits}}&validators={{.validators_to_exit_indexes|join ","}}" \
--webhook-template-body '
  {
    "chain_id": "{{.chain_id}}",
    "total_requested_exits": {{.total_requested_exits}},
    "current_requested_exits": {{.validators_to_exit_pubkeys | len}},
    "indexes": {{.validators_to_exit_indexes | join "," | quote}},
    "pubkeys": {{.validators_to_exit_pubkeys | toJson}},
    "custom": {{env CUSTOM_ENV_VAR}}'
  }
' \
--webhook-template-header="Chain-Id={{.chain_id}}" \
--webhook-template-header="Content-Type=application/json"

Docker

Usage output:

Run exit daemon

Usage:
  lceth exit run [flags]

Flags:
      --eth-cl-addr string                       Address of the Ethereum consensus layer node to connect to [env: ETH_CL_ADDR]
      --operator-idx int                         Index of the operator running the daemon [env: OPERATOR_INDEX]
      --webhook                                  enable 'webhook' [env: WEBHOOK]
      --webhook-endpoint string                  Go template describing the endpoint that the webhook will call [env: WEBHOOK_ENDPOINT]
      --webhook-method string                    HTTP method to query the webhook's endpoint [env: WEBHOOK_METHOD]
      --webhook-template-header stringToString   Go template describing the body to be sent to the webhook's endpoint [env: WEBHOOK_TEMPLATE_HEADER] (default [])
      --webhook-template-body string             Go templates describing the headers to be sent to the webhook [env: WEBHOOK_TEMPLATE_BODY]
      --webhook-retry int                        Retry limit in case of webhook-endpoint returns error with status code 5xx [env: WEBHOOK_RETRY]
      --webhook-timeout duration                 Delay to wait before abort request sent to webhook-endpoint [env: WEBHOOK_TIMEOUT]
      --rpc-call-retry int                       How many time should we retry failed rpc calls [env: RPC_CALL_RETRY] (default 3)
      --block-log-fetch-step int                 How many blocks are fetched at once when fetching logs [env: BLOCK_LOG_FETCH_STEP] (default 100000)
      --max-workers int                          How many goroutines to use on threaded tasks [env: MAX_WORKERS] (default 10)
      --batch-fetch-validators                   Whether to batch when querying the consensus layer to get validators' data (recommended if timeout issues arise) [env: BATCH_FETCH_VALIDATORS]
      --max-get-validators int                   Max size of the batch when querying the consensus layer to get validators' data, only useful if using --batch-fetch-validators [env: MAX_GET_VALIDATORS] (default 1000)
      --loop-sleep-time duration                 How often should the exit-daemon run [env: LOOP_SLEEP_TIME] (default 5m0s)
  -h, --help                                     help for run

Global Flags:
      --allowlist-addr string            Address of the Allowlist contract [env: ALLOWLIST_ADDR]
      --deployment-block uint            Deployment block of the contracts [env: DEPLOYMENT_BLOCK]
      --el-fee-recipient-addr string     Address of the Execution Layer fee recipient contract [env: EL_FEE_RECIPIENT_ADDR]
      --eth-el-addr string               JSON-RPC address of the Ethereum execution layer node to connect to [env: ETH_EL_ADDR]
      --keystore-password string         Password used to encrypt key files [env: KEYSTORE_PASSWORD]
      --keystore-path string             Directory where to store keys [env: KEYSTORE_PATH]
      --log-format string                Log output format (text or json) [env: LOG_FORMAT] (default "text")
      --log-level string                 Log output level [env: LOG_LEVEL] (default "info")
      --operators-registry-addr string   Address of the Operators Registry contract [env: OPERATORS_REGISTRY_ADDR]
      --oracle-addr string               Address of the Oracle contract [env: ORACLE_ADDR]
      --redeem-manager-addr string       Address of the RedeemManager contract [env: REDEEM_MANAGER_ADDR]
      --river-addr string                Address of the River contract [env: RIVER_ADDR]
      --tlc-addr string                  Address of the TLC contract [env: TLC_ADDR]
      --withdraw-addr string             Address of the Withdraw contract [env: WITHDRAW_ADDR]
      --wls-eth-addr string              Address of the WlsEth contract [env: WLSETH_ADDR]