Operate Validator Nodes
This guide is intended for Node Operators participating in the Liquid Collective protocol.
Operate Validator Nodes
Node Operators are validator infrastructure providers responsible for running validator nodes on the Ethereum consensus layer in the name of the protocol. Node Operators must have the capacity to scale to a large number of validator nodes, meet certain compliance requirements, and meet certain performance requirements, including Liquid Collective's Node Operator Performance SLAs.
Node Operators receive ETH delegation from the protocol and they are remunerated proportionally to the delegation they receive.
Operate Validator Nodes
Node Operator Procedure
Node Operators interested in receiving ETH delegation should go through the following flow.
One-time protocol onboarding
Node operator generates Node Operator wallet
Node Operator address gets approved on the Node Operator registry contracts
On-going operations
Pre-registration of validator keys
Node Operator generates validator keys in its infrastructure with expected configuration (withdrawal credentials, execution layer fee recipient, etc.)
Node Operator submits validator keys to the Node Operators registry contract by sending a transaction using the Node Operator wallet
From this point in time, validator keys should be ready to get funded
Protocol administrator reviews and confirms added keys are valid
Validates validator keys, ensuring withdrawal credentials, deposit data, and signatures are correct
Administrator increases operator limit making validator keys eligible for funding. From this point in time, validators keys can be funded at any time
Validator keys are funded and activated
Protocol regularly picks eligible validator keys and deposits them to the official Ethereum deposit contract
Once deposited, validator keys enter the activation queue as per the standard Ethereum staking procedure
Run the Exit Daemon
Node Operators are responsible for exiting their own validators
Node Operators have the possibility of running the
lcethexit daemon in order to be notified when they should exit their validators
CLI
CLI provides various commands to facilitate Node Operators in the process of managing validator keys.
CLI is compatible with the Ethereum deposit CLI and can be used in conjunction with it.
Installation
The recommended installation is to use the public Docker image public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0
It is also possible to build the binary from sources.
Guidelines
Protocol Onboarding
Generate (or Import) Node Operator Wallet
The Node Operator needs a wallet to submit validator keys. The wallet must be approved on the Node Operator Registry Contract.
To generate such a wallet you can use the following command:
env KEYSTORE_PASSWORD={password to encrypt the key file} lceth eth1keys generateOnce generated or imported you should securely store the key file and password for later usage, as you will need it each time you need to register keys.
It is also possible to import an existing wallet:
env KEYSTORE_PASSWORD={password to encrypt the key file} lceth eth1keys import --priv-key {private key in hex formatApprove Node Operator Wallet
A Node Operator should provide administrators with:
name: As the Node Operator will be publicly listed on the Node Operator contract
address: The Node Operator wallet address previously generated
Node Operator Index
Once approved, a Node Operator will get its operator index on the Node Operators Registry. This index will never change over time.
To get your node operator index, run the below command and find your name in the returned list.
env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
lceth operators listOngoing Operations: Pre-Registration of Validator Keys
Pre-registering validators consists of submitting new validators keys to the Node Operators registry contract so they can be picked for validation and funded.
As a Node Operator, you should typically pre-register new validators keys when:
It is the first time you pre-register keys
Most of the keys that you have pre-registered have been funded
It is the responsibility of the Node Operator to make sure validator keys are available for funding.
In the case that a Node Operator has no validator keys available, it will not receive an ETH delegation from the protocol.
Generate validator keys, deposit data and signatures
A Node Operator is responsible for generating validator keys in its own infrastructure.
For each generated key, a Node Operator is also expected to generate the corresponding DepositMessage and BLS12-381 signature as per the Ethereum 2.0 specs
Node Operators are required to set withdrawal credentials to the address of Withdrawal contract.
To get withdrawal credentials you can run:
env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
lceth withdrawal credentialsAs a result of the validator key generation, a Node Operator should obtain a JSON file matching the following format:
[
{
"pubkey": "814dc0f55ac3fb02431668adf6f8fa1c37fb9baa5b87f5be519a373205933dfe742f3df566cba3a35b5be1940e1dffd5",
"signature": "81b24791a89f3596abebb294e13502e398d96cbdd2a70d87b7fa89cc93019080a2905bb7447e0c408f2569a17b0ce628163e8caca4ea5b69af4db458dffcca100f8fcb96edeef30072ca4322721adc5591da8aadffac8b730d78f84c3b9f3f92"
},
{
"pubkey": "88d1ac7f33780fd328bee60957b2325cfa41b3719614b662616d4525e5b478b3a81d490671526936e3ea412428c84451",
"signature": "9985f2eb6f445f5e2bae226333f4796127c01efed7f4a4a35b62719dd5470879d1377e0035c00ccaa1cf892542bb427b072113a8e690e963d3e5b590ffd0051c0680e345cd1ee9318f2559796585445c490b8e9db417e99ac55e99c7e281aced"
}
]Configure validator exec layer fee recipient address
Node Operators are required to set the fee recipient of the validators to the ELFeeRecipient contract that is responsible to flow the execution layer network rewards to the core River contract.
To get the ELFeeRecipient address you can run:
env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
lceth el-fee-recipient addressExec layer fee recipient address and withdrawal address are not the same.
Submit validator keys
Submit validator keys consists of sending a addValidatorKeys(...) transaction to the OperatorsRegistry contract. The transactions should be signed by the Node Operator address:
/// @notice Adds new keys for an operator
/// @dev Only callable by the administrator or the operator address
/// @param _name The name identifying the operator
/// @param _keyCount The amount of keys provided
/// @param _publicKeys Public keys of the validator, concatenated
/// @param _signatures Signatures of the validator keys, concatenated
function addValidatorKeys(
string calldata _name,
uint256 _keyCount,
bytes calldata _publicKeys,
bytes calldata _signatures
)To submit validator keys you can run the following command:
env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
KEYSTORE_PASSWORD={password to encrypt the key file} \
lceth validators add \
--operator-idx {operator index} \
--deposit-data validator_keys.json \
--from {operator address} \
--sendOnce the transaction has been sent to the network and validated, the validator keys are listed on the Node Operators Registry contract and will be reviewed by the administrator.
Deposit and Node Activation
Node Operators should be careful that validator keys can be picked, funded, and deposited to the official Ethereum Deposit Contract at any time.
Once a validator key has been funded and deposited, it enters the activation queue. When this happens, the Node Operator should make sure that the corresponding validator key gets properly deployed to its validator infrastructure so the validator is ready when validator activation occurs.
We strongly recommend Node Operators effectively monitor the Deposit Contract, in particular watching DepositEvent to never miss an activation.
Watching the exit requests
Node Operators are responsible for exiting their own validators
Node Operators can use the provided Exit Daemon
Useful events:
FundedValidatorKeys
RequestedValidatorExits
Exit Daemon
Description
The Validator Exit Daemon is an off-chain application designed to help Liquid Collective Node Operators exit validator keys with ease.
Node Operators are responsible for exiting validator keys from the Consensus Layer, allowing withdrawn funds to be used for satisfying conversions of LsETH for ETH.
Flow
LsETH holders convert LsETH for ETH through the protocol, creating a request for ETH.
The protocol regularly rebalance's ETH positions. When it needs ETH funds to fulfill redeem demands, it withdraws funds from the Consensus Layer and signals Node Operators by emitting an event requesting validator keys to be exited. The protocol is responsible for selecting the Node Operators that need to exit keys, depending on the current validator key allocation.
Upon receiving a validator exit request event, the Node Operator exits the requested number of validator keys, which results in broadcasting ValidatorExit messages to the Consensus Layer. This step is facilitated by the Validator Exit Daemon application.
After validator keys pass through the exit queue, funds are withdrawn to the LC Withdraw contract, and the Liquid Collective protocol takes over the ETH to satisfy the LsETH conversion requests.
Technical considerations & assumptions:
Validator exit is triggered by broadcasting a ValidatorExit message signed with the validator's private key on the Consensus Layer.
We cannot rely on the Dual-Key exit design to also allow triggering the validator key exit from the withdrawal_credentials. Even if this option is preferred, it is not expected to be available on day 1 of the withdrawals.
Validator keys may be slashed, resulting in the keys undergoing the Consensus Layer slashing process and eventually leading to the withdrawal of funds after incurring penalties (this takes at least 36 days).
Existing Liquid Collective validator keys have all been set with a 0x1 prefix and point to the Withdraw contract.
Liquid Collective Node Operators use diverse infrastructure base layers (clouds, bare metal, containers/orchestration), client software (Prysm, Lighthouse, etc.), and may use or not use signers (e.g., Web3Signer). This diversity is expected to grow with staking innovations (e.g., DVT).
Liquid Collective Node Operators have (or are building) infrastructure & processes
Architecture
The Validator Exit Daemon is a long-lived application containerized in a Docker image designed to be run by Node Operators in their infrastructure.
It accesses data from both the Execution Layer and Consensus Layer connected to nodes in the Node Operator infrastructure.
It assumes that Node Operators have an existing system and process in place for proceeding with validator key exits.
The Daemon can interact with the existing exit system through one or both of the following methods:
HTTP hook: Called by the Daemon each time a validator request exit event is emitted, and a validator key should be exited. Node Operators can configure the callback to match any endpoint.
API polling: The Daemon exposes an API that Node Operators can poll to retrieve the key to exit.
The Validator Exit Daemon will provide the following information to the Node Operator:
Number of validator keys to exit
Recommended keys to exit
Diagram
Sequence diagram
Dependencies
A synced Execution Layer client with a JSON-RPC endpoint enabled. All implementations are supported (Geth, Erigon, Besu, etc.)
A synced Consensus Layer client with an API endpoint enabled. All implementations are supported (Prysm, Teku, Lighthouse, etc.)
Installation
The recommended installation is to use the public Docker image public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0
Usage
API
By default, the Exit daemon exposes an API route that Node Operators can use, either in conjunction with or without the HTTP hook.
This allows an operator to retrieve the recommended keys to exit at the current time.
env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
env ETH_CL_ADDR={ethereum consensus layer rpc endpoint} \
lceth exit run \
--operator-idx {operator index}The API meets the following requirements:
paths:
'/':
get:
description: 'Get validator keys to exit'
responses:
200:
description: 'Success.'
schema:
type: object
properties:
chain_id:
type: 'string'
example: '1'
total_requested_exits:
type: 'integer'
description: 'Number of validators requested to exit'
example: 2
validators_to_exit:
type: 'array'
description: 'List of recommended validators to exit'
items:
type: 'object'
properties:
pubkey:
type: 'string'
example: '0x8a9e1213459337631232447e360faf162781aee0a3df533306a02e2eae481b4db4d1313d1756da7a121da437d9964fb0'
index:
type: 'integer'
example: 473944
500:
description: 'Internal unexpected error'Webhook
The Exit daemon can run an HTTP hook system that sends validator exit request callbacks to the operator's existing systems when it receives a Validator Exit request event from the Consensus Layer.
The Exit daemon is stateless. If the webhook is enabled, it will send a call at each start if there is a pending exit request.
Operators can customize the webhook call by providing a template for the body, endpoint, and headers, and use the following macros to pass data related to the exit event:
| Name | Type | Description |
| -------------------------- | -------- | ----------------------------------------- |
| chain_id | string | Network identifier |
| total_requested_exits | int | Number of validators requested to exit |
| validators_to_exit_indexes | int[] | List of validators pubkey to exit |
| validators_to_exit_pubkeys | string[] | List of validators index to exit |
| validators_to_exit | object[] | List of validators pubkey + index to exit |total_requested_exitsrepresents the total requested exits since the beginning of the contract. It’s a incremental-only counter emitted by theRequestedValidatorExitsevent.validators_to_exit(as well asvalidators_to_exit_pubkeys&validators_to_exit_indexes) is an array of the validators to exit at a given time. To get them, the exit daemon fetches all the keys of an operator (via theFundedValidatorKeysevent), check their status on the consensus layer to know those already exited, and then make a diff withtotal_requested_exitsto know how many keys are remaining to exit.
Example of webhook template:
env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
env ETH_CL_ADDR={ethereum consensus layer rpc endpoint} \
lceth exit run \
--operator-idx {operator index} \
--webhook \
--webhook-method="POST" \
--webhook-endpoint="http://127.0.0.1:9080/call?chain_id={{.chain_id}}&count={{.total_requested_exits}}&validators={{.validators_to_exit_indexes|join ","}}" \
--webhook-template-body '
{
"chain_id": "{{.chain_id}}",
"total_requested_exits": {{.total_requested_exits}},
"current_requested_exits": {{.validators_to_exit_pubkeys | len}},
"indexes": {{.validators_to_exit_indexes | join "," | quote}},
"pubkeys": {{.validators_to_exit_pubkeys | toJson}},
"custom": {{env CUSTOM_ENV_VAR}}'
}
' \
--webhook-template-header="Chain-Id={{.chain_id}}" \
--webhook-template-header="Content-Type=application/json"Usage output:
Run exit daemon
Usage:
lceth exit run [flags]
Flags:
--eth-cl-addr string Address of the Ethereum consensus layer node to connect to [env: ETH_CL_ADDR]
--operator-idx int Index of the operator running the daemon [env: OPERATOR_INDEX]
--webhook enable 'webhook' [env: WEBHOOK]
--webhook-endpoint string Go template describing the endpoint that the webhook will call [env: WEBHOOK_ENDPOINT]
--webhook-method string HTTP method to query the webhook's endpoint [env: WEBHOOK_METHOD]
--webhook-template-header stringToString Go template describing the body to be sent to the webhook's endpoint [env: WEBHOOK_TEMPLATE_HEADER] (default [])
--webhook-template-body string Go templates describing the headers to be sent to the webhook [env: WEBHOOK_TEMPLATE_BODY]
--webhook-retry int Retry limit in case of webhook-endpoint returns error with status code 5xx [env: WEBHOOK_RETRY]
--webhook-timeout duration Delay to wait before abort request sent to webhook-endpoint [env: WEBHOOK_TIMEOUT]
--rpc-call-retry int How many time should we retry failed rpc calls [env: RPC_CALL_RETRY] (default 3)
--block-log-fetch-step int How many blocks are fetched at once when fetching logs [env: BLOCK_LOG_FETCH_STEP] (default 100000)
--max-workers int How many goroutines to use on threaded tasks [env: MAX_WORKERS] (default 10)
--batch-fetch-validators Whether to batch when querying the consensus layer to get validators' data (recommended if timeout issues arise) [env: BATCH_FETCH_VALIDATORS]
--max-get-validators int Max size of the batch when querying the consensus layer to get validators' data, only useful if using --batch-fetch-validators [env: MAX_GET_VALIDATORS] (default 1000)
--loop-sleep-time duration How often should the exit-daemon run [env: LOOP_SLEEP_TIME] (default 5m0s)
-h, --help help for run
Global Flags:
--allowlist-addr string Address of the Allowlist contract [env: ALLOWLIST_ADDR]
--deployment-block uint Deployment block of the contracts [env: DEPLOYMENT_BLOCK]
--el-fee-recipient-addr string Address of the Execution Layer fee recipient contract [env: EL_FEE_RECIPIENT_ADDR]
--eth-el-addr string JSON-RPC address of the Ethereum execution layer node to connect to [env: ETH_EL_ADDR]
--keystore-password string Password used to encrypt key files [env: KEYSTORE_PASSWORD]
--keystore-path string Directory where to store keys [env: KEYSTORE_PATH]
--log-format string Log output format (text or json) [env: LOG_FORMAT] (default "text")
--log-level string Log output level [env: LOG_LEVEL] (default "info")
--operators-registry-addr string Address of the Operators Registry contract [env: OPERATORS_REGISTRY_ADDR]
--oracle-addr string Address of the Oracle contract [env: ORACLE_ADDR]
--redeem-manager-addr string Address of the RedeemManager contract [env: REDEEM_MANAGER_ADDR]
--river-addr string Address of the River contract [env: RIVER_ADDR]
--tlc-addr string Address of the TLC contract [env: TLC_ADDR]
--withdraw-addr string Address of the Withdraw contract [env: WITHDRAW_ADDR]
--wls-eth-addr string Address of the WlsEth contract [env: WLSETH_ADDR]Last updated