This guide is intended for Node Operators participating in the Liquid Collective protocol.
Node Operators are validator infrastructure providers responsible for running validator nodes on the Ethereum consensus layer in the name of the protocol. Node Operators must have the capacity to scale to a large number of validator nodes, meet certain compliance requirements, and meet certain performance requirements, including Liquid Collective's Node Operator Performance SLAs.
Node Operators receive ETH delegation from the protocol and they are remunerated proportionally to the delegation they receive.
Node Operators interested in receiving ETH delegation should go through the following flow.
One-time protocol onboarding
Node operator generates Node Operator wallet
Node Operator address gets approved on the Node Operator registry contracts
provides various commands to facilitate Node Operators in the process of managing validator keys.
CLI is compatible with the and can be used in conjunction with it.
The recommended installation is to use the public Docker image public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0
It is also possible to build the binary from sources.
Generate (or Import) Node Operator Wallet
The Node Operator needs a wallet to submit validator keys. The wallet must be approved on the Node Operator Registry Contract.
To generate such a wallet you can use the following command:
Once generated or imported you should securely store the key file and password for later usage, as you will need it each time you need to register keys.
It is also possible to import an existing wallet:
Approve Node Operator Wallet
A Node Operator should provide administrators with:
name: As the Node Operator will be publicly listed on the Node Operator contract
address: The Node Operator wallet address previously generated
Node Operator Index
Once approved, a Node Operator will get its operator index on the Node Operators Registry. This index will never change over time.
To get your node operator index, run the below command and find your name in the returned list.
Pre-registering validators consists of submitting new validators keys to the Node Operators registry contract so they can be picked for validation and funded.
As a Node Operator, you should typically pre-register new validators keys when:
It is the first time you pre-register keys
Most of the keys that you have pre-registered have been funded
It is the responsibility of the Node Operator to make sure validator keys are available for funding.
In the case that a Node Operator has no validator keys available, it will not receive an ETH delegation from the protocol.
Generate validator keys, deposit data and signatures
A Node Operator is responsible for generating validator keys in its own infrastructure.
For each generated key, a Node Operator is also expected to generate the corresponding DepositMessage and BLS12-381 signature as per the
Node Operators are required to set withdrawal credentials to the address of Withdrawal contract.
To get withdrawal credentials you can run:
As a result of the validator key generation, a Node Operator should obtain a JSON file matching the following format:
Configure validator exec layer fee recipient address
Node Operators are required to set the fee recipient of the validators to the ELFeeRecipient contract that is responsible to flow the execution layer network rewards to the core River contract.
To get the ELFeeRecipient address you can run:
Exec layer fee recipient address and withdrawal address are not the same.
Submit validator keys
Submit validator keys consists of sending a addValidatorKeys(...) transaction to the OperatorsRegistry contract. The transactions should be signed by the Node Operator address:
To submit validator keys you can run the following command:
Once the transaction has been sent to the network and validated, the validator keys are listed on the Node Operators Registry contract and will be reviewed by the administrator.
Node Operators should be careful that validator keys can be picked, funded, and deposited to the official Ethereum Deposit Contract at any time.
Once a validator key has been funded and deposited, it enters the activation queue. When this happens, the Node Operator should make sure that the corresponding validator key gets properly deployed to its validator infrastructure so the validator is ready when validator activation occurs.
We strongly recommend Node Operators effectively monitor the Deposit Contract, in particular watching DepositEvent to never miss an activation.
Node Operators are responsible for exiting their own validators
Node Operators can use the provided Exit Daemon
Useful events:
FundedValidatorKeys
RequestedValidatorExits
The Validator Exit Daemon is an off-chain application designed to help Liquid Collective Node Operators exit validator keys with ease.
Node Operators are responsible for exiting validator keys from the Consensus Layer, allowing withdrawn funds to be used for satisfying conversions of LsETH for ETH.
LsETH holders convert LsETH for ETH through the protocol, creating a request for ETH.
The protocol regularly rebalance's ETH positions. When it needs ETH funds to fulfill redeem demands, it withdraws funds from the Consensus Layer and signals Node Operators by emitting an event requesting validator keys to be exited. The protocol is responsible for selecting the Node Operators that need to exit keys, depending on the current validator key allocation.
Upon receiving a validator exit request event, the Node Operator exits the requested number of validator keys, which results in broadcasting ValidatorExit messages to the Consensus Layer. This step is facilitated by the Validator Exit Daemon application.
Validator exit is triggered by broadcasting a ValidatorExit message signed with the validator's private key on the Consensus Layer.
We cannot rely on the Dual-Key exit design to also allow triggering the validator key exit from the withdrawal_credentials. Even if this option is preferred, it is not expected to be available on day 1 of the withdrawals.
Validator keys may be slashed, resulting in the keys undergoing the Consensus Layer slashing process and eventually leading to the withdrawal of funds after incurring penalties (this takes at least 36 days).
The Validator Exit Daemon is a long-lived application containerized in a Docker image designed to be run by Node Operators in their infrastructure.
It accesses data from both the Execution Layer and Consensus Layer connected to nodes in the Node Operator infrastructure.
It assumes that Node Operators have an existing system and process in place for proceeding with validator key exits.
The Daemon can interact with the existing exit system through one or both of the following methods:
HTTP hook: Called by the Daemon each time a validator request exit event is emitted, and a validator key should be exited. Node Operators can configure the callback to match any endpoint.
API polling: The Daemon exposes an API that Node Operators can poll to retrieve the key to exit.
The Validator Exit Daemon will provide the following information to the Node Operator:
Number of validator keys to exit
Recommended keys to exit
Diagram
Sequence diagram
A synced Execution Layer client with a JSON-RPC endpoint enabled. All implementations are supported (Geth, Erigon, Besu, etc.)
A synced Consensus Layer client with an API endpoint enabled. All implementations are supported (Prysm, Teku, Lighthouse, etc.)
The recommended installation is to use the public Docker image public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0
API
By default, the Exit daemon exposes an API route that Node Operators can use, either in conjunction with or without the HTTP hook.
This allows an operator to retrieve the recommended keys to exit at the current time.
The API meets the following requirements:
Webhook
The Exit daemon can run an HTTP hook system that sends validator exit request callbacks to the operator's existing systems when it receives a Validator Exit request event from the Consensus Layer.
The Exit daemon is stateless. If the webhook is enabled, it will send a call at each start if there is a pending exit request.
Operators can customize the webhook call by providing a template for the body, endpoint, and headers, and use the following macros to pass data related to the exit event:
total_requested_exits represents the total requested exits since the beginning of the contract. It’s a incremental-only counter emitted by the RequestedValidatorExits event.
validators_to_exit (as well as validators_to_exit_pubkeys & validators_to_exit_indexes) is an array of the validators to exit at a given time. To get them, the exit daemon fetches all the keys of an operator (via the FundedValidatorKeys event), check their status on the consensus layer to know those already exited, and then make a diff with
Example of webhook template:
Usage output:
Pre-registration of validator keys
Node Operator generates validator keys in its infrastructure with expected configuration (withdrawal credentials, execution layer fee recipient, etc.)
Node Operator submits validator keys to the Node Operators registry contract by sending a transaction using the Node Operator wallet
From this point in time, validator keys should be ready to get funded
Protocol administrator reviews and confirms added keys are valid
Validates validator keys, ensuring withdrawal credentials, deposit data, and signatures are correct
Administrator increases operator limit making validator keys eligible for funding. From this point in time, validators keys can be funded at any time
Validator keys are funded and activated
Protocol regularly picks eligible validator keys and deposits them to the official Ethereum deposit contract
Once deposited, validator keys enter the activation queue as per the standard Ethereum staking procedure
Run the Exit Daemon
Node Operators are responsible for exiting their own validators
Node Operators have the possibility of running the lceth exit daemon in order to be notified when they should exit their validators
After validator keys pass through the exit queue, funds are withdrawn to the LC Withdraw contract, and the Liquid Collective protocol takes over the ETH to satisfy the LsETH conversion requests.
Liquid Collective Node Operators use diverse infrastructure base layers (clouds, bare metal, containers/orchestration), client software (Prysm, Lighthouse, etc.), and may use or not use signers (e.g., Web3Signer). This diversity is expected to grow with staking innovations (e.g., DVT).
Liquid Collective Node Operators have (or are building) infrastructure & processes
total_requested_exitsprinting array via {{validators_to_exit_pubkeys | toJson}}
creating a quoted string from an array via {{.validators_to_exit_indexes | join "," | quote}}
computing array length via {{.validators_to_exit_indexes | len}}
etc.


env KEYSTORE_PASSWORD={password to encrypt the key file} lceth eth1keys generatedocker run \
--env "KEYSTORE_PASSWORD={password to encrypt the key file}" \
-v "keystore:/data/keystore" \
public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 eth1keys generateenv KEYSTORE_PASSWORD={password to encrypt the key file} lceth eth1keys import --priv-key {private key in hex formatdocker run \
--env "KEYSTORE_PASSWORD={password to encrypt the key file}" \
-v "keystore:/data/keystore" \
public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 eth1keys import --priv-key {private key in hex format}env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
lceth operators listdocker run \
--env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 operators listenv ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
lceth withdrawal credentialsdocker run \
--env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 withdrawal credentials[
{
"pubkey": "814dc0f55ac3fb02431668adf6f8fa1c37fb9baa5b87f5be519a373205933dfe742f3df566cba3a35b5be1940e1dffd5",
"signature": "81b24791a89f3596abebb294e13502e398d96cbdd2a70d87b7fa89cc93019080a2905bb7447e0c408f2569a17b0ce628163e8caca4ea5b69af4db458dffcca100f8fcb96edeef30072ca4322721adc5591da8aadffac8b730d78f84c3b9f3f92"
},
{
"pubkey": "88d1ac7f33780fd328bee60957b2325cfa41b3719614b662616d4525e5b478b3a81d490671526936e3ea412428c84451",
"signature": "9985f2eb6f445f5e2bae226333f4796127c01efed7f4a4a35b62719dd5470879d1377e0035c00ccaa1cf892542bb427b072113a8e690e963d3e5b590ffd0051c0680e345cd1ee9318f2559796585445c490b8e9db417e99ac55e99c7e281aced"
}
]env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
lceth el-fee-recipient addressdocker run \
--env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 el-fee-recipient address/// @notice Adds new keys for an operator
/// @dev Only callable by the administrator or the operator address
/// @param _name The name identifying the operator
/// @param _keyCount The amount of keys provided
/// @param _publicKeys Public keys of the validator, concatenated
/// @param _signatures Signatures of the validator keys, concatenated
function addValidatorKeys(
string calldata _name,
uint256 _keyCount,
bytes calldata _publicKeys,
bytes calldata _signatures
)env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
KEYSTORE_PASSWORD={password to encrypt the key file} \
lceth validators add \
--operator-idx {operator index} \
--deposit-data validator_keys.json \
--from {operator address} \
--senddocker run \
--env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
--env "KEYSTORE_PASSWORD={password to encrypt the key file}" \
-v ./keystore:/data/keystore \
-v "./validator_keys.json:/validator_keys.json" \
public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 validators add \
--operator-idx {operator index} \
--deposit-data /validator_keys.json \
--from {operator address} \
--sendenv ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
env ETH_CL_ADDR={ethereum consensus layer rpc endpoint} \
lceth exit run \
--operator-idx {operator index}docker run \
--env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
--env "ETH_CL_ADDR={ethereum consensus layer rpc endpoint}" \
public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 exit run \
--operator-idx {operator index}paths:
'/':
get:
description: 'Get validator keys to exit'
responses:
200:
description: 'Success.'
schema:
type: object
properties:
chain_id:
type: 'string'
example: '1'
total_requested_exits:
type: 'integer'
description: 'Number of validators requested to exit'
example: 2
validators_to_exit:
type: 'array'
description: 'List of recommended validators to exit'
items:
type: 'object'
properties:
pubkey:
type: 'string'
example: '0x8a9e1213459337631232447e360faf162781aee0a3df533306a02e2eae481b4db4d1313d1756da7a121da437d9964fb0'
index:
type: 'integer'
example: 473944
500:
description: 'Internal unexpected error'| Name | Type | Description |
| -------------------------- | -------- | ----------------------------------------- |
| chain_id | string | Network identifier |
| total_requested_exits | int | Number of validators requested to exit |
| validators_to_exit_indexes | int[] | List of validators pubkey to exit |
| validators_to_exit_pubkeys | string[] | List of validators index to exit |
| validators_to_exit | object[] | List of validators pubkey + index to exit |env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
env ETH_CL_ADDR={ethereum consensus layer rpc endpoint} \
lceth exit run \
--operator-idx {operator index} \
--webhook \
--webhook-method="POST" \
--webhook-endpoint="http://127.0.0.1:9080/call?chain_id={{.chain_id}}&count={{.total_requested_exits}}&validators={{.validators_to_exit_indexes|join ","}}" \
--webhook-template-body '
{
"chain_id": "{{.chain_id}}",
"total_requested_exits": {{.total_requested_exits}},
"current_requested_exits": {{.validators_to_exit_pubkeys | len}},
"indexes": {{.validators_to_exit_indexes | join "," | quote}},
"pubkeys": {{.validators_to_exit_pubkeys | toJson}},
"custom": {{env CUSTOM_ENV_VAR}}'
}
' \
--webhook-template-header="Chain-Id={{.chain_id}}" \
--webhook-template-header="Content-Type=application/json"docker run \
--env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
--env "ETH_CL_ADDR={ethereum consensus layer rpc endpoint}" \
public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 exit run \
--operator-idx {operator index} \
--webhook \
--webhook-method="POST" \
--webhook-endpoint="http://127.0.0.1:9080/call?chain_id={{.chain_id}}&count={{.total_requested_exits}}&validators={{.validators_to_exit_indexes|join ","}}" \
--webhook-template-body '
{
"chain_id": "{{.chain_id}}",
"total_requested_exits": {{.total_requested_exits}},
"indexes": {{.validators_to_exit_indexes | join "," | quote}},
"pubkeys": {{.validators_to_exit_pubkeys | toJson}},
"custom": {{env CUSTOM_ENV_VAR}}'
}
' \
--webhook-template-header="Chain-Id={{.chain_id}}" \
--webhook-template-header="Content-Type=application/json"Run exit daemon
Usage:
lceth exit run [flags]
Flags:
--eth-cl-addr string Address of the Ethereum consensus layer node to connect to [env: ETH_CL_ADDR]
--operator-idx int Index of the operator running the daemon [env: OPERATOR_INDEX]
--webhook enable 'webhook' [env: WEBHOOK]
--webhook-endpoint string Go template describing the endpoint that the webhook will call [env: WEBHOOK_ENDPOINT]
--webhook-method string HTTP method to query the webhook's endpoint [env: WEBHOOK_METHOD]
--webhook-template-header stringToString Go template describing the body to be sent to the webhook's endpoint [env: WEBHOOK_TEMPLATE_HEADER] (default [])
--webhook-template-body string Go templates describing the headers to be sent to the webhook [env: WEBHOOK_TEMPLATE_BODY]
--webhook-retry int Retry limit in case of webhook-endpoint returns error with status code 5xx [env: WEBHOOK_RETRY]
--webhook-timeout duration Delay to wait before abort request sent to webhook-endpoint [env: WEBHOOK_TIMEOUT]
--rpc-call-retry int How many time should we retry failed rpc calls [env: RPC_CALL_RETRY] (default 3)
--block-log-fetch-step int How many blocks are fetched at once when fetching logs [env: BLOCK_LOG_FETCH_STEP] (default 100000)
--max-workers int How many goroutines to use on threaded tasks [env: MAX_WORKERS] (default 10)
--batch-fetch-validators Whether to batch when querying the consensus layer to get validators' data (recommended if timeout issues arise) [env: BATCH_FETCH_VALIDATORS]
--max-get-validators int Max size of the batch when querying the consensus layer to get validators' data, only useful if using --batch-fetch-validators [env: MAX_GET_VALIDATORS] (default 1000)
--loop-sleep-time duration How often should the exit-daemon run [env: LOOP_SLEEP_TIME] (default 5m0s)
-h, --help help for run
Global Flags:
--allowlist-addr string Address of the Allowlist contract [env: ALLOWLIST_ADDR]
--deployment-block uint Deployment block of the contracts [env: DEPLOYMENT_BLOCK]
--el-fee-recipient-addr string Address of the Execution Layer fee recipient contract [env: EL_FEE_RECIPIENT_ADDR]
--eth-el-addr string JSON-RPC address of the Ethereum execution layer node to connect to [env: ETH_EL_ADDR]
--keystore-password string Password used to encrypt key files [env: KEYSTORE_PASSWORD]
--keystore-path string Directory where to store keys [env: KEYSTORE_PATH]
--log-format string Log output format (text or json) [env: LOG_FORMAT] (default "text")
--log-level string Log output level [env: LOG_LEVEL] (default "info")
--operators-registry-addr string Address of the Operators Registry contract [env: OPERATORS_REGISTRY_ADDR]
--oracle-addr string Address of the Oracle contract [env: ORACLE_ADDR]
--redeem-manager-addr string Address of the RedeemManager contract [env: REDEEM_MANAGER_ADDR]
--river-addr string Address of the River contract [env: RIVER_ADDR]
--tlc-addr string Address of the TLC contract [env: TLC_ADDR]
--withdraw-addr string Address of the Withdraw contract [env: WITHDRAW_ADDR]
--wls-eth-addr string Address of the WlsEth contract [env: WLSETH_ADDR]