All pages
Powered by GitBook
1 of 1

Loading...

Operate Validator Nodes

This guide is intended for Node Operators participating in the Liquid Collective protocol.

Operate Validator Nodes

Node Operators are validator infrastructure providers responsible for running validator nodes on the Ethereum consensus layer in the name of the protocol. Node Operators must have the capacity to scale to a large number of validator nodes, meet certain compliance requirements, and meet certain performance requirements, including Liquid Collective's Node Operator Performance SLAs.

Node Operators receive ETH delegation from the protocol and they are remunerated proportionally to the delegation they receive.

Operate Validator Nodes

Node Operator Procedure

Node Operators interested in receiving ETH delegation should go through the following flow.

  1. One-time protocol onboarding

    1. Node operator generates Node Operator wallet

    2. Node Operator address gets approved on the Node Operator registry contracts

CLI

provides various commands to facilitate Node Operators in the process of managing validator keys.

CLI is compatible with the and can be used in conjunction with it.

Installation

The recommended installation is to use the public Docker image public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0

It is also possible to build the binary from sources.

Guidelines

Protocol Onboarding

Generate (or Import) Node Operator Wallet

The Node Operator needs a wallet to submit validator keys. The wallet must be approved on the Node Operator Registry Contract.

To generate such a wallet you can use the following command:

Once generated or imported you should securely store the key file and password for later usage, as you will need it each time you need to register keys.

It is also possible to import an existing wallet:

Approve Node Operator Wallet

A Node Operator should provide administrators with:

  • name: As the Node Operator will be publicly listed on the Node Operator contract

  • address: The Node Operator wallet address previously generated

Those values can be updated at any time after first approval.

Node Operator Index

Once approved, a Node Operator will get its operator index on the Node Operators Registry. This index will never change over time.

To get your node operator index, run the below command and find your name in the returned list.

Ongoing Operations: Pre-Registration of Validator Keys

Pre-registering validators consists of submitting new validators keys to the Node Operators registry contract so they can be picked for validation and funded.

As a Node Operator, you should typically pre-register new validators keys when:

  • It is the first time you pre-register keys

  • Most of the keys that you have pre-registered have been funded

It is the responsibility of the Node Operator to make sure validator keys are available for funding.

In the case that a Node Operator has no validator keys available, it will not receive an ETH delegation from the protocol.

Generate validator keys, deposit data and signatures

A Node Operator is responsible for generating validator keys in its own infrastructure.

For each generated key, a Node Operator is also expected to generate the corresponding DepositMessage and BLS12-381 signature as per the

Node Operators are required to set withdrawal credentials to the address of Withdrawal contract.

Per , once a Type 1 (0x01) withdrawal address is set those withdrawal credentials cannot be changed.

To get withdrawal credentials you can run:

As a result of the validator key generation, a Node Operator should obtain a JSON file matching the following format:

It is possible to use the to generate keys, which produces a file in the above format.

Configure validator exec layer fee recipient address

Node Operators are required to set the fee recipient of the validators to the ELFeeRecipient contract that is responsible to flow the execution layer network rewards to the core River contract.

To get the ELFeeRecipient address you can run:

Exec layer fee recipient address and withdrawal address are not the same.

Submit validator keys

Submit validator keys consists of sending a addValidatorKeys(...) transaction to the OperatorsRegistry contract. The transactions should be signed by the Node Operator address:

To submit validator keys you can run the following command:

The container runs with 65532:65532 and the keystore directory should be created and then chown'd to the user.

Once the transaction has been sent to the network and validated, the validator keys are listed on the Node Operators Registry contract and will be reviewed by the administrator.

Deposit and Node Activation

Node Operators should be careful that validator keys can be picked, funded, and deposited to the official Ethereum Deposit Contract at any time.

Once a validator key has been funded and deposited, it enters the activation queue. When this happens, the Node Operator should make sure that the corresponding validator key gets properly deployed to its validator infrastructure so the validator is ready when validator activation occurs.

We strongly recommend Node Operators effectively monitor the Deposit Contract, in particular watching DepositEvent to never miss an activation.

Watching the exit requests

  • Node Operators are responsible for exiting their own validators

  • Node Operators can use the provided Exit Daemon

Useful events:

  • FundedValidatorKeys

  • RequestedValidatorExits

Exit Daemon

Description

The Validator Exit Daemon is an off-chain application designed to help Liquid Collective Node Operators exit validator keys with ease.

Node Operators are responsible for exiting validator keys from the Consensus Layer, allowing withdrawn funds to be used for satisfying conversions of LsETH for ETH.

Flow

  • LsETH holders convert LsETH for ETH through the protocol, creating a request for ETH.

  • The protocol regularly rebalance's ETH positions. When it needs ETH funds to fulfill redeem demands, it withdraws funds from the Consensus Layer and signals Node Operators by emitting an event requesting validator keys to be exited. The protocol is responsible for selecting the Node Operators that need to exit keys, depending on the current validator key allocation.

  • Upon receiving a validator exit request event, the Node Operator exits the requested number of validator keys, which results in broadcasting ValidatorExit messages to the Consensus Layer. This step is facilitated by the Validator Exit Daemon application.

Technical considerations & assumptions:

  • Validator exit is triggered by broadcasting a ValidatorExit message signed with the validator's private key on the Consensus Layer.

  • We cannot rely on the Dual-Key exit design to also allow triggering the validator key exit from the withdrawal_credentials. Even if this option is preferred, it is not expected to be available on day 1 of the withdrawals.

  • Validator keys may be slashed, resulting in the keys undergoing the Consensus Layer slashing process and eventually leading to the withdrawal of funds after incurring penalties (this takes at least 36 days).

Architecture

The Validator Exit Daemon is a long-lived application containerized in a Docker image designed to be run by Node Operators in their infrastructure.

It accesses data from both the Execution Layer and Consensus Layer connected to nodes in the Node Operator infrastructure.

It assumes that Node Operators have an existing system and process in place for proceeding with validator key exits.

The Daemon can interact with the existing exit system through one or both of the following methods:

  • HTTP hook: Called by the Daemon each time a validator request exit event is emitted, and a validator key should be exited. Node Operators can configure the callback to match any endpoint.

  • API polling: The Daemon exposes an API that Node Operators can poll to retrieve the key to exit.

The Validator Exit Daemon will provide the following information to the Node Operator:

  • Number of validator keys to exit

  • Recommended keys to exit

Diagram

Sequence diagram

Dependencies

  • A synced Execution Layer client with a JSON-RPC endpoint enabled. All implementations are supported (Geth, Erigon, Besu, etc.)

  • A synced Consensus Layer client with an API endpoint enabled. All implementations are supported (Prysm, Teku, Lighthouse, etc.)

Installation

The recommended installation is to use the public Docker image public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0

For those running kubernetes and helm you can use this image:public.ecr.aws/alluvial/helm-charts/lceth:1.3.0

Usage

API

By default, the Exit daemon exposes an API route that Node Operators can use, either in conjunction with or without the HTTP hook.

This allows an operator to retrieve the recommended keys to exit at the current time.

Operator index could be retrieved with lceth operators list

The API meets the following requirements:

Webhook

The Exit daemon can run an HTTP hook system that sends validator exit request callbacks to the operator's existing systems when it receives a Validator Exit request event from the Consensus Layer.

The Exit daemon is stateless. If the webhook is enabled, it will send a call at each start if there is a pending exit request.

Operators can customize the webhook call by providing a template for the body, endpoint, and headers, and use the following macros to pass data related to the exit event:

  • total_requested_exits represents the total requested exits since the beginning of the contract. It’s a incremental-only counter emitted by the RequestedValidatorExits event.

  • validators_to_exit (as well as validators_to_exit_pubkeys & validators_to_exit_indexes) is an array of the validators to exit at a given time. To get them, the exit daemon fetches all the keys of an operator (via the FundedValidatorKeys event), check their status on the consensus layer to know those already exited, and then make a diff with

The webhook templating imports the . Which means additional functions are available to format the output.

For instance:

  • using environment variables via {{env "ENV_VAR_NAME"}}

Example of webhook template:

Usage output:

On-going operations
  1. Pre-registration of validator keys

    • Node Operator generates validator keys in its infrastructure with expected configuration (withdrawal credentials, execution layer fee recipient, etc.)

    • Node Operator submits validator keys to the Node Operators registry contract by sending a transaction using the Node Operator wallet

    • From this point in time, validator keys should be ready to get funded

  2. Protocol administrator reviews and confirms added keys are valid

    • Validates validator keys, ensuring withdrawal credentials, deposit data, and signatures are correct

    • Administrator increases operator limit making validator keys eligible for funding. From this point in time, validators keys can be funded at any time

  3. Validator keys are funded and activated

    • Protocol regularly picks eligible validator keys and deposits them to the official Ethereum deposit contract

    • Once deposited, validator keys enter the activation queue as per the standard Ethereum staking procedure

  4. Run the Exit Daemon

    • Node Operators are responsible for exiting their own validators

    • Node Operators have the possibility of running the lceth exit daemon in order to be notified when they should exit their validators

After validator keys pass through the exit queue, funds are withdrawn to the LC Withdraw contract, and the Liquid Collective protocol takes over the ETH to satisfy the LsETH conversion requests.

Existing Liquid Collective validator keys have all been set with a 0x1 prefix and point to the Withdraw contract.
  • Liquid Collective Node Operators use diverse infrastructure base layers (clouds, bare metal, containers/orchestration), client software (Prysm, Lighthouse, etc.), and may use or not use signers (e.g., Web3Signer). This diversity is expected to grow with staking innovations (e.g., DVT).

  • Liquid Collective Node Operators have (or are building) infrastructure & processes

  • total_requested_exits
    to know how many keys are remaining to exit.

    printing array via {{validators_to_exit_pubkeys | toJson}}

  • creating a quoted string from an array via {{.validators_to_exit_indexes | join "," | quote}}

  • computing array length via {{.validators_to_exit_indexes | len}}

  • etc.

  • CLI
    Ethereum deposit CLI
    Ethereum 2.0 specs
    the Ethereum protocol
    Ethereum deposit CLI
    Golang Sprig library
    Validator Exit Daemon
    Validator Exit Daemon Sequence Diagram
    env KEYSTORE_PASSWORD={password to encrypt the key file} lceth eth1keys generate
    docker run \
      --env "KEYSTORE_PASSWORD={password to encrypt the key file}" \
      -v "keystore:/data/keystore" \
      public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 eth1keys generate
    env KEYSTORE_PASSWORD={password to encrypt the key file} lceth eth1keys import --priv-key {private key in hex format
    docker run \
      --env "KEYSTORE_PASSWORD={password to encrypt the key file}" \
      -v "keystore:/data/keystore" \
      public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 eth1keys import --priv-key {private key in hex format}
    env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
     lceth operators list
    docker run \
      --env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
      public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 operators list
    env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
     lceth withdrawal credentials
    docker run \
      --env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
      public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 withdrawal credentials
    validator_keys.json
    [
      {
        "pubkey": "814dc0f55ac3fb02431668adf6f8fa1c37fb9baa5b87f5be519a373205933dfe742f3df566cba3a35b5be1940e1dffd5",
        "signature": "81b24791a89f3596abebb294e13502e398d96cbdd2a70d87b7fa89cc93019080a2905bb7447e0c408f2569a17b0ce628163e8caca4ea5b69af4db458dffcca100f8fcb96edeef30072ca4322721adc5591da8aadffac8b730d78f84c3b9f3f92"
      },
      {
        "pubkey": "88d1ac7f33780fd328bee60957b2325cfa41b3719614b662616d4525e5b478b3a81d490671526936e3ea412428c84451",
        "signature": "9985f2eb6f445f5e2bae226333f4796127c01efed7f4a4a35b62719dd5470879d1377e0035c00ccaa1cf892542bb427b072113a8e690e963d3e5b590ffd0051c0680e345cd1ee9318f2559796585445c490b8e9db417e99ac55e99c7e281aced"
      }
    ]
    env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
     lceth el-fee-recipient address
    docker run \
      --env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
      public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 el-fee-recipient address
    index.sol
    /// @notice Adds new keys for an operator
    /// @dev Only callable by the administrator or the operator address
    /// @param _name The name identifying the operator
    /// @param _keyCount The amount of keys provided
    /// @param _publicKeys Public keys of the validator, concatenated
    /// @param _signatures Signatures of the validator keys, concatenated
    function addValidatorKeys(
        string calldata _name,
        uint256 _keyCount,
        bytes calldata _publicKeys,
        bytes calldata _signatures
    )
    env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
    KEYSTORE_PASSWORD={password to encrypt the key file} \
    lceth validators add \
    --operator-idx {operator index} \
    --deposit-data validator_keys.json \
    --from {operator address} \
    --send
    docker run \
      --env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
      --env "KEYSTORE_PASSWORD={password to encrypt the key file}" \
      -v ./keystore:/data/keystore \
      -v "./validator_keys.json:/validator_keys.json" \
      public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 validators add \
        --operator-idx {operator index} \
        --deposit-data /validator_keys.json \
        --from {operator address} \
        --send
    env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
    env ETH_CL_ADDR={ethereum consensus layer rpc endpoint} \
    lceth exit run \
    --operator-idx {operator index}
    docker run \
      --env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
      --env "ETH_CL_ADDR={ethereum consensus layer rpc endpoint}" \
      public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 exit run \
        --operator-idx {operator index}
    paths:
      '/':
        get:
          description: 'Get validator keys to exit'
          responses:
            200:
              description: 'Success.'
              schema:
                type: object
                properties:
                  chain_id:
                    type: 'string'
                    example: '1'
                  total_requested_exits:
                    type: 'integer'
                    description: 'Number of validators requested to exit'
                    example: 2
                  validators_to_exit:
                    type: 'array'
                    description: 'List of recommended validators to exit'
                    items:
                      type: 'object'
                      properties:
                        pubkey:
                          type: 'string'
                          example: '0x8a9e1213459337631232447e360faf162781aee0a3df533306a02e2eae481b4db4d1313d1756da7a121da437d9964fb0'
                        index:
                          type: 'integer'
                          example: 473944
            500:
              description: 'Internal unexpected error'
    | Name                       | Type     | Description                               |
    | -------------------------- | -------- | ----------------------------------------- |
    | chain_id                   | string   | Network identifier                        |
    | total_requested_exits      | int      | Number of validators requested to exit    |
    | validators_to_exit_indexes | int[]    | List of validators pubkey to exit         |
    | validators_to_exit_pubkeys | string[] | List of validators index to exit          |
    | validators_to_exit         | object[] | List of validators pubkey + index to exit |
    env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
    env ETH_CL_ADDR={ethereum consensus layer rpc endpoint} \
    lceth exit run \
    --operator-idx {operator index} \
    --webhook \
    --webhook-method="POST" \
    --webhook-endpoint="http://127.0.0.1:9080/call?chain_id={{.chain_id}}&count={{.total_requested_exits}}&validators={{.validators_to_exit_indexes|join ","}}" \
    --webhook-template-body '
      {
        "chain_id": "{{.chain_id}}",
        "total_requested_exits": {{.total_requested_exits}},
        "current_requested_exits": {{.validators_to_exit_pubkeys | len}},
        "indexes": {{.validators_to_exit_indexes | join "," | quote}},
        "pubkeys": {{.validators_to_exit_pubkeys | toJson}},
        "custom": {{env CUSTOM_ENV_VAR}}'
      }
    ' \
    --webhook-template-header="Chain-Id={{.chain_id}}" \
    --webhook-template-header="Content-Type=application/json"
    docker run \
      --env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
      --env "ETH_CL_ADDR={ethereum consensus layer rpc endpoint}" \
      public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 exit run \
        --operator-idx {operator index} \
        --webhook \
        --webhook-method="POST" \
        --webhook-endpoint="http://127.0.0.1:9080/call?chain_id={{.chain_id}}&count={{.total_requested_exits}}&validators={{.validators_to_exit_indexes|join ","}}" \
        --webhook-template-body '
          {
            "chain_id": "{{.chain_id}}",
            "total_requested_exits": {{.total_requested_exits}},
            "indexes": {{.validators_to_exit_indexes | join "," | quote}},
            "pubkeys": {{.validators_to_exit_pubkeys | toJson}},
            "custom": {{env CUSTOM_ENV_VAR}}'
          }
        ' \
        --webhook-template-header="Chain-Id={{.chain_id}}" \
        --webhook-template-header="Content-Type=application/json"
    Run exit daemon
    
    Usage:
      lceth exit run [flags]
    
    Flags:
          --eth-cl-addr string                       Address of the Ethereum consensus layer node to connect to [env: ETH_CL_ADDR]
          --operator-idx int                         Index of the operator running the daemon [env: OPERATOR_INDEX]
          --webhook                                  enable 'webhook' [env: WEBHOOK]
          --webhook-endpoint string                  Go template describing the endpoint that the webhook will call [env: WEBHOOK_ENDPOINT]
          --webhook-method string                    HTTP method to query the webhook's endpoint [env: WEBHOOK_METHOD]
          --webhook-template-header stringToString   Go template describing the body to be sent to the webhook's endpoint [env: WEBHOOK_TEMPLATE_HEADER] (default [])
          --webhook-template-body string             Go templates describing the headers to be sent to the webhook [env: WEBHOOK_TEMPLATE_BODY]
          --webhook-retry int                        Retry limit in case of webhook-endpoint returns error with status code 5xx [env: WEBHOOK_RETRY]
          --webhook-timeout duration                 Delay to wait before abort request sent to webhook-endpoint [env: WEBHOOK_TIMEOUT]
          --rpc-call-retry int                       How many time should we retry failed rpc calls [env: RPC_CALL_RETRY] (default 3)
          --block-log-fetch-step int                 How many blocks are fetched at once when fetching logs [env: BLOCK_LOG_FETCH_STEP] (default 100000)
          --max-workers int                          How many goroutines to use on threaded tasks [env: MAX_WORKERS] (default 10)
          --batch-fetch-validators                   Whether to batch when querying the consensus layer to get validators' data (recommended if timeout issues arise) [env: BATCH_FETCH_VALIDATORS]
          --max-get-validators int                   Max size of the batch when querying the consensus layer to get validators' data, only useful if using --batch-fetch-validators [env: MAX_GET_VALIDATORS] (default 1000)
          --loop-sleep-time duration                 How often should the exit-daemon run [env: LOOP_SLEEP_TIME] (default 5m0s)
      -h, --help                                     help for run
    
    Global Flags:
          --allowlist-addr string            Address of the Allowlist contract [env: ALLOWLIST_ADDR]
          --deployment-block uint            Deployment block of the contracts [env: DEPLOYMENT_BLOCK]
          --el-fee-recipient-addr string     Address of the Execution Layer fee recipient contract [env: EL_FEE_RECIPIENT_ADDR]
          --eth-el-addr string               JSON-RPC address of the Ethereum execution layer node to connect to [env: ETH_EL_ADDR]
          --keystore-password string         Password used to encrypt key files [env: KEYSTORE_PASSWORD]
          --keystore-path string             Directory where to store keys [env: KEYSTORE_PATH]
          --log-format string                Log output format (text or json) [env: LOG_FORMAT] (default "text")
          --log-level string                 Log output level [env: LOG_LEVEL] (default "info")
          --operators-registry-addr string   Address of the Operators Registry contract [env: OPERATORS_REGISTRY_ADDR]
          --oracle-addr string               Address of the Oracle contract [env: ORACLE_ADDR]
          --redeem-manager-addr string       Address of the RedeemManager contract [env: REDEEM_MANAGER_ADDR]
          --river-addr string                Address of the River contract [env: RIVER_ADDR]
          --tlc-addr string                  Address of the TLC contract [env: TLC_ADDR]
          --withdraw-addr string             Address of the Withdraw contract [env: WITHDRAW_ADDR]
          --wls-eth-addr string              Address of the WlsEth contract [env: WLSETH_ADDR]