# Operate Validator Nodes

## Operate Validator Nodes

Node Operators are validator infrastructure providers responsible for running validator nodes on the Ethereum consensus layer in the name of the protocol. Node Operators must have the capacity to scale to a large number of validator nodes, meet certain compliance requirements, and meet certain performance requirements, including Liquid Collective's [Node Operator Performance SLAs](https://liquidcollective.io/operator-performance-slas/).

Node Operators receive ETH delegation from the protocol and they are remunerated proportionally to the delegation they receive.

## Operate Validator Nodes

### Node Operator Procedure

Node Operators interested in receiving ETH delegation should go through the following flow.

1. **One-time protocol onboarding**
   1. Node operator generates Node Operator wallet
   2. Node Operator address gets approved on the Node Operator registry contracts
2. **On-going operations**
   1. **Pre-registration of validator keys**
      * Node Operator generates validator keys in its infrastructure with expected configuration (withdrawal credentials, execution layer fee recipient, etc.)
      * Node Operator submits validator keys to the Node Operators registry contract by sending a transaction using the Node Operator wallet
      * From this point in time, validator keys should be ready to get funded
   2. **Protocol administrator reviews and confirms added keys are valid**
      * Validates validator keys, ensuring withdrawal credentials, deposit data, and signatures are correct
      * Administrator increases operator limit making validator keys eligible for funding. From this point in time, validators keys can be funded at any time
   3. **Validator keys are funded and activated**
      * Protocol regularly picks eligible validator keys and deposits them to the official Ethereum deposit contract
      * Once deposited, validator keys enter the activation queue as per the standard Ethereum staking procedure
   4. **Run the Exit Daemon**
      * Node Operators are responsible for exiting their own validators
      * Node Operators have the possibility of running the `lceth` exit daemon in order to be notified when they should exit their validators

### CLI

[CLI](https://docs.liquidcollective.io/eth/technical-reference/cli) provides various commands to facilitate Node Operators in the process of managing validator keys.

CLI is compatible with the [Ethereum deposit CLI](https://github.com/ethereum/staking-deposit-cli) and can be used in conjunction with it.

#### Installation

The recommended installation is to use the public Docker image `public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0`

It is also possible to build the binary from sources.

### Guidelines

#### Protocol Onboarding

**Generate (or Import) Node Operator Wallet**

The Node Operator needs a wallet to submit validator keys. The wallet must be approved on the Node Operator Registry Contract.

To generate such a wallet you can use the following command:

{% tabs %}
{% tab title="CLI" %}

```shell
env KEYSTORE_PASSWORD={password to encrypt the key file} lceth eth1keys generate
```

{% endtab %}

{% tab title="Docker" %}

```shell
docker run \
  --env "KEYSTORE_PASSWORD={password to encrypt the key file}" \
  -v "keystore:/data/keystore" \
  public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 eth1keys generate
```

{% endtab %}
{% endtabs %}

{% hint style="danger" %}
Once generated or imported you should securely store the key file and password for later usage, as you will need it each time you need to register keys.
{% endhint %}

It is also possible to import an existing wallet:

{% tabs %}
{% tab title="CLI" %}

```shell
env KEYSTORE_PASSWORD={password to encrypt the key file} lceth eth1keys import --priv-key {private key in hex format
```

{% endtab %}

{% tab title="Docker" %}

```shell
docker run \
  --env "KEYSTORE_PASSWORD={password to encrypt the key file}" \
  -v "keystore:/data/keystore" \
  public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 eth1keys import --priv-key {private key in hex format}
```

{% endtab %}
{% endtabs %}

**Approve Node Operator Wallet**

A Node Operator should provide administrators with:

* **name**: As the Node Operator will be publicly listed on the Node Operator contract
* **address**: The Node Operator wallet address previously generated

{% hint style="info" %}
Those values can be updated at any time after first approval.
{% endhint %}

**Node Operator Index**

Once approved, a Node Operator will get its operator index on the Node Operators Registry. This index will never change over time.

To get your node operator index, run the below command and find your name in the returned list.

{% tabs %}
{% tab title="CLI" %}

```shell
env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
 lceth operators list
```

{% endtab %}

{% tab title="Docker" %}

```shell
docker run \
  --env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
  public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 operators list
```

{% endtab %}
{% endtabs %}

#### **Ongoing Operations: Pre-Registration of Validator Keys**

Pre-registering validators consists of submitting new validators keys to the Node Operators registry contract so they can be picked for validation and funded.

As a Node Operator, you should typically pre-register new validators keys when:

* It is the first time you pre-register keys
* Most of the keys that you have pre-registered have been funded

{% hint style="warning" %}
It is the responsibility of the Node Operator to make sure validator keys are available for funding.

In the case that a Node Operator has no validator keys available, it will not receive an ETH delegation from the protocol.
{% endhint %}

**Generate validator keys, deposit data and signatures**

A Node Operator is responsible for generating validator keys in its own infrastructure.

For each generated key, a Node Operator is also expected to generate the corresponding *DepositMessage* and BLS12-381 signature as per the [Ethereum 2.0 specs](https://github.com/ethereum/annotated-spec/blob/master/phase0/beacon-chain.md#depositmessage)

Node Operators are required to set withdrawal credentials to the address of Withdrawal contract.

{% hint style="info" %}
Per [the Ethereum protocol](https://launchpad.ethereum.org/en/faq#withdrawal-credentials), once a Type 1 (0x01) withdrawal address is set those withdrawal credentials cannot be changed.
{% endhint %}

To get withdrawal credentials you can run:

{% tabs %}
{% tab title="CLI" %}

```shell
env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
 lceth withdrawal credentials
```

{% endtab %}

{% tab title="Docker" %}

```shell
docker run \
  --env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
  public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 withdrawal credentials
```

{% endtab %}
{% endtabs %}

As a result of the validator key generation, a Node Operator should obtain a JSON file matching the following format:

{% code title="validator\_keys.json" overflow="wrap" lineNumbers="true" %}

```json
[
  {
    "pubkey": "814dc0f55ac3fb02431668adf6f8fa1c37fb9baa5b87f5be519a373205933dfe742f3df566cba3a35b5be1940e1dffd5",
    "signature": "81b24791a89f3596abebb294e13502e398d96cbdd2a70d87b7fa89cc93019080a2905bb7447e0c408f2569a17b0ce628163e8caca4ea5b69af4db458dffcca100f8fcb96edeef30072ca4322721adc5591da8aadffac8b730d78f84c3b9f3f92"
  },
  {
    "pubkey": "88d1ac7f33780fd328bee60957b2325cfa41b3719614b662616d4525e5b478b3a81d490671526936e3ea412428c84451",
    "signature": "9985f2eb6f445f5e2bae226333f4796127c01efed7f4a4a35b62719dd5470879d1377e0035c00ccaa1cf892542bb427b072113a8e690e963d3e5b590ffd0051c0680e345cd1ee9318f2559796585445c490b8e9db417e99ac55e99c7e281aced"
  }
]
```

{% endcode %}

{% hint style="info" %}
It is possible to use the [Ethereum deposit CLI](https://github.com/ethereum/staking-deposit-cli) to generate keys, which produces a file in the above format.
{% endhint %}

***Configure validator exec layer fee recipient address***

Node Operators are required to set the fee recipient of the validators to the ELFeeRecipient contract that is responsible to flow the execution layer network rewards to the core River contract.

To get the ELFeeRecipient address you can run:

{% tabs %}
{% tab title="CLI" %}

```shell
env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
 lceth el-fee-recipient address
```

{% endtab %}

{% tab title="Docker" %}

```shell
docker run \
  --env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
  public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 el-fee-recipient address
```

{% endtab %}
{% endtabs %}

{% hint style="danger" %}
Exec layer fee recipient address and withdrawal address are not the same.
{% endhint %}

***Submit validator keys***

Submit validator keys consists of sending a `addValidatorKeys(...)` transaction to the OperatorsRegistry contract. The transactions should be signed by the Node Operator address:

{% code title="index.sol" overflow="wrap" lineNumbers="true" %}

```solidity
/// @notice Adds new keys for an operator
/// @dev Only callable by the administrator or the operator address
/// @param _name The name identifying the operator
/// @param _keyCount The amount of keys provided
/// @param _publicKeys Public keys of the validator, concatenated
/// @param _signatures Signatures of the validator keys, concatenated
function addValidatorKeys(
    string calldata _name,
    uint256 _keyCount,
    bytes calldata _publicKeys,
    bytes calldata _signatures
)
```

{% endcode %}

To submit validator keys you can run the following command:

{% tabs %}
{% tab title="CLI" %}

```shell
env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
KEYSTORE_PASSWORD={password to encrypt the key file} \
lceth validators add \
--operator-idx {operator index} \
--deposit-data validator_keys.json \
--from {operator address} \
--send
```

{% endtab %}

{% tab title="undefined" %}
{% hint style="info" %}
The container runs with 65532:65532 and the keystore directory should be created and then chown'd to the user.
{% endhint %}
{% endtab %}

{% tab title="Docker" %}

```shell
docker run \
  --env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
  --env "KEYSTORE_PASSWORD={password to encrypt the key file}" \
  -v ./keystore:/data/keystore \
  -v "./validator_keys.json:/validator_keys.json" \
  public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 validators add \
    --operator-idx {operator index} \
    --deposit-data /validator_keys.json \
    --from {operator address} \
    --send
```

{% endtab %}
{% endtabs %}

Once the transaction has been sent to the network and validated, the validator keys are listed on the Node Operators Registry contract and will be reviewed by the administrator.

#### Deposit and Node Activation

Node Operators should be careful that validator keys can be picked, funded, and deposited to the official Ethereum *Deposit Contract* at any time.

Once a validator key has been funded and deposited, it enters the activation queue. When this happens, the Node Operator should make sure that the corresponding validator key gets properly deployed to its validator infrastructure so the validator is ready when validator activation occurs.

We strongly recommend Node Operators effectively monitor the *Deposit Contract,* in particular watching *DepositEvent* to never miss an activation.

#### Watching the exit requests

* Node Operators are responsible for exiting their own validators
* Node Operators can use the provided Exit Daemon

Useful events:

* FundedValidatorKeys
* RequestedValidatorExits

### Exit Daemon

#### Description

The Validator Exit Daemon is an off-chain application designed to help Liquid Collective Node Operators exit validator keys with ease.

Node Operators are responsible for exiting validator keys from the Consensus Layer, allowing withdrawn funds to be used for satisfying conversions of LsETH for ETH.

#### Flow

* LsETH holders convert LsETH for ETH through the protocol, creating a request for ETH.
* The protocol regularly rebalance's ETH positions. When it needs ETH funds to fulfill redeem demands, it withdraws funds from the Consensus Layer and signals Node Operators by emitting an event requesting validator keys to be exited. The protocol is responsible for selecting the Node Operators that need to exit keys, depending on the current validator key allocation.
* Upon receiving a validator exit request event, the Node Operator exits the requested number of validator keys, which results in broadcasting ValidatorExit messages to the Consensus Layer. This step is facilitated by the Validator Exit Daemon application.
* After validator keys pass through the exit queue, funds are withdrawn to the LC Withdraw contract, and the Liquid Collective protocol takes over the ETH to satisfy the LsETH conversion requests.

#### Technical considerations & assumptions:

* Validator exit is triggered by broadcasting a ValidatorExit message signed with the validator's private key on the Consensus Layer.
* We cannot rely on the Dual-Key exit design to also allow triggering the validator key exit from the withdrawal\_credentials. Even if this option is preferred, it is not expected to be available on day 1 of the withdrawals.
* Validator keys may be slashed, resulting in the keys undergoing the Consensus Layer slashing process and eventually leading to the withdrawal of funds after incurring penalties (this takes at least 36 days).
* Existing Liquid Collective validator keys have all been set with a 0x1 prefix and point to the Withdraw contract.
* Liquid Collective Node Operators use diverse infrastructure base layers (clouds, bare metal, containers/orchestration), client software (Prysm, Lighthouse, etc.), and may use or not use signers (e.g., Web3Signer). This diversity is expected to grow with staking innovations (e.g., DVT).
* Liquid Collective Node Operators have (or are building) infrastructure & processes

#### Architecture

The Validator Exit Daemon is a long-lived application containerized in a Docker image designed to be run by Node Operators in their infrastructure.

It accesses data from both the Execution Layer and Consensus Layer connected to nodes in the Node Operator infrastructure.

It assumes that Node Operators have an existing system and process in place for proceeding with validator key exits.

The Daemon can interact with the existing exit system through one or both of the following methods:

* **HTTP hook**: Called by the Daemon each time a validator request exit event is emitted, and a validator key should be exited. Node Operators can configure the callback to match any endpoint.
* **API polling**: The Daemon exposes an API that Node Operators can poll to retrieve the key to exit.

The Validator Exit Daemon will provide the following information to the Node Operator:

* Number of validator keys to exit
* Recommended keys to exit

**Diagram**

![Validator Exit Daemon](https://795605586-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZxJ2nWuxvZaIO7EOU2d0%2Fuploads%2Fgit-blob-38c321a946b0708d8c6ef7d7c672666ffe8e2e1b%2Fvalidator-exit-daemon.png?alt=media)

**Sequence diagram**

![Validator Exit Daemon Sequence Diagram](https://795605586-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZxJ2nWuxvZaIO7EOU2d0%2Fuploads%2Fgit-blob-9079da20e272a825b574d9c93ffac80bcf62517b%2Fvalidator-exit-daemon-sequence.png?alt=media)

#### Dependencies

* A synced Execution Layer client with a JSON-RPC endpoint enabled. All implementations are supported (Geth, Erigon, Besu, etc.)
* A synced Consensus Layer client with an API endpoint enabled. All implementations are supported (Prysm, Teku, Lighthouse, etc.)

#### Installation

The recommended installation is to use the public Docker image public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0

{% hint style="info" %}
For those running kubernetes and helm you can use this image:public.ecr.aws/alluvial/helm-charts/lceth:1.3.0
{% endhint %}

#### Usage

**API**

By default, the Exit daemon exposes an API route that Node Operators can use, either in conjunction with or without the HTTP hook.

This allows an operator to retrieve the recommended keys to exit at the current time.

{% tabs %}
{% tab title="CLI" %}

```shell
env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
env ETH_CL_ADDR={ethereum consensus layer rpc endpoint} \
lceth exit run \
--operator-idx {operator index}
```

{% endtab %}

{% tab title="Docker" %}

```shell
docker run \
  --env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
  --env "ETH_CL_ADDR={ethereum consensus layer rpc endpoint}" \
  public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 exit run \
    --operator-idx {operator index}
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
Operator index could be retrieved with `lceth operators list`
{% endhint %}

The API meets the following requirements:

```yaml
paths:
  '/':
    get:
      description: 'Get validator keys to exit'
      responses:
        200:
          description: 'Success.'
          schema:
            type: object
            properties:
              chain_id:
                type: 'string'
                example: '1'
              total_requested_exits:
                type: 'integer'
                description: 'Number of validators requested to exit'
                example: 2
              validators_to_exit:
                type: 'array'
                description: 'List of recommended validators to exit'
                items:
                  type: 'object'
                  properties:
                    pubkey:
                      type: 'string'
                      example: '0x8a9e1213459337631232447e360faf162781aee0a3df533306a02e2eae481b4db4d1313d1756da7a121da437d9964fb0'
                    index:
                      type: 'integer'
                      example: 473944
        500:
          description: 'Internal unexpected error'
```

**Webhook**

The Exit daemon can run an HTTP hook system that sends validator exit request callbacks to the operator's existing systems when it receives a Validator Exit request event from the Consensus Layer.

{% hint style="warning" %}
The Exit daemon is stateless. If the webhook is enabled, it will send a call at each start if there is a pending exit request.
{% endhint %}

Operators can customize the webhook call by providing a template for the body, endpoint, and headers, and use the following macros to pass data related to the exit event:

```markdown
| Name                       | Type     | Description                               |
| -------------------------- | -------- | ----------------------------------------- |
| chain_id                   | string   | Network identifier                        |
| total_requested_exits      | int      | Number of validators requested to exit    |
| validators_to_exit_indexes | int[]    | List of validators pubkey to exit         |
| validators_to_exit_pubkeys | string[] | List of validators index to exit          |
| validators_to_exit         | object[] | List of validators pubkey + index to exit |
```

* `total_requested_exits` represents the total requested exits since the beginning of the contract. It’s a incremental-only counter emitted by the `RequestedValidatorExits` event.
* `validators_to_exit` (as well as `validators_to_exit_pubkeys` & `validators_to_exit_indexes`) is an array of the validators to exit at a given time. To get them, the exit daemon fetches all the keys of an operator (via the `FundedValidatorKeys` event), check their status on the consensus layer to know those already exited, and then make a diff with `total_requested_exits` to know how many keys are remaining to exit.

{% hint style="info" %}
The webhook templating imports the [Golang Sprig library](https://masterminds.github.io/sprig/). Which means additional functions are available to format the output.

For instance:

* using environment variables via `{{env "ENV_VAR_NAME"}}`
* printing array via `{{validators_to_exit_pubkeys | toJson}}`
* creating a quoted string from an array via `{{.validators_to_exit_indexes | join "," | quote}}`
* computing array length via `{{.validators_to_exit_indexes | len}}`
* etc.
  {% endhint %}

Example of webhook template:

{% tabs %}
{% tab title="CLI" %}

```shell
env ETH_EL_ADDR={ethereum execution layer rpc endpoint} \
env ETH_CL_ADDR={ethereum consensus layer rpc endpoint} \
lceth exit run \
--operator-idx {operator index} \
--webhook \
--webhook-method="POST" \
--webhook-endpoint="http://127.0.0.1:9080/call?chain_id={{.chain_id}}&count={{.total_requested_exits}}&validators={{.validators_to_exit_indexes|join ","}}" \
--webhook-template-body '
  {
    "chain_id": "{{.chain_id}}",
    "total_requested_exits": {{.total_requested_exits}},
    "current_requested_exits": {{.validators_to_exit_pubkeys | len}},
    "indexes": {{.validators_to_exit_indexes | join "," | quote}},
    "pubkeys": {{.validators_to_exit_pubkeys | toJson}},
    "custom": {{env CUSTOM_ENV_VAR}}'
  }
' \
--webhook-template-header="Chain-Id={{.chain_id}}" \
--webhook-template-header="Content-Type=application/json"
```

{% endtab %}

{% tab title="Docker" %}

```shell
docker run \
  --env "ETH_EL_ADDR={ethereum execution layer rpc endpoint}" \
  --env "ETH_CL_ADDR={ethereum consensus layer rpc endpoint}" \
  public.ecr.aws/alluvial/liquid-collective/lceth:v0.37.0 exit run \
    --operator-idx {operator index} \
    --webhook \
    --webhook-method="POST" \
    --webhook-endpoint="http://127.0.0.1:9080/call?chain_id={{.chain_id}}&count={{.total_requested_exits}}&validators={{.validators_to_exit_indexes|join ","}}" \
    --webhook-template-body '
      {
        "chain_id": "{{.chain_id}}",
        "total_requested_exits": {{.total_requested_exits}},
        "indexes": {{.validators_to_exit_indexes | join "," | quote}},
        "pubkeys": {{.validators_to_exit_pubkeys | toJson}},
        "custom": {{env CUSTOM_ENV_VAR}}'
      }
    ' \
    --webhook-template-header="Chain-Id={{.chain_id}}" \
    --webhook-template-header="Content-Type=application/json"
```

{% endtab %}
{% endtabs %}

Usage output:

```
Run exit daemon

Usage:
  lceth exit run [flags]

Flags:
      --eth-cl-addr string                       Address of the Ethereum consensus layer node to connect to [env: ETH_CL_ADDR]
      --operator-idx int                         Index of the operator running the daemon [env: OPERATOR_INDEX]
      --webhook                                  enable 'webhook' [env: WEBHOOK]
      --webhook-endpoint string                  Go template describing the endpoint that the webhook will call [env: WEBHOOK_ENDPOINT]
      --webhook-method string                    HTTP method to query the webhook's endpoint [env: WEBHOOK_METHOD]
      --webhook-template-header stringToString   Go template describing the body to be sent to the webhook's endpoint [env: WEBHOOK_TEMPLATE_HEADER] (default [])
      --webhook-template-body string             Go templates describing the headers to be sent to the webhook [env: WEBHOOK_TEMPLATE_BODY]
      --webhook-retry int                        Retry limit in case of webhook-endpoint returns error with status code 5xx [env: WEBHOOK_RETRY]
      --webhook-timeout duration                 Delay to wait before abort request sent to webhook-endpoint [env: WEBHOOK_TIMEOUT]
      --rpc-call-retry int                       How many time should we retry failed rpc calls [env: RPC_CALL_RETRY] (default 3)
      --block-log-fetch-step int                 How many blocks are fetched at once when fetching logs [env: BLOCK_LOG_FETCH_STEP] (default 100000)
      --max-workers int                          How many goroutines to use on threaded tasks [env: MAX_WORKERS] (default 10)
      --batch-fetch-validators                   Whether to batch when querying the consensus layer to get validators' data (recommended if timeout issues arise) [env: BATCH_FETCH_VALIDATORS]
      --max-get-validators int                   Max size of the batch when querying the consensus layer to get validators' data, only useful if using --batch-fetch-validators [env: MAX_GET_VALIDATORS] (default 1000)
      --loop-sleep-time duration                 How often should the exit-daemon run [env: LOOP_SLEEP_TIME] (default 5m0s)
  -h, --help                                     help for run

Global Flags:
      --allowlist-addr string            Address of the Allowlist contract [env: ALLOWLIST_ADDR]
      --deployment-block uint            Deployment block of the contracts [env: DEPLOYMENT_BLOCK]
      --el-fee-recipient-addr string     Address of the Execution Layer fee recipient contract [env: EL_FEE_RECIPIENT_ADDR]
      --eth-el-addr string               JSON-RPC address of the Ethereum execution layer node to connect to [env: ETH_EL_ADDR]
      --keystore-password string         Password used to encrypt key files [env: KEYSTORE_PASSWORD]
      --keystore-path string             Directory where to store keys [env: KEYSTORE_PATH]
      --log-format string                Log output format (text or json) [env: LOG_FORMAT] (default "text")
      --log-level string                 Log output level [env: LOG_LEVEL] (default "info")
      --operators-registry-addr string   Address of the Operators Registry contract [env: OPERATORS_REGISTRY_ADDR]
      --oracle-addr string               Address of the Oracle contract [env: ORACLE_ADDR]
      --redeem-manager-addr string       Address of the RedeemManager contract [env: REDEEM_MANAGER_ADDR]
      --river-addr string                Address of the River contract [env: RIVER_ADDR]
      --tlc-addr string                  Address of the TLC contract [env: TLC_ADDR]
      --withdraw-addr string             Address of the Withdraw contract [env: WITHDRAW_ADDR]
      --wls-eth-addr string              Address of the WlsEth contract [env: WLSETH_ADDR]
```