§ Wiki · Wiki entry

Swapping a Node in a Subnet

Replace a poorly performing assigned node with an unassigned node owned by the same operator using swap-node-in-subnet-directly — no NNS proposal required.

A node assigned to a subnet does useful work and earns rewards; an unassigned node sits idle and earns nothing. When an assigned node needs maintenance, node swapping lets you move an unassigned node — owned by the same node operator, in the same data center — into the subnet in its place. The subnet keeps its capacity while you service the machine, and because the swap is a direct registry call it needs no NNS proposal.

When the direct swap is not available, the alternative is an NNS proposal that changes the subnet's membership — see When a direct swap is not possible below.

Warning

Do not power the old node off until it has gracefully left the subnet. A node that disappears before completing its graceful departure looks malicious to consensus, which can trigger a rewards penalty. Wait for the log confirmation in step 5 before starting any maintenance.

Before you start

Both nodes have to belong to the same principals and location:

  • The old node is currently assigned to a subnet.
  • The new node is unassigned (its subnet shows - on the dashboard).
  • Both nodes share the same node provider, the same node operator, and the same data center.

The swap is also rate limited, to protect subnet stability:

  • The subnet the old node belongs to must not have had a swap in the last 4 hours.
  • You must not have swapped a node on that same subnet in the last 24 hours.

Procedure

Step 1 — Locate the assigned node

On the IC Dashboard find the assigned node you want to take out of service and note its node ID (principal).

Step 2 — Find an unassigned replacement

Identify an unassigned node in the same data center — one whose subnet column shows - — and note its node ID. This is the node that will take the old node's place in the subnet.

Step 3 — Set up authentication

Define an AUTH array with the node operator key that owns both nodes. Use a PEM key file:

AUTH=(
  "--secret-key-pem"
  <path-to-key-pem>
)

or an HSM:

AUTH=(
  "--use-hsm"
  "--slot"
  <hsm-key-slot>
  "--key-id"
  <hsm-key-id>
  "--pin"
  <hsm-pin>
)

Step 4 — Execute the swap

Run swap-node-in-subnet-directly, passing the old node ID from step 1 and the new node ID from step 2:

ic-admin --nns-urls https://icp-api.io ${AUTH[@]} swap-node-in-subnet-directly \
    --old-node-id <principal-from-step-1> \
    --new-node-id <principal-from-step-2>

Step 5 — Wait for the graceful departure

The old node takes roughly 15–20 minutes to leave the subnet cleanly. Watch its logs for the two confirmations, in order:

  • Node started leaving subnet
  • Node gracefully left a subnet

Step 6 — Perform maintenance

Only once the second log line has appeared is the old node fully out of the subnet. It is now safe to power it off and carry out whatever maintenance you had planned.

When a direct swap is not possible

The direct call only works when the conditions in Before you start hold. If they do not — for example, you have no unassigned node of your own in the same data center, or the rate limits would delay urgent maintenance — the subnet's membership has to be changed by NNS proposal instead.

That path requires a neuron with a hotkey. If you do not have one yet, set it up first with Setting Up a Neuron for NNS Proposals. Once the proposal executes, continue from step 5 above: the old node still needs its graceful departure before you power it off.

Use the DRE tool

Submit these proposals with the DRE tool. It picks the replacement node that best keeps the subnet within its decentralization targets, writes a proposal summary that explains the change to voters, and shows you the proposal for review before submitting it.

Install it (Linux shown; on macOS download dre-x86_64-apple-darwin instead):

mkdir -p $HOME/bin
curl -L https://github.com/dfinity/dre/releases/latest/download/dre-x86_64-unknown-linux -o $HOME/bin/dre
chmod +x $HOME/bin/dre

Then propose the replacement, signing with your hotkey on behalf of your neuron:

dre subnet replace \
    --private-key-pem ~/node-provider-hotkey.pem \
    --neuron-id $NEURON_ID \
    --remove-nodes <old-node-id> \
    --motivation "<why the node is being replaced, with a forum link>"

DRE chooses the replacement itself. Pass --add-nodes <node-id> only if you need a specific node to take the old node's place.

Fallback: ic-admin

If DRE is not available, you can submit the same proposal with ic-admin. You then have to choose a replacement and write the motivation yourself:

ic-admin --nns-urls https://icp-api.io \
    -s ~/node-provider-hotkey.pem \
    propose-to-change-subnet-membership \
    --proposer $NEURON_ID \
    --subnet <subnet-id> \
    --node-ids-remove <old-node-id> \
    --node-ids-add <replacement-node-id> \
    --summary "<why the node is being replaced>"

The replacement must be an unassigned node, but it does not have to be yours. The resulting subnet must still meet its decentralization targets, or expect the proposal to be rejected.