--- title: Troubleshooting Node Deployment Errors slug: node-deployment-troubleshooting description: Diagnostic guide for the common failure modes encountered while installing IC-OS on a node machine — registration errors, blank consoles, missing drives, network failures, and how to get a shell in SetupOS. tags: - node-provider - deployment - ic-os - troubleshooting date: 2026-05-04 related: - node-provider-documentation - node-deployment-gen-2 - node-deployment-gen-1 - node-deployment-config-ini --- This entry collects the error patterns most commonly seen when installing IC-OS on a node machine. **Read it in full before posting a support request** in the Node Provider Matrix channel — the troubleshooting sequence below resolves the vast majority of issues. If you need Dell to service the machine, see the [iDRAC Access and TSR Logs](/wiki/idrac-access-and-tsr-logs/) entry for how to retrieve a Dell TSR log and reset the iDRAC password. ## General troubleshooting steps These steps have been refined over hundreds of deployments. **Complete all of them before messaging in the Matrix channel.** 1. Re-attempt the verification step from your deployment runbook, exactly as written: - [Node Deployment Guide (Gen-2)](/wiki/node-deployment-gen-2/), section "Verify node onboarding". - [Node Deployment Guide (Gen-1, with HSM)](/wiki/node-deployment-gen-1/), section "Verify node onboarding". 2. Confirm you are using the latest IC-OS release from the [Internet Computer Dashboard Releases](https://dashboard.internetcomputer.org/releases). When in doubt, redownload the latest release and retry. 3. Confirm you are following the runbook that matches your hardware generation and onboarding path: - [Node Deployment Guide (Gen-2)](/wiki/node-deployment-gen-2/) — current generation, no HSM. - [Node Deployment Guide (Gen-1, with HSM)](/wiki/node-deployment-gen-1/) — legacy generation, with HSM. 4. Reread every step in your runbook. The instructions are precise and they evolve over time. 5. Verify the networking values written to `config.ini` against the data center's actual configuration. 6. Reread the [Node Provider Networking Guide](/wiki/node-networking-guide/), especially the "What NOT to do" section. 7. Restart **every** piece of equipment between the node and the internet — switch, router, firewall. 8. Restart the deployment from the beginning and try to reproduce the error. 9. Try a different node machine that is not currently in a subnet. If the failure reproduces on multiple machines, that points away from a single-machine hardware fault. ### Support request information requirements If the steps above did not resolve your issue, post in the [IC Node Provider Matrix channel](https://matrix.to/#/#nodeproviders:dfinity.org) with **all** of the following — requests missing this information will be sent back: - Confirmation that you completed every general troubleshooting step above. - Screenshots of the failure and any relevant logs. - The deployment stage at which the failure occurs. - Which runbook you are following: [Gen-2 (no HSM)](/wiki/node-deployment-gen-2/) or [Gen-1 (with HSM)](/wiki/node-deployment-gen-1/). - Whether this is your first IC-OS installation overall. - Whether this is your first IC-OS installation **in this data center**. - Whether this is your first IC-OS installation **with this Node Operator key**. - Whether the issue reproduces. - Hardware details: generation (Gen-1, Gen-2) and server brand. - The data center you are deploying out of. - Whether you are onboarding with an IPv4 address. - Any other relevant networking details. ## Unhealthy nodes The node installed and registered successfully, but the [dashboard](https://dashboard.internetcomputer.org/) shows a status that is **not** "Awaiting Subnet" or "Active in Subnet". Refer to the [Node Provider Troubleshooting](/wiki/node-provider-troubleshooting/) guide for the unhealthy-node diagnostic flow. ## Node registration failures The node installed cleanly. HostOS came up. But the node never appears on the dashboard and you have already worked through the general troubleshooting steps. ### `Node allowance for this node operator is exhausted` The node allowance on your operator record (set during [node provider onboarding](/wiki/node-provider-onboarding/)) is full. Check the dashboard for nodes registered under your principal — if old nodes are still listed that should not be, see [Removing a Node From the Registry](/wiki/removing-node-from-registry/). ### `Cannot remove a node that is a member of a subnet` You are attempting to redeploy a node that is currently assigned to a subnet. The node must be removed from the subnet first. Note: if you are performing a firmware upgrade only, you do **not** need to redeploy. ### `Node reward type is required` The `node_reward_type` field in `config.ini` is unset or invalid. See [Node Deployment config.ini](/wiki/node-deployment-config-ini/) to look up the correct value for your operator record, then redeploy. ### `cannot set up guest memory 'pc.ram': Cannot allocate memory` Full message: ``` QEMU unexpectedly closed the monitor (vm='guestos'): ... qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory ``` HostOS cannot allocate the required RAM to the GuestOS VM. This is almost always a hardware fault. Run `memtest` and confirm every DIMM is seated correctly. ### Blank console after install After SetupOS finishes, the machine reboots and HostOS comes up. After a few minutes you should see a HostOS console log. **This log alone does not signify a successful onboarding** — wait at least 10 minutes for the `Join request successful!` line. If after 10 minutes nothing further has been logged, **leave the node running** and post in the Matrix channel with a screenshot of the console plus all the support request information requirements above. ## IC-OS installation failures ### Blank console If the SetupOS console is blank, **wait 5 minutes**. Some nodes are slow to start logging and this is not necessarily a failure. If the console is still blank after 5 minutes, return to the general troubleshooting steps. ### Missing drives ``` -------------------------------------------------------------------------------- INTERNET COMPUTER - SETUP - FAILED -------------------------------------------------------------------------------- Please consult the wiki guide: Troubleshooting Node Deployment Errors. -------------------------------------------------------------------------------- ERROR -------------------------------------------------------------------------------- Not enough drives found. Are all drives correctly installed? -------------------------------------------------------------------------------- ERROR -------------------------------------------------------------------------------- ``` A variant of this error reads `Aggregate Disk size does not meet requirements`. **Cause.** The installer cannot detect every required drive. Even if all drives appear physically present, one or more may be poorly seated or non-functional. **Resolution.** Reseat every drive. Check drive indicator LEDs to identify any that are not powering up. Install the required number of drives if any are missing. ### Invalid CPU configuration ``` -------------------------------------------------------------------------------- INTERNET COMPUTER - SETUP - FAILED -------------------------------------------------------------------------------- Please consult the wiki guide: Troubleshooting Node Deployment Errors. -------------------------------------------------------------------------------- ERROR -------------------------------------------------------------------------------- Number of threads (16/32) does NOT meet system requirements. -------------------------------------------------------------------------------- ERROR -------------------------------------------------------------------------------- ``` **Cause.** The CPUs are not configured correctly in the BIOS — usually disabled cores or disabled SMT. **Resolution.** Reset the BIOS to factory defaults, then reapply the UEFI configuration for your hardware vendor: - [Gen-2 Dell](/wiki/uefi-config-gen-2-dell/) - [Gen-2 Supermicro](/wiki/uefi-config-gen-2-supermicro/) - [Gen-2 Gigabyte](/wiki/uefi-config-gen-2-gigabyte/) - [Gen-2 ASUS](/wiki/uefi-config-gen-2-asus/) - [Gen-1 Dell (PowerEdge R6525)](/wiki/uefi-config-gen-1-dell/) - [Gen-1 Supermicro](/wiki/uefi-config-gen-1-supermicro/) ### Unable to reach internet ``` -------------------------------------------------------------------------------- INTERNET COMPUTER - SETUP - FAILED -------------------------------------------------------------------------------- Please consult the wiki guide: Troubleshooting Node Deployment Errors. -------------------------------------------------------------------------------- ERROR -------------------------------------------------------------------------------- Unable to ping IPv6 gateway. -------------------------------------------------------------------------------- ERROR -------------------------------------------------------------------------------- ``` **Cause.** The node cannot communicate with the network — either a misconfigured `config.ini`, or a problem somewhere between the node and the internet. **Resolution.** Capture any output that appears before the error. The installer prints the user-defined and system network settings: ``` * Printing user defined network settings... IPv6 Prefix : XXX IPv6 Subnet : XXX IPv6 Gateway: XXX * Printing system's network settings... IPv6 Prefix : XXX IPv6 Subnet : XXX IPv6 Gateway: XXX * Printing IPv6 addresses... SetupOS: XXX HostOS : XXX GuestOS: XXX ``` Compare these against the values you expect. If they don't match, correct `config.ini` and try again. If they do match, the network path between the node and the wider internet is the problem — likely a firewall. **Reboot every device between the node and the internet** even if everything appears to be working. Rebooting an apparently healthy firewall has resolved this class of issue many times. ### Unable to setup PV ``` -------------------------------------------------------------------------------- INTERNET COMPUTER - SETUP - FAILED -------------------------------------------------------------------------------- Please consult the wiki guide: Troubleshooting Node Deployment Errors. -------------------------------------------------------------------------------- ERROR -------------------------------------------------------------------------------- Unable to setup PV on drive '/dev/nvme8n1'. -------------------------------------------------------------------------------- ERROR -------------------------------------------------------------------------------- ``` **Cause.** The node sees the drive but cannot write to it. This usually indicates a hardware fault on that specific drive. **Resolution.** Remove and reinstall every drive. Independently verify that each drive is functioning before retrying the installation. ### Warning: Gen-2 hardware detected but no Node Operator Private Key found This is a **warning**, not an error. The installer is reminding you that Gen-2 hardware should be onboarded via the [Gen-2 deployment guide](/wiki/node-deployment-gen-2/) (no HSM), not the legacy HSM path. - If you are **redeploying** a node that was originally onboarded with an HSM, you may continue — wait 5 minutes for the installation to resume. - If you are onboarding a **new** Gen-2 machine, abort and switch to the [Gen-2 deployment guide](/wiki/node-deployment-gen-2/). ## Getting a shell in SetupOS During the IC-OS installation phase, you can drop to a console shell to diagnose problems directly: 1. Press Enter at any time during installation, or at the error page. 2. Continue pressing Enter until you see a login prompt. 3. Log in as `root` with an empty password. 4. You now have root access for diagnostics. > [!NOTE] > The SetupOS shell is **only available during the IC-OS installation > phase**. After the machine reboots into HostOS, this troubleshooting > console is no longer accessible. To inspect a HostOS-stage failure, > boot the machine from a live USB distribution such as > [Ubuntu Live USB](https://ubuntu.com/tutorials/try-ubuntu-before-you-install#1-getting-started) > and troubleshoot from there. ## Related - [Node Deployment Guide (Gen-2)](/wiki/node-deployment-gen-2/) — the primary deployment runbook. - [Node Deployment Guide (Gen-1, with HSM)](/wiki/node-deployment-gen-1/) — the legacy deployment runbook. - [Node Deployment config.ini](/wiki/node-deployment-config-ini/) — reference for the `node_reward_type` field. - [Node Provider Troubleshooting](/wiki/node-provider-troubleshooting/) — post-deployment troubleshooting once the node is registered. - [iDRAC Access and TSR Logs](/wiki/idrac-access-and-tsr-logs/) — collecting diagnostic data on Dell hardware.