The systemd service for NBD Client doesn't come up on boot

[Thiago Martins]

The `systemd` service for starting the NBD Clients on boot isn't working.

### Steps to Reproduce:

0. Install NBD

-
apt install nbd-server nbd-client
-

1. Prepare an NBD server with a disk image to share. For instance:

-
truncate -s 1T /vol-0.img
-

2. Configure the NBD server by adding the following to `/etc/nbd-server/conf.d/exports.conf`:

-
[vol0]
exportname = /vol-0.img
-

3. Configuring the client connection in `/etc/nbdtab`:

Contents of `/etc/nbdtab`:

-
nbd0 localhost vol0
-

4. Enable it on `systemd`:

-
systemctl enable nbd@nbd0
-

Here's what I see:

-
# systemctl enable nbd@nbd0
Created symlink /<email address hidden> → /lib/systemd/system/nbd@.service.
Unit /lib/systemd/system/nbd@.service is added as a dependency to a non-existent unit dev-nbd0.device.
-

5. Add `nbd` auto-load to system'd boot:

echo nbd > /etc/modules-load.d/nbd.conf

6. Reboot

-
sudo reboot
-

7. The service not coming up on the system's boot. Look:

After rebooting the system, the NBD clien for `nbd0` isn't running (but it's enabled on systemd).

-
root@demo-nbd-test:~# systemctl status nbd@nbd0
○ <email address hidden> - NBD client connection for nbd0
     Loaded: loaded (/lib/systemd/system/nbd@.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:nbd-client

Feb 21 22:54:22 demo-nbd-test systemd[1]: <email address hidden>: Dependency Before=dev-nbd0.device ignored (.device units cannot be delayed)
-

But if you try to start it again. It'll work:

-
root@demo-nbd-test:~# systemctl start nbd@nbd0
root@demo-nbd-test:~# systemctl status nbd@nbd0
● <email address hidden> - NBD client connection for nbd0
     Loaded: loaded (/lib/systemd/system/nbd@.service; enabled; vendor preset: enabled)
     Active: active (exited) since Wed 2024-02-21 22:57:37 UTC; 2s ago
       Docs: man:nbd-client
    Process: 1096 ExecStart=//sbin/nbd-client nbd0 (code=exited, status=0/SUCCESS)
   Main PID: 1096 (code=exited, status=0/SUCCESS)
        CPU: 4ms

Feb 21 22:57:37 demo-nbd-test systemd[1]: Starting NBD client connection for nbd0...
Feb 21 22:57:37 demo-nbd-test nbd-client[1096]: Negotiation: ..size = 3145728MB
Feb 21 22:57:37 demo-nbd-test nbd-client[1096]: Connected /dev/nbd0
Feb 21 22:57:37 demo-nbd-test systemd[1]: Finished NBD client connection for nbd0.
-

So this forces me to have the following `/etc/rc.local` file as a workaround:

-
#!/bin/bash
sleep 6
systemctl start nbd@nbd0
-

Make it executable:

-
chmod +x /etc/rc.local
-

Reboot again, it'll be started on next boot!

I don't think that the `/etc/rc.local` should be part of this procedure... =P

NOTE: Don't need to run `update-initramfs -k all -u`.

[Paride Legovini (paride)]

Hello and thanks for this bug report. I had a quick look at /lib/systemd/system/nbd@.service, and the [Install] section shows that the nbd@*.service units are not meant to just be started automatically, but when a dev-*.device is created:

[Install]
RequiredBy=dev-%i.device
RequiredBy=dev-%ip1.device
[...]

and I believe there are udev rules handling those devices. In other words, I believe we need a more complete picture of what issue to triage this as a bug and begin working on it. Could you please provide a minimal but complete set of steps to configure an Ubuntu system so that the problem reproduces? Using the same system as client and server should be enough, similar to what has been done in LP: #2054470.

Thanks

[Thiago Martins (martinx)]

Done!

[Wouter Verhelst (wouter-debian)]

Please read the systemd section of /usr/share/doc/nbd-client/README.Debian

You have not run

systemctl enable nbd@nbd0

Which is required to instantiate a templated NBD unit from the template.

This is not done automatically, as you may not necessarily want to enable all devices that you define in nbdtab at boot time.

Suggestions on how to document this better are welcome!

[Thiago Martins (martinx)]

Wouter,

Please, read the description carefully. I am running `systemctl enable nbd@nbd0`.

The `status` even shows the service as `enabled`.

Thanks for pointing out the `/usr/share/doc/nbd-client/README.Debian.gz` file! Now I'm sure this is a bug since I'm doing it right.

Thanks.

[Mitchell Dzurick (mitchdz)]

Thiago,

Apologies if I missed it somewhere, but which Ubuntu release are you using for your tests?

[Mitchell Dzurick (mitchdz)]

I've been playing around with noble and I've noticed that sometimes if I try to restart the service right after boot it will fail

root@n-vm:~# systemctl restart nbd@nbd0
Job for <email address hidden> failed because the control process exited with error code.
See "systemctl status <email address hidden>" and "journalctl -xeu <email address hidden>" for details.
root@n-vm:~# systemctl status nbd@nbd0
× <email address hidden> - NBD client connection for nbd0
     Loaded: loaded (/usr/lib/systemd/system/nbd@.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2024-02-27 23:44:35 UTC; 2s ago
       Docs: man:nbd-client
    Process: 871 ExecStart=//sbin/nbd-client nbd0 (code=exited, status=1/FAILURE)
   Main PID: 871 (code=exited, status=1/FAILURE)
        CPU: 5ms

Feb 27 23:44:35 n-vm systemd[1]: Starting <email address hidden> - NBD client connection for nbd0...
Feb 27 23:44:35 n-vm nbd-client[871]: Negotiation: ..size = 1024MB
Feb 27 23:44:35 n-vm nbd_client[871]: Failed to setup device, check dmesg
Feb 27 23:44:35 n-vm nbd-client[871]: Error: Failed to setup device, check dmesg
Feb 27 23:44:35 n-vm nbd-client[871]: Exiting.
Feb 27 23:44:35 n-vm nbd_client[871]: Exiting.
Feb 27 23:44:35 n-vm systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
Feb 27 23:44:35 n-vm systemd[1]: <email address hidden>: Failed with result 'exit-code'.
Feb 27 23:44:35 n-vm systemd[1]: Failed to start <email address hidden> - NBD client connection for nbd0.
root@n-vm:~# dmesg | tail -n 1
[ 12.032858] nbd: nbd0 already in use

But yeah, it just seems that systemd ignores the start because it doesn't want to wait for the devices:

Feb 27 23:44:31 n-vm systemd[1]: <email address hidden>: Dependency Before=dev-nbd0.device ignored (.device units cannot be delayed)

You can force a start (We should not do this, but just showing for debug) by doing something such as:

# echo "WantedBy=multi-user.target" >> /usr/lib/systemd/system/nbd@.service && systemctl enable nbd@nbd0

But then you see the client fails to start, because the devices are not available at that point.

root@n-vm:~# systemctl status nbd@nbd0
× <email address hidden> - NBD client connection for nbd0
     Loaded: loaded (/usr/lib/systemd/system/nbd@.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2024-02-27 23:48:22 UTC; 1s ago
       Docs: man:nbd-client
    Process: 663 ExecStart=//sbin/nbd-client nbd0 (code=exited, status=1/FAILURE)
   Main PID: 663 (code=exited, status=1/FAILURE)
        CPU: 5ms

Feb 27 23:48:22 n-vm systemd[1]: Starting <email address hidden> - NBD client connection for nbd0...
Feb 27 23:48:22 n-vm nbd_client[663]: Socket failed: Connection refused

I suppose the expectation here is that when you run `systemctl enable nbd@nbd0` that the /dev/nbd* devices should be instantiated at boot and the client should run and use those devices, but the devices are not coming up at boot.

root@n-vm:~# ll /dev/nbd*
ls: cannot access '/dev/nbd*': No such file or directory

The interesting thing I am noticing is that the devices are instantiated after the first restart, so even though you see a failure the first time, restarting it a second time makes everything...

[Mitchell Dzurick (mitchdz)]

Wouter, is my assumption correct that ideally we should see the devices available on boot if we enable nbd@nbd0?

[Thiago Martins (martinx)]

Hi Mitchell,

No worries at all!

I forgot to mention another step I took:

```
echo nbd > /etc/modules-load.d/nbd.conf
```

Otherwise it won't load the module on boot.

Perhaps there's also a need to `update-initramfs -k all -u`, not 100% sure.

[Thiago Martins (martinx)]

I'm testing on Ubuntu 22.04.4

[Thiago Martins (martinx)]

I've updated the description to include all the steps necessary to reproduce the issue on Ubuntu 22.04 (Cloud Image - Server Minimal install).

This seems to be a `systemd` thing! lol

[Paride Legovini (paride)]

Thanks for the additional details and for the improved reproducer. I can reproduce the issue and looks like the steps you identified are right. I assume this worked at some point, but the issue is also present in Focal and Bionic.

I don't think this is working as expected, OTOH the RequiredBy= (and, in earlier releases, Before=) systemd dependencies come from the upstream project [1], where I see no bug reports about this issue (not even closed ones). Same goes for Debian: no bugs about this.

I am going to triage this as a bug, however to speed things up I suggest filing an upstream bug [2]. Given that the nbd@. units come from there, the upstream devs may be able to help. @Thiago would you be willing to do so? Thanks!

[1] https://github.com/NetworkBlockDevice/nbd/
[2] https://github.com/NetworkBlockDevice/nbd/issues
[3] https://github.com/NetworkBlockDevice/nbd/tree/master/systemd