Error: only running operations can be connected

[Brian Murray]

This was seen on an armhf test runner with tcl-fitstcl and a trigger of glibc/2.31-0ubuntu9.13.

100s autopkgtest [17:48:52]: rebooting testbed after setup commands that affected boot
140s autopkgtest [17:49:32]: testbed running kernel: Linux 5.4.0-167-generic #184-Ubuntu SMP Tue Oct 31 09:24:51 UTC 2023
157s autopkgtest [17:49:49]: ERROR: "dpkg --print-architecture" failed with stderr "Error: Only running operations can be connected
157s "

I've actually seen this in multiple logs so it bears further investigation.

[Tim Andersson (andersson123)]

I've tried to reproduce this. The "Error: Only running operations can be connected" wasn't an issue for me. Instead it failed like this:

ERROR: "dpkg --print-architecture" failed with stderr "Error: read tcp 10.136.6.103:40086->10.44.124.67:8443: i/o timeout

[Tim Andersson (andersson123)]

10.136.6.103 is the lxd worker so maybe there's a timeout somewhere we need to increase.

[Tim Andersson (andersson123)]

I actually get this also:

ERROR: "uname -srv" failed with stderr "Error: Only running operations can be connected"

The error message isn't consistent

[Tim Andersson (andersson123)]

I was using this to reproduce:

/home/ubuntu/autopkgtest/runner/autopkgtest --output-dir /tmp/autopkgtest-work.nxbirl5u4/out --timeout-copy=6000 --setup-commands 'sed -i "s/ports.ubuntu.com/ftpmaster.internal/; s/ubuntu-ports/ubuntu/" /etc/apt/sources.list `ls /etc/apt/sources.list.d/*.list 2>/dev/null || true`; ln -s /dev/null /etc/systemd/system/bluetooth.service; printf "http_proxy=http://squid.internal:3128\nhttps_proxy=http://squid.internal:3128\nno_proxy=127.0.0.1,127.0.1.1,login.ubuntu.com,localhost,localdomain,novalocal,internal,archive.ubuntu.com,ports.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,launchpad.net,10.24.0.0/24,keystone.ps5.canonical.com,objectstorage.prodstack5.canonical.com\n" >> /etc/environment' --setup-commands 'apt-key adv --keyserver keyserver.ubuntu.com --recv-key 1DAB0B71FE08CDC6BD13E2C0ADCE2AF3A4E0014F' --setup-commands 'REL=$(sed -rn "/^(deb|deb-src) .*(ubuntu.com|ftpmaster)/ { s/^[^ ]+ +(\[.*\] *)?[^ ]* +([^ -]+) +.*$/\2/p; q }" /etc/apt/sources.list); echo "deb http://ppa.launchpad.net/ubuntu-security-proposed/ppa/ubuntu $REL main" > /etc/apt/sources.list.d/autopkgtest-ubuntu-security-proposed-ppa.list; echo "deb-src http://ppa.launchpad.net/ubuntu-security-proposed/ppa/ubuntu $REL main" >> /etc/apt/sources.list.d/autopkgtest-ubuntu-security-proposed-ppa.list;' --apt-upgrade ruby-bert --timeout-short=300 --timeout-copy=20000 --timeout-build=20000 --env=ADT_TEST_TRIGGERS=glibc/2.37-0ubuntu2.2 -- lxd -r lxd-armhf-10.44.124.67 lxd-armhf-10.44.124.67:autopkgtest/ubuntu/lunar/armhf

[Brian Murray (brian-murray)]

Looking in the lxd source code I found out error message:

 $ rg "running operations can be connected"
lxd/operations/operations.go
426: return nil, fmt.Errorf("Only running operations can be connected")

[Brian Murray (brian-murray)]

I found the following upstream lxc issue and running some mtr tests of the connectivity to the lxd-armhf test runners is showing a high amount of latency.

[Tim Andersson (andersson123)]

https://github.com/canonical/lxd/issues/12514

[Brian Murray (brian-murray)]

While the networking issues were resolved by Canonical IS, I think the service should be more robust in the face of latency issues with the network.

[Paride Legovini (paride)]

Apparently this was fixed by IS, marking it as Fix Released.

[Brian Murray (brian-murray)]

I still think we should investigate handling high latency better.