qemu+ssh connections to a remote libvirt fail (from o to n)

[C de-Avillez]

running on Oneiric, connecting to machines running Natty. On these machines, the libvirt/qemu versions are as follows:

ii libvirt-bin 0.8.8-1ubuntu6.5
ii libvirt0 0.8.8-1ubuntu6.5
un qemu <none>
ii qemu-common 0.14.0+noroms-0ubuntu4.4
ii qemu-kvm 0.14.0+noroms-0ubuntu4.4cerdea@aldebaran:/usr/share$

When trying to connect I get a pop-up with the following error:

Unable to open a connection to the libvirt management daemon.

Libvirt URI is: qemu+ssh://cerdea@aldebaran/system

Verify that:
 - The 'libvirt-bin' package is installed
 - The 'libvirtd' daemon has been started
 - You are member of the 'libvirtd' group

packet 1550146349 bytes received from server too large, want 262144

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/connection.py", line 1146, in _open_thread
    self.vmm = self._try_open()
  File "/usr/share/virt-manager/virtManager/connection.py", line 1130, in _try_open
    flags)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 102, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirtError: packet 1550146349 bytes received from server too large, want 262144

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: virt-manager 0.9.0-1ubuntu3
ProcVersionSignature: Ubuntu 3.0.0-12.19-generic 3.0.4
Uname: Linux 3.0.0-12-generic x86_64
NonfreeKernelModules: wl
ApportVersion: 1.23-0ubuntu2
Architecture: amd64
CheckboxSubmission: 24776dc1392ac4f88b6beaa2d7cdfe16
CheckboxSystem: d00f84de8a555815fa1c4660280da308
Date: Wed Oct 5 17:00:44 2011
PackageArchitecture: all
SourcePackage: virt-manager
UpgradeStatus: Upgraded to oneiric on 2011-08-22 (44 days ago)

[C de-Avillez (hggdh2)]

Forgot -- this was working as recently as about one month ago.

[Marc Deslauriers (mdeslaur)]

Looks like a libvirt issue, reassigning.

[Serge Hallyn (serge-hallyn)]

@hggdh,

thanks for submitting this bug. I tried to reproduce it with a new uptodate natty and a new uptodate oneiric host. But from the oneiric host,

   virsh -c qemu+ssh://ubuntu@10.55.60.90/session list --all

worked, and I was able to start and watch and destroy a VM.

Is there anything else unusual about these hosts? Firewalling or custom libvirt configuration?

[C de-Avillez (hggdh2)]

Hi Serge,

These hosts are in the QA lab, and only accessible via a SSH proxy. Firewalls are certainly in place, but no custom libvirt customisation to my knowledge. We can get you access if you need.

Interestingly, via the proxy I get this:

[cerdea-aws]cerdea@xango3:~$ virsh -c qemu+ssh://aldebaran/system list --all
error: packet 1550146349 bytes received from server too large, want 262144
error: failed to connect to the hypervisor
[cerdea-aws]cerdea@xango3:~$

And, if I am local at aldebaran:

cerdea@aldebaran~:$ virsh -c qemu+ssh://aldebaran/system list --all
cerdea@aldebaran's password:
 Id Name State
----------------------------------
1058 8913cfbf-4757-491f-b8c5-02df75d18f6e running

cerdea@aldebaran:~$

[Serge Hallyn (serge-hallyn)]

What is the arch of both? is one 32-bit and one 64?

[C de-Avillez (hggdh2)]

My laptop is a 64-bit core-i7; aldebaran is a 64-bit, probably Xeon-based.

Also, running visrh -c qemu+ssh//localhost/system works on my laptop.

[Serge Hallyn (serge-hallyn)]

Quoting C de-Avillez (<email address hidden>):
> My laptop is a 64-bit core-i7; aldebaran is a 64-bit, probably Xeon-
> based.
>
> Also, running visrh -c qemu+ssh//localhost/system works on my laptop.

Hi,

I've failed to reproduce this. Is there any chance you would have time
to take these two exact same systems, re-install uptodate natty and
oneiric on them, and re-test? Or, could I get access to them (pls
feel free to ping me in irc) if they are still set up? If the oneiric
one is your laptop, I can try setting up a new oneiric client in the
lab.

thanks,
-serge

[C de-Avillez (hggdh2)]

I just installed an Oneiric (Unity) on a local VM. Connecting to the Lab machines results in the same error. There is no option of reinstalling Natty on the Lab machines, these are production machines.

[Dave Walker (davewalker)]

hggdh2, can you confirm the ssh path that is being undertaken.. is ssh being tunnelled over nc?

Thanks.

[C de-Avillez (hggdh2)]

Yes, it is. Before trying it I have to start a SSH session to batuan (the gateway to the machines). Batuan itself is proxied thru yet another proxy. All of them run via nc.

[Serge Hallyn (serge-hallyn)]

The issue turned out to be due to how libvirt uses nc through ssh . hggdh had a line in his .ssh/config entry for the host containing

 LocalCommand printf "whatever"

The text being printed out was interpreted by the virsh client as coming from libvirt, so the first set of characters were read as the (far too large) length of the message.

[C de-Avillez (hggdh2)]

Note that LocalCommand is a valid ssh_config option; if virsh cannot deal with it, it should be documented.

[Serge Hallyn (serge-hallyn)]

The virsh manpage doesn't mention ssh, so it sounds like the file /usr/share/doc/libvirt-doc/remote.html shipped with libvirt-doc could use a patch mentioning this.

[Tim Wieneke (tim-spen)]

hi,
I have a similiar issue but no special config for my ssh:

the ubuntu clients (one is maverick, one is natty) connect via ssh key authentication to libvirtd server.
All servers are fedora, except of one ubuntu (oneiric). The connection to the feodora are all ok for the ubuntu i get:

as normal user:
virsh -c qemu+ssh://<email address hidden>/system list --all
-> after entering the root pass: connected, the list appears

as root:
virsh -c qemu+ssh://<email address hidden>/system list --all
-> error:
error: Connection reset by peer
error: failed to connect to the hypervisor

debug shows:
15:36:04.233: debug : virInitialize:340 : register drivers
15:36:04.233: debug : virRegisterDriver:928 : registering Test as driver 0
15:36:04.233: debug : virRegisterNetworkDriver:734 : registering Test as network driver 0
15:36:04.233: debug : virRegisterInterfaceDriver:765 : registering Test as interface driver 0
15:36:04.233: debug : virRegisterStorageDriver:796 : registering Test as storage driver 0
15:36:04.233: debug : virRegisterDeviceMonitor:827 : registering Test as device driver 0
15:36:04.233: debug : virRegisterSecretDriver:858 : registering Test as secret driver 0
15:36:04.233: debug : virRegisterNWFilterDriver:889 : registering Test as network filter driver 0
15:36:04.233: debug : virRegisterDriver:928 : registering Xen as driver 1
15:36:04.233: debug : virRegisterDriver:928 : registering OPENVZ as driver 2
15:36:04.233: debug : virRegisterDriver:928 : registering remote as driver 3
15:36:04.233: debug : virRegisterNetworkDriver:734 : registering remote as network driver 1
15:36:04.233: debug : virRegisterInterfaceDriver:765 : registering remote as interface driver 1
15:36:04.233: debug : virRegisterStorageDriver:796 : registering remote as storage driver 1
15:36:04.233: debug : virRegisterDeviceMonitor:827 : registering remote as device driver 1
15:36:04.233: debug : virRegisterSecretDriver:858 : registering remote as secret driver 1
15:36:04.233: debug : virRegisterNWFilterDriver:889 : registering remote as network filter driver 1
15:36:04.233: debug : virConnectOpenAuth:1499 : name=qemu+ssh://<email address hidden>/system, auth=0xe7d778, flags=0
15:36:04.233: debug : do_open:1205 : name "qemu+ssh://<email address hidden>/system" to URI components:
  scheme qemu+ssh
  opaque (null)
  authority (null)
  server kavasir.XXXXXXXXXXX
  user rsgadmin
  port 0
  path /system

15:36:04.233: debug : do_open:1244 : trying driver 0 (Test) ...
15:36:04.233: debug : do_open:1250 : driver 0 Test returned DECLINED
15:36:04.233: debug : do_open:1244 : trying driver 1 (Xen) ...
15:36:04.233: debug : do_open:1250 : driver 1 Xen returned DECLINED
15:36:04.233: debug : do_open:1244 : trying driver 2 (OPENVZ) ...
15:36:04.233: debug : do_open:1250 : driver 2 OPENVZ returned DECLINED
15:36:04.233: debug : do_open:1244 : trying driver 3 (remote) ...
15:36:04.233: debug : doRemoteOpen:565 : proceeding with name = qemu:///system
15:36:04.234: debug : virExecWithHook:712 : ssh -l rsgadmin kavasir.XXXXXX sh -c 'nc -q 2>&1 | grep -q "requires an argument";if [ $? -eq 0 ] ; then CMD="nc -q 0 -U /var/run/libvirt/libvirt-sock";else CMD="nc -U /var/run/libvi...

[Serge Hallyn (serge-hallyn)]

@Tim,

please open a new bug for your problem.

[Max Maass (maxjmaass)]

Just wanted to let you know that this bug not only appears when you have the special SSH Option set, but also if you have any form of message inside your .bashrc or similar files. I got a small banner in my .bashrc, and connections via virt-manager / virsh reproducably will only work if I comment it out or rename the .bashrc so it will not get executed.

[Jonathan Carter (jonathan)]

Thanks Max that solved this issue for me as well.