failed to retrieve chardev info with 'info chardev'

[Jordan Rinke]

Fresh server (Ubuntu 10.10), installed from trunk (nova-compute: 2011.1~bzr562-0ubuntu0ppa1~maverick1
), Added qemu-nbd to sudoers list per bug #702723

This occurs when trying to launch a new instance.

(nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): INFO [N/A] instance instance-00000014: Creating image
(nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): INFO [N/A] instance instance-00000014: injecting key into image ami-tty
(nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): WARNING [N/A] instance instance-00000014: ignoring error injecting data into image ami-tty (nbd device /dev/nbd2 did not show up)
libvir: QEMU error : operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.exception 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): ERROR [N/A] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 82, in _wrap
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 382, in spawn
(nova.exception): TRACE: self._conn.createXML(xml, 0)
(nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1289, in createXML
(nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
(nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.exception): TRACE:
(nova.compute.manager 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): ERROR [ZL6YAREVPQG6RLHDA1QX admin admin] instance 20: Failed to spawn
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 208, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 88, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.compute.manager): TRACE:
libvir: QEMU error : Domain not found: no domain with matching name 'instance-00000014'

[Vish Ishaya (vishvananda)]

The nbd issue seems unrelated to the info chardev error. Did you modprobe nbd?

Vish

On Jan 13, 2011, at 9:25 PM, Jordan Rinke wrote:

> Public bug reported:
>
> Fresh server (Ubuntu 10.10), installed from trunk (nova-compute: 2011.1~bzr562-0ubuntu0ppa1~maverick1
> ), Added qemu-nbd to sudoers list per bug #702723
>
> This occurs when trying to launch a new instance.
>
> (nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): INFO [N/A] instance instance-00000014: Creating image
> (nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): INFO [N/A] instance instance-00000014: injecting key into image ami-tty
> (nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): WARNING [N/A] instance instance-00000014: ignoring error injecting data into image ami-tty (nbd device /dev/nbd2 did not show up)
> libvir: QEMU error : operation failed: failed to retrieve chardev info in qemu with 'info chardev'
> (nova.exception 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): ERROR [N/A] Uncaught exception
> (nova.exception): TRACE: Traceback (most recent call last):
> (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 82, in _wrap
> (nova.exception): TRACE: return f(*args, **kw)
> (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 382, in spawn
> (nova.exception): TRACE: self._conn.createXML(xml, 0)
> (nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1289, in createXML
> (nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
> (nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
> (nova.exception): TRACE:
> (nova.compute.manager 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): ERROR [ZL6YAREVPQG6RLHDA1QX admin admin] instance 20: Failed to spawn
> (nova.compute.manager): TRACE: Traceback (most recent call last):
> (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 208, in run_instance
> (nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
> (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 88, in _wrap
> (nova.compute.manager): TRACE: raise Error(str(e))
> (nova.compute.manager): TRACE: Error: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
> (nova.compute.manager): TRACE:
> libvir: QEMU error : Domain not found: no domain with matching name 'instance-00000014'
>
> ** Affects: nova
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/702741
>
> Title:
> nbd device did not show up
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> Fresh server (Ubuntu 10.10), installed from trunk (nova-compute: 2011.1~bzr562-0ubuntu0ppa1~maverick1
> ), Added qemu-nbd to sudoers list per bug #702723
>
> This occurs when...

[Jordan Rinke (jordanrinke)]

lsmod shows nbd is installed:

root@00000r100020001:/etc# lsmod|grep -i "nbd"
nbd 9927 0

[Jordan Rinke (jordanrinke)]

I must have done it while troubleshooting, the nbd error no longer seems to occur so I do believe this is two dif issues as you said.

[Vish Ishaya (vishvananda)]

Seems like a permissions issue. I would suspect:
a) console.log isn't readable due to being a subdirectory that nova doesn't have permissions for (i.e. /root/...)
or
b) there is a permissions issue with the new configuration of nova web console trying to connect to a device
(you could test this by removing the offending line from the libvirt.xml in the instance directory)
or
c) there is some other strange issue with regards to kvm permissions

You may get more info by going to the instance directory and trying to run virsh create libvirt.xml. (if it runs properly as root then also try it as the nova user)

Vish

On Jan 13, 2011, at 10:49 PM, Jordan Rinke wrote:

> I must have done it while troubleshooting, the nbd error no longer seems
> to occur so I do believe this is two dif issues as you said.
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/702741
>
> Title:
> nbd device did not show up
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> Fresh server (Ubuntu 10.10), installed from trunk (nova-compute: 2011.1~bzr562-0ubuntu0ppa1~maverick1
> ), Added qemu-nbd to sudoers list per bug #702723
>
> This occurs when trying to launch a new instance.
>
> (nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): INFO [N/A] instance instance-00000014: Creating image
> (nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): INFO [N/A] instance instance-00000014: injecting key into image ami-tty
> (nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): WARNING [N/A] instance instance-00000014: ignoring error injecting data into image ami-tty (nbd device /dev/nbd2 did not show up)
> libvir: QEMU error : operation failed: failed to retrieve chardev info in qemu with 'info chardev'
> (nova.exception 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): ERROR [N/A] Uncaught exception
> (nova.exception): TRACE: Traceback (most recent call last):
> (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 82, in _wrap
> (nova.exception): TRACE: return f(*args, **kw)
> (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 382, in spawn
> (nova.exception): TRACE: self._conn.createXML(xml, 0)
> (nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1289, in createXML
> (nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
> (nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
> (nova.exception): TRACE:
> (nova.compute.manager 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): ERROR [ZL6YAREVPQG6RLHDA1QX admin admin] instance 20: Failed to spawn
> (nova.compute.manager): TRACE: Traceback (most recent call last):
> (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 208, in run_instance
> (nova.compute.manager): TRACE: self.driver.spawn(i...

[Jordan Rinke (jordanrinke)]

root@00000r100020001:/var/lib/nova/instances/instance-00000017# virsh -d 5 create libvirt.xml
command: "create libvirt.xml "
create: file(DATA): libvirt.xml
error: Failed to create domain from libvirt.xml
error: operation failed: failed to retrieve chardev info in qemu with 'info chardev'

snip
<disk type='file'>
            <driver type='qcow2'/>
            <source file='/var/lib/nova/instances/instance-00000017/disk'/>
            <target dev='vda' bus='virtio'/>
        </disk>
        <disk type='file'>
            <driver type='qcow2'/>
            <source file='/var/lib/nova/instances/instance-00000017/local'/>
            <target dev='vdb' bus='virtio'/>
        </disk>
/snip

The problem was the libvirt.xml had the driver type qcow2, but the disk images were raw. Changing the driver type to raw allowed the specific instance to start manually. Not sure what this implies, if all previous images need to be converted or if there needs to be detection etc.

Suggestion?

[Vish Ishaya (vishvananda)]

So this is actually an app-armor issue. Apparently it blocks access to backing files if the filename of the front file doesn't match its crazy list in /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper. 'disk' is in the list so it works, but 'local' is not.

[Thierry Carrez (ttx)]

I did not reproduce the issue, running bzr569 with use_ipv6=False I could start instances alright (with use_ipv6=True I hit bug 703037)

[Vish Ishaya (vishvananda)]

If you run as root from your home dir it works. Running as nova from
the packages blows it up

On Tuesday, January 18, 2011, Thierry Carrez <email address hidden> wrote:
> I did not reproduce the issue, running bzr569 with use_ipv6=False I
> could start instances alright (with use_ipv6=True I hit bug 703037)
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/702741
>
> Title:
>  failed to retrieve chardev info with 'info chardev'
>
> Status in OpenStack Compute (Nova):
>  In Progress
>
> Bug description:
>  Fresh server (Ubuntu 10.10), installed from trunk (nova-compute: 2011.1~bzr562-0ubuntu0ppa1~maverick1
>  ), Added qemu-nbd to sudoers list per bug #702723
>
>  This occurs when trying to launch a new instance.
>
>  (nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): INFO [N/A] instance instance-00000014: Creating image
>  (nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): INFO [N/A] instance instance-00000014: injecting key into image ami-tty
>  (nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): WARNING [N/A] instance instance-00000014: ignoring error injecting data into image ami-tty (nbd device /dev/nbd2 did not show up)
>  libvir: QEMU error : operation failed: failed to retrieve chardev info in qemu with 'info chardev'
>  (nova.exception 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): ERROR [N/A] Uncaught exception
>  (nova.exception): TRACE: Traceback (most recent call last):
>  (nova.exception): TRACE:   File "/usr/lib/pymodules/python2.6/nova/exception.py", line 82, in _wrap
>  (nova.exception): TRACE:     return f(*args, **kw)
>  (nova.exception): TRACE:   File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 382, in spawn
>  (nova.exception): TRACE:     self._conn.createXML(xml, 0)
>  (nova.exception): TRACE:   File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1289, in createXML
>  (nova.exception): TRACE:     if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
>  (nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
>  (nova.exception): TRACE:
>  (nova.compute.manager 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): ERROR [ZL6YAREVPQG6RLHDA1QX admin admin] instance 20: Failed to spawn
>  (nova.compute.manager): TRACE: Traceback (most recent call last):
>  (nova.compute.manager): TRACE:   File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 208, in run_instance
>  (nova.compute.manager): TRACE:     self.driver.spawn(instance_ref)
>  (nova.compute.manager): TRACE:   File "/usr/lib/pymodules/python2.6/nova/exception.py", line 88, in _wrap
>  (nova.compute.manager): TRACE:     raise Error(str(e))
>  (nova.compute.manager): TRACE: Error: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
>  (nova.compute.manager): TRACE:
>  libvir: QEMU error : Domain not found: no domain with matching name 'instance-00000014'
>
>
>

[Thierry Carrez (ttx)]

Hrm. But I run as nova from the packages :) Ubuntu 10.10, apparmor-loaded libvirt.

[Vish Ishaya (vishvananda)]

Very odd, This was specifically happening on an AMD box, but i can't imagine that made a difference. Maybe it was due to installing libvirt from ubuntu and then upgrading to the ppa version?

Vish

On Jan 18, 2011, at 12:17 PM, Thierry Carrez wrote:

> Hrm. But I run as nova from the packages :) Ubuntu 10.10, apparmor-
> loaded libvirt.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/702741
>
> Title:
> failed to retrieve chardev info with 'info chardev'
>
> Status in OpenStack Compute (Nova):
> In Progress
>
> Bug description:
> Fresh server (Ubuntu 10.10), installed from trunk (nova-compute: 2011.1~bzr562-0ubuntu0ppa1~maverick1
> ), Added qemu-nbd to sudoers list per bug #702723
>
> This occurs when trying to launch a new instance.
>
> (nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): INFO [N/A] instance instance-00000014: Creating image
> (nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): INFO [N/A] instance instance-00000014: injecting key into image ami-tty
> (nova.virt.libvirt_conn 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): WARNING [N/A] instance instance-00000014: ignoring error injecting data into image ami-tty (nbd device /dev/nbd2 did not show up)
> libvir: QEMU error : operation failed: failed to retrieve chardev info in qemu with 'info chardev'
> (nova.exception 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): ERROR [N/A] Uncaught exception
> (nova.exception): TRACE: Traceback (most recent call last):
> (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 82, in _wrap
> (nova.exception): TRACE: return f(*args, **kw)
> (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 382, in spawn
> (nova.exception): TRACE: self._conn.createXML(xml, 0)
> (nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1289, in createXML
> (nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
> (nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
> (nova.exception): TRACE:
> (nova.compute.manager 2011.1-workspace:tarmac-20110114001914-5vbfe8sm72xr2py4): ERROR [ZL6YAREVPQG6RLHDA1QX admin admin] instance 20: Failed to spawn
> (nova.compute.manager): TRACE: Traceback (most recent call last):
> (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 208, in run_instance
> (nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
> (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 88, in _wrap
> (nova.compute.manager): TRACE: raise Error(str(e))
> (nova.compute.manager): TRACE: Error: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
> (nova.compute.manager): TRACE:
> libvir: QEMU error : Domain not found: no domain with matching name 'instance-00000014'
>
>

[Thierry Carrez (ttx)]

Maybe. Don't waste time on understanding why it works for me, if you know why it fails for everyone else :)