virsh error: error: no valid connection error: Cannot create user runtime directory '/run/user/1000/libvirt': Permission denied

[wes hayutin]

Shutting down all the nodes created by quickstart will result in virsh errors and the inability to further control guests from the stack user.

[stack@localhost ~]$ virsh list --all
 Id Name State
----------------------------------------------------
 2 undercloud running
 - compute_0 shut off
 - compute_1 shut off
 - control_0 shut off

[stack@localhost ~]$ virsh shutdown undercloud
Domain undercloud is being shutdown

[stack@localhost ~]$ virsh list --all
 Id Name State
----------------------------------------------------
 2 undercloud running
 - compute_0 shut off
 - compute_1 shut off
 - control_0 shut off

[stack@localhost ~]$ virsh list --all
 Id Name State
----------------------------------------------------
 2 undercloud running
 - compute_0 shut off
 - compute_1 shut off
 - control_0 shut off

[stack@localhost ~]$ virsh list --all
error: failed to connect to the hypervisor
error: no valid connection
error: Cannot create user runtime directory '/run/user/1000/libvirt': Permission denied

[John Trowbridge (trown)]

I have been able to reproduce this behavior. Not sure about the root cause yet.

[Lars Kellogg-Stedman (larsks)]

I am having trouble triggering this behavior. Can (either of) you post a set of steps that reproduces it for you?

[Yolanda Robla (yolanda.robla)]

I have been able to reproduce as well. I only could troubleshoot by rebooting the server.

[James Slagle (james-slagle)]

i've reproduced the same error. my steps were:

new centos 7 install on my virt host

sudo yum -y update
git clone https://git.openstack.org/openstack/tripleo-quickstart
cd tripleo-quickstart/
sudo bash quickstart.sh --install-deps
bash quickstart.sh --config config/general_config/devmode.yml --release master-tripleo localhost

This fails with:
TASK [setup/overcloud : Define volume pool] ************************************
task path: /home/jslagle/tripleo-quickstart/roles/libvirt/setup/overcloud/tasks/main.yml:27
Wednesday 06 July 2016 08:20:27 -0400 (0:00:00.403) 0:02:11.707 ********
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["virsh", "pool-define", "/home/jslagle/.quickstart/volume_pool.xml"], "delta": "0:00:00.013418", "end": "2016-07-06 08:20:27.782413", "failed": true, "rc": 1, "start": "2016-07-06 08:20:27.768995", "stderr": "error: failed to connect to the hypervisor\nerror: no valid connection\nerror: Cannot create user runtime directory '/run/user/1001/libvirt': Permission denied", "stdout": "", "stdout_lines": [], "warnings": []}

I assume using localhost for $VIRTHOST should be ok?

[James Slagle (james-slagle)]

note that logout/login got me past this error. i hit something different on the next run though. filed a new bug for that: https://bugs.launchpad.net/tripleo-quickstart/+bug/1599509

[wes hayutin (weshayutin)]

So to get you past your current error, add --teardown all to your invocation. I'm assuming this is just a testbox and you don't have any other vms you want to save. Using localhost is fine for the virthost

[Ronelle Landy (rlandy)]

I can reproduce this problem.
Working around this problem by changing permissions on the /run/user/

[John Trowbridge (trown)]

Is this still an issue? I have not seen this recently.

[Luke Hinds (lhinds)]

I just ran into this:

[stack@rhlab ~]$ virsh list --all
error: failed to connect to the hypervisor
error: no valid connection
error: Cannot create user runtime directory '/run/user/1001/libvirt': Permission denied

For me the root cause seemed to be related somehow to me destroying / undefining the domains manually (where as I should have used --teardown).

[Luke Hinds (lhinds)]

Update....after another run of quickstart.sh with --teardown, all is well again:

[stack@rhlab ~]$ virsh list --all
 Id Name State
----------------------------------------------------
 2 undercloud running
 - compute_0 shut off
 - control_0 shut off

[Honza Pokorny (hpokorny)]

I'm still experiencing this issue when running quickstart on a clean machine, no VMs defined.

[Ronelle Landy (rlandy)]

This comes up every now and again on IRC questions.
Do we have a suggested workaround? or fix?

[Alex Schultz (alex-schultz)]

This can show up if XDG_RUNTIME_DIR is set in your user's .bashrc. You may need to remove it if you're having problems.

https://github.com/openstack/tripleo-quickstart/blob/eb3c5012e602465b0429bc414210ecb552c6803c/roles/provision/remote/tasks/main.yml#L81-L95

[Ronelle Landy (rlandy)]

Add section to troubleshooting doc:

If you hit permission denied error, run again with --teardown.

[Matthias Runge (mrunge)]

I was able to trigger this with a fresh and baremetal CentOS install

[Matthias Runge (mrunge)]

After removing the mentioned patch in https://github.com/openstack/tripleo-quickstart/blob/eb3c5012e602465b0429bc414210ecb552c6803c/roles/provision/remote/tasks/main.yml#L81-L95
I was able to workaround this.

[Alex Schultz (alex-schultz)]

https://review.openstack.org/#/c/503174/ is up to remove the XDG_RUNTIME_DIR workaround in quickstart. so we should close it after that merges

[Michele Baldessari (michele)]

https://review.openstack.org/#/c/503174/ has merged, so closing this.