grenade nova network (n-net) fails to start

[clayg]

Here we see that n-net never started logging to it's screen:

http://logs.openstack.org/02/91502/1/check/check-grenade-dsvm/912e89e/logs/new/

The errors in n-cpu seem to support that the n-net service never started.

According to http://logs.openstack.org/02/91502/1/check/check-grenade-dsvm/912e89e/logs/grenade.sh.log.2014-05-01-042623, circa "2014-05-01 04:31:15.580" the interesting bits should be in:

/opt/stack/status/stack/n-net.failure

But I don't see that captured.

I'm not sure why n-net did not start.

[Matt Riedemann (mriedem)]

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiTWVzc2FnaW5nVGltZW91dDogVGltZWQgb3V0IHdhaXRpbmcgZm9yIGEgcmVwbHkgdG8gbWVzc2FnZSBJRFwiIEFORCBtZXNzYWdlOlwibmV0d29ya1wiIEFORCB0YWdzOlwic2NyZWVuLW4tY3B1LnR4dFwiICIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5OTE2NzQ0OTY4M30=

[Matt Riedemann (mriedem)]

It's going to be hard to debug any of the service start failures in the grenade jobs without the *.failure logs, which I thought was fixed with this:

https://review.openstack.org/#/c/98885/

But they still don't show up:

http://logs.openstack.org/44/99144/1/gate/gate-grenade-dsvm/3be9510/logs/devstack-gate-cleanup-host.txt

2014-06-13 14:31:00.111 | ++ ls '/opt/stack/status/stack/*.failure'
2014-06-13 14:31:00.112 | ls: cannot access /opt/stack/status/stack/*.failure: No such file or directory

Although the console log says it's writing to them:

http://logs.openstack.org/44/99144/1/gate/gate-grenade-dsvm/3be9510/console.html

Removes them here first:

2014-06-13 14:01:24.631 | 2014-06-13 14:01:24.619 | + rm -f '/opt/stack/status/stack/*.failure'

And we set n-net failed to start here:

http://logs.openstack.org/44/99144/1/gate/gate-grenade-dsvm/3be9510/console.html#_2014-06-13_14_03_24_026

2014-06-13 14:03:24.026 | 2014-06-13 14:03:23.989 | + screen -S stack -p n-net -X stuff 'cd /opt/stack/old/nova && /usr/local/bin/nova-network --config-file /etc/nova/nova.conf & echo $! >/opt/stack/status/stack/n-net.pid; fg || echo "n-net failed to start" | tee "/opt/stack/status/stack/n-net.failure"

[Ian Wienand (iwienand)]

Unfortunately the links referenced are gone, but

---
2014-06-13 14:03:24.026 | 2014-06-13 14:03:23.989 | + screen -S stack -p n-net -X stuff 'cd /opt/stack/old/nova && /usr/local/bin/nova-network --config-file /etc/nova/nova.conf & echo $! >/opt/stack/status/stack/n-net.pid; fg || echo "n-net failed to start" | tee "/opt/stack/status/stack/n-net.failure"
---

will only write out the .failure file if the "fg" brings the started process to the foreground and it exits. Otherwise, I think the assumption must be that nova-network has started and is hung ... it's not logging to it's regular file but the process hasn't exited.

A ps dump in the final output would maybe help here

[Attila Fazekas (afazekas)]

The same logstash query founds this error in another jobs, and they are runtime issues.

http://logs.openstack.org/94/105194/2/check/check-tempest-dsvm-postgres-full/f029225/logs/screen-n-net.txt.gz#_2014-07-07_17_24_19_012

[Sean Dague (sdague)]

I think this is a screen issue

[Sean Dague (sdague)]

This grenade bug was last updated over 180 days ago, as grenade
is a fast moving project and we'd like to get the tracker down to
currently actionable bugs, this is getting marked as Invalid. If the
issue still exists, please feel free to reopen it.