swift unable to start in gate-tempest-dsvm-neutron-large-ops

[Joe Gordon]

http://logstash.openstack.org/#eyJzZWFyY2giOiJcInNvY2tldC5lcnJvcjogW0Vycm5vIDExMV0gQ29ubmVjdGlvbiByZWZ1c2VkXCIgQU5EIGZpbGVuYW1lOlwibG9ncy9zY3JlZW4tcy1jb250YWluZXIudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzODc0OTM4ODM0MTYsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0=

Starting December 19th

"socket.error: [Errno 111] Connection refused" AND filename:"logs/screen-s-container.txt"

http://logs.openstack.org/70/62770/4/check/gate-tempest-dsvm-neutron-large-ops/d5f6251/

[Samuel Merritt (torgomatic)]

Based on that stack trace, it's definitely swift attempting to connect to syslog that's causing the error in screen-s-container.txt.

[Joe Gordon (jogo)]

This is only happening in gate-tempest-dsvm-neutron-large-ops which is why this is filed under neutron.

[Anita Kuno (anteaya)]

In this log file http://logs.openstack.org/70/62770/4/check/gate-tempest-dsvm-neutron-large-ops/d5f6251/console.html#_2013-12-19_22_00_00_881
there is a three second gap in the log. This indicates that the cpu was not available for 3 seconds.

This job ran on hpcloud-az2 Linux 3.2.0-57-virtual (devstack-precise-hpcloud-az2-899237) 12/19/2013 _x86_64_ (4 CPU) which may have been having host issues at this time.

It will be interesting to see the result of running this job on a different host.

[Joe Gordon (jogo)]

After further investigation, it looks like rsyslog is restarted during nova-manage db_sync , and for some reason when running gate-tempest-dsvm-neutron-large-op it doesn't restart properly.

Nov 26 00:45:36 devstack-precise-check-rax-iad-737920 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="3834" x-info="http://www.rsyslog.com"] exiting on signal 15.

[Joe Gordon (jogo)]

marking as critical because this is a gate issue, and it appears to be related to nova (at least indirectly)

[Davanum Srinivas (DIMS) (dims-v)]

setup_host method in [1] seems to have a stop/start of rsyslog and is called twice!

[1] https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh

[James E. Blair (corvus)]

Yes, I noticed that and have a patch up to fix that: https://review.openstack.org/#/c/62983/

However, as all of that happens well before devstack starts (and on every job) I don't believe it's the problem.

[Joe Gordon (jogo)]

Jim got a hold of a box having this problem and we were able to do a bit more digging.

The nova-manage db_sync was a red herring, the timestamps in console.html and syslog.txt our out of sync.

We set up devstack to not use rsyslog, but swift is still using it. So for now the solution is to make swift not use syslog when we set SYSLOG=False in localrc.

It looks like there is possibly a strange interaction between
openvswitch and syslog that is making syslog not shutdown properly

http://paste.openstack.org/show/55760/

http://paste.openstack.org/show/55767/

[Davanum Srinivas (DIMS) (dims-v)]

Strange. Don't see any hits in last 12 hours!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/81663

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/81663
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=ad7e8c63e6891c59eb4387a01d94838f60370930
Submitter: Jenkins
Branch: master

commit ad7e8c63e6891c59eb4387a01d94838f60370930
Author: Sean Dague <email address hidden>
Date: Wed Mar 19 19:13:20 2014 -0400

    move the rsyslogd restart

    it's not clear why swift start is the place where an rsyslogd start
    is happening, we should really only make this change when we actually
    change a file on disk.

    Also, use rsyslogd's -HUP signal directly instead of the system init
    scripts which are typically doing a stop and start, and apparently
    racing under some circumstances.

    Change-Id: I1b9891313d67b1da2ca2582e532b2536a81f9b25
    Closes-Bug: #1262906

[Cedric Brandily (cbrandily)]

This bug is > 365 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[Armando Migliaccio (armando-migliaccio)]

This bug is > 240 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Neutron bug closed due to lack of activity, please feel free to reopen if needed.