traceback is suppressed when deploy.loadapp fails

[Jay Buffington]

I saw this error when attempt to start a relatively recent quantum (setup.py --version says "2013.2.a782.ga36f237"):
     ERROR: Unable to load quantum from configuration file /etc/quantum/api-paste.ini.

After running quantum-server through strace I determined that the error was due to missing mysql client libraries:

    ...
    open("/lib64/tls/libmysqlclient.so.18", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/lib64/libmysqlclient.so.18", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib64/tls/libmysqlclient.so.18", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib64/libmysqlclient.so.18", O_RDONLY) = -1 ENOENT (No such file or directory)
    munmap(0x7ffcd8132000, 34794) = 0
    munmap(0x7ffccd147000, 2153456) = 0
    close(4) = 0
    close(3) = 0
    write(2, "ERROR: Unable to load quantum fr"..., 95ERROR: Unable to load quantum from configuration file /usr/local/csi/etc/quantum/api-paste.ini.) = 95
    write(2, "\n", 1 ) = 1
    rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x3eec80f500}, {0x3eef90db70, [], SA_RESTORER, 0x3eec80f500}, 8) = 0
    exit_group(1)

The error message is completely bogus and the lack of traceback made it difficult to debug.

This is a regression from commit 6869821 which was to fix related bug 1004062

[Jay Buffington (jaybuff)]

This happened to me again today. I had this line in api-paste.ini (note the incorrect capitalisation)

paste.filter_factory = neutron.auth:neutronKeystoneContext.factory

and got this unhelpful error message:

ERROR: Unable to load quantum from configuration file /usr/local/csi/etc/neutron/api-paste.ini.

Instead of this helpful one:

2013-08-22 23:56:33,190 (neutron): CRITICAL log logging_excepthook <module 'neutron.auth' from '/usr/local/csi/share/csi-neutron.venv/lib/python2.6/site-packages/neutron/auth.py'> has no 'neutronKeystoneContext' attribute

[Salvatore Orlando (salvatore-orlando)]

I will triage it. Hopefully should be an easy one.

[ZhiQiang Fan (aji-zqfan)]

there is a typo in your config file

paste.filter_factory = neutron.auth:neutronKeystoneContext.factory
->
paste.filter_factory = neutron.auth:NeutronKeystoneContext.factory

[ZhiQiang Fan (aji-zqfan)]

sorry, i misunderstand your question

[Doug Schaapveld (djschaap)]

I believe I saw the same thing, and can confirm the only error seen (Unable to load quantum from configuration file /etc/quantum/api-paste.ini) and fix (yum install mysql) running RDO grizzly on CentOS 6. This was seen on a freshly-built system dedicated to quantum (no other OpenStack components, using remote database), perhaps explaining why this isn't a common problem.

Making the openstack-quantum-openvswitch package dependent on the MySQL client/libraries would solve this -- but may not be completely correct, because by default ovs_quantum_plugin.ini specifies sqlite://.

And, back to the initial bug description -- logging the fault (missing MySQL library) may be the best answer.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/51696

[Armando Migliaccio (armando-migliaccio)]

or havana backport potential

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/51696
Committed: http://github.com/openstack/neutron/commit/bf1b5b88792ce0505c8ea8b6b50cff1970ce42c5
Submitter: Jenkins
Branch: master

commit bf1b5b88792ce0505c8ea8b6b50cff1970ce42c5
Author: armando-migliaccio <email address hidden>
Date: Mon Oct 14 12:37:24 2013 -0700

    Avoid suppressing underlying error when deploy.loadapp fails

    Raise log level at the first load attempt. If there is a
    fundamental problem with the environment, a Lookup error
    is not helpful in troubleshooting the problem further.

    Fix bug 1210236

    Change-Id: I55aa11a0e5a37dbef3f0c1b55aa4c9c11bbf55d9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/54956

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/54956
Committed: http://github.com/openstack/neutron/commit/58ff51d62ac1b24bd247570da416920f8f89120b
Submitter: Jenkins
Branch: stable/havana

commit 58ff51d62ac1b24bd247570da416920f8f89120b
Author: armando-migliaccio <email address hidden>
Date: Mon Oct 14 12:37:24 2013 -0700

    Avoid suppressing underlying error when deploy.loadapp fails

    Raise log level at the first load attempt. If there is a
    fundamental problem with the environment, a Lookup error
    is not helpful in troubleshooting the problem further.

    Fix bug 1210236

    (cherry picked from commit: 696e9f3fc8ef7e8bd2a8f21071f3ef91fac14ad5)

    Change-Id: I55aa11a0e5a37dbef3f0c1b55aa4c9c11bbf55d9

[ephem (tpiperatgod)]

After i set the app_name='neutron' in func _run_wsgi of neutron.service.py, the neutron-server has be ok