Command: 'openstack service create --format shell identity --name keystone --description OpenStack Identity Service' has been running for more than 10 seconds!

[Anastasia Palkina]

1. Create new environment
2. Add 3 controllers and 2 computes
3. Start deployment. It was successful
4. Run OSTF test. It was successful
5. Add 1 cinder
6. Redeploy environment. It was successful
7. Remove the 'cinder' node and remove a 'controller' node
8. Add 'controller'+'cinder' node
9. Start deployment
10. It has failed with error in puppet.log on primary controller (node-8):

2015-09-08 12:55:01 ERR (/Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone]/ensure) change from absent to present failed: Command: 'openstack service create --format shell identity --name keystone --description OpenStack Identity Service' has been running for more then 10 seconds!

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaSXFOUHNiaEdROHc/view?usp=sharing

{"build_id": "287", "build_number": "287", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "287", "build_number": "287", "api": "1.0", "fuel-library_sha": "43224223dab8cf9627b5ecf737e60216a3fdd114", "nailgun_sha": "46a7a2177a0b7ef91422284c1c90295fee8f5c84", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac", "astute_sha": "a717657232721a7fafc67ff5e1c696c9dbeb0b95", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "43224223dab8cf9627b5ecf737e60216a3fdd114", "nailgun_sha": "46a7a2177a0b7ef91422284c1c90295fee8f5c84", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac", "astute_sha": "a717657232721a7fafc67ff5e1c696c9dbeb0b95", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"}

[Boris Bobrov (bbobrov)]

A failure during deployment has nothing to do with logrotate-related bug. I am removing the duplicate-of.

[Boris Bobrov (bbobrov)]

Right now I am in progress of finding a good sleep interval between launching `apache2ctl graceful`. Results will be discussed in an hour with Sergii Golovatiuk and however joins the call.

[Boris Bobrov (bbobrov)]

No segfaults happened when `apache2ctl graceful` was called every 30 seconds. The recommendation is to run `apache2ctl graceful` not more often than 30 seconds.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/222360

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/222362

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/222360
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=92a4ed5bb811c62263d72cf76b8cbd7a719fed3f
Submitter: Jenkins
Branch: master

commit 92a4ed5bb811c62263d72cf76b8cbd7a719fed3f
Author: Vladimir Kuklin <email address hidden>
Date: Thu Sep 10 23:44:48 2015 +0300

    Add workaround for apache restart during deployment

    As figured out in the following launcpad bug
    https://bugs.launchpad.net/fuel/+bug/1493353
    Apache2 is not very friendly to any types of
    restarts. This may lead to the issue when we
    restart apache to often during deployment and
    it gets into Byzantine unresponsive state
    along with keystone which breaks keystone
    providers operations.

    Change-Id: I11c52089e9598fc6d088c3478c90de3aa853652a
    Closes-bug: #1493372
    Related-bug: #1493353

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/222362
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d4723a87bee9ab1a958bdec174b222a0fcd05d5c
Submitter: Jenkins
Branch: stable/7.0

commit d4723a87bee9ab1a958bdec174b222a0fcd05d5c
Author: Vladimir Kuklin <email address hidden>
Date: Thu Sep 10 23:44:48 2015 +0300

    Add workaround for apache restart during deployment

    As figured out in the following launcpad bug
    https://bugs.launchpad.net/fuel/+bug/1493353
    Apache2 is not very friendly to any types of
    restarts. This may lead to the issue when we
    restart apache to often during deployment and
    it gets into Byzantine unresponsive state
    along with keystone which breaks keystone
    providers operations.

    Change-Id: I11c52089e9598fc6d088c3478c90de3aa853652a
    Closes-bug: #1493372
    Related-bug: #1493353

[Dennis Dmitriev (ddmitriev)]

Reproduced on CI:
https://product-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.ha_neutron_tun/38/testReport/%28root%29/neutron_tun_ha_addremove/
https://product-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.controller_replacement/55/testReport/%28root%29/deploy_ha_neutron_vlan_ctrl_replacement/

Scenario for deploy_ha_neutron_vlan_ctrl_replacement:
  1. Create cluster with neutron vlan
  2. Add 3 node with controller role
  3. Add 1 node with compute
  3. Deploy cluster
  4. Remove a primary controller and add new controller
  5. Deploy changes

On step 5, deploy failed:

node-1 2015-09-13T05:20:08.035454 notice: (Scope(Class[main])) MODULAR: keystone.pp
node-1 2015-09-13T05:20:12.475120 notice: Compiled catalog for node-1.test.domain.local in environment production in 4.47 seconds
node-1 2015-09-13T05:22:04.996102 err: Could not prefetch keystone_service provider 'openstack': Command: 'openstack service list --quiet --format csv --long' has been running for more then 10 seconds!
...
, and than a lot of 10 seconds timeouts in other commands.

[Dennis Dmitriev (ddmitriev)]

One more test was failed with the same issue on CI:
https://product-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.ha_neutron_tun_scale/3/testReport/%28root%29/neutron_tun_scalability/

Scenario:
            1. Create cluster
            2. Add 1 controller node
            3. Deploy the cluster
            4. Add 2 controller nodes
            5. Deploy changes
            6. Run network verification
            7. Add 2 controller 1 compute nodes
            8. Deploy changes

Deploy failed on step 8:

node-1 2015-09-14T02:56:09.859026 notice: (Scope(Class[main])) MODULAR: apache.pp
node-1 2015-09-14T02:56:11.994354 notice: Compiled catalog for node-1.test.domain.local in environment production in 2.14 seconds
node-1 2015-09-14T02:56:38.754152 err: (/Stage[main]/Apache::Service/Service[httpd]) Failed to call refresh: Could not start Service[httpd]: Execution of '/etc/init.d/apache2 start' returned 1:

...

node-1 2015-09-14T03:00:37.022528 notice: (Scope(Class[main])) MODULAR: keystone.pp
node-1 2015-09-14T03:00:40.441348 notice: Compiled catalog for node-1.test.domain.local in environment production in 3.45 seconds
node-1 2015-09-14T03:02:33.392631 err: Could not prefetch keystone_service provider 'openstack': Command: 'openstack service list --quiet --format csv --long' has been running for more then 10 seconds!

...

[Vladimir Kuklin (vkuklin)]

According to the logs apache keeps segfaulting its keystone childs even if we have 30 seconds sleep between graceful restarts. This has something to do either with keystone application or with apache itself. My suggestion is that it behaves this way due to issues with processing of changed memcached servers configuration during scaling of the nodes.

[Boris Bobrov (bbobrov)]

> My suggestion is that it behaves this way due to issues with processing of changed memcached servers configuration during scaling of the nodes.

I was asked to check whether presence of unreacheable memcache hosts has stomething to do with the bug. I tried adding ip 10.109.254.254 to the list of memcache hosts and to the list of rabbit hosts, verified that it's not reachable and restarted apache multiple times every 30 seconds. Nothing failed.

I am returning this to MOS-Linux.

[Boris Bobrov (bbobrov)]

We have reproduced it. Now we are trying get core dumps.

[Boris Bobrov (bbobrov)]

Root cause is still unknown, now we are trying some old versions of apache2.

We can reproduce it in 100% cases when adding/removing a controller, but cannot reproduce manually. We tried to get a core dump, but no luck.

[Boris Bobrov (bbobrov)]

Downgrade of some of he packages failed (Vladimir Kuklin can tell more about the reasons)

Tomorrow we are going to:

1. Try to get coredumps with packages that Vladimir is going to prepare for tomorrow;
2. Downgrading apache2 and mod-wsgi packages. We have a feeling that the issue might happen due to one of CVE-fixing patches and we are goint to try packages that were built before that patch. If that doesn't help, try packages from Ubuntu 12.04.

[Boris Bobrov (bbobrov)]

Got a coredump!

http://paste.openstack.org/show/461857/ -- gdb session. wsgi_server_config is null at the moment of referencing.

Thanks to Vladimir Kuklin who recompiled mod-wsgi for us.

[Artem Silenkov (asilenkov)]

I've started an issue in project's homeland
https://github.com/GrahamDumpleton/mod_wsgi/issues/90

[Boris Bobrov (bbobrov)]

There is bugreport https://trac.macports.org/ticket/44762 . It says that ``apache2ctl graceful'' is not enough and that apache2 should be stopped and then started. I asked Vasyl Saienko to verify this. Update is expected in an hour.

[Boris Bobrov (bbobrov)]

Still verifying, udpate in an hour.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Artem Silenkov <email address hidden>
Review: https://review.fuel-infra.org/11572

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11572
Submitter: Andrey Nikitin <email address hidden>
Branch: master

Commit: 3e4c67b65ff4abb18118e36380e2302fce0fce96
Author: Artem Silenkov <email address hidden>
Date: Tue Sep 15 12:04:37 2015

mod-wsgi added for trusty

Attempt to fix core dumps during reload operations

Change-Id: Ie50139272e12b79da0c1d6068b02864d2e6b7865
Related-Bug: #1493372

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 7.0
Change author: Artem Silenkov <email address hidden>
Review: https://review.fuel-infra.org/11574

[Vasyl Saienko (vsaienko)]

command "apachectl graceful" leads to apache hung[0], if we call puppet apply -vd /etc/puppet/modules/osnailyfacter/modular/apache/apache.pp; puppet apply -vd /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp
It is buggy apache code.

[0] https://paste.mirantis.net/show/1122/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/223733

[Vasyl Saienko (vsaienko)]

also I noticed that apachectl graceful leads to apache hung, only if previously apache was start with "service apache2 start".
Other words it is possible to add tweak to osnaylifacter/modular/apache/apache.pp and it should fix problem

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/223775

[Davanum Srinivas (DIMS) (dims-v)]

Vasyl,

Can you please tell me the manual steps you used to verify the two reviews? i understand it takes about 5-10 mins to recreate the problem and these fixes have been verified by hand already

Thanks,
Dims

[Vasyl Saienko (vsaienko)]

to quickly reproduce it call the following command on controller node:
puppet apply -vd /etc/puppet/modules/osnailyfacter/modular/apache/apache.pp; puppet apply -vd /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/223943

[Vasyl Saienko (vsaienko)]

Anastasia,

I've build two ISO with fixes for this problem:

1) http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_7.0_iso/1217/
2) http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_7.0_iso/1218/

Please check if you can reproduce this problem on provided ISO's

Thanks,
Vasyl

[Nastya Urlapova (aurlapova)]

http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/7.0.custom_system_test/755/console

Nastya Urlapova [12:16 PM]
http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/7.0.custom_system_test/756/console

[Davanum Srinivas (DIMS) (dims-v)]

The following bug showed up after the fixes for puppet in keystone:
https://bugs.launchpad.net/mos/+bug/1496512

Please see the url for a possible fix.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vasyl Saienko (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/223733
Reason: replaced with https://review.openstack.org/#/c/223943/

[Vasyl Saienko (vsaienko)]

After fixing: https://bugs.launchpad.net/mos/+bug/1496512
new bugs showed up:
https://bugs.launchpad.net/fuel/+bug/1496709
https://bugs.launchpad.net/fuel/+bug/1484066

[Eugene Bogdanov (ebogdanov)]

We need MOS-QA help to verify fixes. Timur, can you please help.

[Davanum Srinivas (DIMS) (dims-v)]

Notes that will be useful for MOS 8.0, we should stick to the puppet reviews above for MOS 7.0

So, digging into root cause of why mod_wsgi segfaults to have a better fix for MOS 8.0 led to the learning of intricacies of how "apachectl graceful" works with the modules.

mod_wsgi uses ap_hook_post_config to register a wsgi_hook_init which is supposed to fire when the module is initially loaded
https://github.com/GrahamDumpleton/mod_wsgi/blob/develop/src/server/mod_wsgi.c#L15087

in wsgi_hook_init, a global data structure wsgi_server_config is loaded with information:
https://github.com/GrahamDumpleton/mod_wsgi/blob/develop/src/server/mod_wsgi.c#L12360

Note that wsgi_hook_init has a hack to smartly side step the case where wsgi_hook_init is called twice (during apache start)
https://github.com/GrahamDumpleton/mod_wsgi/blob/develop/src/server/mod_wsgi.c#L12307

wsgi_hook_child_init is supposed to be called after wsgi_hook_init and uses the information in wsgi_server_config. Problem is that when graceful is triggered, the mod_wsgi.so is unloaded and reloaded BUT wsgi_hook_init is called just once! As documented in the apache wiki [1] this smart hack fails miserably for our case. As the wiki page eloquently says

"In this case, the post_config hook is executed once after restart and will only set the flag. Module initialization will not be performed, and when child_init and other hooks are called, bad things may happen."

Which is exactly why we fail when we try to use the wsgi_server_config datastructure in wsgi_hook_child_init during the graceful scenario.

Note that all pointers above are to the latest git of mod_wsgi, so it's still not been fixed in mod_wsgi. We need to provide a work around to mod_wsgi community that allows the behavior we need which is load everything the first time wsgi_hook_init is called and don't do anything if it is called again.

[1] https://wiki.apache.org/httpd/ModuleLife

[Vladimir Kuklin (vkuklin)]

Just to provide the latest info on what to do after that epic battle with apache - the best fix for apache is to remove apache. I think we need to switch to uwsgi + nginx instead of apache and use twemproxy as a unique memcached/Redis proxy in order to not use openstack buggy key-value pooling code.

[Timur Nurlygayanov (tnurlygayanov)]

On custom ISO with the fixes: http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_7.0_iso/1225/

Steps:
1. Deploy cluster with 3 controllers, 1 compute node, Ceph OSD nodes on controllers, Neutron VLANs. Other configuration by default.
2. Run Smoke and sanity OSTF tests - it will pass.
3. Run the following commands on master node:
puppet apply -vd /etc/puppet/modules/osnailyfacter/modular/apache/apache.pp; puppet apply -vd /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp

Observed Result:
Info: /Stage[main]/Apache/Concat::Fragment[Apache ports header]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_Apache ports header]: Scheduling refresh of Exec[concat_/etc/httpd/conf/ports.conf]
Debug: /Stage[main]/Apache/Concat::Fragment[Apache ports header]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_Apache ports header]: The container Concat::Fragment[Apache ports header] will propagate my refresh event
Debug: Concat::Fragment[Apache ports header]: The container Class[Apache] will propagate my refresh event
Notice: /Stage[main]/Osnailyfacter::Apache/Osnailyfacter::Apache::Apache_port[80]/Apache::Listen[80]/Concat::Fragment[Listen 80]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_Listen 80]/ensure: defined content as '{md5}d5fcefc335117f400d451de47efeca87'
Info: /Stage[main]/Osnailyfacter::Apache/Osnailyfacter::Apache::Apache_port[80]/Apache::Listen[80]/Concat::Fragment[Listen 80]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_Listen 80]: Scheduling refresh of Exec[concat_/etc/httpd/conf/ports.conf]
Debug: /Stage[main]/Osnailyfacter::Apache/Osnailyfacter::Apache::Apache_port[80]/Apache::Listen[80]/Concat::Fragment[Listen 80]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_Listen 80]: The container Concat::Fragment[Listen 80] will propagate my refresh event
Notice: /Stage[main]/Apache/Concat[/etc/httpd/conf/ports.conf]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments.concat]/ensure: created
Debug: /Stage[main]/Apache/Concat[/etc/httpd/conf/ports.conf]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments.concat]: The container Concat[/etc/httpd/conf/ports.conf] will propagate my refresh event
Notice: /Stage[main]/Osnailyfacter::Apache/Osnailyfacter::Apache::Apache_port[80]/Apache::Namevirtualhost[*:80]/Concat::Fragment[NameVirtualHost *:80]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_NameVirtualHost *_80]/ensure: defined content as '{md5}a6144f5815141f7fa494ea04970d6b11'
Info: /Stage[main]/Osnailyfacter::Apache/Osnailyfacter::Apache::Apache_port[80]/Apache::Namevirtualhost[*:80]/Concat::Fragment[NameVirtualHost *:80]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_NameVirtualHost *_80]: Scheduling refresh of Exec[concat_/etc/httpd/conf/ports.conf]
Debug: /Stage[main]/Osnailyfacter::Apache/Osnailyfacter::Apache::Apache_port[80]/Apache::Namevirtualhost[*:80]/Concat::Fragment[NameVirtualHost *:80]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_NameVirtualHost *_80]: The container Concat::Fragment[NameVirtualHost *:80] will propag...

[Timur Nurlygayanov (tnurlygayanov)]

By the way, initial problem doesn't reproduced withe custom ISO: http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_7.0_iso/1225/

Steps To Verify:
root@node-1:~# . openrc
root@node-1:~# time openstack service create --format shell identity --name keystone --description "OpenStack Identity Service"
description="OpenStack Identity Service"
enabled="True"
id="c5b6bfd9826f4ee497ac682abccbd8b5"
name="keystone"
type="identity"

real 0m0.990s
user 0m0.440s
sys 0m0.119s
root@node-1:~#

But "quick repro scenario" with execution of puppet manifests failed:

Steps:
Execute on master node:
puppet apply -vd /etc/puppet/modules/osnailyfacter/modular/apache/apache.pp; puppet apply -vd /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp

Observed Result:
Error: No matching Packages to list
Could not retrieve libvirt_package_version: undefined method `split' for nil:NilClass
Error: No matching Packages to list
Could not retrieve libvirt_package_version: undefined method `split' for nil:NilClass
Warning: Unrecognised escape sequence '\(' in file /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp at line 203
Warning: Unrecognised escape sequence '\)' in file /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp at line 203
Warning: Unrecognised escape sequence '\1' in file /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp at line 203
Notice: Scope(Class[main]): MODULAR: keystone.pp
Debug: hiera(): Hiera YAML backend starting
Debug: hiera(): Looking up network_scheme in YAML backend
Debug: hiera(): Looking for data source common
Debug: hiera(): Looking up network_metadata in YAML backend
Debug: hiera(): Looking for data source common
Debug: hiera(): Looking up node_name in YAML backend
Debug: hiera(): Looking for data source common
Error: Could not find data item node_name in any Hiera data file and no default supplied at /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp:7 on node fuel.domain.tld
Error: Could not find data item node_name in any Hiera data file and no default supplied at /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp:7 on node fuel.domain.tld

[Timur Nurlygayanov (tnurlygayanov)]

root@node-1:~# ps ax | grep apache
 8452 pts/23 R+ 0:00 grep --color=auto apache
23900 ? Ss 0:01 /usr/sbin/apache2 -k start
28157 ? Sl 0:01 /usr/sbin/apache2 -k start
28158 ? Sl 0:01 /usr/sbin/apache2 -k start
28159 ? Sl 0:16 /usr/sbin/apache2 -k start
28160 ? Sl 0:15 /usr/sbin/apache2 -k start

[Timur Nurlygayanov (tnurlygayanov)]

After manual "woraround" on all controller nodes:
service apache2 stop
apachectl graceful

I tested it one more time:

1. On master node run the following command:
puppet apply -vd /etc/puppet/modules/osnailyfacter/modular/apache/apache.pp; puppet apply -vd /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp

Observed Result:
Error: No matching Packages to list
Could not retrieve libvirt_package_version: undefined method `split' for nil:NilClass
Error: No matching Packages to list
Could not retrieve libvirt_package_version: undefined method `split' for nil:NilClass
Warning: Unrecognised escape sequence '\(' in file /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp at line 203
Warning: Unrecognised escape sequence '\)' in file /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp at line 203
Warning: Unrecognised escape sequence '\1' in file /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp at line 203
Notice: Scope(Class[main]): MODULAR: keystone.pp
Debug: hiera(): Hiera YAML backend starting
Debug: hiera(): Looking up network_scheme in YAML backend
Debug: hiera(): Looking for data source common
Debug: hiera(): Looking up network_metadata in YAML backend
Debug: hiera(): Looking for data source common
Debug: hiera(): Looking up node_name in YAML backend
Debug: hiera(): Looking for data source common
Error: Could not find data item node_name in any Hiera data file and no default supplied at /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp:7 on node fuel.domain.tld
Error: Could not find data item node_name in any Hiera data file and no default supplied at /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp:7 on node fuel.domain.tld

[Davanum Srinivas (DIMS) (dims-v)]

upstream issue: https://github.com/GrahamDumpleton/mod_wsgi/issues/94

[Pavel Boldin (pboldin)]

Proposed upstream fix:
https://github.com/paboldin/mod_wsgi/commit/2b97d3e6f5c0a1ef3c5de6b1e06b2af5ac443a33

[Artem Silenkov (asilenkov)]

Package is merged and placed in primary repo.

[Artem Silenkov (asilenkov)]

https://review.fuel-infra.org/#/c/11896/

[Vasyl Saienko (vsaienko)]

Fixed with: https://review.fuel-infra.org/#/c/11896/

[Davanum Srinivas (DIMS) (dims-v)]

I understand that we may be doing something else with uWSGI etc in 8.0. but until then we should use the same fix we had in mod_wsgi in 8.0 as well

[Artem Silenkov (asilenkov)]

^^DIMS

I'm going to cherrypick this very fixed package to 8.0 on `fix-released`.

[Andrey Sledzinskiy (asledzinskiy)]

Verified on all aforementioned tests scenarios

build_id": "301",
"build_number": "301",
"release_versions":
{

    "2015.1.0-7.0":

{

    "VERSION":

{

    "build_id": "301",
    "build_number": "301",
    "api": "1.0",
    "fuel-library_sha": "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd",
    "nailgun_sha": "4162b0c15adb425b37608c787944d1983f543aa8",
    "feature_groups":

        [
            "mirantis"
        ],
        "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
        "openstack_version": "2015.1.0-7.0",
        "fuel-agent_sha": "50e90af6e3d560e9085ff71d2950cfbcca91af67",
        "production": "docker",
        "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3",
        "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012",
        "fuel-ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c",
        "release": "7.0",
        "fuelmain_sha": "a65d453215edb0284a2e4761be7a156bb5627677"
    }

}

[Davanum Srinivas (DIMS) (dims-v)]

Thanks Artem.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vasyl Saienko (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/223775
Reason: it was fixed in wsgi module

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vasyl Saienko (<email address hidden>) on branch: master
Review: https://review.openstack.org/223943
Reason: it was fixed in wsgi module

[Vasyl Saienko (vsaienko)]

For 8.0 was fixed in the following commit:
https://review.fuel-infra.org/#/c/12084/

[Alex Schultz (alex-schultz)]

Should the updated package be available? We were testing a custom iso built after this package fix was created and the test failed because apache was hung.

http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/8.0.custom_system_test/134/console

That evironment ran into this problem but it does not appear that the updated mod-wsgi module package was installed.

root@node-1:/var/log/apache2# dpkg -l | grep wsgi
ii libapache2-mod-wsgi 3.4-4ubuntu2.1.14.04.2 amd64 Python WSGI adapter module for Apache

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Artem Silenkov <email address hidden> on branch: 7.0
Review: https://review.fuel-infra.org/11574

[Anastasia Palkina (apalkina)]

Verified on ISO #303

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "303"
  build_id: "303"
  fuel-nailgun_sha: "352548386007399a95a6f21b1fcd9c48a0726325"
  python-fuelclient_sha: "b2bbcdf1c0f38adb34cff01cb6040006911f2ea5"
  fuel-agent_sha: "49bb78675b749d15ae8f0f045dc2b0811777a9d6"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "c56dfde2da034151a7e707b381c4cf9d213b4ba2"
  fuel-library_sha: "14576a3dbb3be5e4013c306776dc2eaefe0c15e0"
  fuel-ostf_sha: "9910a4726cbd038c257582b429527e40c4c3cb20"
  fuel-mirror_sha: "dbbe9ddc2c8a336aa7ab62952761bd079e374d1d"
  fuelmenu_sha: "680b720291ff577f4c058cee25f85e563c96312e"
  shotgun_sha: "cacb93cbc28910ff0dc38f30a855efa9af50d8ce"
  network-checker_sha: "d443ef47abeda58d319bc8d33d5005dd09440a02"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "74e9affd54e5a31fd55ed75a3402940dd186a621"