restarting apache sometimes fails

[Matthew J Black]

In the CI jobs, apache is being restarted by an exec call. It looks like apache sometimes fails to restart because of the port being held open.

An excerpt from the log showing where the issue occurs.

2016-04-12 13:49:50.903 | Info: /Stage[main]/Keystone::Disable_admin_token_auth/Ini_subsetting[api_v3/admin_token_auth]: Scheduling refresh of Exec[restart_keystone]
2016-04-12 13:49:50.903 | Notice: /Stage[main]/Openstack_extras::Auth_file/File[/root/openrc]/ensure: defined content as '{md5}bf13aac1561aa7c3aab3fc78f12ae961'
2016-04-12 13:49:51.020 | Notice: /Stage[main]/Keystone::Disable_admin_token_auth/Ini_subsetting[admin_api/admin_token_auth]/ensure: removed
2016-04-12 13:49:51.021 | Info: /Stage[main]/Keystone::Disable_admin_token_auth/Ini_subsetting[admin_api/admin_token_auth]: Scheduling refresh of Exec[restart_keystone]
2016-04-12 13:50:11.812 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: * Restarting web server apache2
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: (98)Address already in use: AH00072: make_sock: could not bind to address 127.0.0.1:35357
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: no listening sockets available, shutting down
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: AH00015: Unable to open logs
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: Action 'start' failed.
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: The Apache error log may have more information.
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: ...fail!
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: * The apache2 instance did not start within 20 seconds. Please read the log files to discover problems
2016-04-12 13:50:11.818 | Error: /Stage[main]/Keystone/Exec[restart_keystone]: Failed to call refresh: service apache2 restart returned 1 instead of one of [0]
2016-04-12 13:50:11.818 | Error: /Stage[main]/Keystone/Exec[restart_keystone]: service apache2 restart returned 1 instead of one of [0]

and here is the line in the manifest that it occurs at. https://github.com/openstack/puppet-keystone/blob/master/manifests/init.pp#L957-L963

[Emilien Macchi (emilienm)]

for the record, here is how Fuel is fixing it:
https://github.com/openstack/fuel-library/blob/c3491543decb9f5600e5a17e4a82c6617f49d6c0/deployment/puppet/tweaks/manifests/apache_wrappers.pp

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/307912

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matthew J Black (<email address hidden>) on branch: master
Review: https://review.openstack.org/307912

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/308012

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/308012
Committed: https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=a90175e620470ab0722ab8b735916cfeccb8cf77
Submitter: Jenkins
Branch: master

commit a90175e620470ab0722ab8b735916cfeccb8cf77
Author: Matthew Black <email address hidden>
Date: Tue Apr 19 11:53:52 2016 -0400

    Prevent keystone eventlet from starting

    On debian based hosts, when a package is installed, the packager
    has the service automatically start. This causes a problem,
    especially in CI, where the eventlet doesnt quit fast enough
    and apache is started but cant bind to port.

    Change-Id: I6b07a70f0729639f12ebc19f9d3cd405d8be214b
    Closes-Bug: 1569390

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/309685

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309685
Committed: https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=78ef8e9ad49249587bcf399d9076a084d8ceea7f
Submitter: Jenkins
Branch: stable/mitaka

commit 78ef8e9ad49249587bcf399d9076a084d8ceea7f
Author: Matthew Black <email address hidden>
Date: Tue Apr 19 11:53:52 2016 -0400

    Prevent keystone eventlet from starting

    On debian based hosts, when a package is installed, the packager
    has the service automatically start. This causes a problem,
    especially in CI, where the eventlet doesnt quit fast enough
    and apache is started but cant bind to port.

    Change-Id: I6b07a70f0729639f12ebc19f9d3cd405d8be214b
    Closes-Bug: 1569390
    (cherry picked from commit a90175e620470ab0722ab8b735916cfeccb8cf77)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/323470

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/puppet-keystone 8.1.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/323470
Committed: https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=2adced3aa5e9b48e5e91bf429f8e5584edfde958
Submitter: Jenkins
Branch: stable/liberty

commit 2adced3aa5e9b48e5e91bf429f8e5584edfde958
Author: Matthew Black <email address hidden>
Date: Tue Apr 19 11:53:52 2016 -0400

    Prevent keystone eventlet from starting

    On debian based hosts, when a package is installed, the packager
    has the service automatically start. This causes a problem,
    especially in CI, where the eventlet doesnt quit fast enough
    and apache is started but cant bind to port.

    Change-Id: I6b07a70f0729639f12ebc19f9d3cd405d8be214b
    Closes-Bug: 1569390
    (cherry picked from commit 78ef8e9ad49249587bcf399d9076a084d8ceea7f)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/puppet-keystone 9.0.0 release.