restarting apache sometimes fails

Bug #1569390 reported by Matthew J Black
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
puppet-keystone
Fix Released
High
Emilien Macchi

Bug Description

In the CI jobs, apache is being restarted by an exec call. It looks like apache sometimes fails to restart because of the port being held open.

An excerpt from the log showing where the issue occurs.

2016-04-12 13:49:50.903 | Info: /Stage[main]/Keystone::Disable_admin_token_auth/Ini_subsetting[api_v3/admin_token_auth]: Scheduling refresh of Exec[restart_keystone]
2016-04-12 13:49:50.903 | Notice: /Stage[main]/Openstack_extras::Auth_file/File[/root/openrc]/ensure: defined content as '{md5}bf13aac1561aa7c3aab3fc78f12ae961'
2016-04-12 13:49:51.020 | Notice: /Stage[main]/Keystone::Disable_admin_token_auth/Ini_subsetting[admin_api/admin_token_auth]/ensure: removed
2016-04-12 13:49:51.021 | Info: /Stage[main]/Keystone::Disable_admin_token_auth/Ini_subsetting[admin_api/admin_token_auth]: Scheduling refresh of Exec[restart_keystone]
2016-04-12 13:50:11.812 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: * Restarting web server apache2
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: (98)Address already in use: AH00072: make_sock: could not bind to address 127.0.0.1:35357
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: no listening sockets available, shutting down
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: AH00015: Unable to open logs
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: Action 'start' failed.
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: The Apache error log may have more information.
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: ...fail!
2016-04-12 13:50:11.818 | Notice: /Stage[main]/Keystone/Exec[restart_keystone]/returns: * The apache2 instance did not start within 20 seconds. Please read the log files to discover problems
2016-04-12 13:50:11.818 | Error: /Stage[main]/Keystone/Exec[restart_keystone]: Failed to call refresh: service apache2 restart returned 1 instead of one of [0]
2016-04-12 13:50:11.818 | Error: /Stage[main]/Keystone/Exec[restart_keystone]: service apache2 restart returned 1 instead of one of [0]

and here is the line in the manifest that it occurs at. https://github.com/openstack/puppet-keystone/blob/master/manifests/init.pp#L957-L963

Revision history for this message
Emilien Macchi (emilienm) wrote :
Changed in puppet-keystone:
importance: Undecided → High
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/307912

Changed in puppet-keystone:
assignee: nobody → Matthew J Black (mjblack)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on puppet-keystone (master)

Change abandoned by Matthew J Black (<email address hidden>) on branch: master
Review: https://review.openstack.org/307912

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/308012

Changed in puppet-keystone:
assignee: Matthew J Black (mjblack) → Emilien Macchi (emilienm)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-keystone (master)

Reviewed: https://review.openstack.org/308012
Committed: https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=a90175e620470ab0722ab8b735916cfeccb8cf77
Submitter: Jenkins
Branch: master

commit a90175e620470ab0722ab8b735916cfeccb8cf77
Author: Matthew Black <email address hidden>
Date: Tue Apr 19 11:53:52 2016 -0400

    Prevent keystone eventlet from starting

    On debian based hosts, when a package is installed, the packager
    has the service automatically start. This causes a problem,
    especially in CI, where the eventlet doesnt quit fast enough
    and apache is started but cant bind to port.

    Change-Id: I6b07a70f0729639f12ebc19f9d3cd405d8be214b
    Closes-Bug: 1569390

Changed in puppet-keystone:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-keystone (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/309685

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-keystone (stable/mitaka)

Reviewed: https://review.openstack.org/309685
Committed: https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=78ef8e9ad49249587bcf399d9076a084d8ceea7f
Submitter: Jenkins
Branch: stable/mitaka

commit 78ef8e9ad49249587bcf399d9076a084d8ceea7f
Author: Matthew Black <email address hidden>
Date: Tue Apr 19 11:53:52 2016 -0400

    Prevent keystone eventlet from starting

    On debian based hosts, when a package is installed, the packager
    has the service automatically start. This causes a problem,
    especially in CI, where the eventlet doesnt quit fast enough
    and apache is started but cant bind to port.

    Change-Id: I6b07a70f0729639f12ebc19f9d3cd405d8be214b
    Closes-Bug: 1569390
    (cherry picked from commit a90175e620470ab0722ab8b735916cfeccb8cf77)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-keystone (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/323470

Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/puppet-keystone 8.1.0

This issue was fixed in the openstack/puppet-keystone 8.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-keystone (stable/liberty)

Reviewed: https://review.openstack.org/323470
Committed: https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=2adced3aa5e9b48e5e91bf429f8e5584edfde958
Submitter: Jenkins
Branch: stable/liberty

commit 2adced3aa5e9b48e5e91bf429f8e5584edfde958
Author: Matthew Black <email address hidden>
Date: Tue Apr 19 11:53:52 2016 -0400

    Prevent keystone eventlet from starting

    On debian based hosts, when a package is installed, the packager
    has the service automatically start. This causes a problem,
    especially in CI, where the eventlet doesnt quit fast enough
    and apache is started but cant bind to port.

    Change-Id: I6b07a70f0729639f12ebc19f9d3cd405d8be214b
    Closes-Bug: 1569390
    (cherry picked from commit 78ef8e9ad49249587bcf399d9076a084d8ceea7f)

tags: added: in-stable-liberty
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/puppet-keystone 9.0.0

This issue was fixed in the openstack/puppet-keystone 9.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers