Upgrades, puppet fails because it cannot start supervisor inside of container

[Evgeniy L]

1. install 5.0
2. run upgrade to 5.1

It happens 3 runs out of 10

Actual result:
It fails with errors

2014-08-26 16:20:27 INFO 31163 (health_checker) Checkers which are still fail ['integration_postgres_nailgun_nginx']
Or
2014-08-26 16:20:27 INFO 31163 (health_checker) Checkers which are still fail ['ostf']

Here a logs from containers
http://pastebin.com/7DPBjakF

    Error: Could not start Service[supervisord]: Execution of '/sbin/service supervisord start' returned 1:

http://pastebin.com/5vUVtfex

    Error: Could not start Service[supervisord]: Execution of '/sbin/service supervisord start' returned 1:

Here is debugging information http://pastebin.com/k7qUvrHQ
The reason is there is lock file /var/run/supervisord.pid from previous run

When I added removing of pid file in the beginning of start.sh script, I got another error:

    Error: Another program is already listening on a port that one of our HTTP servers is configured to use. Shut this program down first before starting supervisord.

http://paste.openstack.org/show/100549/

It looks like there is supervisor which was started by puppet, and then start.sh tries to start another supervisor and fails.

Expected result:
upgrade succeed

*5.1*

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "478"
  build_id: "2014-08-26_00-01-17"
  astute_sha: "bc60b7d027ab244039f48c505ac52ab8eb0a990c"
  fuellib_sha: "ca5aa450ea3da771c2d5f1e82511450f0b8faf28"
  ostf_sha: "907f25f8fad39b177bf6a66fba9785afa7dd8008"
  nailgun_sha: "44876ddae29823449e0cbc59428aafa466cbbbc2"
  fuelmain_sha: "74ad3dd68020aac3042f62c59c137498474ecbee"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/117250

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/117250
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=36bebf494f66d63310706e0532a492b0dff286c3
Submitter: Jenkins
Branch: master

commit 36bebf494f66d63310706e0532a492b0dff286c3
Author: Matthew Mosesohn <email address hidden>
Date: Wed Aug 27 17:55:37 2014 +0400

    Fix supervisord start with stale pidfile

    supervisord has a stale pidfile sometimes
    during Fuel container upgrades.
    Also corrected whitespace issues.

    Reduced wait time for processes to 5s so that
    init script doesn't exceed its max timeout.

    Change-Id: I6e5511f5fcce0205283594a290e72084b20e6aa7
    Closes-Bug: #1361756

[Matthew Mosesohn (raytrac3r)]

Please note that https://review.openstack.org/#/c/117278/ was also applied to address this bug. This patch is necessary in combination with https://review.openstack.org/#/c/117250.

[Andrey Sledzinskiy (asledzinskiy)]

verified on fuel-5.1-upgrade-11-2014-09-17_21-40-34.tar.lrz