some docker containers remain unconfigured

[Vladimir Kuklin]

deploy ISO, e.g.

{"build_id": "2014-05-04_01-00-26", "mirantis": "yes", "build_number": "179", "ostf_sha": "134765fcb5a07dce0cd1bb399b2290c988c3c63b", "nailgun_sha": "2de1dcf9fa3fc1521999bff6377eaa6f01d825aa", "production": "docker", "api": "1.0", "fuelmain_sha": "95c35c199c2efc03fb105d090c5a42525430b7b3", "astute_sha": "3cffebde1e5452f5dbf8f744c6525fc36c7afbf3", "release": "5.0", "fuellib_sha": "c414bd7e49e7cfb6c5d66b37b55ae06f05dbecc3"}

go into, e.g. cobbler and see:

# Cobbler generated configuration file for dnsmasq
# Sat May 3 21:14:26 2014

read-ethers
log-dhcp
log-queries
log-facility=LOCAL0
addn-hosts = /var/lib/cobbler/cobbler_hosts
domain=domain.tld
dhcp-lease-max=1000
server=/domain.tld/
resolv-file=/etc/dnsmasq.upstream
dhcp-match=gpxe,175
interface=eth0

# This is one of the key options. dnsmasq tries to move out servername
# and PXE filename from special fields into DHCP options.
# Some old clients can't understand those DHCP options, so they
# will not be able to boot via PXE without this option enabled.
# For example gPXE will not work while iPXE works fine.
dhcp-no-override

dhcp-option=6,172.17.0.2

dhcp-range=internal,10.20.0.130,10.20.0.254,255.255.255.0
dhcp-option=net:internal,option:router,172.17.0.2
pxe-service=net:#gpxe,x86PC,"Install",pxelinux,172.17.0.2
dhcp-boot=net:internal,pxelinux.0,boothost,172.17.0.2

dnsmasq is unconfigured for some reason

this leads to failure of PXE boot of slave nodes

[Vladimir Kuklin (vkuklin)]

It seems the problem that the environment is snapshotted during system tests BEFORE all the puppet configuration of all the containers is finished, e.g. the one for cobbler.

[Vladimir Kuklin (vkuklin)]

more precisely, before all the start scripts of all containers finish their run

[Vladimir Kuklin (vkuklin)]

we need a way to check if all the containers finished their, e.g. by echoing some staff in start.sh scripts or Dockerfiles of containers, and only then we need to check if nailgun succesfully responds to curl

[Vladimir Kuklin (vkuklin)]

this affects only specific case with env snapshot/revert and currently affects only system tests

[Openstack Gerrit (openstack-gerrit)]

Related fix proposed to branch: master
Review: https://review.openstack.org/92025

[Vladimir Kuklin (vkuklin)]

there was a mistake in commit message

actual proposed fix is here:

https://review.openstack.org/#/c/92110/

[Matthew Mosesohn (raytrac3r)]

It's okay. Thanks for getting it merged.

On Tue, May 6, 2014 at 5:32 AM, Vladimir Kuklin <email address hidden>wrote:

> there was a mistake in commit message
>
> actual proposed fix is here:
>
> https://review.openstack.org/#/c/92110/
>
> --
> You received this bug notification because you are a member of Fuel
> Library Team, which is a bug assignee.
> https://bugs.launchpad.net/bugs/1315940
>
> Title:
> some docker containers remain unconfigured
>
> Status in Fuel: OpenStack installer that works:
> Confirmed
>
> Bug description:
> deploy ISO, e.g.
>
> {"build_id": "2014-05-04_01-00-26", "mirantis": "yes", "build_number":
> "179", "ostf_sha": "134765fcb5a07dce0cd1bb399b2290c988c3c63b",
> "nailgun_sha": "2de1dcf9fa3fc1521999bff6377eaa6f01d825aa",
> "production": "docker", "api": "1.0", "fuelmain_sha":
> "95c35c199c2efc03fb105d090c5a42525430b7b3", "astute_sha":
> "3cffebde1e5452f5dbf8f744c6525fc36c7afbf3", "release": "5.0",
> "fuellib_sha": "c414bd7e49e7cfb6c5d66b37b55ae06f05dbecc3"}
>
> go into, e.g. cobbler and see:
>
> # Cobbler generated configuration file for dnsmasq
> # Sat May 3 21:14:26 2014
>
> read-ethers
> log-dhcp
> log-queries
> log-facility=LOCAL0
> addn-hosts = /var/lib/cobbler/cobbler_hosts
> domain=domain.tld
> dhcp-lease-max=1000
> server=/domain.tld/
> resolv-file=/etc/dnsmasq.upstream
> dhcp-match=gpxe,175
> interface=eth0
>
> # This is one of the key options. dnsmasq tries to move out servername
> # and PXE filename from special fields into DHCP options.
> # Some old clients can't understand those DHCP options, so they
> # will not be able to boot via PXE without this option enabled.
> # For example gPXE will not work while iPXE works fine.
> dhcp-no-override
>
> dhcp-option=6,172.17.0.2
>
> dhcp-range=internal,10.20.0.130,10.20.0.254,255.255.255.0
> dhcp-option=net:internal,option:router,172.17.0.2
> pxe-service=net:#gpxe,x86PC,"Install",pxelinux,172.17.0.2
> dhcp-boot=net:internal,pxelinux.0,boothost,172.17.0.2
>
>
> dnsmasq is unconfigured for some reason
>
> this leads to failure of PXE boot of slave nodes
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1315940/+subscriptions
>

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/92330
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d389cbd51c137e7dbaae8617def1277b624d8d45
Submitter: Jenkins
Branch: master

commit d389cbd51c137e7dbaae8617def1277b624d8d45
Author: Vladimir Kuklin <email address hidden>
Date: Tue May 6 13:41:09 2014 +0400

    Change cobbler container checker

    Check cobblerd process existence, not puppet
    process absence.

    Change-Id: I5254a44e4a234ab3c77a183008240dd7e1c85f9c
    Partial-Bug: #1315940

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/92353
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=75aade2f74cc7782581bff7013f1dd45f51d8922
Submitter: Jenkins
Branch: master

commit 75aade2f74cc7782581bff7013f1dd45f51d8922
Author: Vladimir Kuklin <email address hidden>
Date: Tue May 6 15:59:57 2014 +0400

    Check for foreground cobbler

    It seems that cobblerd also starts during
    puppet run. What we really need to check
    is cobbler running in the foreground inside
    docker container

    Change-Id: I0f46cb363deb6fc10d6ab8eebc0f8a835633d0df
    Partial-Bug: #1315940

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/92602

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/92602
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=61d42a35f25aee3d2351c735fa0619aa04450def
Submitter: Jenkins
Branch: master

commit 61d42a35f25aee3d2351c735fa0619aa04450def
Author: Vladimir Kuklin <email address hidden>
Date: Wed May 7 18:36:05 2014 +0400

    Add checkers for all containers

    check containers services status
    1) for postgres do select 1 in nailgun
    2) for rabbitmq use AMQP management plugin
    and fetch nodes list
    3) for other containers: check listening on ports
    or check processes in process list
    4) add check function to dockerctl for convinience
    5) add check function call to container start
    Closes-Bug: #1315940
    Related-Bug: #1316044

    Change-Id: Id21a75718b4ad789c18aa248efca8bf16c90f6b8

[Andrey Sledzinskiy (asledzinskiy)]

verified on {"build_id": "2014-05-08_01-10-31", "mirantis": "yes", "build_number": "188", "ostf_sha": "fe718434f88f2ab167779770828a195f06eb29f8", "nailgun_sha": "82b9d42a7a5e9aa1caf6b2779c45ca045cad0ad2", "production": "docker", "api": "1.0", "fuelmain_sha": "97d7f6d5461db3afc27f58160cf9f6985230d255", "astute_sha": "9c83d3ecec69df03cd94620e2df92249ba4ec786", "release": "5.0", "fuellib_sha": "fd31d9a8f85136347b60377df00df6728eda14ca"}