tripleo

CI: Undercloud second install (idempotency step) fails on pereiodic nonha job

Bug #1666459 reported by Sagi (Sergey) Shnaidman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Jiří Stránský

Bug Description

Second install of undercloud fails (idempotency step).
http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/944a03e/

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/944a03e/logs/undercloud/var/log/undercloud_install_idempotent.txt.gz

2017-02-21 07:42:17,630 INFO: Notice: /Stage[main]/Tripleo::Profile::Base::Docker_registry/File[/etc/docker-distribution/registry/config.yml]/ensure: defined content as '{md5}944be5b9a5850417c064a8ff5005e522'
2017-02-21 07:42:17,800 INFO: Error: Systemd start for docker-distribution failed!
2017-02-21 07:42:17,800 INFO: journalctl log for docker-distribution:
2017-02-21 07:42:17,800 INFO: -- Logs begin at Tue 2017-02-21 01:41:22 UTC, end at Tue 2017-02-21 07:42:17 UTC. --
2017-02-21 07:42:17,800 INFO: Feb 21 07:42:17 tripleo-centos-7-tripleo-test-cloud-rh1-7395270 systemd[1]: Stopping v2 Registry server for Docker...
2017-02-21 07:42:17,801 INFO: Feb 21 07:42:17 tripleo-centos-7-tripleo-test-cloud-rh1-7395270 systemd[1]: Stopped v2 Registry server for Docker.
2017-02-21 07:42:17,801 INFO: 
2017-02-21 07:42:17,801 INFO: Error: /Stage[main]/Tripleo::Profile::Base::Docker_registry/Service[docker-distribution]/ensure: change from stopped to running failed: Systemd start for docker-distribution failed!
2017-02-21 07:42:17,801 INFO: journalctl log for docker-distribution:
2017-02-21 07:42:17,801 INFO: -- Logs begin at Tue 2017-02-21 01:41:22 UTC, end at Tue 2017-02-21 07:42:17 UTC. --
2017-02-21 07:42:17,801 INFO: Feb 21 07:42:17 tripleo-centos-7-tripleo-test-cloud-rh1-7395270 systemd[1]: Stopping v2 Registry server for Docker...
2017-02-21 07:42:17,801 INFO: Feb 21 07:42:17 tripleo-centos-7-tripleo-test-cloud-rh1-7395270 systemd[1]: Stopped v2 Registry server for Docker.
2017-02-21 07:42:17,801 INFO: 

Sagi (Sergey) Shnaidman (sshnaidm)
Changed in tripleo:
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
Jiří Stránský (jistr) wrote :

Looking at the environment, the journal for the docker-distribution service looks clean of errors:

-bash-4.2# journalctl -u docker-distribution
-- Logs begin at Mon 2017-02-20 10:50:02 UTC, end at Tue 2017-02-21 10:16:48 UTC. --
Feb 21 07:32:34 sshnaidm-undercloud systemd[1]: Started v2 Registry server for Docker.
Feb 21 07:32:34 sshnaidm-undercloud systemd[1]: Starting v2 Registry server for Docker...
Feb 21 07:32:34 sshnaidm-undercloud registry[7085]: time="2017-02-21T07:32:34Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries a
Feb 21 07:32:34 sshnaidm-undercloud registry[7085]: time="2017-02-21T07:32:34Z" level=info msg="redis not configured" go.version=go1.6.3 instance.id=ed8ecb2e-c20a-455a-bdc8-ba0870a83331 version="v2.4.1+unknown"
Feb 21 07:32:34 sshnaidm-undercloud registry[7085]: time="2017-02-21T07:32:34Z" level=info msg="Starting upload purge in 47m0s" go.version=go1.6.3 instance.id=ed8ecb2e-c20a-455a-bdc8-ba0870a83331 version="v2.4.1
Feb 21 07:32:34 sshnaidm-undercloud registry[7085]: time="2017-02-21T07:32:34Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.6.3 instance.id=ed8ecb2e-c20a-455a-bdc8-ba0870a83331 version="
Feb 21 07:32:34 sshnaidm-undercloud registry[7085]: time="2017-02-21T07:32:34Z" level=info msg="listening on 192.168.24.1:8787" go.version=go1.6.3 instance.id=ed8ecb2e-c20a-455a-bdc8-ba0870a83331 version="v2.4.1
Feb 21 08:19:34 sshnaidm-undercloud registry[7085]: time="2017-02-21T08:19:34Z" level=info msg="PurgeUploads starting: olderThan=2017-02-14 08:19:34.411389844 +0000 UTC, actuallyDelete=true"
Feb 21 08:19:34 sshnaidm-undercloud registry[7085]: time="2017-02-21T08:19:34Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=1"
Feb 21 08:19:34 sshnaidm-undercloud registry[7085]: time="2017-02-21T08:19:34Z" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.6.3 instance.id=ed8ecb2e-c20a-455a-bdc8-ba0870a83331 version="v2.4
Feb 21 09:27:15 sshnaidm-undercloud systemd[1]: Stopping v2 Registry server for Docker...
Feb 21 09:27:15 sshnaidm-undercloud systemd[1]: Stopped v2 Registry server for Docker.

But the docker-distribution RPM is not present anymore:

-bash-4.2# systemctl status docker-distribution
Unit docker-distribution.service could not be found.

-bash-4.2# rpm -qa | grep docker
docker-common-1.12.5-14.el7.centos.x86_64
python-docker-py-1.9.0-1.el7.noarch
docker-client-1.12.5-14.el7.centos.x86_64
docker-1.12.5-14.el7.centos.x86_64

Revision history for this message
Jiří Stránský (jistr) wrote :

There's only one line mentioning docker-distribution in yum.log. ATM i'm not exactly sure how come the package disappeared.

-bash-4.2# grep docker-distribution /var/log/yum.log
Feb 21 07:31:13 Installed: docker-distribution-2.5.1-1.el7.x86_64

Revision history for this message
Emilien Macchi (emilienm) wrote :

https://review.openstack.org/#/c/436442/ and https://review.openstack.org/436441 will help to fix this one.

OpenStack Infra (hudson-openstack)
Changed in tripleo:
assignee: nobody → Jiří Stránský (jistr)
status: Triaged → In Progress
Jiří Stránský (jistr)
tags: added: ocata-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (master)

Reviewed: https://review.openstack.org/436442
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=d12c004bc9c630c756a6b0df351916b9e04b9778
Submitter: Jenkins
Branch: master

commit d12c004bc9c630c756a6b0df351916b9e04b9778
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 21 13:54:06 2017 +0100

    Stop accidentally removing docker-distribution

    By default Puppet does virtual package matching if precise name matching
    fails. Docker-distribution RPM "provides" docker-registry:

    bash-4.2# rpm -q --whatprovides docker-registry
    docker-distribution-2.5.1-1.el7.x86_64

    This means that when we wanted to make docker-registry package absent,
    we were actually removing docker-distribution instead. This is now fixed
    by allow_virtual => false. Only name matching is performed.

    Change-Id: I1f93b404085f0bc2b6c063f573c801db6409c0bb
    Closes-Bug: #1666459

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/436520

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/ocata)

Reviewed: https://review.openstack.org/436520
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=ce03c7f2359d2b974dc35d3fdad0fb88134ac84f
Submitter: Jenkins
Branch: stable/ocata

commit ce03c7f2359d2b974dc35d3fdad0fb88134ac84f
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 21 13:54:06 2017 +0100

    Stop accidentally removing docker-distribution

    By default Puppet does virtual package matching if precise name matching
    fails. Docker-distribution RPM "provides" docker-registry:

    bash-4.2# rpm -q --whatprovides docker-registry
    docker-distribution-2.5.1-1.el7.x86_64

    This means that when we wanted to make docker-registry package absent,
    we were actually removing docker-distribution instead. This is now fixed
    by allow_virtual => false. Only name matching is performed.

    Change-Id: I1f93b404085f0bc2b6c063f573c801db6409c0bb
    Closes-Bug: #1666459
    (cherry picked from commit d12c004bc9c630c756a6b0df351916b9e04b9778)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/441130

Jiří Stránský (jistr)
tags: added: newton-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/newton)

Reviewed: https://review.openstack.org/441130
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=915f271134655eb580425fe64e7bc896a24da853
Submitter: Jenkins
Branch: stable/newton

commit 915f271134655eb580425fe64e7bc896a24da853
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 21 13:54:06 2017 +0100

    Stop accidentally removing docker-distribution

    By default Puppet does virtual package matching if precise name matching
    fails. Docker-distribution RPM "provides" docker-registry:

    bash-4.2# rpm -q --whatprovides docker-registry
    docker-distribution-2.5.1-1.el7.x86_64

    This means that when we wanted to make docker-registry package absent,
    we were actually removing docker-distribution instead. This is now fixed
    by allow_virtual => false. Only name matching is performed.

    Related BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1428826

    Change-Id: I1f93b404085f0bc2b6c063f573c801db6409c0bb
    Closes-Bug: #1666459
    (cherry picked from commit d12c004bc9c630c756a6b0df351916b9e04b9778)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo 6.3.0

This issue was fixed in the openstack/puppet-tripleo 6.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo 7.0.0

This issue was fixed in the openstack/puppet-tripleo 7.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo 5.6.0

This issue was fixed in the openstack/puppet-tripleo 5.6.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.