docker-registry fails to start when installing undercloud on rhel7.3 with rdo-newton

[David Hill]

docker-registry fails to start when installing undercloud on rhel7.3 with rdo-ocata thus failing the undercloud installation

Here is the output:
2017-03-24 15:46:08,520 INFO: ESC[1;31mError: Systemd start for docker-registry failed!
2017-03-24 15:46:08,521 INFO: journalctl log for docker-registry:
2017-03-24 15:46:08,521 INFO: -- No entries --
2017-03-24 15:46:08,521 INFO: ESC[0m
2017-03-24 15:46:08,521 INFO: ESC[1;31mError: /Stage[main]/Main/Service[docker-registry]/ensure: change from stopped to running failed: Systemd start for docker-registry failed!
2017-03-24 15:46:08,521 INFO: journalctl log for docker-registry:
2017-03-24 15:46:08,522 INFO: -- No entries --
2017-03-24 15:46:08,522 INFO: ESC[0m

Troubleshooting:
[root@undercloud-0-ocata ~]# systemctl status docker-distribution
● docker-distribution.service - v2 Registry server for Docker
   Loaded: loaded (/usr/lib/systemd/system/docker-distribution.service; disabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Fri 2017-03-24 16:14:08 EDT; 4s ago
  Process: 1325 ExecStart=/usr/bin/registry serve /etc/docker-distribution/registry/config.yml (code=exited, status=1/FAILURE)
 Main PID: 1325 (code=exited, status=1/FAILURE)

Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: Unit docker-distribution.service entered failed state.
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: docker-distribution.service failed.
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: docker-distribution.service holdoff time over, scheduling restart.
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: start request repeated too quickly for docker-distribution.service
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: Failed to start v2 Registry server for Docker.
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: Unit docker-distribution.service entered failed state.
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: docker-distribution.service failed.
[root@undercloud-0-ocata ~]# journalctl -f
-- Logs begin at Fri 2017-03-24 15:02:52 EDT. --
Mar 24 16:14:08 undercloud-0-ocata.localdomain registry[1325]: time="2017-03-24T16:14:08-04:00" level=info msg="using inmemory blob descriptor cache" go.version=go1.7.4 instance.id=b236b438-4be7-4ff2-b03a-45b8fddf22da version="v2.6.0+unknown"
Mar 24 16:14:08 undercloud-0-ocata.localdomain registry[1325]: time="2017-03-24T16:14:08-04:00" level=fatal msg="listen tcp :5000: bind: address already in use"
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: docker-distribution.service: main process exited, code=exited, status=1/FAILURE
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: Unit docker-distribution.service entered failed state.
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: docker-distribution.service failed.
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: docker-distribution.service holdoff time over, scheduling restart.
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: start request repeated too quickly for docker-distribution.service
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: Failed to start v2 Registry server for Docker.
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: Unit docker-distribution.service entered failed state.
Mar 24 16:14:08 undercloud-0-ocata.localdomain systemd[1]: docker-distribution.service failed.
Mar 24 16:14:18 undercloud-0-ocata.localdomain container-server[32700]: Begin container update sweep
Mar 24 16:14:18 undercloud-0-ocata.localdomain container-server[32700]: Container update sweep completed: 0.00s
^C
[root@undercloud-0-ocata ~]# netstat -nap | grep 5000
tcp 0 0 192.168.24.1:5000 0.0.0.0:* LISTEN 20146/httpd

[Alex Schultz (alex-schultz)]

I believe this to be a duplicate of 1666459 which was caused by the docker package updates. This was fixed by https://review.openstack.org/#/c/440842/ to use the puppet-tripleo profile for the docker updates. Can you please provide a package list with versions?

[Mark Hamzy (mark-hamzy)]

@Alex, is this what you are looking for?

[stack@oscloud5 ~]$ sudo find /usr/ -name puppet-stack-config.pp
/usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.pp
[stack@oscloud5 ~]$ rpm -qf /usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.pp
instack-undercloud-5.2.0-1.el7.noarch

[stack@oscloud5 ~]$ cat /etc/yum.repos.d/CentOS-OpenStack-newton.repo
# CentOS-OpenStack-newton.repo
#
# Please see http://wiki.centos.org/SpecialInterestGroup/Cloud for more
# information

[centos-openstack-newton]
name=CentOS-7 - OpenStack newton
baseurl=http://mirror.centos.org/centos/7/cloud/$basearch/openstack-newton/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud

[centos-openstack-newton-test]
name=CentOS-7 - OpenStack newton Testing
baseurl=http://buildlogs.centos.org/centos/7/cloud/$basearch/openstack-newton/
gpgcheck=0
enabled=0

[centos-openstack-newton-debuginfo]
name=CentOS-7 - OpenStack newton - Debug
baseurl=http://debuginfo.centos.org/centos/7/cloud/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud

[centos-openstack-newton-source]
name=CentOS-7 - OpenStack newton - Source
baseurl=http://vault.centos.org/centos/7/cloud/Source/openstack-newton/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud

[rdo-trunk-newton-tested]
name=OpenStack newton Trunk Tested
baseurl=http://buildlogs.centos.org/centos/7/cloud/$basearch/rdo-trunk-newton-tested/
gpgcheck=0
enabled=0

[Mark Hamzy (mark-hamzy)]

Alex, on #tripleo, mentioned using https://trunk.rdoproject.org/centos7-newton/current-passed-ci/ as a repo instead.

[David Hill (david-hill-ubisoft)]

So when will the official mirror end up using the latest built packages? As I'm using this and it doesn't work:

[centos-openstack-newton]
name=CentOS-7 - OpenStack newton
baseurl=http://mirror.centos.org/centos/7/cloud/$basearch/openstack-newton/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud

I have the same issue with mitaka and liberty btw. As soon as we drop a build environments, I no longer am able to build an environment even though it used to work. If I'm using trunk.rdoproject.org, this will work as long as we don't drop the support for that release ... my point here is that if it worked at point A in time, it should also work at point B later on which is actually not the case.

[David Hill (david-hill-ubisoft)]

BTW, I have this issue with ocata and not newton.

[David Hill (david-hill-ubisoft)]

I'm sorry , this bug was opened in march and was fixed since then so I'm closing it.

[David Hill (david-hill-ubisoft)]

I spoke to fast here... i still have the issue with newton on the official centos mirrors:

2017-05-24 09:18:58 - ESC[1;31mError: Could not start Service[docker-registry]: Execution of '/bin/systemctl start docker-registry' returned 5: Failed to start docker-registry.service: Unit not found.ESC[0m
2017-05-24 09:18:58 - ESC[1;31mError: /Stage[main]/Main/Service[docker-registry]/ensure: change from stopped to running failed: Could not start Service[docker-registry]: Execution of '/bin/systemctl start docker-registry' returned 5: Failed to start docker-registry.service: Unit not found.ESC[0m

That's what I'm using:

[openstack-newton]
name=OpenStack newton - Official
baseurl=http://mirror.centos.org/centos/7/cloud/$basearch/openstack-newton/
gpgcheck=0
enabled=1

If I use newton trunk, it works as expected. So my question here is when will the mirror be updated with the latest newton packages?

[David Hill (david-hill-ubisoft)]

Mitaka might have the same issue where it works in trunk but not on the centos mirror.

[Alex Schultz (alex-schultz)]

We need to cut new tags for the stable versions for them to get published. It's on the list of things to do in the near future. In the mean time, use the trunk.rdoproject.org as a work around

[Alex Schultz (alex-schultz)]

Evidently it's currently in the process of being tested http://cbs.centos.org/koji/buildinfo?buildID=17107 but there were additional packages needed.

[Emilien Macchi (emilienm)]

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (FUTURE, PIKE, QUEENS, ROCKY, STEIN).
  Valid example: CONFIRMED FOR: FUTURE