Build Fails Using Trunk

[Daneyon Hansen]

I follow these instructions: http://docs.openstack.org/developer/kolla/quickstart.html

The build fails at:

$ kolla-build --type source --registry 10.30.118.91:4000 --push
.....
INFO:kolla.cmd.build:horizon:Installed:
INFO:kolla.cmd.build:horizon: mod_wsgi.x86_64 0:3.4-12.el7_0
INFO:kolla.cmd.build:horizon:Dependency Installed:
INFO:kolla.cmd.build:horizon: apr.x86_64 0:1.4.8-3.el7
INFO:kolla.cmd.build:horizon: apr-util.x86_64 0:1.5.2-6.el7
INFO:kolla.cmd.build:horizon: centos-logos.noarch 0:70.0.6-3.el7.centos
INFO:kolla.cmd.build:horizon: httpd-tools.x86_64 0:2.4.6-40.el7.centos
INFO:kolla.cmd.build:horizon: mailcap.noarch 0:2.1.41-2.el7
INFO:kolla.cmd.build:horizon:Failed:
INFO:kolla.cmd.build:horizon: httpd.x86_64 0:2.4.6-40.el7.centos
INFO:kolla.cmd.build:horizon:Complete!
INFO:kolla.cmd.build:horizon:Removing intermediate container 0acca1130bfd
ERROR:kolla.cmd.build:horizon:Error'd with the following message
ERROR:kolla.cmd.build:horizon:The command '/bin/sh -c yum install -y httpd mod_wsgi && yum clean all && sed -i -r 's,^(Listen 80),#\1,' /etc/httpd/conf/httpd.conf' returned a non-zero code: 1

# git log --oneline
e081c74 Merge "Added Kibana instructions to quickstart.rst"
a69977c Merge "Fix the Neutron reconfigure"
...

[Daneyon Hansen (danehans)]

Build still failing

INFO:kolla.cmd.build:aodh-base:Built
INFO:kolla.cmd.build:manila-base:Building
INFO:kolla.cmd.build:aodh-base:Pushed successfully
ERROR:kolla.cmd.build:Make sure Docker is running and that you have the correct privileges to run Docker (root)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/kolla/cmd/build.py", line 154, in run
    self.builder(image)
  File "/usr/local/lib/python2.7/dist-packages/kolla/cmd/build.py", line 267, in builder
    self.process_source(image, image['source'])
  File "/usr/local/lib/python2.7/dist-packages/kolla/cmd/build.py", line 173, in process_source
    r = requests.get(source['source'], timeout=self.conf.timeout)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 608, in send
    r.content
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 737, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 667, in generate
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='tarballs.openstack.org', port=80): Read timed out.
^C

root@k03-13:~/kolla# whoami
root

root@k03-13:~/kolla# docker --version
Docker version 1.10.3, build 20f81dd

root@k03-13:~/kolla# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
10.30.118.91:4000/kollaglue/centos-source-aodh-base 2.0.0 2b457243c9dd 6 minutes ago 706.2 MB
...

[Daneyon Hansen (danehans)]

Latest build error using f0c18db as my tip:

INFO:kolla.cmd.build:neutron-base:Pushed successfully
ERROR:kolla.cmd.build:Make sure Docker is running and that you have the correct privileges to run Docker (root)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/kolla/cmd/build.py", line 154, in run
    self.builder(image)
  File "/usr/local/lib/python2.7/dist-packages/kolla/cmd/build.py", line 267, in builder
    self.process_source(image, image['source'])
  File "/usr/local/lib/python2.7/dist-packages/kolla/cmd/build.py", line 173, in process_source
    r = requests.get(source['source'], timeout=self.conf.timeout)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 608, in send
    r.content
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 737, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 667, in generate
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='tarballs.openstack.org', port=80): Read timed out.

[Daneyon Hansen (danehans)]

This is a more complete view of the error in #2

INFO:kolla.cmd.build:neutron-base:Successfully built d763f5bfa103
INFO:kolla.cmd.build:neutron-base:Built
INFO:kolla.cmd.build:magnum-base:Building
INFO:kolla.cmd.build:neutron-base:Pushed successfully
ERROR:kolla.cmd.build:Make sure Docker is running and that you have the correct privileges to run Docker (root)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/kolla/cmd/build.py", line 154, in run
    self.builder(image)
  File "/usr/local/lib/python2.7/dist-packages/kolla/cmd/build.py", line 267, in builder
    self.process_source(image, image['source'])
  File "/usr/local/lib/python2.7/dist-packages/kolla/cmd/build.py", line 173, in process_source
    r = requests.get(source['source'], timeout=self.conf.timeout)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 608, in send
    r.content
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 737, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 667, in generate
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='tarballs.openstack.org', port=80): Read timed out.

[Daneyon Hansen (danehans)]

My keystone image build continues to fail. Here is the command and associated errors:

# kolla-build --type source --registry 10.30.118.91:4000 --push keystone

...
INFO:kolla.cmd.build:keystone:Error unpacking rpm package httpd-2.4.6-40.el7.centos.x86_64
INFO:kolla.cmd.build:keystone:
INFO:kolla.cmd.build:keystone:
....
INFO:kolla.cmd.build:keystone:Failed:
INFO:kolla.cmd.build:keystone: httpd.x86_64 0:2.4.6-40.el7.centos
INFO:kolla.cmd.build:keystone:Complete!
INFO:kolla.cmd.build:keystone:Removing intermediate container d70b425a1713
ERROR:kolla.cmd.build:keystone:Error'd with the following message
ERROR:kolla.cmd.build:keystone:The command '/bin/sh -c yum -y install httpd mod_wsgi && yum clean all && sed -i -r 's,^(Listen 80),#\1,' /etc/httpd/conf/httpd.conf' returned a non-zero code: 1
INFO:kolla.cmd.build:Successfully built images
INFO:kolla.cmd.build:=========================
INFO:kolla.cmd.build:openstack-base
INFO:kolla.cmd.build:base
INFO:kolla.cmd.build:Images that failed to build
INFO:kolla.cmd.build:===========================
ERROR:kolla.cmd.build:keystone Failed with status: error
({'keystone': 'error'}, {'openstack-base': 'built', 'base': 'built'}, {'aodh-expirer': 'unmatched', 'heat-api-cfn': 'unmatched', 'cinder-base': 'unmatched', 'swift-container': 'unmatched', 'designate-api': 'unmatched', 'trove-api': 'unmatched', 'heat-engine': 'unmatched', 'gnocchi-base': 'unmatched', 'openvswitch-vswitchd': 'unmatched', 'nova-network': 'unmatched', 'trove-base': 'unmatched', 'designate-central': 'unmatched', 'aodh-base': 'unmatched', 'murano-engine': 'unmatched', 'neutron-server': 'unmatched', 'magnum-base': 'unmatched', 'magnum-conductor': 'unmatched', 'murano-api': 'unmatched', 'mistral-executor': 'unmatched', 'mesos-dns': 'unmatched', 'nova-spicehtml5proxy': 'unmatched', 'aodh-api': 'unmatched', 'nova-libvirt': 'unmatched', 'cinder-api': 'unmatched', 'rabbitmq': 'unmatched', 'manila-api': 'unmatched', 'nova-base': 'unmatched', 'zookeeper': 'unmatched', 'kibana': 'unmatched', 'cron': 'unmatched', 'swift-proxy-server': 'unmatched', 'aodh-evaluator': 'unmatched', 'tempest': 'unmatched', 'trove-guestagent': 'unmatched', 'neutron-linuxbridge-agent': 'unmatched', 'nova-ssh': 'unmatched', 'magnum-api': 'unmatched', 'keepalived': 'unmatched', 'ceph-mon': 'unmatched', 'memcached': 'unmatched', 'mesos-slave': 'unmatched', 'heka': 'unmatched', 'haproxy': 'unmatched', 'ceilometer-collector': 'unmatched', 'glance-api': 'unmatched', 'swift-base': 'unmatched', 'elasticsearch': 'unmatched', 'horizon': 'unmatched', 'cinder-scheduler': 'unmatched', 'designate-sink': 'unmatched', 'murano-base': 'unmatched', 'dind': 'unmatched', 'kolla-toolbox': 'unmatched', 'designate-poolmanager': 'unmatched', 'neutron-openvswitch-agent': 'unmatched', 'ceilometer-compute': 'unmatched', 'manila-scheduler': 'unmatched', 'ceph-osd': 'unmatched', 'ceilometer-central': 'unmatched', 'nova-novncproxy': 'unmatched', 'mesos-master': 'unmatched', 'designate-backend-bind9': 'unmatched', 'gnocchi-api': 'unmatched', 'cinder-rpcbind': 'unmatched', 'heat-api': 'unmatched', 'aodh-notifier': 'unmatched', 'cinder-bac...

[John Dilts (hybridindie)]

This is effecting the stable/Mitaka branch as of this morning, 4-15-2016.

There are multiple accounts of this being related to outdated AUFS and effects a lot of RedHat based distros, Centos and Amazon for sure. You can see a pretty extensive discussion here https://github.com/docker/hub-feedback/issues/461 with mutiple links to additional issues that are similar or due to the same underlying issue.

One proposed workaround is installing a dummy libcap but it's not a Docker provided package

[Steven Dake (sdake)]

Daneyon,

Are you sure you don't have a proxy problem? Dave Mccowan has solutions or the proxy issues.

Daneyon, could you try using a btrfs storage backend driver? To do so requires a partition on which to run btrfs

I think the solution to either of these scenarios is documentation fixes.

ping me via cell phone and I'll log into IRC and see if I can get you going. I am on PTO monday/tuesday which is why I mention the celll phone.

[Daneyon Hansen (danehans)]

Steve,

I was able to build the images using ubuntu base. For example:

$ kolla-build --type source --base ubuntu --registry 10.30.118.91:4000 --no-cache --push magnum

I have several other machines in the same lab, on the same network. None of them are configured to use a proxy and are able to pull packages and generally hit the Internet.

[Vikram Hosakote (vhosakot)]

Moved to invalid state as rebuilding resolves the issue, and intermittent network issues cause build breakage due to unreachable mirrors.

Please re-open this bug if the issue is seen again.

[elrondwang (elrondwong)]

I didn't understand how to resolve. I have encountered this problem too.
My rabbitmq image build continues to fail.Here is the command and associated errors:

INFO:kolla.cmd.build:ironic-pxe
INFO:kolla.cmd.build:gnocchi-statsd
INFO:kolla.cmd.build:trove-conductor
INFO:kolla.cmd.build:Images that failed to build
INFO:kolla.cmd.build:===========================
ERROR:kolla.cmd.build:heka Failed with status: error
ERROR:kolla.cmd.build:rabbitmq Failed with status: error
({'heka': 'error', 'rabbitmq': 'error'}, {'aodh-expirer': 'built', 'nova-api': 'built', 'cinder-base': 'built', 'swift-container': 'built', 'designate-api': 'built', 'kolla-toolbox': 'built', 'heat-engine': 'built', 'gnocchi-base': 'built', 'openvswitch-vswitchd': 'built', 'nova-network': 'built', 'trove-base': 'built', 'heat-api-cfn': 'built', 'aodh-base': 'built', 'murano-engine': 'built', 'neutron-server': 'built', 'magnum-base': 'built', 'magnum-conductor': 'built', 'murano-api': 'built', 'mistral-executor': 'built', 'nova-spicehtml5proxy': 'built', 'aodh-api': 'built', 'nova-libvirt': 'built', 'cinder-api': 'built', 'manila-api': 'built', 'nova-base': 'built', 'kibana': 'built', 'cron': 'built', 'swift-proxy-server': 'built', 'aodh-evaluator': 'built', 'tempest': 'built', 'trove-guestagent': 'built', 'neutron-linuxbridge-agent': 'built', 'nova-ssh': 'built', 'magnum-api': 'built', 'keepalived': 'built', 'iscsid': 'built', 'openstack-base': 'built', 'keystone': 'built', 'mongodb': 'built', 'ceph-mon': 'built', 'memcached': 'built', 'base': 'built', 'tgtd': 'built', 'haproxy': 'built', 'ceilometer-collector': 'built', 'glance-api': 'built', 'swift-base': 'built', 'elasticsearch': 'built', 'horizon': 'built', 'cinder-scheduler': 'built', 'designate-sink': 'built', 'murano-base': 'built', 'dind': 'built', 'trove-api': 'built', 'designate-poolmanager': 'built', 'neutron-openvswitch-agent': 'built', 'ceilometer-compute': 'built', 'manila-scheduler': 'built', 'ceilometer-central': 'built', 'nova-novncproxy': 'built', 'designate-backend-bind9': 'built', 'gnocchi-api': 'built', 'ceilometer-base': 'built', 'heat-api': 'built', 'aodh-notifier': 'built', 'cinder-backup': 'built', 'nova-conductor': 'built', 'neutron-base': 'built', 'designate-central': 'built', 'neutron-l3-agent': 'built', 'ironic-api': 'built', 'mistral-base': 'built', 'mistral-api': 'built', 'cinder-rpcbind': 'built', 'swift-object': 'built', 'openvswitch-base': 'built', 'ceph-base': 'built', 'cinder-volume': 'built', 'ceilometer-api': 'built', 'trove-taskmanager': 'built', 'swift-rsyncd': 'built', 'aodh-listener': 'built', 'glance-registry': 'built', 'openvswitch-db-server': 'built', 'ceph-rgw': 'built', 'ironic-base': 'built', 'mistral-engine': 'built', 'zaqar': 'built', 'nova-consoleauth': 'built', 'designate-base': 'built', 'neutron-dhcp-agent': 'built', 'heat-base': 'built', 'designate-mdns': 'built', 'ceilometer-notification': 'built', 'ironic-inspector': 'built', 'ironic-conductor': 'built', 'nova-scheduler': 'built', 'manila-base': 'built', 'manila-share': 'built', 'ceph-osd': 'built', 'mariadb': 'built', 'swift-account': 'built', 'nova-compute': 'built', 'neutron-metadata-agent': 'built', 'nova-compute-ironic...