Unreliability when retrieving external resources (repository, signing keys)

[David Moreau Simard]

I've attempted 5 times to build CentOS Binary Kolla containers (through tox -e build-centos-binary). All 5 times failed even before starting to really build anything.
These attempts were done in three different locations to try and see if it was not a problem on my end.

Four times failed due to the Percona repository not being available:
===
INFO:kolla.image.build.base:Step 16 : RUN yum -y install http://repo.percona.com/release/7/RPMS/x86_64/percona-release-0.1-4.noarch.rpm && yum clean all
INFO:kolla.image.build.base: ---> Running in 2a97105fdfbb
INFO:kolla.image.build.base:Loaded plugins: fastestmirror, ovl
INFO:kolla.image.build.base:Cannot open: http://repo.percona.com/release/7/RPMS/x86_64/percona-release-0.1-4.noarch.rpm. Skipping.
INFO:kolla.image.build.base:Error: Nothing to do
INFO:kolla.image.build.base:
INFO:kolla.image.build.base:Removing intermediate container 2a97105fdfbb
ERROR:kolla.image.build.base:Error'd with the following message
===

And the other time it failed due to an error in fetching (one?) of the signing keys:
===
INFO:kolla.image.build.base:Step 17 : RUN true && rpm --import http://yum.mariadb.org/RPM-GPG-KEY-MariaDB && rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-Percona && rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch && rpm --import https://repos.influxdata.com/influxdb.key && rpm --import https://packagecloud.io/gpg.key && rpm --import https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
INFO:kolla.image.build.base: ---> Running in f1f9d9c45871
INFO:kolla.image.build.base:curl: (28) Operation timed out after 300508 milliseconds with 0 out of 0 bytes received
INFO:kolla.image.build.base:
INFO:kolla.image.build.base:error: https://repos.influxdata.com/influxdb.key: import read failed(2).
INFO:kolla.image.build.base:
INFO:kolla.image.build.base:Removing intermediate container f1f9d9c45871
ERROR:kolla.image.build.base:Error'd with the following message
===

If resources are not reliably available from the internet, they should be cached/stored locally.

[Jeffrey Zhang (jeffrey4l)]

could u try again?

If is hard for kolla to cache the data. another solution is set up a mirror site for each resource.

[David Moreau Simard (dmsimard)]

Steven, Jeffrey, I ended up figuring out the root cause of the issue and it boils down to a MTU misconfiguration.

I am trying to build containers in OpenStack clouds that provide a default MTU of 1400 to the virtual machines. So the compute node has 1500, the VM has 1400 and the docker container (on the VM) defaults to 1500. The VMs are set up to use 1400 to account for GRE/VXLAN overhead (http://codesearch.openstack.org/?q=dhcp-option-force%3D26%2C1400&i=nope&files=&repos=).
This lead some network transactions to fail.

I am able to work around the issue by "manually" lowering the docker container MTU below 1400 but since I'm using the baremetal role to set this up, it gets in my way and I filed a bug about it here: https://bugs.launchpad.net/kolla-ansible/+bug/1654395

You can probably close this bug.