cloud-archive is missing cache-control, leading to apparent signature issues

[Francis Ginther]

Found during an automated landscape autopilot CI: https://ci.lscape.net/job/landscape-system-tests/5109/

Using landscape: 17.01~bzr10851+jenkins3405-2
openstack: newton
keystone-charm: cs:xenial/keystone-261
charm config: {
    "openstack-origin": openstack_origin,
    "admin-token": self.region_config.admin_token,
    "admin-password": self.region_config.admin_password
}
where openstack_origin should be "cloud:xenial-updates/newton".

The following log shows a successful installation of ubuntu-cloud-keyring, but then a BADSIG on the cloud-archive when trying to update sources. The later apt-get commands fail due to unauthenticated sources.

[from landscape-0-inner-logs/keystone-1/var/log/juju/unit-keystone-1.log]
2017-01-26 16:49:59 INFO juju-log Installing ubuntu-cloud-keyring with options: ['--option=Dpkg::Options::=--force-confold']
2017-01-26 16:49:59 INFO install Reading package lists...
2017-01-26 16:49:59 INFO install Building dependency tree...
2017-01-26 16:49:59 INFO install Reading state information...
2017-01-26 16:49:59 INFO install The following NEW packages will be installed:
2017-01-26 16:49:59 INFO install ubuntu-cloud-keyring
2017-01-26 16:50:00 INFO install 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
2017-01-26 16:50:00 INFO install Need to get 5086 B of archives.
2017-01-26 16:50:00 INFO install After this operation, 34.8 kB of additional disk space will be used.
2017-01-26 16:50:00 INFO install Get:1 http://archive.ubuntu.com/ubuntu xenial/universe amd64 ubuntu-cloud-keyring all 2012.08.14 [5086 B]
2017-01-26 16:50:00 INFO install Fetched 5086 B in 0s (471 kB/s)
2017-01-26 16:50:00 INFO install Selecting previously unselected package ubuntu-cloud-keyring.
2017-01-26 16:50:00 INFO install (Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 32334 files and directories currently installed.)
2017-01-26 16:50:00 INFO install Preparing to unpack .../ubuntu-cloud-keyring_2012.08.14_all.deb ...
2017-01-26 16:50:00 INFO install Unpacking ubuntu-cloud-keyring (2012.08.14) ...
2017-01-26 16:50:00 INFO install Setting up ubuntu-cloud-keyring (2012.08.14) ...
2017-01-26 16:50:00 INFO install Importing ubuntu-cloud.archive.canonical.com keyring
2017-01-26 16:50:00 INFO install OK
2017-01-26 16:50:00 INFO install Processing ubuntu-cloud.archive.canonical.com removal keyring
2017-01-26 16:50:00 INFO install gpg: /etc/apt/trustdb.gpg: trustdb created
2017-01-26 16:50:00 INFO install OK
2017-01-26 16:50:02 INFO install Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
2017-01-26 16:50:02 INFO install Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
2017-01-26 16:50:03 INFO install Ign:3 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton InRelease
2017-01-26 16:50:03 INFO install Hit:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
2017-01-26 16:50:03 INFO install Get:5 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton Release [7881 B]
2017-01-26 16:50:03 INFO install Hit:6 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
2017-01-26 16:50:03 INFO install Get:7 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton Release.gpg [543 B]
2017-01-26 16:50:03 INFO install Ign:7 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton Release.gpg
2017-01-26 16:50:03 INFO install Get:8 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton/main amd64 Packages [119 kB]
2017-01-26 16:50:03 INFO install Fetched 128 kB in 0s (151 kB/s)
2017-01-26 16:50:05 INFO install Reading package lists...
2017-01-26 16:50:05 INFO install W: GPG error: http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton Release: The following signatures were invalid: BADSIG 5EDB1B62EC4926EA Canonical Cloud Archive Signing Key <email address hidden>
2017-01-26 16:50:05 INFO install W: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton Release' is not signed.
2017-01-26 16:50:05 INFO juju-log Installing ['apache2', 'haproxy', 'keystone', 'libapache2-mod-wsgi', 'openssl', 'pwgen', 'python-keystoneclient', 'python-mysqldb', 'python-psycopg2', 'python-six', 'unison', 'uuid'] with options: ['--option=Dpkg::Options::=--force-confold']
2017-01-26 16:50:05 INFO install Reading package lists...
2017-01-26 16:50:06 INFO install Building dependency tree...
2017-01-26 16:50:06 INFO install Reading state information...
2017-01-26 16:50:06 INFO install python-six is already the newest version (1.10.0-3).
2017-01-26 16:50:06 INFO install openssl is already the newest version (1.0.2g-1ubuntu4.5).
2017-01-26 16:50:06 INFO install The following additional packages will be installed:
2017-01-26 16:50:06 INFO install alembic apache2-bin apache2-data apache2-utils docutils-common docutils-doc
2017-01-26 16:50:06 INFO install formencode-i18n javascript-common libapr1 libaprutil1
2017-01-26 16:50:06 INFO install libaprutil1-dbd-sqlite3 libaprutil1-ldap libfreetype6 libjbig0
2017-01-26 16:50:06 INFO install libjpeg-turbo8 libjpeg8 libjs-jquery libjs-sphinxdoc libjs-underscore
2017-01-26 16:50:06 INFO install liblcms2-2 liblua5.1-0 liblua5.3-0 libmysqlclient20 libossp-uuid16
2017-01-26 16:50:06 INFO install libpaper-utils libpaper1 libpq5 libtiff5 libwebp5 libwebpmux1 libxmlsec1
2017-01-26 16:50:06 INFO install libxmlsec1-openssl libxslt1.1 mysql-common pycadf-common python-alembic
2017-01-26 16:50:06 INFO install python-amqp python-anyjson python-babel python-babel-localedata
2017-01-26 16:50:06 INFO install python-blinker python-bs4 python-cachetools python-cffi-backend
2017-01-26 16:50:06 INFO install python-chardet python-concurrent.futures python-contextlib2 python-crypto
2017-01-26 16:50:06 INFO install python-cryptography python-dateutil python-dbus python-debtcollector
2017-01-26 16:50:06 INFO install python-decorator python-docutils python-dogpile.cache
2017-01-26 16:50:06 INFO install python-egenix-mxdatetime python-egenix-mxtools python-enum34 python-eventlet
2017-01-26 16:50:06 INFO install python-extras python-fasteners python-fixtures python-formencode
2017-01-26 16:50:06 INFO install python-funcsigs python-functools32 python-futurist python-gi python-greenlet
2017-01-26 16:50:06 INFO install python-html5lib python-idna python-ipaddress python-iso8601
2017-01-26 16:50:06 INFO install python-jsonschema python-jwt python-keyring python-keystone
2017-01-26 16:50:06 INFO install python-keystoneauth1 python-keystonemiddleware python-kombu
2017-01-26 16:50:06 INFO install python-linecache2 python-lxml python-mako python-memcache python-migrate
2017-01-26 16:50:06 INFO install python-mimeparse python-mock python-monotonic python-msgpack
2017-01-26 16:50:06 INFO install python-ndg-httpsclient python-oauthlib python-openid python-openssl
2017-01-26 16:50:06 INFO install python-oslo.cache python-oslo.concurrency python-oslo.config
2017-01-26 16:50:06 INFO install python-oslo.context python-oslo.db python-oslo.i18n python-oslo.log
2017-01-26 16:50:06 INFO install python-oslo.messaging python-oslo.middleware python-oslo.policy
2017-01-26 16:50:06 INFO install python-oslo.serialization python-oslo.service python-oslo.utils
2017-01-26 16:50:06 INFO install python-osprofiler python-passlib python-paste python-pastedeploy
2017-01-26 16:50:06 INFO install python-pastedeploy-tpl python-pastescript python-pbr python-pika
2017-01-26 16:50:06 INFO install python-pika-pool python-pil python-positional python-posix-ipc python-pyasn1
2017-01-26 16:50:06 INFO install python-pycadf python-pygments python-pyinotify python-pymysql
2017-01-26 16:50:06 INFO install python-pyparsing python-pysaml2 python-repoze.lru python-requests
2017-01-26 16:50:06 INFO install python-retrying python-rfc3986 python-roman python-routes python-scgi
2017-01-26 16:50:06 INFO install python-secretstorage python-sqlalchemy python-sqlalchemy-ext python-sqlparse
2017-01-26 16:50:06 INFO install python-stevedore python-tempita python-testtools python-traceback2 python-tz
2017-01-26 16:50:06 INFO install python-unittest2 python-urllib3 python-webob python-wrapt
2017-01-26 16:50:06 INFO install python-zope.interface ssl-cert xmlsec1
2017-01-26 16:50:06 INFO install Suggested packages:
2017-01-26 16:50:06 INFO install www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom
2017-01-26 16:50:06 INFO install vim-haproxy haproxy-doc liblcms2-utils python-editor python-amqp-doc
2017-01-26 16:50:06 INFO install python-blinker-doc python-crypto-dbg python-crypto-doc
2017-01-26 16:50:06 INFO install python-cryptography-doc python-cryptography-vectors python-dbus-doc
2017-01-26 16:50:06 INFO install python-dbus-dbg python-debtcollector-doc texlive-latex-recommended
2017-01-26 16:50:06 INFO install texlive-latex-base texlive-lang-french fonts-linuxlibertine
2017-01-26 16:50:06 INFO install | ttf-linux-libertine python-egenix-mxdatetime-dbg
2017-01-26 16:50:06 INFO install python-egenix-mxdatetime-doc python-egenix-mxtools-dbg
2017-01-26 16:50:06 INFO install python-egenix-mxtools-doc python-enum34-doc python-funcsigs-doc
2017-01-26 16:50:06 INFO install python-futurist-doc python-gi-cairo python-greenlet-doc python-greenlet-dev
2017-01-26 16:50:06 INFO install python-greenlet-dbg python-genshi gir1.2-gnomekeyring-1.0 python-fs
2017-01-26 16:50:06 INFO install python-gdata python-kde4 python-keyczar python-ldap python-ldappool
2017-01-26 16:50:06 INFO install python-keystoneauth1-doc python-requests-kerberos python-beanstalkc
2017-01-26 16:50:06 INFO install python-boto python-django python-kombu-doc python-pymongo python-redis
2017-01-26 16:50:06 INFO install python-lxml-dbg python-lxml-doc python-beaker python-mako-doc memcached
2017-01-26 16:50:06 INFO install python-mock-doc mysql-server python-mysqldb-dbg python-openssl-doc
2017-01-26 16:50:06 INFO install python-openssl-dbg python-oslo.cache-doc python-oslo.concurrency-doc
2017-01-26 16:50:06 INFO install python-oslo.log-doc python-kafka python-zmq python-oslo.middleware-doc
2017-01-26 16:50:06 INFO install python-oslo.policy-doc python-oslo.service-doc python-pastewebkit
2017-01-26 16:50:06 INFO install libjs-mochikit libapache2-mod-python libapache2-mod-scgi python-pgsql
2017-01-26 16:50:06 INFO install python-flup python-cherrypy python-cheetah python-pika-doc python-pil-doc
2017-01-26 16:50:06 INFO install python-pil-dbg python-positional-doc python-psycopg2-doc doc-base
2017-01-26 16:50:06 INFO install ttf-bitstream-vera python-pyinotify-doc python-pymysql-doc python-repoze.who
2017-01-26 16:50:06 INFO install python-socks gnome-keyring python-secretstorage-doc python-sqlalchemy-doc
2017-01-26 16:50:06 INFO install python-fdb python-pymssql python-sqlparse-doc python-testtools-doc
2017-01-26 16:50:06 INFO install python-twisted python-ntlm python-webob-doc openssl-blacklist unison-all
2017-01-26 16:50:06 INFO install The following NEW packages will be installed:
2017-01-26 16:50:06 INFO install alembic apache2 apache2-bin apache2-data apache2-utils docutils-common
2017-01-26 16:50:06 INFO install docutils-doc formencode-i18n haproxy javascript-common keystone
2017-01-26 16:50:06 INFO install libapache2-mod-wsgi libapr1 libaprutil1 libaprutil1-dbd-sqlite3
2017-01-26 16:50:06 INFO install libaprutil1-ldap libfreetype6 libjbig0 libjpeg-turbo8 libjpeg8 libjs-jquery
2017-01-26 16:50:06 INFO install libjs-sphinxdoc libjs-underscore liblcms2-2 liblua5.1-0 liblua5.3-0
2017-01-26 16:50:06 INFO install libmysqlclient20 libossp-uuid16 libpaper-utils libpaper1 libpq5 libtiff5
2017-01-26 16:50:06 INFO install libwebp5 libwebpmux1 libxmlsec1 libxmlsec1-openssl libxslt1.1 mysql-common
2017-01-26 16:50:06 INFO install pwgen pycadf-common python-alembic python-amqp python-anyjson python-babel
2017-01-26 16:50:06 INFO install python-babel-localedata python-blinker python-bs4 python-cachetools
2017-01-26 16:50:06 INFO install python-cffi-backend python-chardet python-concurrent.futures
2017-01-26 16:50:06 INFO install python-contextlib2 python-crypto python-cryptography python-dateutil
2017-01-26 16:50:06 INFO install python-dbus python-debtcollector python-decorator python-docutils
2017-01-26 16:50:06 INFO install python-dogpile.cache python-egenix-mxdatetime python-egenix-mxtools
2017-01-26 16:50:06 INFO install python-enum34 python-eventlet python-extras python-fasteners python-fixtures
2017-01-26 16:50:06 INFO install python-formencode python-funcsigs python-functools32 python-futurist
2017-01-26 16:50:06 INFO install python-gi python-greenlet python-html5lib python-idna python-ipaddress
2017-01-26 16:50:06 INFO install python-iso8601 python-jsonschema python-jwt python-keyring python-keystone
2017-01-26 16:50:06 INFO install python-keystoneauth1 python-keystoneclient python-keystonemiddleware
2017-01-26 16:50:06 INFO install python-kombu python-linecache2 python-lxml python-mako python-memcache
2017-01-26 16:50:06 INFO install python-migrate python-mimeparse python-mock python-monotonic python-msgpack
2017-01-26 16:50:06 INFO install python-mysqldb python-ndg-httpsclient python-oauthlib python-openid
2017-01-26 16:50:06 INFO install python-openssl python-oslo.cache python-oslo.concurrency python-oslo.config
2017-01-26 16:50:06 INFO install python-oslo.context python-oslo.db python-oslo.i18n python-oslo.log
2017-01-26 16:50:06 INFO install python-oslo.messaging python-oslo.middleware python-oslo.policy
2017-01-26 16:50:06 INFO install python-oslo.serialization python-oslo.service python-oslo.utils
2017-01-26 16:50:06 INFO install python-osprofiler python-passlib python-paste python-pastedeploy
2017-01-26 16:50:06 INFO install python-pastedeploy-tpl python-pastescript python-pbr python-pika
2017-01-26 16:50:06 INFO install python-pika-pool python-pil python-positional python-posix-ipc
2017-01-26 16:50:06 INFO install python-psycopg2 python-pyasn1 python-pycadf python-pygments python-pyinotify
2017-01-26 16:50:06 INFO install python-pymysql python-pyparsing python-pysaml2 python-repoze.lru
2017-01-26 16:50:06 INFO install python-requests python-retrying python-rfc3986 python-roman python-routes
2017-01-26 16:50:06 INFO install python-scgi python-secretstorage python-sqlalchemy python-sqlalchemy-ext
2017-01-26 16:50:06 INFO install python-sqlparse python-stevedore python-tempita python-testtools
2017-01-26 16:50:06 INFO install python-traceback2 python-tz python-unittest2 python-urllib3 python-webob
2017-01-26 16:50:06 INFO install python-wrapt python-zope.interface ssl-cert unison uuid xmlsec1
2017-01-26 16:50:06 INFO install 0 upgraded, 157 newly installed, 0 to remove and 9 not upgraded.
2017-01-26 16:50:06 INFO install Need to get 19.7 MB of archives.
2017-01-26 16:50:06 INFO install After this operation, 101 MB of additional disk space will be used.
2017-01-26 16:50:06 INFO install WARNING: The following packages cannot be authenticated!
2017-01-26 16:50:06 INFO install python-alembic alembic pycadf-common python-babel-localedata python-babel
2017-01-26 16:50:06 INFO install python-cryptography python-pbr python-funcsigs python-dogpile.cache
2017-01-26 16:50:06 INFO install python-fixtures python-mock python-positional python-urllib3 python-requests
2017-01-26 16:50:06 INFO install python-stevedore python-keystoneauth1 python-oslo.i18n python-oslo.config
2017-01-26 16:50:06 INFO install python-oslo.utils python-oslo.serialization python-keystoneclient
2017-01-26 16:50:06 INFO install python-oslo.context python-pycadf python-webob python-keystonemiddleware
2017-01-26 16:50:06 INFO install python-openssl python-oslo.log python-oslo.cache python-oslo.concurrency
2017-01-26 16:50:06 INFO install python-oslo.db python-oslo.middleware python-oslo.service
2017-01-26 16:50:06 INFO install python-oslo.messaging python-oslo.policy python-osprofiler python-keystone
2017-01-26 16:50:06 INFO install keystone
2017-01-26 16:50:06 INFO install E: There were unauthenticated packages and -y was used without --allow-unauthenticated

[James Page (james-page)]

I'm not able to reproduce this using 16.04 and the Newton UCA sources; the archive is signed.

Its possible this was some sort of transient signing issue with the files in the cloud archive, but its definitely not a charm specific problem.

[Francis Ginther (fginther)]

I suspect the maas squid proxy may have gotten in the way and sent a stale file. I'll comb through the proxy logs if this comes up again. For now, incomplete/invalid sounds appropriate.

[Jason Hobbs (jason-hobbs)]

We hit this last night: http://paste.ubuntu.com/26506640/

[Jason Hobbs (jason-hobbs)]

The run from last night was an OpenStack deploy against MAAS, and would have been using MAAS's proxy.

[Jason Hobbs (jason-hobbs)]

And we hit it again a few hours later, on a different MAAS install.

http://paste.ubuntu.com/26506664/

[Jason Hobbs (jason-hobbs)]

and again a few hours later on another MAAS: http://paste.ubuntu.com/26506708/

[Jason Hobbs (jason-hobbs)]

And again http://paste.ubuntu.com/26506742/

[Jason Hobbs (jason-hobbs)]

Each of these failures was on a fresh MAAS deployment, so the MAAS proxy used there doesn't seem like it could be to blame. However, each MAAS deployment was configured to us an upstream IS provided proxy (squid.internal) - maybe something with that proxy was to blame?

[Jason Hobbs (jason-hobbs)]

It looks like the problem is that ubuntu-cloud.archive.canonical.com does not specify any cache-control header to prevent the proxy from caching Release or Release.gpg. So, the proxy is caching a Release that does not match Release.gpg.

See the difference in response headers between ubuntu-cloud.archive.canonical.com and archive.ubuntu.com:

http://paste.ubuntu.com/26508687/

[Jason Hobbs (jason-hobbs)]

Canonical IS has configured cache-control on ubuntu-cloud.archive.canonical.com to use proxy-revalidate, so this should be fixed.