Landscape Charm

add_source() might fail due to network errors

Bug #1370933 reported by Björn Tillenius on 2014-09-18

This bug affects 3 people

	Status	Importance	Assigned to
Autopilot Log Analyser	Fix Released	Low	Francis Ginther
Charm Helpers	Fix Released	Undecided	Chad Smith
Glance - Simplestreams Sync Charm	Fix Committed	Undecided	Chad Smith
Landscape Charm	Fix Released	Undecided	Chad Smith
landscape-client-charm	Fix Released	Undecided	Chad Smith

Bug Description

I saw the following error when installing landscape-client, which uses charm-helper:

2014-09-18 07:34:50 INFO install Cannot add PPA: 'ppa:landscape/trunk'.
2014-09-18 07:34:50 INFO install Please check that the PPA name or format is correct.
2014-09-18 07:34:50 INFO install Traceback (most recent call last):
2014-09-18 07:34:50 INFO install File "/var/lib/juju/agents/unit-landscape-client-0/charm/hooks/install", line 93, in <module>
2014-09-18 07:34:50 INFO install sys.exit(hooks.execute(sys.argv))
2014-09-18 07:34:50 INFO install File "/var/lib/juju/agents/unit-landscape-client-0/charm/hooks/charmhelpers/core/hookenv.py", line 381, in execute
2014-09-18 07:34:50 INFO install self._hooks[hook_name]()
2014-09-18 07:34:50 INFO install File "/var/lib/juju/agents/unit-landscape-client-0/charm/hooks/install", line 34, in install
2014-09-18 07:34:50 INFO install add_apt_source(origin)
2014-09-18 07:34:50 INFO install File "/var/lib/juju/agents/unit-landscape-client-0/charm/hooks/install", line 89, in add_apt_source
2014-09-18 07:34:50 INFO install return add_source(url, key)
2014-09-18 07:34:50 INFO install File "/var/lib/juju/agents/unit-landscape-client-0/charm/hooks/charmhelpers/fetch/__init__.py", line 198, in add_source
2014-09-18 07:34:50 INFO install subprocess.check_call(['add-apt-repository', '--yes', source])
2014-09-18 07:34:50 INFO install File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
2014-09-18 07:34:50 INFO install raise CalledProcessError(retcode, cmd)
2014-09-18 07:34:50 INFO install subprocess.CalledProcessError: Command '['add-apt-repository', '--yes', u'ppa:landscape/trunk']' returned non-zero exit status 1
2014-09-18 07:34:50 ERROR juju.worker.uniter uniter.go:486 hook failed: exit status 1

When I retried the hook it succeeded, so it was probably a networking error. add_source() should retry adding it, in case there are network failures.

Tags:

Related branches

lp:~chad.smith/charm-helpers/retry-add-apt-repository

Merged into lp:charm-helpers at revision 705

Eric Snow (community): Approve on 2017-03-03

David Britton (community): Approve on 2017-03-03

lp:charm-helpers

lp:~chad.smith/charms/trusty/glance-simplestreams-sync/sync-charmhelpers-for-add-apt-repo-retry

Ready for review for merging into lp:~openstack-charmers/charms/trusty/glance-simplestreams-sync/next

OpenStack Charmers: Pending requested 2017-03-04

Diff: 10125 lines (+5920/-1725)

74 files modified

charm-helpers-sync.yaml (+1/-0)
charmhelpers/__init__.py (+11/-13)
charmhelpers/contrib/__init__.py (+11/-13)
charmhelpers/contrib/charmsupport/__init__.py (+11/-13)
charmhelpers/contrib/charmsupport/nrpe.py (+103/-34)
charmhelpers/contrib/charmsupport/volumes.py (+11/-13)
charmhelpers/contrib/hahelpers/__init__.py (+11/-13)
charmhelpers/contrib/hahelpers/apache.py (+30/-17)
charmhelpers/contrib/hahelpers/cluster.py (+70/-23)
charmhelpers/contrib/network/__init__.py (+11/-13)
charmhelpers/contrib/network/ip.py (+99/-42)
charmhelpers/contrib/openstack/__init__.py (+11/-13)
charmhelpers/contrib/openstack/alternatives.py (+11/-13)
charmhelpers/contrib/openstack/amulet/__init__.py (+11/-13)
charmhelpers/contrib/openstack/amulet/deployment.py (+200/-69)
charmhelpers/contrib/openstack/amulet/utils.py (+354/-38)
charmhelpers/contrib/openstack/context.py (+313/-127)
charmhelpers/contrib/openstack/exceptions.py (+21/-0)
charmhelpers/contrib/openstack/files/__init__.py (+11/-13)
charmhelpers/contrib/openstack/files/check_haproxy.sh (+7/-5)
charmhelpers/contrib/openstack/ha/__init__.py (+13/-0)
charmhelpers/contrib/openstack/ha/utils.py (+139/-0)
charmhelpers/contrib/openstack/ip.py (+60/-25)
charmhelpers/contrib/openstack/keystone.py (+178/-0)
charmhelpers/contrib/openstack/neutron.py (+57/-23)
charmhelpers/contrib/openstack/templates/__init__.py (+11/-13)
charmhelpers/contrib/openstack/templates/haproxy.cfg (+19/-11)
charmhelpers/contrib/openstack/templates/memcached.conf (+53/-0)
charmhelpers/contrib/openstack/templates/openstack_https_frontend (+5/-0)
charmhelpers/contrib/openstack/templates/openstack_https_frontend.conf (+5/-0)
charmhelpers/contrib/openstack/templates/section-keystone-authtoken (+8/-5)
charmhelpers/contrib/openstack/templates/section-keystone-authtoken-legacy (+10/-0)
charmhelpers/contrib/openstack/templates/section-keystone-authtoken-mitaka (+20/-0)
charmhelpers/contrib/openstack/templates/wsgi-openstack-api.conf (+100/-0)
charmhelpers/contrib/openstack/templating.py (+19/-15)
charmhelpers/contrib/openstack/utils.py (+1163/-151)
charmhelpers/contrib/python/__init__.py (+11/-13)
charmhelpers/contrib/python/packages.py (+59/-26)
charmhelpers/contrib/storage/__init__.py (+11/-13)
charmhelpers/contrib/storage/linux/__init__.py (+11/-13)
charmhelpers/contrib/storage/linux/ceph.py (+766/-72)
charmhelpers/contrib/storage/linux/loopback.py (+21/-13)
charmhelpers/contrib/storage/linux/lvm.py (+11/-13)
charmhelpers/contrib/storage/linux/utils.py (+16/-18)
charmhelpers/core/__init__.py (+11/-13)
charmhelpers/core/decorators.py (+11/-13)
charmhelpers/core/files.py (+11/-13)
charmhelpers/core/fstab.py (+11/-13)
charmhelpers/core/hookenv.py (+157/-19)
charmhelpers/core/host.py (+482/-150)
charmhelpers/core/host_factory/centos.py (+56/-0)
charmhelpers/core/host_factory/ubuntu.py (+56/-0)
charmhelpers/core/hugepage.py (+13/-13)
charmhelpers/core/kernel.py (+34/-30)
charmhelpers/core/kernel_factory/centos.py (+17/-0)
charmhelpers/core/kernel_factory/ubuntu.py (+13/-0)
charmhelpers/core/services/__init__.py (+11/-13)
charmhelpers/core/services/base.py (+11/-13)
charmhelpers/core/services/helpers.py (+25/-18)
charmhelpers/core/strutils.py (+11/-13)
charmhelpers/core/sysctl.py (+11/-13)
charmhelpers/core/templating.py (+40/-24)
charmhelpers/core/unitdata.py (+11/-14)
charmhelpers/fetch/__init__.py (+43/-302)
charmhelpers/fetch/archiveurl.py (+12/-14)
charmhelpers/fetch/bzrurl.py (+48/-50)
charmhelpers/fetch/centos.py (+171/-0)
charmhelpers/fetch/giturl.py (+33/-37)
charmhelpers/fetch/snap.py (+122/-0)
charmhelpers/fetch/ubuntu.py (+364/-0)
charmhelpers/osplatform.py (+25/-0)
charmhelpers/payload/__init__.py (+11/-13)
charmhelpers/payload/archive.py (+11/-13)
charmhelpers/payload/execd.py (+14/-15)

lp:~chad.smith/landscape-charm/add-apt-repository-retries

Merged into lp:~landscape/landscape-charm/trunk at revision 383

🤖 Landscape Builder: Approve (test results) on 2017-03-04

Eric Snow (community): Approve on 2017-03-03

Adam Collard (adam-collard) on 2014-09-18

tags:

added: cloud-installer landscape

Revision history for this message

Jorge Niedbalski (niedbalski) wrote on 2014-12-12:

Hello @adam-collard,

How recurring does this happens? Do you think the helper library should deal with possible networking errors?

Thanks

Revision history for this message

Andreas Hasenack (ahasenack) wrote on 2016-06-22:

It happens every now and then. I just got another cloud deploy where this failed, this time in the glance-simplestreams-sync charm:

2016-06-22 15:19:05 INFO install Cannot add PPA: 'ppa:landscape/glance-simplestreams-sync'.
2016-06-22 15:19:05 INFO install Please check that the PPA name or format is correct.
2016-06-22 15:19:05 INFO install Traceback (most recent call last):
2016-06-22 15:19:05 INFO install File "/var/lib/juju/agents/unit-glance-simplestreams-sync-0/charm/hooks/install", line 281, in <module>
2016-06-22 15:19:05 INFO install hooks.execute(sys.argv)
2016-06-22 15:19:05 INFO install File "/var/lib/juju/agents/unit-glance-simplestreams-sync-0/charm/hooks/charmhelpers/core/hookenv.py", line 704, in execute
2016-06-22 15:19:05 INFO install self._hooks[hook_name]()
2016-06-22 15:19:05 INFO install File "/var/lib/juju/agents/unit-glance-simplestreams-sync-0/charm/hooks/install", line 199, in install
2016-06-22 15:19:05 INFO install add_source(hookenv.config('source'), hookenv.config('key'))
2016-06-22 15:19:05 INFO install File "/var/lib/juju/agents/unit-glance-simplestreams-sync-0/charm/hooks/charmhelpers/fetch/__init__.py", line 281, in add_source
2016-06-22 15:19:05 INFO install subprocess.check_call(['add-apt-repository', '--yes', source])
2016-06-22 15:19:05 INFO install File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
2016-06-22 15:19:05 INFO install raise CalledProcessError(retcode, cmd)
2016-06-22 15:19:05 INFO install subprocess.CalledProcessError: Command '['add-apt-repository', '--yes', u'ppa:landscape/glance-simplestreams-sync']' returned non-zero exit status 1

Ursula Junque (ursinha) on 2016-08-30

Changed in autopilot-log-analyser:
status:	New → In Progress
importance:	Undecided → Low
assignee:	nobody → Ursula Junque (ursinha)

🤖 Landscape Builder (landscape-builder) on 2016-08-30

Changed in autopilot-log-analyser:
status:	In Progress → Fix Committed

Revision history for this message

Francis Ginther (fginther) wrote on 2017-01-06:

Need to update analyser check to cover https://bugs.launchpad.net/autopilot-log-analyser/+bug/1654640.

Revision history for this message

Francis Ginther (fginther) wrote on 2017-02-06:

Another flavor of this: https://ci.lscape.net/job/landscape-system-tests/5198

Revision history for this message

Francis Ginther (fginther) wrote on 2017-02-06:

lp-1370933-ppa-not-found.tar.gz Edit (7.2 KiB, application/x-tar)

Download full text (3.7 KiB)

Here's a better comment.

Another flavor of this: https://ci.lscape.net/job/landscape-system-tests/5198

[from landscape-0-inner-logs/glance-simplestreams-sync-0/var/log/juju/unit-glance-simplestreams-sync-0.log]
2017-02-06 15:25:41 INFO install Cannot add PPA: 'ppa:~landscape/ubuntu/glance-simplestreams-sync'.
2017-02-06 15:25:41 INFO install The team named '~landscape' has no PPA named 'ubuntu/glance-simplestreams-sync'
2017-02-06 15:25:41 INFO install Please choose from the following available PPAs:
2017-02-06 15:25:41 INFO install * '14.10': LDS 14.10
2017-02-06 15:25:41 INFO install * '15.01': LDS 15.01
2017-02-06 15:25:41 INFO install * '15.01-proxy': LDS 15.01 with proxy test
2017-02-06 15:25:41 INFO install * '15.10': LDS 15.10
2017-02-06 15:25:41 INFO install * '15.10-proxy': LDS 15.10 with proxy test
2017-02-06 15:25:41 INFO install * '15.11': LDS 15.11
2017-02-06 15:25:41 INFO install * '16.03': LDS 16.03
2017-02-06 15:25:41 INFO install * '16.06': LDS 16.06
2017-02-06 15:25:41 INFO install * 'fake-kernel': fake kernel
2017-02-06 15:25:41 INFO install * 'glance-simplestreams-sync': glance simplestreams sync
2017-02-06 15:25:41 INFO install * 'landscape-api': Landscape API client
2017-02-06 15:25:41 INFO install * 'landscape-api-staging': Landscape API client staging
2017-02-06 15:25:41 INFO install * 'landscape-client-cloud-installer': Landscape Client for the LDS Cloud Installer
2017-02-06 15:25:41 INFO install * 'lds-demo': LDS Demo
2017-02-06 15:25:41 INFO install * 'lds-preview': LDS Preview
2017-02-06 15:25:41 INFO install * 'maas-1.8.2-fix-1576253': MAAS 1.8.2 packages for #1576253
2017-02-06 15:25:41 INFO install * 'opendaylight-0.2.3': OpenDaylight Helium 0.2.3
2017-02-06 15:25:41 INFO install * 'opendaylight-0.2.4': OpenDalight Helium 0.2.4 SR4
2017-02-06 15:25:41 INFO install * 'production': Landscape client production packages
2017-02-06 15:25:41 INFO install * 'staging': Landscape client staging packages
2017-02-06 15:25:41 INFO install * 'trunk': Landscape client trunk packages
2017-02-06 15:25:41 INFO install * 'txjuju-daily': txjuju daily builds
2017-02-06 15:25:41 INFO install Traceback (most recent call last):
2017-02-06 15:25:41 INFO install File "/var/lib/juju/agents/unit-glance-simplestreams-sync-0/charm/hooks/install.real", line 290, in <module>
2017-02-06 15:25:41 INFO install hooks.execute(sys.argv)
2017-02-06 15:25:41 INFO install File "/var/lib/juju/agents/unit-glance-simplestreams-sync-0/charm/hooks/charmhelpers/core/hookenv.py", line 704, in execute
2017-02-06 15:25:41 INFO install self._hooks[hook_name]()
2017-02-06 15:25:41 INFO install File "/var/lib/juju/agents/unit-glance-simplestreams-sync-0/charm/hooks/install.real", line 206, in install
2017-02-06 15:25:41 INFO install add_source(hookenv.config('source'), hookenv.config('key'))
2017-02-06 15:25:41 INFO install File "/var/lib/juju/agents/unit-glance-simplestreams-sync-0/charm/hooks/charmhelpers/fetch/__init__.py", line 281, in add_source
2017-02-06 15:25:41 INFO install subprocess.check_call(['add-apt-repository', '--yes', source])
2017-02-06 15:25:41 INFO install File "/usr/lib/...

And yet another flavor. This one from https://ci.lscape.net/job/landscape-system-tests/5125.

[from landscape-0-inner-logs/ceilometer-0/var/log/juju/unit-landscape-client-57.log]
2017-01-28 10:08:04 INFO install gpg: keyring `/tmp/tmp9p2uy5f6/secring.gpg' created
2017-01-28 10:08:04 INFO install gpg: keyring `/tmp/tmp9p2uy5f6/pubring.gpg' created
2017-01-28 10:08:04 INFO install gpg: requesting key 4652B4E6 from hkp server keyserver.ubuntu.com
2017-01-28 10:08:14 INFO install gpgkeys: key 35F77D63B5CEC106C577ED856E85A86E4652B4E6 can't be retrieved
2017-01-28 10:08:14 INFO install gpg: no valid OpenPGP data found.
2017-01-28 10:08:14 INFO install gpg: Total number processed: 0
2017-01-28 10:08:14 INFO install gpg: keyserver communications error: keyserver helper general error
2017-01-28 10:08:14 INFO install gpg: keyserver communications error: unknown pubkey algorithm
2017-01-28 10:08:14 INFO install gpg: keyserver receive failed: unknown pubkey algorithm
2017-01-28 10:08:14 INFO install Ign:1 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton InRele
ase
2017-01-28 10:08:14 INFO install Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
2017-01-28 10:08:14 INFO install Hit:3 http://archive.ubuntu.com/ubuntu xenial InRelease
2017-01-28 10:08:14 INFO install Get:4 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton Releas
e [7881 B]
2017-01-28 10:08:14 INFO install Get:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
2017-01-28 10:08:15 INFO install Get:6 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton Releas
e.gpg [543 B]
2017-01-28 10:08:15 INFO install Get:7 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
2017-01-28 10:08:15 INFO install Get:8 http://ppa.launchpad.net/landscape/landscape-client-cloud-installer/ubuntu xe
nial InRelease [17.6 kB]
2017-01-28 10:08:15 INFO install Get:9 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [202 kB
]
017-01-28 10:08:16 INFO install Ign:8 http://ppa.launchpad.net/landscape/landscape-client-cloud-installer/ubuntu xenial InRelease
2017-01-28 10:08:16 INFO install Get:10 http://ppa.launchpad.net/landscape/landscape-client-cloud-installer/ubuntu xenial/main amd64 Packages [864 B]
2017-01-28 10:08:16 INFO install Get:11 http://ppa.launchpad.net/landscape/landscape-client-cloud-installer/ubuntu xenial/main Translation-en [364 B]
2017-01-28 10:08:22 INFO install Fetched 536 kB in 2s (246 kB/s)
2017-01-28 10:08:24 INFO install Reading package lists...
2017-01-28 10:08:24 INFO install W: GPG error: http://ppa.launchpad.net/landscape/landscape-client-cloud-installer/ubuntu xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6E85A86E4652B4E6
2017-01-28 10:08:24 INFO install W: The repository 'http://ppa.launchpad.net/landscape/landscape-client-cloud-installer/ubuntu xenial InRelease' is not signed.
2017-01-28 10:08:25 INFO juju-log Installing ['landscape-client'] with options: ['--option=Dpkg::Options::=--force-confold']
2017-01-28 10:08:25 INFO install Reading package lists...
2017-01-28 10:08:25 INFO install Building dependency tree...
2017-01-28 10:08:25 INFO install Reading state information...
2017-01-28 10:08:26 INFO install The following additional packages will be installed:
2017-01-28 10:08:26 INFO install   bc landscape-common python-configobj python-gdbm python-pycurl
2017-01-28 10:08:26 INFO install   python-twisted-web
2017-01-28 10:08:26 INFO install Suggested packages:
2017-01-28 10:08:26 INFO install   python-configobj-doc python-gdbm-dbg libcurl4-gnutls-dev python-pycurl-dbg
2017-01-28 10:08:26 INFO install   python-pycurl-doc
2017-01-28 10:08:26 INFO install The following NEW packages will be installed:
2017-01-28 10:08:26 INFO install   bc landscape-client landscape-common python-configobj python-gdbm
2017-01-28 10:08:26 INFO install   python-pycurl python-twisted-web
2017-01-28 10:08:26 INFO install 0 upgraded, 7 newly installed, 0 to remove and 8 not upgraded.
2017-01-28 10:08:26 INFO install Need to get 449 kB of archives.
2017-01-28 10:08:26 INFO install After this operation, 2279 kB of additional disk space will be used.
2017-01-28 10:08:26 INFO install WARNING: The following packages cannot be authenticated!
2017-01-28 10:08:26 INFO install   landscape-common landscape-client
2017-01-28 10:08:26 INFO install E: There were unauthenticated packages and -y was used without --allow-unauthenticated
2017-01-28 10:08:26 INFO install Traceback (most recent call last):
2017-01-28 10:08:26 INFO install   File "/var/lib/juju/agents/unit-landscape-client-57/charm/hooks/install", line 92, in <module>
2017-01-28 10:08:26 INFO install     sys.exit(hooks.execute(sys.argv))
2017-01-28 10:08:26 INFO install   File "/var/lib/juju/agents/unit-landscape-client-57/charm/hooks/charmhelpers/core/hookenv.py", line 598, in execute
2017-01-28 10:08:26 INFO install     self._hooks[hook_name]()
2017-01-28 10:08:26 INFO install   File "/var/lib/juju/agents/unit-landscape-client-57/charm/hooks/install", line 42, in install
2017-01-28 10:08:26 INFO install     result = subprocess.call(["landscape-config", "--init", "-d", data_path])
2017-01-28 10:08:26 INFO install   File "/usr/lib/python3.5/subprocess.py", line 557, in call
2017-01-28 10:08:26 INFO install     with Popen(*popenargs, **kwargs) as p:
2017-01-28 10:08:26 INFO install   File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
2017-01-28 10:08:26 INFO install     restore_signals, start_new_session)
2017-01-28 10:08:26 INFO install   File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
2017-01-28 10:08:26 INFO install     raise child_exception_type(errno_num, err_msg)
2017-01-28 10:08:26 INFO install FileNotFoundError: [Errno 2] No such file or directory: 'landscape-config'
2017-01-28 10:08:27 ERROR juju.worker.uniter.operation runhook.go:107 hook "install" failed: exit status 1

In this example, the outcome is different. Here the error is complaining about 'landscape-config' not being found, but the failure was caused above when apt-add-repository failed to retrieve the public key and only partially worked. As a result, the necessary landscape-client package was not installed the charm failed a little later.

It's also incorrect behavior to continue with the charm when a necessary package failed to install.

Revision history for this message

Francis Ginther (fginther) wrote on 2017-02-06:

Actually, comment #6 fits better under lp:1587606 which applies to transient add-key calls being fatal to charm completion.

Francis Ginther (fginther) on 2017-02-22

Changed in autopilot-log-analyser:
status:	Fix Committed → In Progress
assignee:	Ursula Junque (ursinha) → Francis Ginther (fginther)

🤖 Landscape Builder (landscape-builder) on 2017-02-22

Changed in autopilot-log-analyser:
status:	In Progress → Fix Committed

Francis Ginther (fginther) on 2017-02-22

Changed in autopilot-log-analyser:
status:	Fix Committed → In Progress

Chad Smith (chad.smith) on 2017-03-03

Changed in charm-helpers:
assignee:	nobody → Chad Smith (chad.smith)
status:	New → In Progress

David Britton (dpb) on 2017-03-03

Changed in charm-helpers:
status:	In Progress → Fix Committed
status:	Fix Committed → Fix Released
Changed in autopilot-log-analyser:
status:	In Progress → Fix Released
Changed in landscape-client-charm:
assignee:	nobody → Chad Smith (chad.smith)
status:	New → In Progress

Chad Smith (chad.smith) on 2017-03-03

Changed in landscape-charm:
assignee:	nobody → Chad Smith (chad.smith)
status:	New → In Progress

Chad Smith (chad.smith) on 2017-03-04

Changed in glance-simplestreams-sync-charm:
status:	New → In Progress
assignee:	nobody → Chad Smith (chad.smith)
Changed in landscape-client-charm:
status:	In Progress → Fix Released