Resources are sometimes missing during charm install hook

Bug #1763175 reported by George Kraft
36
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
Low
Unassigned

Bug Description

Every once in a while, we see a deployment where one of the units goes through initial hooks (install, leader-elected, config-changed) without any resources, despite having resources provided to `juju deploy`.

An upgrade-charm hook occurs shortly after, at which point the resources become available.

This behavior is problematic for me as a charm developer. Our users need to tightly control when individual units perform their upgrade with a new resource; we've introduced an `upgrade` action for this. In order to provide a good user experience, we have to change the status message to inform the user that the upgrade action needs to be invoked.

The problem is that the charm can't distinguish between initial deployment (where the resource should be fetched and used right away) or a `juju attach` (where the resource should -not- be used yet, and a status message printed instead). Originally we thought the invocation of an `upgrade-charm` hook would signal this to the charm, but given the behavior described above, that isn't always the case.

Attachments incoming...

Revision history for this message
George Kraft (cynerva) wrote :

Attached output of juju status - this is a minimal test charm with an install hook that sets the status message to the output of `resource-get test0`. Note that the leader is the one who was missing a resource.

Revision history for this message
George Kraft (cynerva) wrote :

Status log for test/1. Note that the upgrade-charm hook occurs; I never called `juju attach` or `juju upgrade-charm`.

Revision history for this message
George Kraft (cynerva) wrote :

Status log for test/0, for comparison. No upgrade-charm occurs.

Revision history for this message
George Kraft (cynerva) wrote :

Attached juju debug-log output. I didn't see anything useful in here, but just in case.

Revision history for this message
George Kraft (cynerva) wrote :

Scratch that about juju debug-log being useless. The output of `resource-get` is in there:

ERROR could not download resource: HTTP request failed: already staged

Revision history for this message
George Kraft (cynerva) wrote :

Attached controller logs. The 'failed to track progress: setting the resource failed' and 'failed to track progress: resource "test0" not found' around 19:12:41 look interesting.

Revision history for this message
George Kraft (cynerva) wrote :

$ juju version
2.3.5-xenial-amd64

$ lxc version
Client version: 3.0.0
Server version: 3.0.0

I have seen this on aws before as well.

Revision history for this message
George Kraft (cynerva) wrote :

Attached tarball with the charm, bundle, and script I used to repro. The script deploys from the charm store however. I have only been able to reproduce this by deploying a bundle pointing to the charm store, but given how rare this behavior is, I can't rule anything out.

Tim Penhey (thumper)
tags: added: race-condition resources
Changed in juju:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Canonical Juju QA Bot (juju-qa-bot) wrote :

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.

Changed in juju:
importance: High → Low
tags: added: expirebugs-bot
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.