resource-get hangs when trying to deploy a charm with resource from the store

[Casey Marshall]

Steps to reproduce:

1. juju deploy cs:~cmars/mattermost

This charm was published with a resource:

> $ charm push ~/trusty/mattermost/ cs:~cmars/mattermost --resource
> bdist=bdist/mattermost.tar.gz
>
> $ charm publish cs:~cmars/mattermost-8 --resource bdist-1

> $ charm list-resources cs:~cmars/mattermost
> [Service]
> RESOURCE REVISION
> bdist 1
>
> - name: bdist
> type: file
> path: mattermost.tar.gz
> description: Binary distribution of Mattermost, obtained from
> https://www.mattermost.org/download/
> origin: store
> revision: 1
> fingerprint: ...

$ juju list-resources mattermost --format yaml
resources:
- resourceid: mattermost/bdist
  serviceid: mattermost
  name: bdist
  type: file
  path: mattermost.tar.gz
  description: Binary distribution of Mattermost, obtained from https://www.mattermost.org/download/
  revision: 1
  fingerprint: becda1bfaad2637b2c7be9d0300aa38d906a244289c0f82c685715bbabcea9807c78b525df99fcbb1e3122142958f561
  size: 31205420
  origin: store
  used: false

2. Install hook hangs. Logsink: https://paste.ubuntu.com/16135024/

$ juju status

[Services]
NAME STATUS EXPOSED CHARM
mattermost maintenance false cs:~cmars/mattermost-8

[Units]
ID WORKLOAD-STATUS JUJU-STATUS VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE
mattermost/0 maintenance executing 2.0-beta7.1 0 10.10.86.243 (install) installing charm software

[Machines]
ID STATE DNS INS-ID SERIES AZ
0 started 10.10.86.243 juju-ed88e08e-f0f4-43bf-8028-46445cd43da5-machine-0 trusty

If I deploy specifying a local file with --resource, `resource-get` works and the installation proceeds. I'm only having the problem with the controller downloading the resource from the charmstore.

[Jay R. Wren (evarlast)]

Here is the tail of an strace. The SIGINT is my ctrl-c after waiting a significant number of seconds. It seems that the agent never responds.

stat(".", {st_mode=S_IFDIR|0755, st_size=31, ...}) = 0
stat("/var/lib/juju/agents/unit-juju-gui-0/charm", {st_mode=S_IFDIR|0755, st_size=31, ...}) = 0
stat(".", {st_mode=S_IFDIR|0755, st_size=31, ...}) = 0
stat("/var/lib/juju/agents/unit-juju-gui-0/charm", {st_mode=S_IFDIR|0755, st_size=31, ...}) = 0
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
connect(3, {sa_family=AF_LOCAL, sun_path=@"/var/lib/juju/agents/unit-juju-gui-0/agent.socket"}, 52) = 0
epoll_create1(EPOLL_CLOEXEC) = 4
epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=59658832, u64=140067533115984}}) = 0
getsockname(3, {sa_family=AF_LOCAL, NULL}, [2]) = 0
getpeername(3, {sa_family=AF_LOCAL, sun_path=@"/var/lib/juju/agents/unit-juju-gui-0/agent.socket"}, [52]) = 0
futex(0xc820193508, FUTEX_WAKE, 1) = 1
write(3, "/\377\201\3\1\1\7Request\1\377\202\0\1\2\1\rServiceMethod\1\f\0\1\3Seq\1\6\0\0\0\17\377\202\1\nJujuc.Main\0\\\377\203\3\1\1\7Request\1\377\204\0\1\6\1\tContextId\1\f\0\1\3Dir\1\f\0\1\vCommandName\1\f\0\1\4Args\1\377\206\0\1\10StdinSet\1\2\0\1\5Stdin\1\n\0\0\0\26\377\205\2\1\1\10[]string\1\377\206\0\1\f\0\0p\377\204\1&juju-gui/0-install-7831898561028304465\1*/var/lib/juju/agents/unit-juju-gui-0/charm\1\fresource-get\1\1\10juju-gui\0", 293) = 293
read(3, 0xc8200f0000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
epoll_wait(4, {{EPOLLOUT, {u32=59658832, u64=140067533115984}}}, 128, 0) = 1
epoll_wait(4, 7fff53dec9c0, 128, -1) = -1 EINTR (Interrupted system call)
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
rt_sigaction(SIGINT, {SIG_DFL, ~[], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x479ad0}, NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
gettid() = 16765
tkill(16765, SIGINT) = 0
--- SIGINT {si_signo=SIGINT, si_code=SI_TKILL, si_pid=16765, si_uid=0} ---
+++ killed by SIGINT +++

[Nell Jerram (neil-jerram)]

How can I most easily work around this, if I'm deploying a bundle with Juju 2?

For me the resource-using charm is cs:~lazypower/etcd-6. I'm using

    juju deploy bundle.yaml

to deploy a moderately complex system, where bundle.yaml includes

  etcd:
    charm: "cs:~lazypower/etcd-6"

And I'm seeing that the machine installing the etcd charm is hanging on resource-get. Is there a straightforward workaround?

[Nate Finch (natefinch)]

PR with fix here: https://github.com/juju/charmrepo/pull/92 (we'll need to update juju to reference the change in that repo when it lands).