juju deploy hangs for a long time and then fails

[Jay R. Wren]

$ juju deploy local:blues-browser --debug --show-log
2014-10-27 20:03:36 INFO juju.cmd supercommand.go:37 running juju [1.21-alpha2-utopic-amd64 gc]
2014-10-27 20:03:36 DEBUG juju.api api.go:153 trying cached API connection settings
2014-10-27 20:03:36 INFO juju.api api.go:251 connecting to API addresses: [juju-azure-14p4bwrjf6.cloudapp.net:17070 10.0.0.4:17070]
2014-10-27 20:03:36 INFO juju.api apiclient.go:252 dialing "wss://juju-azure-14p4bwrjf6.cloudapp.net:17070/environment/da0e7f3c-9dfb-48f9-853e-ba3b2eb04e01/api"
2014-10-27 20:03:36 INFO juju.api apiclient.go:252 dialing "wss://10.0.0.4:17070/environment/da0e7f3c-9dfb-48f9-853e-ba3b2eb04e01/api"
2014-10-27 20:03:36 INFO juju.api apiclient.go:175 connection established to "wss://juju-azure-14p4bwrjf6.cloudapp.net:17070/environment/da0e7f3c-9dfb-48f9-853e-ba3b2eb04e01/api"
2014-10-27 20:03:39 DEBUG juju.api apiclient.go:258 error dialing "wss://10.0.0.4:17070/environment/da0e7f3c-9dfb-48f9-853e-ba3b2eb04e01/api", will retry: websocket.Dial wss://10.0.0.4:17070/environment/da0e7f3c-9dfb-48f9-853e-ba3b2eb04e01/api: dial tcp 10.0.0.4:17070: no route to host
2014-10-27 20:04:11 INFO juju.utils http.go:66 hostname SSL verification disabled
2014-10-27 20:09:47 INFO juju.rpc server.go:334 error closing codec: EOF
2014-10-27 20:09:47 ERROR juju.cmd supercommand.go:323 cannot upload charm: Post https://juju-azure-14p4bwrjf6.cloudapp.net:17070/charms?series=trusty: EOF

Notice the long time - 5min 35s between log INFO messages.

While this deploy was happening, I tried to ssh to node 0 (the only other machine)

$ juju ssh 0
Warning: Permanently added 'juju-azure-14p4bwrjf6.cloudapp.net,23.99.67.251' (ECDSA) to the list of
 known hosts.
Warning: Permanently added '10.0.0.4' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-36-generic x86_64)

 * Documentation: https://help.ubuntu.com/

 System information disabled due to load higher than 1.0

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

-bash: fork: Cannot allocate memory
ubuntu@jujunklnyeyowic4ucqh1kj02hdzs6pwksb7o4d2b92hqluv7r:~$ top
top - 20:09:03 up 24 min, 1 user, load average: 1.90, 1.71, 1.33
Tasks: 224 total, 2 running, 222 sleeping, 0 stopped, 0 zombie
%Cpu(s): 15.6 us, 1.3 sy, 0.0 ni, 0.0 id, 82.4 wa, 0.0 hi, 0.7 si, 0.0 st
KiB Mem: 1718608 total, 1614344 used, 104264 free, 3252 buffers
KiB Swap: 0 total, 0 used, 0 free. 52484 cached Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 2095 root 20 0 2180472 1.328g 4412 S 16.6 81.0 2:09.52 jujud

jujud was using 1.3 GB of resident memory!

Apparently I need at least 5 times the memory free of the largest charm I plan to deploy.

This environment was just created:
$ juju status
environment: azure
machines:
  "0":
    agent-state: started
    agent-version: 1.21-alpha2
    dns-name: juju-azure-14p4bwrjf6.cloudapp.net
    instance-id: juju-azure-14p4bwrjf6-jujunklnyeyowic4ucqh1kj02hdzs6pwksb7o4d2b92hqluv7r
    instance-state: ReadyRole
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1792M root-disk=130048M
    state-server-member-status: has-vote
services: {}

Expected: I can deploy large charms even with state server on a small node.

[Kapil Thangavelu (hazmat)]

thats very sad if the state server is storing the entire charm in memory on the way to storing it in gridfs..

[Jay R. Wren (evarlast)]

Not just entire charm in memory, but 3X the charm size in memory.

I have been reducing this charms footprint, but even at 70MB, the jujud and
mongod processes both grow from initial size to > 210MB after deploying
this charm.

On Tue, Oct 28, 2014 at 7:44 PM, Kapil Thangavelu <
<email address hidden>> wrote:

> thats very sad if the state server is storing the entire charm in memory
> on the way to storing it in gridfs..
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1386405
>
> Title:
> juju deploy hangs for a long time and then fails
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1386405/+subscriptions
>

[Ian Booth (wallyworld)]

The blob store does the right thing; it doesn't read the entire blob into memory. It streams it to a temp file so the hash can be calculated, and then passes the hash and file reference to GridFS to be persisted.

Deploying local charms does have an issue - this code does appear to read the entire charm into memory.