deploying large charms fail because vagrant needs more RAM

Bug #1371665 reported by Kevin W Monroe on 2014-09-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juju Vagrant images
High
Unassigned
juju-core
Medium
Unassigned

Bug Description

Deploying a fat charm under Vagrant 1.6.5 on OSX 10.9.5 fails if the charm is large (around 600MB).

$ juju version
1.20.7-trusty-amd64

To reproduce, add some filler to ./files to get the charm over 600MB:

$ cd ~/charms/precise/mycharm
$ dd if=/dev/zero of=./files/foo bs=1M count=600
$ du -sh
601M .
$ juju deploy --repository ~/charms local:precise/mycharm --debug
2014-09-19 15:36:31 INFO juju.cmd supercommand.go:37 running juju [1.20.7-trusty-amd64 gc]
2014-09-19 15:36:31 DEBUG juju.conn api.go:187 trying cached API connection settings
2014-09-19 15:36:31 INFO juju.conn api.go:270 connecting to API addresses: [localhost:17070 10.0.3.1:17070 10.0.2.15:17070 172.16.250.15:17070]
2014-09-19 15:36:31 INFO juju.state.api apiclient.go:242 dialing "wss://localhost:17070/environment/aee94ed2-f800-4705-81ed-50921a2a1259/api"
2014-09-19 15:36:31 INFO juju.state.api apiclient.go:176 connection established to "wss://localhost:17070/environment/aee94ed2-f800-4705-81ed-50921a2a1259/api"
2014-09-19 15:36:46 INFO juju.utils http.go:66 hostname SSL verification disabled
2014-09-19 15:36:50 INFO juju.rpc server.go:334 error closing codec: EOF
2014-09-19 15:36:50 ERROR juju.cmd supercommand.go:323 cannot upload charm: Post https://localhost:17070/charms?series=precise: EOF

Fwiw, I did a few rounds and found that a 575MB charm works, but 600MB does not. Note, this is not the same error that occurs when the filesystem fills up. That scenario fails (expectedly) like this:

$ juju deploy --repository ~/charms local:precise/mycharm
ERROR error uploading charm: cannot extract uploaded charm: cannot extract "files/foo": write /tmp/charm-download150588614/extracted/files/foo: no space left on device

John George (jog) wrote :

Kevin,

Would you please attach the Juju environment .jenv and the Juju logs?

Changed in juju-core:
status: New → Incomplete
Kevin W Monroe (kwmonroe) wrote :

Sure jog, local.jenv and logs are on their way. I didn't see anything happening in /var/log/juju-*log during deploy, so I'm not sure if I'm grabbing the right stuff. Let me know if you want others. Thanks!

Kevin W Monroe (kwmonroe) wrote :
Changed in juju-core:
status: Incomplete → New
Abel Deuring (adeuring) on 2014-09-22
Changed in juju-core:
status: New → Triaged
importance: Undecided → Medium
Abel Deuring (adeuring) wrote :

Confirmed using the "ubuntu" charm with an additional large file as described by Kevin. The usual log files do not show anything interesting. The same charm can be deploy to a regular local/LXC environment

Abel Deuring (adeuring) wrote :

This seems to be a memory issue: Vagrnat creates a VirtualBox instance with 2GB memory. If this is increased to 4GB, the "fat charm" can be deployed.

Curtis Hovey (sinzui) wrote :

This bug is not in juju nor is it in the juju vagrant test suite. The issue is really in the juju vagrant defaults or docs. We need to know where we can make a change to address this issue.

Changed in juju-core:
status: Triaged → Invalid
Changed in juju-vagrant-images:
status: New → Triaged
importance: Undecided → High
Curtis Hovey (sinzui) on 2014-09-22
summary: - juju deploy to vagrant container fails with large charms
+ deploying large charms fail because vagrant needs more RAM
Abel Deuring (adeuring) wrote :

I think the main problem here is the misleading error message:

vagrant@vagrant-ubuntu-trusty-64:~$ juju deploy --repository=/vagrant/charms local:precise/ubuntu --debug
2014-09-23 08:05:52 INFO juju.cmd supercommand.go:37 running juju [1.20.7-trusty-amd64 gc]
2014-09-23 08:05:52 DEBUG juju.conn api.go:187 trying cached API connection settings
2014-09-23 08:05:52 INFO juju.conn api.go:270 connecting to API addresses: [localhost:17070 10.0.3.1:17070 10.0.2.15:17070 172.16.250.15:17070]
2014-09-23 08:05:52 INFO juju.state.api apiclient.go:242 dialing "wss://localhost:17070/environment/f44ec1be-0feb-4fa4-838d-49456888735d/api"
2014-09-23 08:05:52 INFO juju.state.api apiclient.go:176 connection established to "wss://localhost:17070/environment/f44ec1be-0feb-4fa4-838d-49456888735d/api"
2014-09-23 08:06:03 INFO juju.utils http.go:66 hostname SSL verification disabled
2014-09-23 08:06:17 ERROR juju.cmd supercommand.go:323 error uploading charm: cannot access provider storage: cannot access environment: failed verification of local provider prerequisites:
juju-local must be installed to enable the local provider:

    sudo apt-get install juju-local
vagrant@vagrant-ubuntu-trusty-64:~$ sudo apt-get install juju-local
Reading package lists... Done
Building dependency tree
Reading state information... Done
juju-local is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

There is no hint that the failure is in any way related to a "memory shortage".

Kevin W Monroe (kwmonroe) wrote :

I updated my Vagrantfile to use 3GB of memory:

+ config.vm.provider "virtualbox" do |vb|
+ # Use VBoxManage to customize the VM. For example to change memory:
+ vb.customize ["modifyvm", :id, "--memory", "3072"]
+ end

And deployed my 600MB 'mycharm' successfully. It fails if I beef up ./files/foo to 1000MB. Perhaps documentation could note the max charm sizes for various memory constraints (eg: a 2GB Vagrant environment supports 500MB charms; a 3GB Vagrant env supports 900MB charms; etc). I am not sure if the same constraints apply to other vm/cloud providers.

Also, I agree with Abel's note on the confusing error message. I've seen both the EOF error noted in the description and the strange juju-local error noted in comment 10. The EOF makes at least some sense to me, though it would be nice if this failure could be caught earlier and tell the user about the memory limit. If this issue affects other providers, I think it would fall to juju-core to trap/explain the failure. I'll try to check non-vagrant providers next.

Kevin W Monroe (kwmonroe) wrote :

One other quick vagrant data point.. With --memory=1024 in Vagrantfile, 200MB charms succeed while 300MB charms fail.

This begs me to ask why we are buffering the entire charm payload in memory.

On Mon, Sep 29, 2014 at 1:26 PM, Kevin W Monroe <email address hidden>
wrote:

> One other quick vagrant data point.. With --memory=1024 in Vagrantfile,
> 200MB charms succeed while 300MB charms fail.
>
> --
> You received this bug notification because you are subscribed to Juju
> Vagrant images.
> Matching subscriptions: vagrant-images
> https://bugs.launchpad.net/bugs/1371665
>
> Title:
> deploying large charms fail because vagrant needs more RAM
>
> Status in juju-core:
> Invalid
> Status in Juju Vagrant images:
> Triaged
>
> Bug description:
> Deploying a fat charm under Vagrant 1.6.5 on OSX 10.9.5 fails if the
> charm is large (around 600MB).
>
> $ juju version
> 1.20.7-trusty-amd64
>
> To reproduce, add some filler to ./files to get the charm over 600MB:
>
> $ cd ~/charms/precise/mycharm
> $ dd if=/dev/zero of=./files/foo bs=1M count=600
> $ du -sh
> 601M .
> $ juju deploy --repository ~/charms local:precise/mycharm --debug
> 2014-09-19 15:36:31 INFO juju.cmd supercommand.go:37 running juju
> [1.20.7-trusty-amd64 gc]
> 2014-09-19 15:36:31 DEBUG juju.conn api.go:187 trying cached API
> connection settings
> 2014-09-19 15:36:31 INFO juju.conn api.go:270 connecting to API
> addresses: [localhost:17070 10.0.3.1:17070 10.0.2.15:17070
> 172.16.250.15:17070]
> 2014-09-19 15:36:31 INFO juju.state.api apiclient.go:242 dialing
> "wss://localhost:17070/environment/aee94ed2-f800-4705-81ed-50921a2a1259/api"
> 2014-09-19 15:36:31 INFO juju.state.api apiclient.go:176 connection
> established to
> "wss://localhost:17070/environment/aee94ed2-f800-4705-81ed-50921a2a1259/api"
> 2014-09-19 15:36:46 INFO juju.utils http.go:66 hostname SSL verification
> disabled
> 2014-09-19 15:36:50 INFO juju.rpc server.go:334 error closing codec: EOF
> 2014-09-19 15:36:50 ERROR juju.cmd supercommand.go:323 cannot upload
> charm: Post https://localhost:17070/charms?series=precise: EOF
>
> Fwiw, I did a few rounds and found that a 575MB charm works, but 600MB
> does not. Note, this is not the same error that occurs when the
> filesystem fills up. That scenario fails (expectedly) like this:
>
> $ juju deploy --repository ~/charms local:precise/mycharm
> ERROR error uploading charm: cannot extract uploaded charm: cannot
> extract "files/foo": write
> /tmp/charm-download150588614/extracted/files/foo: no space left on device
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1371665/+subscriptions
>

Kevin W Monroe (kwmonroe) wrote :

+1 to lazy's question above. If we can avoid exhausting our memory between reading the charm from our local repo and writing it to /tmp, that would be ideal. If not, at least catch this scenario and throw a nice out-of-memory error instead of the cryptic EOF or worse, "juju-local must be installed" (wat?).

If neither fix makes the cut this cycle, I've created PR https://github.com/juju/docs/pull/202 to at least mention this limitation and how to work around it in the docs.

Antonio Rosales (arosales) wrote :

Aside from vagrant we are getting charms that have workloads up to 5GB that we need to upload on deploy. These charms will either deploy from a local copy of the workload (the client) or download from a public URL. In any case the download or upload is going to be quite large and will hit this problem.

Suggestion on how we can proceed here?

-thanks,
Antonio

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers