Juju 2.0 Resource Error - cannot add resource failed to write data: read tcp : i/o timeout

[Suchitra Venugopal]

Hello Team,

I am facing the below issue in Juju 2.0 while uploading packages to the controller using juju attach command and also juju inline deployment command using resources.

Error code : http://pastebin.ubuntu.com/18097919/

Below are the details:

Package Size tried to upload : 1.8 GB
Juju Version Used: 2.0-beta7-xenial-amd64
Command which i have used: juju deploy /root/charms/trusty/ibm-platform-symphony-storage --series trusty --resource ibm_ps_installer=/root/repo/pssasetup2015_linux-x86_64.bin --resource ibm_ps_entitlement=/root/repo/platform_sym_adv_entitlement.dat

To Reproduce the same try to upload huge size of package to the local controller using juju attach command or deploy a charm by passing resources using inline juju deploy command.

Thanks

[Shruthima (salmavar)]

Hello Team,
Products like Http-server has multiple resources like 3 install packages and 2 fix packs.
so, when we ran juju-attach command for each resource it is taking 20-30min and to fetch the resource into the container it is taking 20-30 min for each resource.
  Overall it is taking 2-2.30 hours for each deployment which is not efficient.Is these related to the same bug?? We are really stuck with testing due to these.
Can you suggest any other methods for uploading and downloading resources from controller so that it will help us a lot while testing the charm,until this bug fixes.

[Cheryl Jennings (cherylj)]

@salmavar - can you please attach the following from the controller machine (machine-0 in the controller model)

1 - /var/log/syslog
2 - /var/log/juju/machine-0.log

If you can reproduce this reliably, it would be helpful to set the logging level to DEBUG and recreate before you send the above files. You can set the logging level with `juju set-model-config logging-config="<root>=DEBUG" -m controller`

[Matt Bruzek (mbruzek)]

It appears this is Juju 2.0 with the local (LXD) provider. As @suchvenu points out, the controller is a node on the host machine and transferring large data between containers on the same system should not take very long.

Is there a workaround that the IBM team could copy the images to a specific location without using the `juju attach` command? Can someone on the juju-core team please advise on this or another workaround?

[Eric Snow (ericsnowcurrently)]

Well, the bits have to get there no matter what. However, I would not expect it to take *that* long using the LXD provider.

The files do get cached for each unit in the "resources/<resource name>" directory under the unit agent's data dir. The file name is the one specified in the charm's metadata.

So directly copying each file there should work as a work-around. However, we should definitely determine why it's taking so long.

[shilpa (shilkaul)]

The syslog and machine-0.log is attached

[Suchitra Venugopal (suchvenu)]

Logs from the machine where its taking long time to attach the package.

[Suchitra Venugopal (suchvenu)]

Both these logs are from the same controller.

[Antonio Rosales (arosales)]

@Eric,

Agreed the initial upload will take some time, but one of the main benefits of Resources is subsequent deployments from models, once the resource is on the controller, should be significantly faster.

The IBM charms are a great use case as the file sizes tend to be quite large. Thus the recommendation is to upload once to the controller, and then deploy multiple times in models. However, it seems the transfer in a local network via Juju is still taking significantly longer than expected.

Any ideas on how to debug this would be greatly appreciated.

-thanks,
Antonio

[Shruthima (salmavar)]

@arosales,

After initial upload of the resources to ibm-http charm.

juju-list resources <charm-name> shown as:

 http://paste.ubuntu.com/18214977/

Once after removing the service and when we try to deploy once again without running juju-attach command,
charm deployment doesn't proceed further and the log shows:

http://paste.ubuntu.com/18216315/

juju-list resources <charm-name> will now shown as:

http://paste.ubuntu.com/18215273/

After removing service if resources are present in controller, then we need not run juju-attach command for the next deployment right ? But we are not able to proceed further without running juju attach for second time.

Is my understanding correct ? Or am i missing something here?

Also could you please provide the exact path of resources stored in the controller, so that we can go and verify once before deploying it second time to check whether the packages are still present there.