juju reporting cryptic error when trying to upgrade a machine with a full filesystem

[Junien F]

Hi,

Controller is running 2.3.8, machine agent on a hosted model is running 2.3.7

The controller is logging the following messages :

2018-07-18 13:50:21 DEBUG juju.apiserver httpcontext.go:298 sending error: 400 failed to write agent binaries: write tcp 10.X.Y.Z:17070->1.2.3.4:46658: write: connection reset by peer
2018-07-18 13:50:21 WARNING juju.apiserver log.go:172 http: multiple response.WriteHeader calls

2018-07-18 13:50:21 ERROR juju.apiserver tools.go:62 cannot write response: http: wrote more than the declared Content-Length

It turns out that this be because the juju machine agent on 1.2.3.4, currently 2.3.7, is trying to download the 2.3.8 agent but fails because its filesystem is full.

As soon as I cleared some space on the filesystem, the machine agent upgraded itself to 2.3.8.

I believe juju should handle full filesystems better, including logging.

Thanks

[Anastasia (anastasia-macmood)]

@Junien Fridrick,

This a nice report - thank you!

Any suggestions on how juju could "handle full filesystems"? :D

[Paul Gear (paulgear)]

Checking the disk space on the file system where /var/lib/juju/tools lives beforehand and comparing it to the amount of space required by the agent would be a good start.

Also, returning a more understandable message than "2018-07-18 13:50:21 DEBUG juju.apiserver httpcontext.go:298 sending error: 400 failed to write agent binaries: write tcp 10.X.Y.Z:17070->1.2.3.4:46658: write: connection reset by peer".

[Richard Harding (rharding)]

Thanks, the logging can definitely be improved there. Thanks for the report and will see how detailed we can get around the file size expectations.

[Richard Harding (rharding)]

I also can't help but wonder if we can do something with the machine status around disk size issues. We hit an array of issues around logging/etc when the disk if full and feel like that should raise a message to the user in the machine status output.

[Christian Muirhead (2-xtian)]

PR for fix: https://github.com/juju/juju/pull/9180