juju reporting cryptic error when trying to upgrade a machine with a full filesystem

Bug #1782367 reported by Junien Fridrick on 2018-07-18
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju
High
Christian Muirhead

Bug Description

Hi,

Controller is running 2.3.8, machine agent on a hosted model is running 2.3.7

The controller is logging the following messages :

2018-07-18 13:50:21 DEBUG juju.apiserver httpcontext.go:298 sending error: 400 failed to write agent binaries: write tcp 10.X.Y.Z:17070->1.2.3.4:46658: write: connection reset by peer
2018-07-18 13:50:21 WARNING juju.apiserver log.go:172 http: multiple response.WriteHeader calls

2018-07-18 13:50:21 ERROR juju.apiserver tools.go:62 cannot write response: http: wrote more than the declared Content-Length

It turns out that this be because the juju machine agent on 1.2.3.4, currently 2.3.7, is trying to download the 2.3.8 agent but fails because its filesystem is full.

As soon as I cleared some space on the filesystem, the machine agent upgraded itself to 2.3.8.

I believe juju should handle full filesystems better, including logging.

Thanks

Junien Fridrick (axino) on 2018-07-18
tags: added: canonical-is
Anastasia (anastasia-macmood) wrote :

@Junien Fridrick,

This a nice report - thank you!

Any suggestions on how juju could "handle full filesystems"? :D

Changed in juju:
status: New → Incomplete
Paul Gear (paulgear) wrote :

Checking the disk space on the file system where /var/lib/juju/tools lives beforehand and comparing it to the amount of space required by the agent would be a good start.

Also, returning a more understandable message than "2018-07-18 13:50:21 DEBUG juju.apiserver httpcontext.go:298 sending error: 400 failed to write agent binaries: write tcp 10.X.Y.Z:17070->1.2.3.4:46658: write: connection reset by peer".

Changed in juju:
status: Incomplete → New
Richard Harding (rharding) wrote :

Thanks, the logging can definitely be improved there. Thanks for the report and will see how detailed we can get around the file size expectations.

Changed in juju:
importance: Undecided → High
milestone: none → 2.3.9
Richard Harding (rharding) wrote :

I also can't help but wonder if we can do something with the machine status around disk size issues. We hit an array of issues around logging/etc when the disk if full and feel like that should raise a message to the user in the machine status output.

Changed in juju:
status: New → Triaged
Changed in juju:
assignee: nobody → Christian Muirhead (2-xtian)
Changed in juju:
status: Triaged → In Progress
Changed in juju:
status: In Progress → Fix Committed
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers