Improve http timeout handling in LXD

[Barry Price]

Hi,

I have successfully synced the ubuntu:14.04, ubuntu:16.04 and ubuntu:16.10 images from cloud-images.ubuntu.com but for some reason the ubuntu:12.04 image can't be retrieved - the process consistently hangs before even making the request from the remote server.

http://pastebin.ubuntu.com/24057592/plain/

ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: lxd 2.9.2-0ubuntu1
ProcVersionSignature: Ubuntu 4.10.0-8.10-generic 4.10.0-rc8
Uname: Linux 4.10.0-8-generic x86_64
NonfreeKernelModules: zfs zunicode zavl zcommon znvpair
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
CurrentDesktop: i3
Date: Fri Feb 24 14:35:10 2017
InstallationDate: Installed on 2016-11-08 (107 days ago)
InstallationMedia: Ubuntu 16.10 "Yakkety Yak" - Release amd64 (20161012.2)
SourcePackage: lxd
UpgradeStatus: Upgraded to zesty on 2017-01-24 (31 days ago)

[Stéphane Graber (stgraber)]

stgraber@dakara:~/data/code/lxc/lxd/test (stgraber/master)$ lxc image copy ubuntu:12.04 local: --copy-aliases
Image copied successfully!

[Stéphane Graber (stgraber)]

It could be that a download operation was initiated by the daemon and is stuck due to network problems. If that's the case "systemctl restart lxd" should unstick things.

If the problem persists after restarting the daemon, then my guess would be some MTU problem which is the usual suspect for such random but repetitive problems.

You can also try "lxc image export ubuntu:12.04 ." which will perform the same image download directly in the client rather than through the LXD daemon.

[Barry Price (barryprice)]

Thanks, the systemctl restart suggestion did the trick. It's definitely possible that the first attempt ran into network issues. Should there perhaps be a timeout for download operations to avoid this?

[Stéphane Graber (stgraber)]

Yeah. I'm pretty sure that Go http has some kind of timeout but it seems to be a bit generous :)

[Stéphane Graber (stgraber)]

So Go's http library really sucks as far as timeouts...

We can set timeouts for obvious things like TCP connection, TLS handshake and HTTP headers reception. But we can't have the body send/receive itself timeout in a sane way...

The only thing that Go seems to offer there is a non-resetable timeout, meaning that we have to guess ahead of time how long the client will take to download or upload the data as even if bytes are successfully sent or received, the timeout will not be reset and when it hits, will just close the connection.

So it looks like we'll need to look into some kind of custom solution to this so we can have the connection timeout if it fails to receive data in the past 30s or so, resetting the timer whenever data is received.

[Stéphane Graber (stgraber)]

We've improved this somewhat by at least making it possible to cancel image downloads. That's not quite the full solution to this problem though.

[Stéphane Graber (stgraber)]

A number more timeouts are now set in recent LXDs and I've not run into such a situation ever since, so I think we're good to close this.