Fix race on getting close notifier channel

[Xiang Hui]

[OS]
Ubuntu Xenial
etcd 2.2.5

[Error Log]
Sep 2 16:08:16 ubuntu etcd[18180]: panic: net/http: CloseNotify called after ServeHTTP finished
Sep 2 16:08:16 ubuntu etcd[18180]: goroutine 421 [running]:
Sep 2 16:08:16 ubuntu etcd[18180]: net/http.(*response).CloseNotify(0xc8202f64e0, 0x0)
Sep 2 16:08:16 ubuntu etcd[18180]: #011/usr/lib/go/src/net/http/server.go:1535 +0x9d
Sep 2 16:08:16 ubuntu etcd[18180]: github.com/coreos/etcd/proxy.(*reverseProxy).ServeHTTP.func1(0x7fb0024d1a80, 0xc8202f64e0, 0xc8203e4f78, 0xc8202e8a80, 0xc8203e4fa0, 0xc820464f50)
Sep 2 16:08:16 ubuntu etcd[18180]: #011/build/etcd-tG_CNV/etcd-2.2.5+dfsg/obj-x86_64-linux-gnu/src/github.com/coreos/etcd/proxy/reverse.go:107 +0x39
Sep 2 16:08:16 ubuntu etcd[18180]: created by github.com/coreos/etcd/proxy.(*reverseProxy).ServeHTTP
Sep 2 16:08:16 ubuntu etcd[18180]: #011/build/etcd-tG_CNV/etcd-2.2.5+dfsg/obj-x86_64-linux-gnu/src/github.com/coreos/etcd/proxy/reverse.go:113 +0x691
Sep 2 16:08:16 ubuntu systemd[1]: etcd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 2 16:08:16 ubuntu systemd[1]: etcd.service: Unit entered failed state.
Sep 2 16:08:16 ubuntu systemd[1]: etcd.service: Failed with result 'exit-code'.

etcd proxy process dies and is not restarted by systemd, Components that depend on etcd report connection errors like this:

- Request to server http://127.0.0.1:4001 failed: MaxRetryError("HTTPConnectionPool(host='127.0.0.1', port=4001): Max retries exceeded with url: /v2/keys/calico/dhcp/v1/subnet?waitIndex=1625&recursive=true&wait=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6f0c5cae10>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))",)

[Root Cause]
etcd proxy panic occur in etcd 2.2.5, which is fixed in etcd 2.3.4 (https://github.com/coreos/etcd/pull/5269/files#r62072134), currently only have 2.2.5 in xenial, we need it backported.

-------------------------------------------------------

[Impact]
This patch fix race on getting close notifier channel when a panic reported as 'net/http: CloseNotify called after ServeHTTP finished'.

[Test Case]
No special configuration, running etcd in proxy mode.

[Regression Potential]
etcd no longer crashes every so often while in proxy mode...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "xenial-lp1620897.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Andy Whitcroft (apw)]

Hello Xiang, or anyone else affected,

Accepted etcd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/etcd/2.2.5+dfsg-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Xiang Hui (xianghui)]

The panic is not shown up after updating from xenial-proposed branch.

[Martin Pitt (pitti)]

FTR, release of this is blocked by yakkety never getting the fix (it's stuck in yakkety-proposed on FTBFS)

[Martin Pitt (pitti)]

@Xiang: Last warning: please upload the fix to zesty. Removing verification-done tag for now as this must not accidentally land.

[Steve Langasek (vorlon)]

I don't see any reason to block this SRU on the broken version of etcd that's in zesty-proposed (broken because of a too-new golang-github-coreos-pkg-dev in the archive). I'll release this to xenial-updates, and forward-copy it to yakkety-updates and zesty. The package in zesty-proposed will sort itself out (or not) when someone has the interest to look at it.

[Launchpad Janitor (janitor)]

This bug was fixed in the package etcd - 2.2.5+dfsg-1ubuntu1

---------------
etcd (2.2.5+dfsg-1ubuntu1) xenial; urgency=medium

  * Fix race on getting close notifier channel (LP: #1620897)

 -- Hui Xiang <email address hidden> Wed, 07 Sep 2016 03:55:36 +0000

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for etcd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package etcd - 2.2.5+dfsg-1ubuntu1

---------------
etcd (2.2.5+dfsg-1ubuntu1) xenial; urgency=medium

  * Fix race on getting close notifier channel (LP: #1620897)

 -- Hui Xiang <email address hidden> Wed, 07 Sep 2016 03:55:36 +0000