intermittent panic in 1.6.0-3 due to race in WriteStatus

[Tiit Pikma]

[Impact]

The version of golang-google-grpc packaged in bionic and cosmic (1.6.0-3) crashes intermittently due to a race condition. Upstream bug report and pull requests:

https://github.com/grpc/grpc-go/issues/1111
https://github.com/grpc/grpc-go/pull/1546
https://github.com/grpc/grpc-go/pull/1687

This can cause all packages Built-Using: golang-google-grpc=1.6.0-3 to crash as well (see https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1800973).

The attached debdiff contains two upstream fixes squashed together:

https://github.com/grpc/grpc-go/commit/22c3f92f5faea8db492fb0f5ae4daf0d2752b19e
https://github.com/grpc/grpc-go/commit/c6b46087ab923e9f453ec433f99174cdd45b9b89

These fix the race by synchronizing the entire function.

[Test Case]
The patch includes patches to test cases to exercise the race condition. One test case would fail sometimes without the fix; the second test case would always fail without the fix.

If the package builds successfully with the patch, then the test cases have passed.

[Regression Potential]

Regressions are unlikely due to the size and scope of the patch. The fix has been applied upstream since 30 Nov 2017 and is still in use in the latest release:

https://github.com/grpc/grpc-go/blob/v1.19.0/internal/transport/handler_server.go#L193

A possible source of regressions would be the squashing of the two commits into one. Luckily they were also applied upstream immediately one after the other so it should work without problems.

Another source would be if these patches depended on some other changes made since 1.6.0-3. However testing has not revealed this to be the case.

Issues might also arise from my inexperience in updating Ubuntu packages and submitting patches for Stable Release Updates.

[Other Info]

As golang-google-grpc is a Go library it is linked into other packages statically. This means that simply updating this package is not enough and dependents need to be rebuilt to benefit from these fixes.

If this patch is accepted then I plan to open another SRU request to set the fixed version as a minimal build dependency for etcd to have it also rebuilt and resolve https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1800973.

Release: Ubuntu 18.04.2 LTS
Package: golang-google-grpc 1.6.0-3

[Simon Quigley (tsimonq2)]

I have uploaded the debdiff to Bionic, Cosmic, and Disco.

Thank you for your contribution to Ubuntu, Tiit!

[Steve Langasek (vorlon)]

Hello Tiit, or anyone else affected,

Accepted golang-google-grpc into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/golang-google-grpc/1.6.0-3ubuntu0.19.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Tiit, or anyone else affected,

Accepted golang-google-grpc into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/golang-google-grpc/1.6.0-3ubuntu0.18.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Tiit, or anyone else affected,

Accepted golang-google-grpc into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/golang-google-grpc/1.6.0-3ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Tiit Pikma (thsnr)]

First up, thank you Simon for sponsoring this patch and Steve for accepting it into proposed.

The test case described in the bug report passed for bionic and cosmic when the packages were built successfully. I performed additional testing by building etcd 3.2.17+dfsg-1 using golang-google-grpc 1.6.0-3ubuntu0.18.04.1 on bionic and etcd 3.2.18+dfsg-1 using golang-google-grpc 1.6.0-3ubuntu0.18.10.1 on cosmic. I ran overnight stress tests on both and neither crashed, indicating that the issue seems to be resolved.

As for disco, the package failed to build from source (https://launchpad.net/ubuntu/+source/golang-google-grpc/1.6.0-3ubuntu0.19.04.1/+build/16729837). However when I tried building golang-google-grpc 1.6.0-3 locally on a clean install of disco, then it failed with the same error WITHOUT the patch. So golang-google-grpc seems to FTBFS on disco at the moment. I will investigate a bit to see if I can find the cause for this. I am hesitant to mark this as verification-failed-disco, since the failure was not the patch's fault.

[Tiit Pikma (thsnr)]

I reported the disco FTBFS at https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1828230.

Hopefully this unrelated disco bug will not block the fixed bionic and cosmic packages from moving into -updates.

[Łukasz Zemczak (sil2100)]

Thank you for your verification. The disco failure is indeed quite a shame, but since you are actively working on all the issues, I'll release the bionic and cosmic parts regardless. Let's just make sure that the FTBFS is resolved with some follow-up upload for disco as soon as possible.

[Launchpad Janitor (janitor)]

This bug was fixed in the package golang-google-grpc - 1.6.0-3ubuntu0.18.10.1

---------------
golang-google-grpc (1.6.0-3ubuntu0.18.10.1) cosmic; urgency=medium

  * Backport upstream fixes for intermittent panic due to a race condition
    when sending RPC status (LP: #1819936)
    - transport/handler_server.go: wrap status sending function in a mutex
    - transport/handler_server_test.go: add test cases for the races
    - Thanks to Gyuho Lee for the patches.

 -- Tiit Pikma <email address hidden> Wed, 13 Mar 2019 15:32:22 +0000

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for golang-google-grpc has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package golang-google-grpc - 1.6.0-3ubuntu0.18.04.1

---------------
golang-google-grpc (1.6.0-3ubuntu0.18.04.1) bionic; urgency=medium

  * Backport upstream fixes for intermittent panic due to a race condition
    when sending RPC status (LP: #1819936)
    - transport/handler_server.go: wrap status sending function in a mutex
    - transport/handler_server_test.go: add test cases for the races
    - Thanks to Gyuho Lee for the patches.

 -- Tiit Pikma <email address hidden> Wed, 13 Mar 2019 15:32:22 +0000

[Brian Murray (brian-murray)]

Hello Tiit, or anyone else affected,

Accepted golang-google-grpc into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/golang-google-grpc/1.6.0-3ubuntu0.19.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

All autopkgtests for the newly accepted golang-google-grpc (1.6.0-3ubuntu0.19.04.2) for disco have finished running.
There have been regressions in tests triggered by the package. Please visit the sru report page and investigate the failures.

https://people.canonical.com/~ubuntu-archive/pending-sru.html#disco

[Tiit Pikma (thsnr)]

Hello again, all,

I performed the same SRU verification steps as I described in comment #7 on disco using 1.6.0-3ubuntu0.19.04.2. The tests were successful and I changed the verification-needed tags to verification-done.

The autopkgtest failure was caused by a temporary network issue and has been resolved since.

[Launchpad Janitor (janitor)]

This bug was fixed in the package golang-google-grpc - 1.6.0-3ubuntu0.19.04.2

---------------
golang-google-grpc (1.6.0-3ubuntu0.19.04.2) disco; urgency=medium

  * Fix FTBFS caused by golang-goprotobuf-dev >= 1.2.0 (LP: #1828230)
    - Add Invoke and NewStream wrappers to ClientConn.
    - Use keyed fields for struct initializers (thanks to Doug Fawley).

golang-google-grpc (1.6.0-3ubuntu0.19.04.1) disco; urgency=medium

  * Backport upstream fixes for intermittent panic due to a race condition
    when sending RPC status (LP: #1819936)
    - transport/handler_server.go: wrap status sending function in a mutex
    - transport/handler_server_test.go: add test cases for the races
    - Thanks to Gyuho Lee for the patches.

 -- Tiit Pikma <email address hidden> Thu, 16 May 2019 10:54:50 +0000