intermittent panic in 1.6.0-3 due to race in WriteStatus

Bug #1819936 reported by Tiit Pikma on 2019-03-13
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
golang-google-grpc (Ubuntu)
Medium
Unassigned
Bionic
Medium
Unassigned
Cosmic
Medium
Unassigned
Disco
Medium
Unassigned

Bug Description

[Impact]

The version of golang-google-grpc packaged in bionic and cosmic (1.6.0-3) crashes intermittently due to a race condition. Upstream bug report and pull requests:

https://github.com/grpc/grpc-go/issues/1111
https://github.com/grpc/grpc-go/pull/1546
https://github.com/grpc/grpc-go/pull/1687

This can cause all packages Built-Using: golang-google-grpc=1.6.0-3 to crash as well (see https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1800973).

The attached debdiff contains two upstream fixes squashed together:

https://github.com/grpc/grpc-go/commit/22c3f92f5faea8db492fb0f5ae4daf0d2752b19e
https://github.com/grpc/grpc-go/commit/c6b46087ab923e9f453ec433f99174cdd45b9b89

These fix the race by synchronizing the entire function.

[Test Case]
The patch includes patches to test cases to exercise the race condition. One test case would fail sometimes without the fix; the second test case would always fail without the fix.

If the package builds successfully with the patch, then the test cases have passed.

[Regression Potential]

Regressions are unlikely due to the size and scope of the patch. The fix has been applied upstream since 30 Nov 2017 and is still in use in the latest release:

https://github.com/grpc/grpc-go/blob/v1.19.0/internal/transport/handler_server.go#L193

A possible source of regressions would be the squashing of the two commits into one. Luckily they were also applied upstream immediately one after the other so it should work without problems.

Another source would be if these patches depended on some other changes made since 1.6.0-3. However testing has not revealed this to be the case.

Issues might also arise from my inexperience in updating Ubuntu packages and submitting patches for Stable Release Updates.

[Other Info]

As golang-google-grpc is a Go library it is linked into other packages statically. This means that simply updating this package is not enough and dependents need to be rebuilt to benefit from these fixes.

If this patch is accepted then I plan to open another SRU request to set the fixed version as a minimal build dependency for etcd to have it also rebuilt and resolve https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1800973.

Release: Ubuntu 18.04.2 LTS
Package: golang-google-grpc 1.6.0-3

Tiit Pikma (thsnr) wrote :
Tiit Pikma (thsnr) on 2019-03-13
description: updated
Tiit Pikma (thsnr) wrote :
description: updated
tags: added: patch patch-accepted-upstream
Changed in etcd (Ubuntu):
importance: Undecided → Medium
Changed in golang-google-grpc (Ubuntu):
importance: Undecided → Medium
no longer affects: etcd (Ubuntu)
Simon Quigley (tsimonq2) wrote :

I have uploaded the debdiff to Bionic, Cosmic, and Disco.

Thank you for your contribution to Ubuntu, Tiit!

Changed in golang-google-grpc (Ubuntu Bionic):
importance: Undecided → Medium
Changed in golang-google-grpc (Ubuntu Cosmic):
importance: Undecided → Medium
Changed in golang-google-grpc (Ubuntu Disco):
importance: Undecided → Medium
Changed in golang-google-grpc (Ubuntu):
status: New → Fix Committed
Changed in golang-google-grpc (Ubuntu Bionic):
status: New → In Progress
Changed in golang-google-grpc (Ubuntu Cosmic):
status: New → In Progress
Changed in golang-google-grpc (Ubuntu Disco):
status: New → In Progress

Hello Tiit, or anyone else affected,

Accepted golang-google-grpc into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/golang-google-grpc/1.6.0-3ubuntu0.19.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

description: updated
Changed in golang-google-grpc (Ubuntu Disco):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-disco
Changed in golang-google-grpc (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed-cosmic
Steve Langasek (vorlon) wrote :

Hello Tiit, or anyone else affected,

Accepted golang-google-grpc into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/golang-google-grpc/1.6.0-3ubuntu0.18.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in golang-google-grpc (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Steve Langasek (vorlon) wrote :

Hello Tiit, or anyone else affected,

Accepted golang-google-grpc into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/golang-google-grpc/1.6.0-3ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Tiit Pikma (thsnr) wrote :

First up, thank you Simon for sponsoring this patch and Steve for accepting it into proposed.

The test case described in the bug report passed for bionic and cosmic when the packages were built successfully. I performed additional testing by building etcd 3.2.17+dfsg-1 using golang-google-grpc 1.6.0-3ubuntu0.18.04.1 on bionic and etcd 3.2.18+dfsg-1 using golang-google-grpc 1.6.0-3ubuntu0.18.10.1 on cosmic. I ran overnight stress tests on both and neither crashed, indicating that the issue seems to be resolved.

As for disco, the package failed to build from source (https://launchpad.net/ubuntu/+source/golang-google-grpc/1.6.0-3ubuntu0.19.04.1/+build/16729837). However when I tried building golang-google-grpc 1.6.0-3 locally on a clean install of disco, then it failed with the same error WITHOUT the patch. So golang-google-grpc seems to FTBFS on disco at the moment. I will investigate a bit to see if I can find the cause for this. I am hesitant to mark this as verification-failed-disco, since the failure was not the patch's fault.

tags: added: verification-done-bionic verification-done-cosmic
removed: verification-needed-bionic verification-needed-cosmic
Tiit Pikma (thsnr) wrote :

I reported the disco FTBFS at https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1828230.

Hopefully this unrelated disco bug will not block the fixed bionic and cosmic packages from moving into -updates.

Łukasz Zemczak (sil2100) wrote :

Thank you for your verification. The disco failure is indeed quite a shame, but since you are actively working on all the issues, I'll release the bionic and cosmic parts regardless. Let's just make sure that the FTBFS is resolved with some follow-up upload for disco as soon as possible.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package golang-google-grpc - 1.6.0-3ubuntu0.18.10.1

---------------
golang-google-grpc (1.6.0-3ubuntu0.18.10.1) cosmic; urgency=medium

  * Backport upstream fixes for intermittent panic due to a race condition
    when sending RPC status (LP: #1819936)
    - transport/handler_server.go: wrap status sending function in a mutex
    - transport/handler_server_test.go: add test cases for the races
    - Thanks to Gyuho Lee for the patches.

 -- Tiit Pikma <email address hidden> Wed, 13 Mar 2019 15:32:22 +0000

Changed in golang-google-grpc (Ubuntu Cosmic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for golang-google-grpc has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package golang-google-grpc - 1.6.0-3ubuntu0.18.04.1

---------------
golang-google-grpc (1.6.0-3ubuntu0.18.04.1) bionic; urgency=medium

  * Backport upstream fixes for intermittent panic due to a race condition
    when sending RPC status (LP: #1819936)
    - transport/handler_server.go: wrap status sending function in a mutex
    - transport/handler_server_test.go: add test cases for the races
    - Thanks to Gyuho Lee for the patches.

 -- Tiit Pikma <email address hidden> Wed, 13 Mar 2019 15:32:22 +0000

Changed in golang-google-grpc (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers