[Hyper-V] vmbus: Fix a bug in hv_need_to_signal_on_read()

[Joshua R. Poulson]

The following patch has been submitted upstream in response to investigation of customer issues with network connections hanging under high load.

On the consumer side, we have interrupt driven flow management of the producer. It is sufficient to base the signalling decision on the amount of space that is available to write after the read is complete. The current code samples the previous available space and uses this in making the signalling decision. This state can be stale and is unnecessary. Since the state can be stale, we end up not signalling the host (when we should) and this can result in a hang. Fix this problem by removing the unnecessary check.

I would like to thank Arseney Romanenko <email address hidden>
for pointing out this bug.

Signed-off-by: K. Y. Srinivasan <email address hidden>
Tested-by: Dexuan Cui <email address hidden>
Cc: <email address hidden>
---
 drivers/hv/ring_buffer.c | 7 +++----
 1 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
index 5613e2b..085003a 100644
--- a/drivers/hv/ring_buffer.c
+++ b/drivers/hv/ring_buffer.c
@@ -103,8 +103,7 @@ static bool hv_need_to_signal(u32 old_write, struct hv_ring_buffer_info *rbi)
  * there is room for the producer to send the pending packet.
  */

-static bool hv_need_to_signal_on_read(u32 prev_write_sz,
- struct hv_ring_buffer_info *rbi)
+static bool hv_need_to_signal_on_read(struct hv_ring_buffer_info *rbi)
 {
        u32 cur_write_sz;
        u32 r_size;
@@ -120,7 +119,7 @@ static bool hv_need_to_signal_on_read(u32 prev_write_sz,
        cur_write_sz = write_loc >= read_loc ? r_size - (write_loc - read_loc) :
                        read_loc - write_loc;

- if ((prev_write_sz < pending_sz) && (cur_write_sz >= pending_sz))
+ if (cur_write_sz >= pending_sz)
                return true;

        return false;
@@ -455,7 +454,7 @@ int hv_ringbuffer_read(struct hv_ring_buffer_info *inring_info,
        /* Update the read index */
        hv_set_next_read_location(inring_info, next_read_location);

- *signal = hv_need_to_signal_on_read(bytes_avail_towrite, inring_info);
+ *signal = hv_need_to_signal_on_read(inring_info);

        return ret;
 }

We have customers who are encountering this issue. Although this patch is not yet accepted upstream, we would like to get them a test kernel as soon as we can.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1556264

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

Is this bug affecting all releases or only affecting some of them?

[Joshua R. Poulson (jrp)]

Xenial, Wily, and Vivid for now.

[Joshua R. Poulson (jrp)]

No logs required, upstream submission.

[Joshua R. Poulson (jrp)]

The customers run a spectrum of kernels, but we will put Wily as the highest priority. It is possible we will have to iterate on this patch as well.

[Joseph Salisbury (jsalisbury)]

I'll have a Wily and Xenial test kernel ready shortly, since the patch applies cleanly to them. I'll have to modify the patch or identify prereq commits for it to apply to Trusty.

[Joseph Salisbury (jsalisbury)]

The Wily and Xenal test kernels with this patch are ready. They can be downloaded from:

Wily: http://kernel.ubuntu.com/~jsalisbury/lp1556264/wily/

Xenial: http://kernel.ubuntu.com/~jsalisbury/lp1556264/xenial

[Joseph Salisbury (jsalisbury)]

The Trusty and linux-lts-vivid test kernels with this patch are ready. They can be downloaded from:

Trusty: http://kernel.ubuntu.com/~jsalisbury/lp1556264/trusty

linux-lts-vivid: http://kernel.ubuntu.com/~jsalisbury/lp1556264/linux-lts-vivid

Both the Trusty and lts-vivid test kernels required commit a5cca686 as a prerequisite. This commit was in mainline as of 4.3-rc1, so Xenial already had it. Commit a5cca686 was applied to Wily via bug 1519917 when a 4.3 rebase was done.

[Joshua R. Poulson (jrp)]

Thank you very much, Joe, we'll get them to test it as soon as possible.

[Joseph Salisbury (jsalisbury)]

I can submit an SRU request for each release once we have testing feedback.

[Joshua R. Poulson (jrp)]

Initial feedback is promising. I think we can move forward with this.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-15.31

---------------
linux (4.4.0-15.31) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1559252

  * Xilinx KU3 Capi card does not show up in Ubuntu 16.04 (LP: #1557001)
    - SAUCE: (noup) cxl: Allow initialization on timebase sync failures

  * policy namespace stacking (LP: #1379535)
    - Revert "UBUNTU: SAUCE: Move replacedby allocation into label_alloc"
    - Revert "UBUNTU: SAUCE: Fixup: __label_update() still doesn't handle some cases correctly."
    - Revert "UBUNTU: SAUCE: fix: audit "no_new_privs" case for exec failure"
    - Revert "UBUNTU: SAUCE: fixup: warning about aa_label_vec_find_or_create not being static"
    - Revert "UBUNTU: SAUCE: apparmor: fix refcount race when finding a child profile"
    - Revert "UBUNTU: SAUCE: fixup: cast poison values to remove warnings"
    - Revert "UBUNTU: SAUCE: fixup: get rid of unused var build warning"
    - Revert "UBUNTU: SAUCE: fixup: 20/23 locking issue around in __label_update"
    - Revert "UBUNTU: SAUCE: fixup: make __share_replacedby private to get rid of build warning"
    - Revert "UBUNTU: SAUCE: fix: replacedby forwarding is not being properly update when ns is destroyed"
    - Revert "UBUNTU: SAUCE: apparmor: fix log of apparmor audit message when kern_path() fails"
    - Revert "UBUNTU: SAUCE: fixup: cleanup return handling of labels"
    - Revert "UBUNTU: SAUCE: apparmor: fix: ref count leak when profile sha1 hash is read"
    - Revert "UBUNTU: SAUCE: apparmor: Fix: query label file permission"
    - Revert "UBUNTU: SAUCE: apparmor: Don't remove label on rcu callback if the label has already been removed"
    - Revert "UBUNTU: SAUCE: apparmor: Fix: break circular refcount for label that is directly freed."
    - Revert "UBUNTU: SAUCE: apparmor: Fix: refcount bug when inserting label update that transitions ns"
    - Revert "UBUNTU: SAUCE: apparmor: Fix: now that insert can force replacement use it instead of remove_and_insert"
    - Revert "UBUNTU: SAUCE: apparmor Fix: refcount bug in pivotroot mediation"
    - Revert "UBUNTU: SAUCE: apparmor: ensure that repacedby sharing is done correctly"
    - Revert "UBUNTU: SAUCE: apparmor: Fix: update replacedby allocation to take a gfp parameter"
    - Revert "UBUNTU: SAUCE: apparmor: Fix: convert replacedby update to be protected by the labelset lock"
    - Revert "UBUNTU: SAUCE: apparmor: Fix: add required locking of __aa_update_replacedby on merge path"
    - Revert "UBUNTU: SAUCE: apparmor: Fix: deadlock in aa_put_label() call chain"
    - Revert "UBUNTU: SAUCE: apparmor: Fix: label_vec_merge insertion"
    - Revert "UBUNTU: SAUCE: apparmor: Fix: ensure new labels resulting from merge have a replacedby"
    - Revert "UBUNTU: SAUCE: apparmor: Fix: refcount leak in aa_label_merge"
    - Revert "UBUNTU: SAUCE: apparmor: Fix: refcount race between locating in labelset and get"
    - Revert "UBUNTU: SAUCE: apparmor: Fix: label merge handling of marking unconfined and stale"
    - Revert "UBUNTU: SAUCE: apparmor: add underscores to indicate aa_label_next_not_in_set() use needs locking"
    - Revert "UBUNTU: SAUCE: apparmor: debug: POISON label and replaceby ...

[Joshua R. Poulson (jrp)]

A more comprehensive fix has been submitted upstream, but in the meantime, we'd like the first fix to be considered for inclusion in Wily.

The more comprehensive change is a 7-part series submitted Mar 23 in lkml, and uses full memory barrier before making signaling decisions:

"Drivers: hv: vmbus: Cleanup the ring buffer code"
Cleanup and fix a bug in the ring buffer code. Also implement
APIs for in place consumption of received packets.

K. Y. Srinivasan (7):
  Drivers: hv: vmbus: Introduce functions for estimating room in the
    ring buffer
  Drivers: hv: vmbus: Use READ_ONCE() to read variables that are
    volatile
  Drivers: hv: vmbus: Fix a bug in hv_need_to_signal_on_read()
  Drivers: hv: vmbus: Use the new virt_xx barrier code
  Drivers: hv: vmbus: Export the vmbus_set_event() API
  Drivers: hv: vmbus: Move some ring buffer functions to hyperv.h
  Drivers: hv: vmbus: Implement APIs to support "in place" consumption
    of vmbus packets

 drivers/hv/connection.c | 1 +
 drivers/hv/hyperv_vmbus.h | 2 -
 drivers/hv/ring_buffer.c | 79 ++++-------------------
 include/linux/hyperv.h | 156 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 169 insertions(+), 69 deletions(-)

We will test this new change and respond to upstream feedback. The first fix in this bug works for our mutual customers at the moment.

[Joseph Salisbury (jsalisbury)]

An SRU request has been submitted for all stable releases to include the first fix. I can build a test kernel with the new 7 patches when needed.

[Joshua R. Poulson (jrp)]

Sorry, I know we have a version of this fix in proposed now, but based on upstream feedback, we should switch to the following patch:

From: K. Y. Srinivasan [mailto:<email address hidden>]
Sent: Tuesday, April 5, 2016 11:50 AM
To: <email address hidden>; <email address hidden>; <email address hidden>; <email address hidden>; <email address hidden>; <email address hidden>; <email address hidden>
Cc: KY Srinivasan <email address hidden>; <email address hidden>
Subject: [PATCH 1/1] Drivers: hv: vmbus: Fix signaling logic in hv_need_to_signal_on_read()

On the consumer side, we have interrupt driven flow management of the producer. It is sufficient to base the signaling decision on the amount of space that is available to write after the read is complete.
The current code samples the previous available space and uses this in making the signaling decision. This state can be stale and is unnecessary. Since the state can be stale, we end up not signaling the host (when we should) and this can result in a hang. Fix this problem by removing the unnecessary check. I would like to thank Arseney Romanenko <email address hidden> for pointing out this issue.

Also, issue a full memory barrier before making the signaling descision to correctly deal with potential reordering of the write (read index) followed by the read of pending_sz.

Signed-off-by: K. Y. Srinivasan <email address hidden>
Tested-by: Dexuan Cui <email address hidden>
Cc: <email address hidden>
---
 drivers/hv/ring_buffer.c | 26 ++++++++++++++++++++------
 1 files changed, 20 insertions(+), 6 deletions(-)

diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c index 5613e2b..a40a73a 100644
--- a/drivers/hv/ring_buffer.c
+++ b/drivers/hv/ring_buffer.c
@@ -103,15 +103,29 @@ static bool hv_need_to_signal(u32 old_write, struct hv_ring_buffer_info *rbi)
  * there is room for the producer to send the pending packet.
  */

-static bool hv_need_to_signal_on_read(u32 prev_write_sz,
- struct hv_ring_buffer_info *rbi)
+static bool hv_need_to_signal_on_read(struct hv_ring_buffer_info *rbi)
 {
  u32 cur_write_sz;
  u32 r_size;
- u32 write_loc = rbi->ring_buffer->write_index;
+ u32 write_loc;
  u32 read_loc = rbi->ring_buffer->read_index;
- u32 pending_sz = rbi->ring_buffer->pending_send_sz;
+ u32 pending_sz;

+ /*
+ * Issue a full memory barrier before making the signaling decision.
+ * Here is the reason for having this barrier:
+ * If the reading of the pend_sz (in this function)
+ * were to be reordered and read before we commit the new read
+ * index (in the calling function) we could
+ * have a problem. If the host were to set the pending_sz after we
+ * have sampled pending_sz and go to sleep before we commit the
+ * read index, we could miss sending the interrupt. Issue a full
+ * memory barrier to address this.
+ */
+ mb();
+
+ pending_sz = rbi->ring_buffer->pending_send_sz;
+ write_loc = rbi->ring_buffer->write_index;
  /* If the other end is not blocked on write don't bother. */
  if (pending_sz == 0)
   return false;
@@ -120,7 +134,7 @@ static bool hv_need_to_signal_on_read(u32 prev_write_sz,
  cur_write_sz = write_loc >= read_loc ? r_size - (write_loc - read_loc) :
...

[Joseph Salisbury (jsalisbury)]

Would it be best to wait until the patch lands in mainline? That way we ensure we are getting the correct version of the patch.

KY filed the first version of that patch on April 2, but then replied Greg to drop the patch, it was the wrong version:
http://marc.info/?t=145963120100001&r=1&w=2

On April 5, another version of the patch was submitted(same title, different contents).

Also, there is another April 2 version with the title "[PATCH v2 1/1]" here:
http://marc.info/?t=145963331800024&r=1&w=2

If this "new version of the fix" is necessary to fix a bug, we may want to open a new bug to ensure the fix gets tracked properly.

[Kamal Mostafa (kamalmostafa)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Kamal Mostafa (kamalmostafa)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-vivid' to 'verification-done-vivid'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Kamal Mostafa (kamalmostafa)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-wily' to 'verification-done-wily'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Joseph Salisbury (jsalisbury)]

Joshua,

Would it be possible to have the fixes in -proposed tested?

[Chris Valean (cvalean)]

Hi Joe,

I will have this verified asap, thanks for the notice!

[Chris Valean (cvalean)]

I've done a quick test pass and also measured the network throughput for the below 2 proposed kernels. The results are looking good, marking this as good.

Trusty - 3.13.0.86.92
Wily - 4.2.0.36.39

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.2.0-36.41

---------------
linux (4.2.0-36.41) wily; urgency=low

  [ Kamal Mostafa ]

  * Release Tracking Bug
    - LP: #1571667

  [ Benjamin Tissoires ]

  * SAUCE: Input: synaptics - handle spurious release of trackstick
    buttons, again
    - LP: #1553811

  [ dann frazier ]

  * Revert "SAUCE: arm64, numa, dt: adding dt based numa support using dt
    node property arm, associativity"
    - LP: #1558828
  * Revert "SAUCE: Documentation: arm64/arm: dt bindings for numa."
    - LP: #1558828
  * Revert "SAUCE: arm64, numa: adding numa support for arm64 platforms."
    - LP: #1558828
  * Revert "[Config] Enable NUMA on ARM64"
    - LP: #1558828

  [ K. Y. Srinivasan ]

  * SAUCE: (noup): Drivers: hv: vmbus: Fix a bug in
    hv_need_to_signal_on_read()
    - LP: #1556264

  [ Kamal Mostafa ]

  * [debian] BugLink: close LP: bugs only for Launchpad urls
  * [Config] updateconfigs after v4.2.8-ckt7

  [ Upstream Kernel Changes ]

  * Revert "jffs2: Fix lock acquisition order bug in jffs2_write_begin"
    - LP: #1561677
  * tipc: fix connection abort during subscription cancel
    - LP: #1561677
  * tipc: fix nullptr crash during subscription cancel
    - LP: #1561677
  * s390/mm: four page table levels vs. fork
    - LP: #1561677
  * Input: aiptek - fix crash on detecting device without endpoints
    - LP: #1561677
  * wext: fix message delay/ordering
    - LP: #1561677
  * cfg80211/wext: fix message ordering
    - LP: #1561677
  * mac80211: fix use of uninitialised values in RX aggregation
    - LP: #1561677
  * mac80211: minstrel: Change expected throughput unit back to Kbps
    - LP: #1561677
  * libata: fix HDIO_GET_32BIT ioctl
    - LP: #1561677
  * iwlwifi: mvm: inc pending frames counter also when txing non-sta
    - LP: #1561677
  * [media] adv7604: fix tx 5v detect regression
    - LP: #1561677
  * ahci: add new Intel device IDs
    - LP: #1561677
  * ahci: Order SATA device IDs for codename Lewisburg
    - LP: #1561677
  * Adding Intel Lewisburg device IDs for SATA
    - LP: #1561677
  * ASoC: samsung: Use IRQ safe spin lock calls
    - LP: #1561677
  * mac80211: minstrel_ht: set default tx aggregation timeout to 0
    - LP: #1561677
  * usb: chipidea: otg: change workqueue ci_otg as freezable
    - LP: #1561677
  * jffs2: Fix page lock / f->sem deadlock
    - LP: #1561677
  * Fix directory hardlinks from deleted directories
    - LP: #1561677
  * iommu/amd: Fix boot warning when device 00:00.0 is not iommu covered
    - LP: #1561677
  * iommu/amd: Apply workaround for ATS write permission check
    - LP: #1561677
  * libata: Align ata_device's id on a cacheline
    - LP: #1561677
  * can: gs_usb: fixed disconnect bug by removing erroneous use of kfree()
    - LP: #1561677
  * fbcon: set a default value to blink interval
    - LP: #1561677
  * KVM: x86: fix root cause for missed hardware breakpoints
    - LP: #1561677
  * arm64: vmemmap: use virtual projection of linear region
    - LP: #1561677
  * vfio: fix ioctl error handling
    - LP: #1561677
  * ALSA: ctl: Fix ioctls for X32 ABI
    - LP: #1561677
  * ALSA: pcm: Fix ioctls for X32 ABI
    - LP: #1561677
  * ALSA: rawmidi: Fix ioct...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.13.0-86.130

---------------
linux (3.13.0-86.130) trusty; urgency=low

  [ Kamal Mostafa ]

  * Release Tracking Bug
    - LP: #1571718

  [ Benjamin Tissoires ]

  * SAUCE: Input: synaptics - handle spurious release of trackstick
    buttons, again
    - LP: #1553811

  [ K. Y. Srinivasan ]

  * SAUCE: (noup): Drivers: hv: vmbus: Fix a bug in
    hv_need_to_signal_on_read()
    - LP: #1556264

  [ Kamal Mostafa ]

  * [debian] BugLink: close LP: bugs only for Launchpad urls
  * [Config] updateconfigs after v3.13.11-ckt38

  [ Tim Gardner ]

  * [Debian] Fix linux-doc dangling symlinks
    - LP: #661306

  [ Upstream Kernel Changes ]

  * Revert "jffs2: Fix lock acquisition order bug in jffs2_write_begin"
    - LP: #1562900
  * [stable-only] AIO: properly check iovec sizes
    - LP: #1562900
  * Input: aiptek - fix crash on detecting device without endpoints
    - LP: #1562900
  * wext: fix message delay/ordering
    - LP: #1562900
  * cfg80211/wext: fix message ordering
    - LP: #1562900
  * mac80211: fix use of uninitialised values in RX aggregation
    - LP: #1562900
  * libata: fix HDIO_GET_32BIT ioctl
    - LP: #1562900
  * mac80211: minstrel_ht: set default tx aggregation timeout to 0
    - LP: #1562900
  * jffs2: Fix page lock / f->sem deadlock
    - LP: #1562900
  * Fix directory hardlinks from deleted directories
    - LP: #1562900
  * iommu/amd: Fix boot warning when device 00:00.0 is not iommu covered
    - LP: #1562900
  * libata: Align ata_device's id on a cacheline
    - LP: #1562900
  * vfio: fix ioctl error handling
    - LP: #1562900
  * ALSA: ctl: Fix ioctls for X32 ABI
    - LP: #1562900
  * ALSA: rawmidi: Fix ioctls X32 ABI
    - LP: #1562900
  * ALSA: timer: Fix broken compat timer user status ioctl
    - LP: #1562900
  * ALSA: timer: Fix ioctls for X32 ABI
    - LP: #1562900
  * cifs: fix out-of-bounds access in lease parsing
    - LP: #1562900
  * CIFS: Fix SMB2+ interim response processing for read requests
    - LP: #1562900
  * ALSA: hdspm: Fix wrong boolean ctl value accesses
    - LP: #1562900
  * ALSA: hdspm: Fix zero-division
    - LP: #1562900
  * ALSA: hdsp: Fix wrong boolean ctl value accesses
    - LP: #1562900
  * USB: cp210x: Add ID for Parrot NMEA GPS Flight Recorder
    - LP: #1562900
  * ASoC: wm8958: Fix enum ctl accesses in a wrong type
    - LP: #1562900
  * ASoC: wm8994: Fix enum ctl accesses in a wrong type
    - LP: #1562900
  * ASoC: wm_adsp: Fix enum ctl accesses in a wrong type
    - LP: #1562900
  * USB: serial: option: add support for Telit LE922 PID 0x1045
    - LP: #1562900
  * USB: serial: option: add support for Quectel UC20
    - LP: #1562900
  * ALSA: seq: oss: Don't drain at closing a client
    - LP: #1562900
  * drm/ast: Fix incorrect register check for DRAM width
    - LP: #1562900
  * drm/radeon/pm: update current crtc info after setting the powerstate
    - LP: #1562900
  * PM / sleep / x86: Fix crash on graph trace through x86 suspend
    - LP: #1562900
  * ALSA: hda - Fix mic issues on Acer Aspire E1-472
    - LP: #1562900
  * MIPS: traps: Fix SIGFPE information leak from `do_ov' and
    `do_trap_or_bp'
    - LP: #1562900
  * ubi: Fix...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-lts-vivid - 3.19.0-59.65~14.04.1

---------------
linux-lts-vivid (3.19.0-59.65~14.04.1) trusty; urgency=low

  [ Kamal Mostafa ]

  * Release Tracking Bug
    - LP: #1572258

  [ Benjamin Tissoires ]

  * SAUCE: Input: synaptics - handle spurious release of trackstick
    buttons, again
    - LP: #1553811

  [ K. Y. Srinivasan ]

  * SAUCE: (noup): Drivers: hv: vmbus: Fix a bug in
    hv_need_to_signal_on_read()
    - LP: #1556264

  [ Kamal Mostafa ]

  * [debian] BugLink: close LP: bugs only for Launchpad urls
  * [Config] updateconfigs after v3.19.8-ckt18

  [ Upstream Kernel Changes ]

  * Revert "jffs2: Fix lock acquisition order bug in jffs2_write_begin"
    - LP: #1561714
  * [stable-only] AIO: properly check iovec sizes
    - LP: #1561714
  * Input: aiptek - fix crash on detecting device without endpoints
    - LP: #1561714
  * wext: fix message delay/ordering
    - LP: #1561714
  * cfg80211/wext: fix message ordering
    - LP: #1561714
  * mac80211: fix use of uninitialised values in RX aggregation
    - LP: #1561714
  * libata: fix HDIO_GET_32BIT ioctl
    - LP: #1561714
  * iwlwifi: mvm: inc pending frames counter also when txing non-sta
    - LP: #1561714
  * [media] adv7604: fix tx 5v detect regression
    - LP: #1561714
  * ahci: add new Intel device IDs
    - LP: #1561714
  * ahci: Order SATA device IDs for codename Lewisburg
    - LP: #1561714
  * Adding Intel Lewisburg device IDs for SATA
    - LP: #1561714
  * mac80211: minstrel_ht: set default tx aggregation timeout to 0
    - LP: #1561714
  * usb: chipidea: otg: change workqueue ci_otg as freezable
    - LP: #1561714
  * jffs2: Fix page lock / f->sem deadlock
    - LP: #1561714
  * Fix directory hardlinks from deleted directories
    - LP: #1561714
  * iommu/amd: Fix boot warning when device 00:00.0 is not iommu covered
    - LP: #1561714
  * iommu/amd: Apply workaround for ATS write permission check
    - LP: #1561714
  * libata: Align ata_device's id on a cacheline
    - LP: #1561714
  * can: gs_usb: fixed disconnect bug by removing erroneous use of kfree()
    - LP: #1561714
  * vfio: fix ioctl error handling
    - LP: #1561714
  * ALSA: ctl: Fix ioctls for X32 ABI
    - LP: #1561714
  * ALSA: rawmidi: Fix ioctls X32 ABI
    - LP: #1561714
  * ALSA: timer: Fix broken compat timer user status ioctl
    - LP: #1561714
  * ALSA: timer: Fix ioctls for X32 ABI
    - LP: #1561714
  * cifs: fix out-of-bounds access in lease parsing
    - LP: #1561714
  * CIFS: Fix SMB2+ interim response processing for read requests
    - LP: #1561714
  * Fix cifs_uniqueid_to_ino_t() function for s390x
    - LP: #1561714
  * arm/arm64: KVM: Fix ioctl error handling
    - LP: #1561714
  * ALSA: hdspm: Fix wrong boolean ctl value accesses
    - LP: #1561714
  * ALSA: hdspm: Fix zero-division
    - LP: #1561714
  * ALSA: hdsp: Fix wrong boolean ctl value accesses
    - LP: #1561714
  * USB: qcserial: add Dell Wireless 5809e Gobi 4G HSPA+ (rev3)
    - LP: #1561714
  * USB: cp210x: Add ID for Parrot NMEA GPS Flight Recorder
    - LP: #1561714
  * ASoC: wm8958: Fix enum ctl accesses in a wrong type
    - LP: #1561714
  * ASoC: wm8994: Fix enum ctl accesses in ...