Multipathd hangs with long iscsi target names in Ubuntu 18.04

[Fernando Ferraz]

[Impact]

 * Users of some storage devices (for instance NetApp E-Series RDAC) due
   to very long names increasing the VDP page and and cause multipathd to
   hang

 * Introduced as side effect of other changes in 7.4 and fixed later this
   is backporting the HW exploitation to read an enhanced VDP in this case
   which in such cases works fine again.

[Test Case]

 * #1 Case: Test on affected hardware. The NetApp E-Series RDAC is an
   example, the reporter has such HW and is able to do so.

 * #2 regression tests: since this touches multipath discovery any such
   cases would be interesting tests. From installers using multipath on
   mainframes to multipath-iscsi disks on an raspi. The Range is huge and
   unclear how much exactly should be done. I can myself do said mainframe
   checks and utilize the pkgtests to run through iscsid.
   In addition we should keep this one a bit longer in -proposed IMHO.

[Regression Potential]

 * All cases asking for vpd 0x83 should be unchanged and only those asking
   for an 0xc9 should get the new improved behavior. But I'm not a 100%
   sure on potential side effects yet. Changes and thereby regressions
   would be limited to the device discovery of multipathd, so that is what
   one needs to take an extra eye on e.g. pre/post list of multipath
   devices (and attributes as those are populated via vpd).

[Other Info]

 * n/a

---

I've notice recently that multipathd started hanging when trying to handle long iscsi target names. In my case these targets are generated by the OpenStack cinder driver for SolidFire.

My guess is that the problem is occurring due to a bug introduced in multipath-tools 0.7.4, by commit "limbultipath: prefer RDAC checker with detect_checker*"*[1].

This change adds a routine to check RDAC compatibility during the detect_checker procedure, which runs into an infinite loop in the piece of code that tries to gather SCSI VPD pages larger than 254 bytes (this is my case due to long iSCSI Target names).

This issue was later addressed by several commits in 0.7.5 and 0.7.6: "libmultipath: sgio_get_vpd: add page argument" [2], "libmultipath: Fix sgio_get_vpd()" [3], "libmultipath: fix return code of sgio_get_vpd()" [4], "libmultipath: get_vpd_sgio: support VPD 0xc9" [5].

I've found a temporary work around for this by using select_checker = “no” in Ubuntu 18. This appears to avoid the problem by skipping RDAC compatibility check code.

Also, I've also tested 0.8.3 in a RH environment and found no issues running default select_checker = “yes” (Ubuntu 20 also delivers 0.8.3). ‌

So, in short:

- Looks like Ubuntu multipath-tools is broken,
- There is a fix in newer versions that aren’t in the ubuntu 18 repo yet.
- The OpenStack Cinder driver for Solidfire just happen to hit this bug because of the long names
- Setting select_checker = “no” is an possible work-around until Ubuntu publishes 0.7.6

Bellow "multipath -r -v3" output from my env:

:/$ cat /etc/multipath.conf

devices {
device {
vendor "SolidFir" product "SSD SAN" detect_checker "no"
}
}

:/$ sudo multipath -r -v3 Jul 09 19:06:24 | loading //lib/multipath/libchecktur.so checker Jul 09 19:06:24 | loading //lib/multipath/libprioconst.so prioritizer Jul 09 19:06:24 | vda: blacklisted, udev property missing Jul 09 19:06:24 | sda: udev property ID_WWN whitelisted Jul 09 19:06:24 | sda: mask = 0x1f Jul 09 19:06:24 | sda: dev_t = 8:0 Jul 09 19:06:24 | sda: size = 10485760 Jul 09 19:06:24 | sda: vendor = SolidFir Jul 09 19:06:24 | sda: product = SSD SAN Jul 09 19:06:24 | sda: rev = 0001 Jul 09 19:06:24 | sda: h:b:t:l = 2:0:0:0 Jul 09 19:06:24 | sda: tgt_node_name = iqn.2010-01.com.solidfire:xpsi.uuid-7b8a927e-4dbc-4a2b-b47a-1bde324895eb.53 Jul 09 19:06:24 | sda: path state = running Jul 09 19:06:24 | sda: 1018 cyl, 166 heads, 62 sectors/track, start at 0 Jul 09 19:06:24 | sda: serial = 7870736900000035f47acc0100000000 Jul 09 19:06:24 | sda: get_state Jul 09 19:06:24 | sda: detect_checker = no (setting: storage device configuration) Jul 09 19:06:24 | sda: path_checker = tur (setting: multipath internal) Jul 09 19:06:24 | sda: checker timeout = 30 s (setting: multipath internal) Jul 09 19:06:24 | sda: tur state = up Jul 09 19:06:24 | sda: uid_attribute = ID_SERIAL (setting: multipath internal) Jul 09 19:06:24 | sda: uid = 36f47acc1000000007870736900000035 (udev) Jul 09 19:06:24 | sda: detect_prio = yes (setting: multipath internal) Jul 09 19:06:24 | sda: prio = const (setting: multipath internal) Jul 09 19:06:24 | sda: prio args = "" (setting: multipath internal) Jul 09 19:06:24 | sda: const prio = 1 Jul 09 19:06:24 | loop0: blacklisted, udev property missing Jul 09 19:06:24 | loop1: blacklisted, udev property missing Jul 09 19:06:24 | loop2: blacklisted, udev property missing Jul 09 19:06:24 | loop3: blacklisted, udev property missing Jul 09 19:06:24 | loop4: blacklisted, udev property missing Jul 09 19:06:24 | loop5: blacklisted, udev property missing Jul 09 19:06:24 | loop6: blacklisted, udev property missing Jul 09 19:06:24 | loop7: blacklisted, udev property missing ===== paths list ===== uuid hcil dev dev_t pri dm_st chk_st vend/prod 36f47acc1000000007870736900000035 2:0:0:0 sda 8:0 1 undef undef SolidFir, Jul 09 19:06:24 | libdevmapper version 1.02.145 (2017-11-03) Jul 09 19:06:24 | DM multipath kernel driver v1.13.0 Jul 09 19:06:24 | sda: udev property ID_WWN whitelisted Jul 09 19:06:24 | sda: (SolidFir:SSD SAN) vendor/product blacklisted Jul 09 19:06:24 | sda: orphan path, wwid blacklisted Jul 09 19:06:24 | const prioritizer refcount 1 Jul 09 19:06:24 | tur checker refcount 1 Jul 09 19:06:24 | unloading const prioritizer Jul 09 19:06:24 | unloading tur checker

[1]https://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=84e16fbdb6648e685b65781d6ef0742469bd4b18

[2]https://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=d74a79f3590f3f28d34287b64c565fd5bf9705d0

[3]https://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=1f401b69e478b35d4b54b1980be67943c6d61d9f

[4]https://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=c6140952d1b682261c07cdb7138c944ee83d8809

[5]https://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=9d0e6bffbb3e490e92cf91444cfeb4177d540707

[Rafael David Tinoco (rafaeldtinoco)]

Thanks a lot for reporting this Fernando. Let me take a deeper look in what you provided and will get back to you soon.

[Christian Ehrhardt  (paelzer)]

Assuming the referred patches match they are all in >=7.5 which means >Bionic is fixed.

[Fernando Ferraz (fernando-ferraz)]

Hi Christian, you are correct. I Can't reproduce this issue in Focal (0.8.3).

Could multipath-tools be upgraded to 0.8.3 for Bionic aswell?

[Christian Ehrhardt  (paelzer)]

Hi Fernando,
Bionic will stay at 0.7.4 for SRU Policy reasons [1] but we should check if we can backport just the fixes in a way that works for your case without regressing others.

Thanks for your near instant reply BTW, If I get this backported would you be able to test a PPA to check if the fix would work for you?

[1]: https://wiki.ubuntu.com/StableReleaseUpdates

[Fernando Ferraz (fernando-ferraz)]

Hi Christian, sounds good to me. I can test it for sure.

[Christian Ehrhardt  (paelzer)]

7.6 1f401b69e libmultipath: Fix sgio_get_vpd()
7.5 9d0e6bffb libmultipath: get_vpd_sgio: support VPD 0xc9
7.5 d74a79f35 libmultipath: sgio_get_vpd: add page argument
7.5 c6140952d libmultipath: fix return code of sgio_get_vpd()

All applied as-is without any noise and they also look small and reasonable.
I already started to build it in:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4282/

@Fernando: If you could give those - once built - a try and let me know if/how they work that would be awesome.

[Fernando Ferraz (fernando-ferraz)]

Hi Christian, I've tested the PPA and seems to work just fine. I will post the results here by the and of this week.

[Christian Ehrhardt  (paelzer)]

@Fernando - I beg your pardon, but do not want to get this be lost. Are there any further updates on the PPA-test?

[Fernando Ferraz (fernando-ferraz)]

Hi Christian, sorry for the late reply.

After applying the PPA, multipath no longer hangs when attaching a SolidFire volumes. The problem with long iscsi target names seems to be fixed now.

See the output bellow:

stack@60-sfernand-development:~$ sudo multipath -v6 -ll
Oct 13 20:22:27 | loading //lib/multipath/libchecktur.so checker
Oct 13 20:22:27 | loading //lib/multipath/libprioconst.so prioritizer
Oct 13 20:22:27 | Discover device /sys/devices/pci0000:00/0000:00:06.0/virtio3/block/vda
Oct 13 20:22:27 | vda: dev not found in pathvec
Oct 13 20:22:27 | 252:0: dev_t not found in pathvec
Oct 13 20:22:27 | vda: blacklisted, udev property missing
Oct 13 20:22:27 | Discover device /sys/devices/pci0000:00/0000:00:06.0/virtio3/block/vda/vda1
Oct 13 20:22:27 | Discover device /sys/devices/pci0000:00/0000:00:06.0/virtio3/block/vda/vda14
Oct 13 20:22:27 | Discover device /sys/devices/pci0000:00/0000:00:06.0/virtio3/block/vda/vda15
Oct 13 20:22:27 | Discover device /sys/devices/platform/host4/session228/target4:0:0/4:0:0:0/block/sdc
Oct 13 20:22:27 | sdc: dev not found in pathvec
Oct 13 20:22:27 | 8:32: dev_t not found in pathvec
Oct 13 20:22:27 | sdc: udev property ID_WWN whitelisted
Oct 13 20:22:27 | sdc: mask = 0x7
Oct 13 20:22:27 | sdc: dev_t = 8:32
Oct 13 20:22:27 | open '/sys/devices/platform/host4/session228/target4:0:0/4:0:0:0/block/sdc/size'
Oct 13 20:22:27 | sdc: size = 10485760
Oct 13 20:22:27 | sdc: vendor = SolidFir
Oct 13 20:22:27 | sdc: product = SSD SAN
Oct 13 20:22:27 | sdc: rev = 0001
Oct 13 20:22:27 | sdc: h:b:t:l = 4:0:0:0
Oct 13 20:22:27 | sdc: tgt_node_name = iqn.2010-01.com.solidfire:xpsi.uuid-e100a0c6-f8d6-4927-b6ea-213248083965.3409
Oct 13 20:22:27 | open '/sys/devices/platform/host4/session228/target4:0:0/4:0:0:0/state'
Oct 13 20:22:27 | sdc: path state = running
Oct 13 20:22:27 | sdc: 1018 cyl, 166 heads, 62 sectors/track, start at 0
Oct 13 20:22:27 | open '/sys/devices/platform/host4/session228/target4:0:0/4:0:0:0/vpd_pg80'
Oct 13 20:22:27 | sdc: serial = 7870736900000d51f47acc0100000000
Oct 13 20:22:27 | sdc: get_state
Oct 13 20:22:27 | sdc: detect_checker = yes (setting: storage device configuration)
Oct 13 20:22:27 | failed to issue vpd inquiry for pgc9
Oct 13 20:22:27 | sdc: path_checker = tur (setting: multipath internal)
Oct 13 20:22:27 | sdc: checker timeout = 30 s (setting: multipath internal)
Oct 13 20:22:27 | sdc: tur state = up
Oct 13 20:22:27 | Discover device /sys/devices/virtual/block/loop0
Oct 13 20:22:27 | loop0: dev not found in pathvec
Oct 13 20:22:27 | 7:0: dev_t not found in pathvec
Oct 13 20:22:27 | loop0: blacklisted, udev property missing
Oct 13 20:22:27 | Discover device /sys/devices/virtual/block/loop1
Oct 13 20:22:27 | loop1: dev not found in pathvec
Oct 13 20:22:27 | 7:1: dev_t not found in pathvec
Oct 13 20:22:27 | loop1: blacklisted, udev property missing
Oct 13 20:22:27 | Discover device /sys/devices/virtual/block/loop2
Oct 13 20:22:27 | loop2: dev not found in pathvec
Oct 13 20:22:27 | 7:2: dev_t not found in pathvec
Oct 13 20:22:27 | loop2: blacklisted, udev property missing
Oct 13 20:22:27 | Discover device /sys/devices/virtual/block/loop3
Oct 13 20:22:27 | loop3: dev not found in pathvec
...

[Christian Ehrhardt  (paelzer)]

Uploaded to Bionic-unapproved to enter SRU processing.
@Fernando - when this lands in -proposed we will need your help again to test it on an affected setup.

[Fernando Ferraz (fernando-ferraz)]

Hi Christian, just to make it sure we are in sync:

Problem is not related to users using NetApp E-Series. I've reproduced this issue in a NetApp SolidFire system that has no relation with E-Series. My understanding is that any storage system our platform that make use of target names larger than 254 bytes will hit this problem.

[Christian Ehrhardt  (paelzer)]

>
> Problem is not related to users using NetApp E-Series. I've reproduced
> this issue in a NetApp SolidFire system that has no relation with
> E-Series. My understanding is that any storage system our platform that
> make use of target names larger than 254 bytes will hit this problem.
>

Yeah - thanks for the info, that does not invalidate the SRU. It actually
makes it slightly more important as more systems are affected.

[Robie Basak (racb)]

> + len = buff_len < 8 ? -ENODATA :
> + (buff_len <= maxlen ? buff_len : maxlen);
> + memcpy (str, buff, len);

This looked concerning to me. There's a code path that will call memcpy with a len of -ENODATA which seemed quite dangerous to me.

Upstream seems to have already fixed this here: https://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=blob;f=libmultipath/discovery.c;h=193d7127addbd3ad3feec30d1f67e20dabfb1293;hb=HEAD#l1303

Could you incorporate this please? Are there any other changes made upstream to this code since the cherry-picks which are also relevant?

[Robie Basak (racb)]

https://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commitdiff;h=0f5e45b02508ca7ac93b63e07491edaf2e60cbc7 seems relevant.

[Christian Ehrhardt  (paelzer)]

On Wed, Oct 28, 2020 at 7:11 PM Robie Basak <email address hidden>
wrote:

> > + len = buff_len < 8 ? -ENODATA :
> > + (buff_len <= maxlen ? buff_len : maxlen);
> > + memcpy (str, buff, len);
>
> This looked concerning to me. There's a code path that will call memcpy
> with a len of -ENODATA which seemed quite dangerous to me.
>
>
Great catch Robie,
I'll incorporate that.

[Christian Ehrhardt  (paelzer)]

So far: c6140952d 1f401b69e 9d0e6bffb d74a79f35
Unfortunately there are no direct Fixes mentioned, but I've tracked further related changes to sgio_get_vpd and get_vpd_sgio:

Related but not needed:
3a309966 libmultipath: "fix sgio_get_vpd looping"
  fixes a different issue, not related to the SRU, is in 8.4 (>=Groovy)
d1571b3f "libmultipath: add code to get vendor specific vpd data":
3e60ab1c "libmultipath: add vend_id to get_vpd_sgio"
 feature add, not related to the SRU, is in 8.4 (>=Groovy)

Related and needed:
2c562423 "libmultipath: decrease log level for failed VPD c9"
  avoids log burst due to the change we intend to SRU
  Already in >=Focal
0f5e45b0 "libmultipath: fix length issues in get_vpd_sgio"
  fix buffer overwrite in the change we SRU
  Already in >=Focal

...
Then the changes we already integrated follow.

[Christian Ehrhardt  (paelzer)]

Improved upload tested from PPA [1] as ~ppa2 version.
Then I uploaded the improved version to -unapproved for re-review.
Again - thanks @Robie for the catch.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4282/+packages

[Christian Ehrhardt  (paelzer)]

While clearing out things for the new year I found the old initial upload rejected (as expected) in [1] but my re-upload seems to have been lost (I can't find it anywhere).
So I re-uploaded it to bionic-unapproved again.
I double checked - and at least right now it is in [2] as it should be.

[1]: https://launchpad.net/ubuntu/bionic/+queue?queue_state=4&queue_text=multipath-tools
[2]: https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text=multipath-tools

[Robie Basak (racb)]

Hello Fernando, or anyone else affected,

Accepted multipath-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.7.4-2ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

The fix for this bug has been awaiting testing feedback in the -proposed repository for bionic for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

[Fernando Ferraz (fernando-ferraz)]

Hi folks, I will test that again using SolidFire hardware and get back soon with the results. We don't a have E-series storage to test but my understanding is that isn't the goal of the fix anyway.

[Fernando Ferraz (fernando-ferraz)]

Folks would like to suggest two changes to the description:

1)

 * Users of a NetApp E-Series RDAC due to very long names increasing
   the VDP page and and cause multipathd to hang

The above is not exactly true, in fact any target (can be any storage vendor) with long names may hit if select_checker is enabled.

2)

[Test Case]

 * #1 Case: Test on a NetApp E-Series RDAC, the reporter has such HW and
   is able to do so.

Reporter has only SolidFire storage to test with. Would like to know for you if the test output with a SolidFire device (will hit the long names issue) is actually enough to validate the issue.

[Lucas Kanashiro (lucaskanashiro)]

@fernando-ferraz I updated the bug description to try to make it more general.

For the verification work needed by the SRU process you need to trigger the bug, update the package to the version in -proposed, and then demonstrate that the bug was fixed. If you can do this using your SolidFire device that would be enough, just make sure you can trigger the issue with this new hardware. TIA for your work!

[Fábio Oliveira (fabiooliveira1)]

Hi

@fernando-ferraz asked me to proceed with the tests.

I used this /etc/multipath.conf file:
devices {
  device {
    vendor "SolidFir"
    product "SSD SAN"
    detect_checker "yes"
  }
}
blacklist {
  device {
    vendor "SolidFir"
   product "SSD SAN"
  }
}

Without the correction patch I got the same error.

Then I installed the patch 0.7.4 (proposed enabled).

My steps were:
1. Set multipath config
2. Stop multipath-tools service
3. Create a solidfire volume
4. Attach volume to a existent instance
5. Start multipath-tools service
6. run sudo multipath -v6 -ll

Output:
Jun 24 12:11:56 | loading //lib/multipath/libchecktur.so checker
Jun 24 12:11:56 | loading //lib/multipath/libprioconst.so prioritizer
Jun 24 12:11:56 | Discover device /sys/devices/pci0000:00/0000:00:06.0/virtio3/block/vda
Jun 24 12:11:56 | vda: dev not found in pathvec
Jun 24 12:11:56 | 252:0: dev_t not found in pathvec
Jun 24 12:11:56 | vda: blacklisted, udev property missing
Jun 24 12:11:56 | Discover device /sys/devices/pci0000:00/0000:00:06.0/virtio3/block/vda/vda1
Jun 24 12:11:56 | Discover device /sys/devices/pci0000:00/0000:00:06.0/virtio3/block/vda/vda14
Jun 24 12:11:56 | Discover device /sys/devices/pci0000:00/0000:00:06.0/virtio3/block/vda/vda15
Jun 24 12:11:56 | Discover device /sys/devices/platform/host2/session3/target2:0:0/2:0:0:0/block/sda
Jun 24 12:11:56 | sda: dev not found in pathvec
Jun 24 12:11:56 | 8:0: dev_t not found in pathvec
Jun 24 12:11:56 | sda: udev property ID_WWN whitelisted
Jun 24 12:11:56 | sda: mask = 0x7
Jun 24 12:11:56 | sda: dev_t = 8:0
Jun 24 12:11:56 | open '/sys/devices/platform/host2/session3/target2:0:0/2:0:0:0/block/sda/size'
Jun 24 12:11:56 | sda: size = 2097152
Jun 24 12:11:56 | sda: vendor = SolidFir
Jun 24 12:11:56 | sda: product = SSD SAN
Jun 24 12:11:56 | sda: rev = 0001
Jun 24 12:11:56 | sda: h:b:t:l = 2:0:0:0
Jun 24 12:11:56 | sda: tgt_node_name = iqn.2010-01.com.solidfire:xpsi.uuid-9a54f710-4104-4bd8-a53d-59df79a9ac1e.8441
Jun 24 12:11:56 | open '/sys/devices/platform/host2/session3/target2:0:0/2:0:0:0/state'
Jun 24 12:11:56 | sda: path state = running
Jun 24 12:11:56 | sda: 1011 cyl, 34 heads, 61 sectors/track, start at 0
Jun 24 12:11:56 | open '/sys/devices/platform/host2/session3/target2:0:0/2:0:0:0/vpd_pg80'
Jun 24 12:11:56 | sda: serial = 78707369000020f9f47acc0100000000
Jun 24 12:11:56 | sda: get_state
Jun 24 12:11:56 | sda: detect_checker = yes (setting: storage device configuration)
Jun 24 12:11:56 | failed to issue vpd inquiry for pgc9
Jun 24 12:11:56 | sda: path_checker = tur (setting: multipath internal)
Jun 24 12:11:56 | sda: checker timeout = 30 s (setting: multipath internal)
Jun 24 12:11:56 | sda: tur state = up
Jun 24 12:11:56 | Discover device /sys/devices/virtual/block/loop0
Jun 24 12:11:56 | loop0: dev not found in pathvec
Jun 24 12:11:56 | 7:0: dev_t not found in pathvec
Jun 24 12:11:56 | loop0: blacklisted, udev property missing
Jun 24 12:11:56 | Discover device /sys/devices/virtual/block/loop1
Jun 24 12:11:56 | loop1: dev not found in pathvec
Jun 24 12:11:56 | 7:1: dev_t not found in pathvec
Jun 24 12:11:56 | loop1: blacklisted, udev property missing
Jun 24 12:11:56 | Discover device /sys/devices...

[Launchpad Janitor (janitor)]

This bug was fixed in the package multipath-tools - 0.7.4-2ubuntu3.1

---------------
multipath-tools (0.7.4-2ubuntu3.1) bionic; urgency=medium

  * d/p/lp-1891202-*: fix handling of sgio_get_vpd for e.g. long iscsi target
    names as generated by NetApp E-Series RDAC (LP: #1891202)

 -- Christian Ehrhardt <email address hidden> Mon, 28 Sep 2020 16:14:24 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for multipath-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.