MAAS can't deploy Ubuntu if ID_SERIAL of any block device is broken (USB pendrive in this case).

[Patricia Domingues]

[Impact]

 * sg3-utils-udev package introduces 55-scsi-sg3_id.rules udev rule that take places BEFORE rules either finding correct ID_SERIAL for the USB block device, or calculating it from "usb_id" parameter:

    - this rule changes USB ID_SERIAL parameter for the ones given by the Vital Product Data from the block device.

    - USB memory sticks might not provide VPD pages (0x80, 0x83) and, because of that, the USB ID_SERIAL will be broken.

    - because the failure in sg_inq command from the 55-XXX udev rules file, to obtain the VPDs, the udev ACTION(add) takes long time and consumers might not have considered a "settle" need.

 * Anything depending on ID_SERIAL udev attribute from USB block devices might not find appropriate device.

 * Any type of udev consumer not calculating the need for a big time-consuming "udevadm settle" right after a pen drive is inserted
might not be able to see all USB block devices attributes from udev.

[Test Case]

 * Install package "sg3-utils-udev", monitor journalctl -f while introducing a memory stick in your machine and watch sg_inq command failing:

systemd-udevd[1021]: Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg83 --raw' failed with exit code 15.

 * Compare udevadm attributes with and without "sg3-utils-udev" package is installed:

Without sg3-utils 55-scsi-sg3_id.rules:

$ udevadm info --query=all --name=/dev/sdb | grep -i serial
E: ID_SERIAL=Corsair_Voyager_Mini_3.0_070851D0E490C776-0:0
E: ID_SERIAL_SHORT=070851D0E490C776

With 55-scsi-sg3_id.rules:

$ udevadm info --query=all --name=/dev/sdb | grep -i serial
E: ID_SERIAL=3
E: ID_SERIAL_SHORT=2000acde48234567

[Regression Potential]

 * Minor: either I brake functionality for sg3-utils packages on USB block devices. Since sg3-utils is used mostly for SCSI debugging, management and investigations, its a very minor risk.

 * The change is a 1 line change in a udev rule file, very easy to test following the test case.

[Other Info]

ORIGINAL DESCRIPTION:

* MAAS version: 2.4.2 (7034-g2f5deb8b8-0ubuntu1)

MAAS UI changes:

```
Node changed status - From 'Deploying' to 'Failed deployment'
Node installation failure - 'curtin' curtin command install
```

when console_log shows:

```

[ 244.588201] cloud-init[3139]: File "/curtin/curtin/commands/block_meta.py", line 1515, in get_path_if_present
[ 244.588389] cloud-init[3139]: return get_path_to_storage_volume(disk, config)
[ 244.588588] cloud-init[3139]: File "/curtin/curtin/commands/block_meta.py", line 433, in get_path_to_storage_volume
[ 244.588774] cloud-init[3139]: specified to identify disk")
[ 244.588963] cloud-init[3139]: ValueError: serial, wwn or path to block dev must be specified to identify disk
[ 244.589153] cloud-init[3139]: serial, wwn or path to block dev must be specified to identify disk
[ 244.589338] cloud-init[3139]:
[ 244.589523] cloud-init[3139]: Stderr: ''
[ 244.589708] cloud-init[3139]: Unexpected error while running command.
[ 244.589905] cloud-init[3139]: Command: ['curtin', 'block-meta', 'custom']
[ 244.590088] cloud-init[3139]: Exit code: 3
[ 244.590271] cloud-init[3139]: Reason: -
[ 244.590459] cloud-init[3139]: Stdout: start: cmd-install/stage-partitioning/builtin/cmd-block-meta: curtin command block-meta
```

Reproducible: yes

aarch64, arm64 machine:
Manufacturer: CN8890-2000BG2601-CP-Y-G
Part Number: CN88xx

Cavium SOC
=========================
Cavium THUNDERX Boot Stub
=========================
BDK Version: 04.02.14, 2.38 Branch: gigabyte-2.38, Built: Fri Aug 18 01:16:58 UTC 2017

Board Model: gbt-mt30
Board name: MT30-GS1-00
System name: R120-T33-00
Board Revision: 1.x
Board SKU: 04 SATA
SKU version: 1.1

Node: 0 (Fixed)
Chip: 0xa1 Pass 2.1
SKU: CN8890-2000BG2601-CP-Y-G

*MAAS get-curtin-config: https://paste.ubuntu.com/p/k54HHN55ZF/

[Ryan Harper (raharper)]

 - id: sdc
    model: USB Flash Drive
    name: sdc
    serial: '1510615541270083'
    type: disk
    wipe: superblock

Curtin will look for a symlink in /dev/disk/by-id/*1510615541270083* and it doesn't appear to be present during the deployment.

I wonder if the USB device is now no longer present in the system when you deployed?

If so, you should recommission the node so update MAAS's view of storage configuration and it will not send configuration for that disk if it's not present in the system.

[Patricia Domingues (patriciasd)]

It has a USB stick plugged in (same serial as config shows `serial: '1510615541270083' `):
```
$ udevadm info --query=all --name=/dev/sdd | grep -i serial
E: ID_SERIAL=ADATA_USB_Flash_Drive_1510615541270083-0:0
E: ID_SERIAL_SHORT=1510615541270083
```

-anyway, I've recommissioned the machine and the error persists.

```
Node changed status - From 'Deploying' to 'Failed deployment' Fri, 21 Jun. 2019 23:51:25
Node installation failure - 'curtin' curtin command install Fri, 21 Jun. 2019 23:51:25
Node changed status - From 'Allocated' to 'Deploying' Fri, 21 Jun. 2019 23:44:42
Node changed status - From 'Commissioning' to 'Ready' Fri, 21 Jun. 2019 23:32:08
Node changed status - From 'Ready' to 'Commissioning' Fri, 21 Jun. 2019 23:25:17
User starting node commissioning - (patriciasd) Fri, 21 Jun. 2019 23:25:16
```

- adding new log, after the machine was re-commissioned:
*MAAS get-curtin-config: https://paste.ubuntu.com/p/CycNkPGGXh/

[Andrew Cloke (andrew-cloke)]

One observation that I don't know is relevant, but originally the USB drive was showing up as sdc, but now (after recommissioning) appears to be sdd.

[Ryan Harper (raharper)]

Thanks for checking out the recomissioning.

If possible, could you capture all output and attach from:

1. ls -al /dev/disk/by-id/*
2. udevadm info --query=all --name=/dev/<usb disk>

Thanks

[Patricia Domingues (patriciasd)]

Sure, attaching it.

[Ryan Harper (raharper)]

I'm pretty stumped. The output you provided suggests that we found the device.

curtin.block.lookup_disk() will glob for the serial, which returns

'usb-ADATA_USB_Flash_Drive_1510615541270083-0:0'

And then we

os.path.realpath('/dev/disk/by-id/usb-ADATA_USB_Flash_Drive_1510615541270083-0:0')

And then log the link and realpath. (which we don't see in the logs)

We check udevinfo to see if it's a multipath device via checking if 'DM_UUID' is present
in the info (it's not).

Verify that the realpath exists, if not raise ValueError.

We should have returned the path to block_meta, somewhere between the call into lookup_disk
and block-meta we're getting a none-ish value.

[Andrew Cloke (andrew-cloke)]

@raharper we can get you access to the machine (if you don't have it already) if that would help.

[Ryan Harper (raharper)]

On Wed, Jun 26, 2019 at 11:16 AM Andrew Cloke <email address hidden>
wrote:

> @raharper we can get you access to the machine (if you don't have it
> already) if that would help.
>

Sure. lp:raharper for ssh keys.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1833618
>
> Title:
> failing to deploy Ubuntu Disco
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions
>

[Patricia Domingues (patriciasd)]

Ok, I've added your keys, `ssh ubuntu@10.229.71.18`s

[Ryan Harper (raharper)]

On Wed, Jun 26, 2019 at 2:30 PM Patricia Domingues <
<email address hidden>> wrote:

> Ok, I've added your keys, `ssh ubuntu@10.229.71.18`s
>

OK. I'm on. It's currently deployed with bionic via maas and curtin, so
something is slightly different with disco. That feels kernel related to
me.\

I;m going to see if I can manually install disco while deployed under
18.04. Do you have the rsyslog from the host that's failing deployment? I
wonder if there might be a clue in those logs.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1833618
>
> Title:
> failing to deploy Ubuntu Disco
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions
>

[dann frazier (dannf)]

On Tue, Jul 2, 2019 at 9:00 AM Ryan Harper <email address hidden> wrote:
>
> On Wed, Jun 26, 2019 at 2:30 PM Patricia Domingues <
> <email address hidden>> wrote:
>
> > Ok, I've added your keys, `ssh ubuntu@10.229.71.18`s
> >
>
> OK. I'm on. It's currently deployed with bionic via maas and curtin, so
> something is slightly different with disco. That feels kernel related to
> me.\

Patricia: Do we know if 18.04 + hwe-edge deploys OK on the systems
that fail to deploy disco?

  -dann

[Patricia Domingues (patriciasd)]

Dann,
Yes, no issue to deploy it:
```
Linux doerfel 5.0.0-20-generic #21~18.04.1-Ubuntu SMP Thu Jun 27 04:05:19 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@doerfel:~$ lsb_release -d
Description: Ubuntu 18.04.2 LTS
```

[Patricia Domingues (patriciasd)]

Ryan, sorry I didn't notice you've asked the rsyslog - I don't have it.
Do you still need to use the system?

[Ryan Harper (raharper)]

I don;t have a direct need. However, capturing the syslogs for each variation where things work and where they don't should help us narrow down where the trouble comes from.

[Patricia Domingues (patriciasd)]

Adding syslog when it fails with disco deployment

[Patricia Domingues (patriciasd)]

Adding syslog for the same system as bug description and Comment#16 when Bionic is successfully deployed:
```
Ubuntu 18.04.3 LTS, 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:58 UTC 2019 aarch64
```

[Ryan Harper (raharper)]

On Fri, Aug 30, 2019 at 10:06 AM Patricia Domingues <
<email address hidden>> wrote:

> Adding syslog when it fails with disco deployment
>
> ** Attachment added: "syslog_failed_disco"
>
> https://bugs.launchpad.net/curtin/+bug/1833618/+attachment/5285802/+files/syslog_failed_disco

Curtin never runs in this one... also networking looks flaky.

>
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1833618
>
> Title:
> failing to deploy Ubuntu Disco
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions
>

[Patricia Domingues (patriciasd)]

Right. sorry. it had an BMC issue, I've fixed it. Attaching the correct log now

[Ryan Harper (raharper)]

Thanks, diffing those, I can see on disco, we see this kernel/udev message.

% grep sdd syslog_failed_disco.txt
Aug 30 20:33:56 seidel-FLAKYMEMORY systemd-udevd[1021]: Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdd/device/vpd_pg80 --raw' failed with exit code 15.
Aug 30 20:33:56 seidel-FLAKYMEMORY systemd-udevd[1021]: Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdd/device/vpd_pg83 --raw' failed with exit code 15.
Aug 30 20:33:56 seidel-FLAKYMEMORY multipath: sdd: can't store path info
Aug 30 20:33:56 seidel-FLAKYMEMORY kernel: [ 11.221535] sd 0:0:0:0: [sdd] 30344192 512-byte logical blocks: (15.5 GB/14.5 GiB)
Aug 30 20:33:56 seidel-FLAKYMEMORY kernel: [ 11.222236] sd 0:0:0:0: [sdd] Write Protect is off
Aug 30 20:33:56 seidel-FLAKYMEMORY kernel: [ 11.222239] sd 0:0:0:0: [sdd] Mode Sense: 23 00 00 00
Aug 30 20:33:56 seidel-FLAKYMEMORY kernel: [ 11.222908] sd 0:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Aug 30 20:33:56 seidel-FLAKYMEMORY kernel: [ 11.232297] sd 0:0:0:0: [sdd] Attached SCSI removable disk
Aug 30 20:34:48 seidel-FLAKYMEMORY systemd-udevd[2208]: Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdd/device/vpd_pg80 --raw' failed with exit code 15.
Aug 30 20:34:48 seidel-FLAKYMEMORY systemd-udevd[2208]: Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdd/device/vpd_pg83 --raw' failed with exit code 15.
Aug 30 20:34:48 seidel-FLAKYMEMORY multipath: sdd: can't store path info
Aug 30 20:34:48 seidel-FLAKYMEMORY systemd-udevd[2208]: Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdd/device/vpd_pg80 --raw' failed with exit code 15.
Aug 30 20:34:48 seidel-FLAKYMEMORY systemd-udevd[2208]: Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdd/device/vpd_pg83 --raw' failed with exit code 15.
Aug 30 20:34:48 seidel-FLAKYMEMORY multipath: sdd: can't store path info
Aug 30 20:34:49 seidel-FLAKYMEMORY multipathd[1259]: sdd: spurious uevent, path not found
Aug 30 20:34:49 seidel-FLAKYMEMORY multipathd[1259]: sdd: spurious uevent, path not found

And I can see that is is now related to _multipath_ package being present in the disco images, but not in bionic...

Let's add a multipath task here; somehow due to the multipath package which adds new udev rules, it runs different commands to query the ID_SERIAL value from devices...

[Ryan Harper (raharper)]

Possibly related:

https://bugzilla.redhat.com/show_bug.cgi?id=1380272
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1468897

[Ryan Harper (raharper)]

It's actually the sg3-utils-udev package which provides new udev rules to inquery about scsi device's ID_SERIAL.

[Patricia Domingues (patriciasd)]

I think it worth mentioning, as the machine above that we are not able to deploy Disco is a Gigabyte(gbt-mt30)Cavium ThunderX 88XX, I've decided to test Disco in other machine, not Gigabyte, but Cavium ThunderX 88XX(CRB-2s) and I've noticed that in one machine Disco is successfully deployed and in another it fails. The only difference between these machines are the disks:

[Patricia Domingues (patriciasd)]

- it is deployed in a machine where it has just one disk. (machine nominated as wright)
curtin: curtin_wright_success_disco https://paste.ubuntu.com/p/7CmYw57Rn6/

[Patricia Domingues (patriciasd)]

- it fails in a machine where it has more than one disk - in this case an USB stick. (machine nominated as recht)
curtin: curtin_recht_failed_disco https://paste.ubuntu.com/p/NVkPnxVDn5/

[Andrew Cloke (andrew-cloke)]

@Ryan, do you know if the Server team looks after sg3-utils-udev, or should we be chasing the Foundations team to look into this?
Thanks.

[Ryan Harper (raharper)]

On Tue, Sep 3, 2019 at 9:31 AM Andrew Cloke <email address hidden>
wrote:

> @Ryan, do you know if the Server team looks after sg3-utils-udev, or
> should we be chasing the Foundations team to look into this?
>

Not quite sure, I'll follow up.

Also, Patricia, can you try an Eoan install to see if this is maybe fix in
newer release?

Ryan

> Thanks.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1833618
>
> Title:
> failing to deploy Ubuntu Disco
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions
>

[Patricia Domingues (patriciasd)]

Ryan,
Sorry for delay, we had a BMC issue and it required a manual intervention to fix it.
I've tested Eoan in that machine and it also fails, it doens't show the same curtin error as Disco shows, but Eoan is not deployed.
It get stuck and MAAS shows `Marking node failed - Node operation 'Deploying' timed out after 40 minutes.`
Attaching the console log for it (machine nominated as seidel).
Let me know if I can provide any other info.

[Ryan Harper (raharper)]

On Thu, Sep 12, 2019 at 5:00 PM Patricia Domingues <
<email address hidden>> wrote:

> Ryan,
> Sorry for delay, we had a BMC issue and it required a manual intervention
> to fix it.
> I've tested Eoan in that machine and it also fails, it doens't show the
> same curtin error as Disco shows, but Eoan is not deployed.
> It get stuck and MAAS shows `Marking node failed - Node operation
> 'Deploying' timed out after 40 minutes.`
> Attaching the console log for it (machine nominated as seidel).
> Let me know if I can provide any other info.
>
> ** Attachment added: "eoan_console_seidel"
>
> https://bugs.launchpad.net/curtin/+bug/1833618/+attachment/5288410/+files/eoan_console_seidel

Thanks. I expected it would fail as well. We'll need to dig into the
sg-utils udev rules.

Ryan

[Andrew Cloke (andrew-cloke)]

@Ryan, do you know who owns sg-utils? Is there any further information or system access that we can provide?
Many thanks, Andy.

[Andrew Cloke (andrew-cloke)]

MAAS deployments of disco to Marvell "Sabre" boards (CN99XX SoC) with more than one disk also fails.
Internal node:
* fails on "starbuck" with 2 disks
* passes on "hotdog" and "apollo" server that both only have 1 disk

[Rafael David Tinoco (rafaeldtinoco)]

@Patricia,

I'll concentrate efforts into the sg3-utils issue with disco for now. For Eoan, based on the console output you attached to the bug, it seems that one of the services listed as "<email address hidden>" dependencies did not start before timeout occurred. You can check systemd output after the failure to know which one was the one to blame.

In a short list, the dependencies for getty on ttyAMA0 would be:

<email address hidden>
● ├─system-getty.slice
● └─sysinit.target
● ├─blk-availability.service
● ├─dev-hugepages.mount
● ├─dev-mqueue.mount
● ├─grub-initrd-fallback.service
● ├─keyboard-setup.service
● ├─kmod-static-nodes.service
● ├─proc-sys-fs-binfmt_misc.automount
● ├─resolvconf.service
● ├─setvtrgb.service
● ├─sys-fs-fuse-connections.mount
● ├─sys-kernel-config.mount
● ├─sys-kernel-debug.mount
● ├─systemd-ask-password-console.path
● ├─systemd-binfmt.service
● ├─systemd-hwdb-update.service
● ├─systemd-journal-flush.service
● ├─systemd-journald.service
● ├─systemd-machine-id-commit.service
● ├─systemd-modules-load.service
● ├─systemd-random-seed.service
● ├─systemd-sysctl.service
● ├─systemd-sysusers.service
● ├─systemd-tmpfiles-setup-dev.service
● ├─systemd-tmpfiles-setup.service
● ├─systemd-udev-trigger.service
● ├─systemd-udevd.service
● ├─systemd-update-utmp.service
● ├─cryptsetup.target
● ├─local-fs.target
● │ ├─-.mount
● │ ├─boot-efi.mount
● │ ├─systemd-fsck-root.service
● │ └─systemd-remount-fs.service
● └─swap.target

In a recursive list, you can see it as:

https://pastebin.ubuntu.com/p/qx27jrJwFh/

So, for now, concentrating efforts in the following output:

"""
start: cmd-install/stage-partitioning/builtin/cmd-block-meta: curtin command block-meta
get_path_to_storage_volume for volume sdb
Processing serial 1510611512340057 via udev to 1510611512340057
TIMED BLOCK_META: 0.001
finish: cmd-install/stage-partitioning/builtin/cmd-block-meta: FAIL: curtin command block-meta

Traceback (most recent call last):
  File "/curtin/curtin/commands/main.py", line 202, in main
    ret = args.func(args)
  File "/curtin/curtin/log.py", line 97, in wrapper
    return log_time("TIMED %s: " % msg, func, *args, **kwargs)
  File "/curtin/curtin/log.py", line 79, in log_time
    return func(*args, **kwargs)
  File "/curtin/curtin/commands/block_meta.py", line 71, in block_meta
    extract_storage_ordered_dict(cfg))
  File "/curtin/curtin/commands/block_meta.py", line 1518, in get_disk_paths_from_storage_config
    for (k, v) in storage_config.items()
  File "/curtin/curtin/commands/block_meta.py", line 1521, in <listcomp>
    not config.value_as_boolean(v.get('preserve'))]
  File "/curtin/curtin/commands/block_meta.py", line 1515, in get_path_if_present
    return get_path_to_storage_volume(disk, config)
  File "/curtin/curtin/commands/block_meta.py", line 433, in get_path_to_storage_volume
    specified to identify disk")

ValueError: serial, wwn or path to block dev must be specified to identify disk
serial, wwn or path to block dev must be specified to identify disk
builtin command failed
"""

For this bug.

[Ryan Harper (raharper)]

@Rafael,

the critical path would be to check the output from:

udevadm info --query=all --name=/dev/sd[cd] | grep -i serial

ID_SERIAL does not get set on Disco where we have sg3-utils package install, which adds the s3g-utils-udev package, and includes the rules file 55-scsi-sg3_id.rules.

This udev rule uses sg_inq --export /path/to/device to query for device properties.

I suspect you can reproduce the failure on the device by running

sg_inq --export on the usb block device.

Note that in 55-scsi-sg3 rules, if no ID_SERIAL is found via the previous import, then the rules construct an ID_SERIAL from various scsi id values;

This is the core issue since normally 60-persistent-storage.rules would use 'ata_id', 'usb_id' and 'scsi_id' to extract the ID_SERIAL value, but since 55 rules run before 60, the sg3-utils-udev package sets ID_SERIAL, and prevents the 60-persistent-storage.rules from setting the value to match.

[Rafael David Tinoco (rafaeldtinoco)]

@Rhyan, thanks for the quick pointers, helped a lot =), I think I found out what is going on.

Linux mylab 5.0.0-30-generic #32-Ubuntu SMP Wed Sep 18 00:24:43 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

Using a memstick in an aarch64 server:

$ sudo sg_inq -d /dev/sdb

standard INQUIRY:
  PQual=0 Device_type=0 RMB=1 LU_CONG=0 version=0x06 [SPC-4]
  [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2
  SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 [BQue=0]
  EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0
  [RelAdr=0] WBus16=0 Sync=0 [Linked=0] [TranDis=0] CmdQue=0
    length=36 (0x24) Peripheral device type: disk
 Vendor identification: Corsair
 Product identification: Voyager Mini 3.0
 Product revision level: 000B
 Unit serial number: 0C7707865010

command sg_inq returns right away with proper info and, with 60-persistent-storage.rules only:

$ sudo udevadm info --query=all --name=/dev/sdb | grep -i serial
E: ID_SERIAL=Corsair_Voyager_Mini_3.0_070851D0E490C776-0:0
E: ID_SERIAL_SHORT=070851D0E490C776

--------

$ sudo sg_inq -d /dev/sdb

<hangs for quite sometime>

$ sudo udevadm info --query=all --name=/dev/sdb | grep -i serial

<no serial is given>

AFTER sg_inq command returns (10 seconds or more) AND with 55-scsi-sg3.rules installed, then we get:

$ sudo udevadm info --query=all --name=/dev/sdb | grep -i serial
E: ID_SERIAL=3
E: ID_SERIAL_SHORT=2000acde48234567

Which means that its likely that curtin did not wait enough (or settled) for the device VPD to "arrive". I haven't found out why the "USB" ("SCSI") vpd gets so long to be returned when sg-utils-udev is installed, will investigate.

[Ryan Harper (raharper)]

The _real_ concern is:

do you get the _same_ ID_SERIAL value with and without the sg3-utils-udev package installed?

Can you compare: /dev/disk/by-id/*<memstick serial*

Which is what curtin is doing; it's been told a disk has a serial number, curtin looks for this out in /dev/disk/by-id/*serial* to find the mapping; the serial number isn't present.

Looking at comment #20 the disco syslog, you can see that the 55 rules are running commands which don't completely work (returns errors):

Aug 30 20:33:56 seidel-FLAKYMEMORY systemd-udevd[1021]: Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdd/device/vpd_pg80 --raw' failed with exit code 15.
Aug 30 20:33:56 seidel-FLAKYMEMORY kernel: [ 0.000000] Memory: 65690396K/67088384K available (11964K kernel code, 1734K rwdata, 5100K rodata, 5568K init, 1160K bss, 1365220K reserved, 32768K cma-reserved)
Aug 30 20:33:56 seidel-FLAKYMEMORY kernel: [ 0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=48, Nodes=1
Aug 30 20:33:56 seidel-FLAKYMEMORY kernel: [ 0.000000] ftrace: allocating 42707 entries in 167 pages
Aug 30 20:33:56 seidel-FLAKYMEMORY kernel: [ 0.000000] rcu: Hierarchical RCU implementation.
Aug 30 20:33:56 seidel-FLAKYMEMORY kernel: [ 0.000000] rcu: #011RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=48.
Aug 30 20:33:56 seidel-FLAKYMEMORY kernel: [ 0.000000] #011Tasks RCU enabled.
Aug 30 20:33:56 seidel-FLAKYMEMORY kernel: [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
Aug 30 20:33:56 seidel-FLAKYMEMORY systemd-udevd[1021]: Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdd/device/vpd_pg83 --raw' failed with exit code 15.
Aug 30 20:33:56 seidel-FLAKYMEMORY systemd[1]: Started Flush Journal to Persistent Storage.
Aug 30 20:33:56 seidel-FLAKYMEMORY multipath: sdd: can't store path info
Aug 30 20:33:56 seidel-FLAKYMEMORY systemd[1]: Started udev Wait for Complete Device Initialization.

And you can see the multipath rules coming in to play as well.

[Rafael David Tinoco (rafaeldtinoco)]

This also happens in amd64 (and likely other architectures), only when having "sg3-utils-udev" package installed.

Command "udevadm settle" only returns after the initial "sg_inq" command also returns, raising the question:

(1) Would call "udevmadm settle" before "udevadm info" be enough to get A USB device serial ? (not saying it would be the same serial as it would have been without sg3-utils-udev rules in place).

Another question:

(2) Is this device initialization taking long because of bad "sg3-utils-udev" rules being executed before the initial I/O commands can be submitted to the device ? (very likely).

Stack shows:

[<0>] blk_execute_rq+0x75/0xb0
[<0>] sg_io+0x176/0x460
[<0>] scsi_cmd_ioctl+0x1b9/0x3f0
[<0>] scsi_cmd_blk_ioctl+0x51/0x61
[<0>] sd_ioctl+0xcd/0x1c0
[<0>] blkdev_ioctl+0x4c1/0x9f0
[<0>] block_ioctl+0x3d/0x50
[<0>] do_vfs_ioctl+0xa9/0x640
[<0>] ksys_ioctl+0x67/0x90
[<0>] __x64_sys_ioctl+0x1a/0x20
[<0>] do_syscall_64+0x5a/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] 0xffffffffffffffff

we are blocked waiting for the VPD (as it was sent through sd_ioctl command to block device layer). I'll trace kernel to check why blk_execute_rq() is taking so long when sg3-utils-udev package is installed (likely command might still be held because device it not yet initialized, due to bad udev rules, "to check").

--

Regarding your concern: YES, it IS A different serial, AND... sometimes, udevadm returns no serial at all (until "udev settle" returns, and this behavior only happens when sg3-utils-udev rules are in place and during the first sg_inq command, suggesting the device is initializing with the bad udev rules).

I'll "bisect" the rules to find the guilty one(s), and, yes, this might be orthogonal to the issue you need, if you need "same serials" in place. For that, you would need a higher priority rule setting SCSI SERIAL based on anything you want (since SCSI SERIAL is not present in USB devices, not the VITAL PRODUCT PAGE, like regular SCSI devices). Likely, the correct thing to do here is to keep the USB serial as the "emulated" (by udev rules) SCSI SERIAL.

[Rafael David Tinoco (rafaeldtinoco)]

BTW, you got:

       15 the utility is unable to open, close or use the given DEVICE or some other file. The given file name could be incorrect or there may be permission problems. Adding the '-v' option may give more information.

And this is likely because USB devices don't have VPD, so sysfs is not showing "/sys/block/XXX/device/vpd_pg83":

From SCSI SPC-3 standard:
"""
Device Identification VPD page (page number: 0x83). Since SPC-3, support for this page has been flagged as mandatory. This page can be fetched by using the --ident option.

The reference document used for interpreting VPD pages (and the INQUIRY standard response) is T10/1713-D Revision 36e (SPC-4, 24 August 2012) found at http://www.t10.org . Obsolete and reserved items in the standard INQUIRY response output are displayed in square brackets.
"""

BUT it only serves for SCSI devices for RAW DEVICE MAPPING technique.

[Rafael David Tinoco (rafaeldtinoco)]

Well, checking udev rules, its exactly what you said.

I guess we could change sg3 udev rules not to execute sg_inq --export --inhex.... to /sys/block/$kernel/device/vpd_pg83 for SCSI devices and that would be "enough" for this issue.

    --inhex=FN|-I FN read ASCII hex from file FN instead of DEVICE;
                        if used with --raw then read binary from FN

Looks like sg_inq gets the VPD 0×83 page directly from in-cache sysfs kernel file - from the initial device initialization made by kernel - instead of submitting an I/O to the device like this:

$ sudo sg_vpd /dev/sdc
Supported VPD pages VPD page:
invalid VPD response; probably a STANDARD INQUIRY response
fetching VPD page failed: Malformed response

which would also give us an error.

[Rafael David Tinoco (rafaeldtinoco)]

I meant "for USB devices and that would be "enough"" in previous comment.

[Rafael David Tinoco (rafaeldtinoco)]

Alright, I'll change the udev rules from sg3-utils-udev package to ignore USB devices. From the initial ACTION (add) event, I only get simple data from udev:

S: disk/by-id/usb-Corsair_Voyager_Mini_3.0_070851D0E490C776-0:0
S: disk/by-path/pci-0000:00:14.0-usb-0:2:1.0-scsi-0:0:0:0
S: disk/by-label/Ubuntu\x2019.04\x20amd64
S: disk/by-uuid/2019-04-16-19-19-59-00
E: DEVPATH=/devices/pci0000:00/0000:00:14.0/usb2/2-2/2-2:1.0/host2/target2:0:0/2:0:0:0/block/sdc
E: SUBSYSTEM=block
E: DEVNAME=/dev/sdc
E: DEVTYPE=disk
E: MAJOR=8
E: MINOR=32
E: USEC_INITIALIZED=376735479267

Before the device is fully initialized. I will try to ignore DEVPATH containing *usb*, for example, at least for the VPD pages 0x80 and 0x83, I'll get back soon.

[Rafael David Tinoco (rafaeldtinoco)]

Adding:

# Ignore USB block devices as they don't contain VPD 0x80 and 0x83
# and their SERIAL is calculated from usb_id in persistent-storage rules
DEVPATH=="*usb*", GOTO="sg3_utils_id_end"

before

#SCSI EVPD page 0x80 values

in 55-scsi-sg3_id.rules file fixes the issue.

[Rafael David Tinoco (rafaeldtinoco)]

$ rmadison sg3-utils-udev
 sg3-utils-udev | 1.40-0ubuntu1 | xenial | all
 sg3-utils-udev | 1.42-2ubuntu1 | bionic | all
 sg3-utils-udev | 1.42-2ubuntu1 | disco | all
 sg3-utils-udev | 1.44-1ubuntu1 | eoan | all

And:

  [ Mauricio Faria de Oliveira ]
  * sg3-utils-udev (new binary package): provide the udev rules for
    SCSI-ID mappings and symlinks at boot time for multipath-tools:
    - debian/control: add sg3-utils-udev
    - debian/rules: build it with override_dh_{install,clean}
    - debian/initramfs/hooks: add_udev_rules() and copy_exec() 'sg_inq'
      to initramfs
    - debian/sg3-utils-udev.post{inst,rm}: call update-initramfs

Marking this affecting Bionic, Disco and Eoan.
Will also provide a MR do Debian/Salsa.

[Rafael David Tinoco (rafaeldtinoco)]

I have asked upstream to consider the following merge:

https://salsa.debian.org/linux-blocks-team/sg3-utils/merge_requests/2

[Rafael David Tinoco (rafaeldtinoco)]

Summary:

Merge Requests (to be reviewed):

# Debian
MR: https://salsa.debian.org/linux-blocks-team/sg3-utils/merge_requests/2

# Eoan
https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/sg3-utils/+git/sg3-utils/+merge/373438
# Disco
https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/sg3-utils/+git/sg3-utils/+merge/373439
# Bionic
https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/sg3-utils/+git/sg3-utils/+merge/373440

PPA (to be tested: amd64, arm64 and s390x):

https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1833618

I have already asked for server review on merge requests, I would appreciate, if possible that the packages provided in PPA are tested.

Thank you very much!

Rafael D. Tinoco

[Rafael David Tinoco (rafaeldtinoco)]

Note:

I have to re-submit Disco MR and a new PPA package, as the compiler in Disco is likely more recent and treating lots of "new warnings" as errors (-Wimplicit-fallthrough, etc...). I'll do it tomorrow!

[Rafael David Tinoco (rafaeldtinoco)]

Disco PPA package (and MR) is good.

[dann frazier (dannf)]

I MAAS deployed an impacted system w/ the attached preseed to include your PPA, and curtin successfully installed. Afterward, the system entered a deploy loop, but there's no sign that this was related to your fix - rather, it may just be a boot order issue. Interrupting the loop and telling it to boot from disk caused it to successfully reach deployed state.

[dann frazier (dannf)]

@rafaeldtinoco: Also, I wonder if it wouldn't make sense to submit your patch upstream?
  https://github.com/hreinecke/sg3_utils

[Rafael David Tinoco (rafaeldtinoco)]

Hey Dann, nope, the file changed is Debian related only. I'm glad it worked, I'll ask server team review and merge then! Tks for testing it!!

[dann frazier (dannf)]

On Thu, Oct 3, 2019 at 4:42 AM Rafael David Tinoco
<email address hidden> wrote:
>
> Hey Dann, nope, the file changed is Debian related only.

hmm.. I see the file in the upstream source, and Debian hasn't patched
it. But, perhaps you mean that upstream is not impacted. And yeah, I
can see that this may be the case. Upstream appear to have applied a
change post 1.44 to only run sg_inq on known devices. I'll comment
further on that via your MR.

> I'm glad it
> worked, I'll ask server team review and merge then! Tks for testing it!!

np - thanks for working this!

[Rafael David Tinoco (rafaeldtinoco)]

No no, you are definitely right, I totally missed upstream change, I did "on my own", not sure why I thought upstream didn't have any change/fix (too many bugs in my head perhaps). I'll reply you in the MR, but, we should go with whatever upstream has done, so delta is reduced. I might re-do the MRs, even the one for Debian.

[dann frazier (dannf)]

A follow-up on comment #47 - I've proven that the reboot loop had nothing to do with your change, I just missed a required section of the preseed. Updated file attached just as an fyi.

[Rafael David Tinoco (rafaeldtinoco)]

Thanks for confirming that!

On the other hand, patch seems to be good, even for upstream.
From the merge request comment:

"""
ACTUALLY, it made little difference because the new rules:

Oct 04 09:34:46 macbook systemd-udevd[30592]: Process '/usr/bin/sg_inq --export --inhex=/sys/class/scsi_device/2:0:0:0/device/vpd_pg80 --raw' failed with exit code 15.
Oct 04 09:34:46 macbook systemd-udevd[30592]: Process '/usr/bin/sg_inq --export --inhex=/sys/class/scsi_device/2:0:0:0/device/vpd_pg83 --raw' failed with exit code 15.

Still don't consider USB bus exception. The rule you mentioned checks for anything (else) then /dev/sd* or /dev/sr* which is not the case for pendrives, for example. Thus, the need for me to have added:

DEVPATH=="*usb*", GOTO="sg3_utils_id_end"

because:

E: DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1

it the only attribute I could differentiate USB from SCSI BEFORE the actual VPD reading.

I'll suggest my change to upstream now, possibly mentioning a Forwarded item in the patches.
"""

[Rafael David Tinoco (rafaeldtinoco)]

This it the upstream pull request:

https://github.com/hreinecke/sg3_utils/pull/47

And I have updated Debian pull request as well:

https://salsa.debian.org/linux-blocks-team/sg3-utils/merge_requests/2/commits

I'm preparing updates for Eoan, Disco and Bionic fixes.

[Rafael David Tinoco (rafaeldtinoco)]

All good for a final review and Eoan upload (followed by the SRUs then).

[Rafael David Tinoco (rafaeldtinoco)]

16:47 <rafaeldtinoco> sp.. 60-persistent-storage.rules sets the ID_SERIAL for USB devices
16:47 <rafaeldtinoco> it checks first for ATA then ATAPI (with ata_id command)
16:47 <rafaeldtinoco> if it does not work, it falls back to usb_id (from USB device descriptor)
16:48 <rafaeldtinoco> IF you install sg3-utils, it does not check if it is USB or not, it simply tries to put a serial in all SCSI devices (that might be using SCSI only for commands, and not the full SPEC) based on VPDs
16:48 <rafaeldtinoco> so ignoring USB devices for not having VPDs seems ok
16:48 <rafaeldtinoco> as they will be ATA/ATAPI devices (using SCSI commands only)
16:48 <rafaeldtinoco> now, not sure if there is a SCSI device (full implementation) over USB serial
16:49 <rafaeldtinoco> (like SAS disk over USB3 <- not sure in this case)
16:49 <rafaeldtinoco> but it looks broken for those cases either way
16:49 <rafaeldtinoco> as sg3-utils dont make any decision for different USB devices

[Rafael David Tinoco (rafaeldtinoco)]

Alright, I found a much better solution for this:

# SCSI EVPD page 0x80 values
KERNEL=="sd*[!0-9]|sr*", ENV{ID_SCSI}=="1", IMPORT{program}="/usr/bin/sg_inq --export --inhex=/sys/block/$kernel/device/vpd_pg80 --raw", ENV{ID_SCSI_SN}="1"
KERNEL=="sd*[!0-9]|sr*", ENV{ID_SCSI_SN}!="1" GOTO="sg3_utils_id_end"
KERNEL=="sd*[!0-9]|sr*", ENV{ID_SCSI}=="1", ENV{ID_SCSI_SN}!="1", IMPORT{program}="/usr/bin/sg_inq --export --page=sn $tempnode", ENV{ID_SCSI_SN}="1"

I fail the USB SCSI device only if it does not contain VPD_PG80.. and then it keeps its serial as it won't support SCSI commands anyway. This is better because it keeps compatibility with SCSI protocol, for USB drives supporting not only SCSI commands but also SCSI protocol itself:

Example for a memory stick (only SCSI commands):

$ sudo sg_inq --export --inhex=/sys/block/sdc/device/vpd_pg80 --raw
unable to open binary file /sys/block/sdc/device/vpd_pg80: No such file or directory

This will make udev to stop all other rules (and stop udev for running another INQ directly to the device, which would fail).

Now, for a device that supports SCSI protocol:

$ sudo sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg80 --raw
SCSI_IDENT_SERIAL=NA7EMXLG

then udev will keep running all other udev rules (and get a new SCSI_SERIAL_SN, for example).

[Rafael David Tinoco (rafaeldtinoco)]

Output from udevtest rules in a USB drive supporting VPDs and SCSI protocol:

----
Starting '/usr/bin/sg_inq --export /dev/sdb'
'/usr/bin/sg_inq --export /dev/sdb'(out) 'SCSI_TPGS=0'
'/usr/bin/sg_inq --export /dev/sdb'(out) 'SCSI_TYPE=disk'
'/usr/bin/sg_inq --export /dev/sdb'(out) 'SCSI_VENDOR=Seagate'
'/usr/bin/sg_inq --export /dev/sdb'(out) 'SCSI_VENDOR_ENC=Seagate\x20'
'/usr/bin/sg_inq --export /dev/sdb'(out) 'SCSI_MODEL=Backup+_Desk'
'/usr/bin/sg_inq --export /dev/sdb'(out) 'SCSI_MODEL_ENC=Backup+\x20\x20Desk\x20\x20\x20'
'/usr/bin/sg_inq --export /dev/sdb'(out) 'SCSI_REVISION=0406'
Process '/usr/bin/sg_inq --export /dev/sdb' succeeded.
Starting '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg80 --raw'
'/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg80 --raw'(out) 'SCSI_IDENT_SERIAL=NA7EMXLG'
Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg80 --raw' succeeded.
Starting '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg83 --raw'
'/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg83 --raw'(out) 'SCSI_IDENT_LUN_NAA_REG=5000000000000001'
Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg83 --raw' succeeded.
Starting 'probe-bcache -o udev /dev/sdb'
Process 'probe-bcache -o udev /dev/sdb' succeeded.
Starting '/lib/udev/vdev_id -d sdb'
Process '/lib/udev/vdev_id -d sdb' succeeded.
----

Output from udevtest in a memory stick supporting only SCSI commands:

----
Process '/usr/bin/sg_inq --export /dev/sdc' succeeded.
Starting '/usr/bin/sg_inq --export --inhex=/sys/block/sdc/device/vpd_pg80 --raw'
'/usr/bin/sg_inq --export --inhex=/sys/block/sdc/device/vpd_pg80 --raw'(err) 'unable to open binary file /sys/block/sdc/device/vpd_pg80: No such file or directory'
Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdc/device/vpd_pg80 --raw' failed with exit code 15.
Starting 'probe-bcache -o udev /dev/sdc'
Process 'probe-bcache -o udev /dev/sdc' succeeded.
Starting '/lib/udev/vdev_id -d sdc'
Process '/lib/udev/vdev_id -d sdc' succeeded.
----

[Rafael David Tinoco (rafaeldtinoco)]

Okay, so I discovered the upstream version is doing something "similar" to what I just proposed, but in a different and more complete way (not suitable for a SRU for example). I explained it here:

https://github.com/hreinecke/sg3_utils/pull/47#issuecomment-539545275

And, because of that, I'll probably go with my suggestion (or close to it) as a SRU for Eoan, but will keep it opened for F-series and tag this as server-next, as it will require more attention for the next release and sg3-utils merges.

[Rafael David Tinoco (rafaeldtinoco)]

Quick comment: its opened for F-series because the "fix upstream" is actually a "way of deviating from bad execution path" but we still need a udev rule to make the USB pendrives (not supporting 0x80) as ENV{ID_SCSI_INQUIRY}="0", so the logic is good (still have to test this, next merge though).

[Rafael David Tinoco (rafaeldtinoco)]

From, at least, commit:

commit b6e681fea79d3d9b479acca1806f32310199053a
Author: Doug Gilbert <email address hidden>
Date: Mon Sep 18 16:15:01 2017

    https://github.com/hreinecke/sg3_utils branch sles15 synced 20170914; change sg_ll_*() function's 'int noisy' to bool

and beyond we haven't updated sg3-tools. This commit has changed how 55-scsi-sg3_id.rules logic works but not as much as commit:

commit c4c6af020963d11b624da6abb53f792074f943b2
Author: Doug Gilbert <email address hidden>
Date: Tue Mar 26 00:54:53 2019

    sg_inq: update version descriptors to spc5r21; scripts/scsi-sg3_id: update rules; testing folder work

    git-svn-id: svn://localhost/trunk@814 6180dd3e-e324-4e3e-922d-17de1ae2f315

which introduced the latest logic that makes sure not to query - or at least give us a way not to query - devices that don't support VPD 0x80/0x83 :

...

The following comment:

"""
# SCSI INQUIRY values
# If the 'inquiry' sysfs attribute is present the kernel will already
# have scanned for VPD pages, so if the vpd page attribute is not
# present it is not supported (or deemed unsafe to access).
# Hence we can skip the call to sg_inq and avoid I/O altogether.
# Set 'ID_SCSI_INQUIRY=0' in an earlier udev rule if the kernel
# fails to scan VPD pages correctly; the rules will then fall
# back to calling sg_vpd directly.
LABEL="scsi_inquiry"
ENV{ID_SCSI_INQUIRY}=="0", GOTO="sg_inquiry"

# As of 2018/4.15, the kernel doesn't provide VPD pages for "SPC" devices
# (SCSI version 0x03, ANSI INCITS 301-1997) in sysfs.
# It's usually safe to try though (no counter-example is known),
# and for scsi_id compatibility, we have to try.
SUBSYSTEMS=="scsi", ATTRS{scsi_level}=="4", GOTO="sg_inquiry"
"""

in 55-scsi-sg3_id.rules suggests that it is "SAFE" to try getting the VPD pages out of "SPC" devices (SCSI version 0x03, ANSI INCITS 301-1997) - our case - saying no counter-example is known (maybe this is the first ?).

It even says its a "scsi_id" compatibility. Because of that, what I'm suggesting for SRU is to do the VPD 0x80 and fail fast, instead of trying VPD 0x83 right after if 0x80 is not supported. This will keep the same behavior - as we have today - for USB drives supporting VPD pages - external SCSI-enabled USB disk devices - and will keep SPC-only USB devices - pendrives/memory sticks - with correct serial (different that we have today).

Nevertheless, I still have to merge sg3-tools in the next release - as the version we have seems old - and make sure the correct behavior (not attempting 0x83 after a failed 0x80) is done (thus the F-series will continue open).

[Rafael David Tinoco (rafaeldtinoco)]

Both, upstream version and Debian version already have the following change:

commit 988e967513fc9dc2be35dcb5bbc6b888f853538f
Author: Douglas Gilbert <email address hidden>
Date: Wed May 11 17:20:52 2016

    rescan-scsi-bus.sh + 55-scsi-sg3_id.rules: fixes from HR at Suse

    git-svn-id: svn://localhost/trunk@703 6180dd3e-e324-4e3e-922d-17de1ae2f315

which fix the SERIAL issue (but not the delay in insertion caused by errors of sg_inq commands).

That will be enough for this particular case (MAAS related). I can, during the next merge (F-series) propose to upstream a change not to execute sg_inq 0x83 if 0x80 wasn't successful, but, for now, taking in consideration the phase we are for Eoan (almost freeze), I'll base the changes in the upstream commit (at least the changes for 55-scsi-sg3_id.rules changes).

Marking F-series and Eoan as "Fix Released" as they keep the serial like it should be:

rafaeldtinoco@lenovo:~/work/sources/ubuntu$ sudo udevadm test -a -p /sys/block/sdb
...
Starting '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/inquiry --raw'
'/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/inquiry --raw'(out) 'SCSI_TPGS=0'
'/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/inquiry --raw'(out) 'SCSI_TYPE=disk'
'/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/inquiry --raw'(out) 'SCSI_VENDOR=Corsair'
'/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/inquiry --raw'(out) 'SCSI_VENDOR_ENC=Corsair\x20'
'/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/inquiry --raw'(out) 'SCSI_MODEL=Voyager_Mini_3.0'
'/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/inquiry --raw'(out) 'SCSI_MODEL_ENC=Voyager\x20Mini\x203.0'
'/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/inquiry --raw'(out) 'SCSI_REVISION=000B'
Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/inquiry --raw' succeeded.
Starting '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg80 --raw'
'/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg80 --raw'(err) 'unable to open binary file /sys/block/sdb/device/vpd_pg80: No such file or directory'
Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg80 --raw' failed with exit code 52.
Starting '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg83 --raw'
'/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg83 --raw'(err) 'unable to open binary file /sys/block/sdb/device/vpd_pg83: No such file or directory'
Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg83 --raw' failed with exit code 52.
Starting 'probe-bcache -o udev /dev/sdb'
Process 'probe-bcache -o udev /dev/sdb' succeeded.
Starting '/lib/udev/vdev_id -d sdb'
Process '/lib/udev/vdev_id -d sdb' succeeded.
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host2/target2:0:0/2:0:0:0/block/sdb
DEVNAME=/dev/sdb
DEVTYPE=disk
MAJOR=8
MINOR=16
ACTION=-p
SUBSYSTEM=block
SCSI_TPGS=0
SCSI_TYPE=disk
SCSI_VENDOR=Corsair
SCSI_VENDOR_ENC=Corsair\x20
SCSI_MODEL=Voyager_Mini_3.0
SCSI_MODEL_ENC=Voyager\x20Mini\x203.0
SCSI_REVISION=000B
ID_SCSI=1
ID_SCSI_INQUIRY=1
ID_VENDOR=Corsair
ID_VENDOR_ENC=Corsair\x20
ID_MODEL=Voyager_Mini_3.0
ID_MODEL_ENC=Voyage...

[Rafael David Tinoco (rafaeldtinoco)]

Will provide SRU proposal (Disco and Bionic) later today.

[Rafael David Tinoco (rafaeldtinoco)]

Provided a fix for Ubuntu Disco at the PPA and pushed changes into the merge request.

-----------

# Ubuntu Disco (showing the problem)

kernel: usb 2-6: new SuperSpeed Gen 1 USB device number 3 using xhci_hcd
kernel: usb 2-6: New USB device found, idVendor=1b1c, idProduct=1a0c, bcdDevice= 1.00
kernel: usb 2-6: New USB device strings: Mfr=1, Product=2, SerialNumber=3
kernel: usb 2-6: Product: Voyager Mini 3.0
kernel: usb 2-6: Manufacturer: Corsair
kernel: usb 2-6: SerialNumber: 070851D0E490C776
mtp-probe[17351]: checking bus 2, device 3: "/sys/devices/pci0000:00/0000:00:14.0/usb2/2-6"
mtp-probe[17351]: bus: 2, device: 3 was not an MTP device
kernel: usb-storage 2-6:1.0: USB Mass Storage device detected
kernel: scsi host2: usb-storage 2-6:1.0
kernel: usbcore: registered new interface driver usb-storage
kernel: usbcore: registered new interface driver uas
mtp-probe[17360]: checking bus 2, device 3: "/sys/devices/pci0000:00/0000:00:14.0/usb2/2-6"
mtp-probe[17360]: bus: 2, device: 3 was not an MTP device
kernel: scsi 2:0:0:0: Direct-Access Corsair Voyager Mini 3.0 000B PQ: 0 ANSI: 6
kernel: sd 2:0:0:0: Attached scsi generic sg1 type 0
kernel: sd 2:0:0:0: [sdb] 30283008 512-byte logical blocks: (15.5 GB/14.4 GiB)
kernel: sd 2:0:0:0: [sdb] Write Protect is off
kernel: sd 2:0:0:0: [sdb] Mode Sense: 23 00 00 00
kernel: sd 2:0:0:0: [sdb] No Caching mode page found
kernel: sd 2:0:0:0: [sdb] Assuming drive cache: write through
kernel: sdb: sdb1
kernel: sd 2:0:0:0: [sdb] Attached SCSI removable disk
systemd-udevd[17357]: Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg80 --raw' failed with exit code 15.
kernel: usb 2-6: reset SuperSpeed Gen 1 USB device number 3 using xhci_hcd
systemd-udevd[17357]: Spawned process '/usr/bin/sg_inq --export --page=sn /dev/sdb' [17363] is taking longer than 59s to complete
systemd-udevd[17357]: Process '/usr/bin/sg_inq --export --page=sn /dev/sdb' failed with exit code 99.
systemd-udevd[17357]: Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg83 --raw' failed with exit code 15.

and the serial:

P: /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host2/target2:0:0/2:0:0:0/block/sdb/sdb1
N: sdb1
L: 0
S: disk/by-id/scsi-32000acde48234567-part1
S: disk/by-label/ESD-USB
S: disk/by-id/scsi-3-part1
S: disk/by-id/scsi-28765432ab567abcd-part1
S: disk/by-uuid/B068-2D19
S: disk/by-id/wwn-0x2000acde48234567-part1
S: disk/by-id/scsi-1USB_DISK_2.0-part1
S: disk/by-partuuid/2ff58786-01
S: disk/by-path/pci-0000:00:14.0-usb-0:6:1.0-scsi-0:0:0:0-part1
E: DEVPATH=/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/host2/target2:0:0/2:0:0:0/block/sdb/sdb1
E: SUBSYSTEM=block
E: DEVNAME=/dev/sdb1
E: DEVTYPE=partition
E: PARTN=1
E: MAJOR=8
E: MINOR=17
E: USEC_INITIALIZED=1623657380
E: SCSI_TPGS=0
E: SCSI_TYPE=disk
E: SCSI_VENDOR=Corsair
E: SCSI_VENDOR_ENC=Corsair\x20
E: SCSI_MODEL=Voyager_Mini_3.0
E: SCSI_MODEL_ENC=Voyager\x20Mini\x203.0
E: SCSI_REVISION=000B
E: SCSI_IDENT_LUN_T10=USB_DISK_2.0
E: SCSI_IDENT_LUN_EUI64=8765432ab567abcd
E: SCSI_IDENT_LUN_NAA_EXT=2000acde48234567
E: ID_SCSI=1
E: ID_VENDOR=Corsair
E: ID_VENDOR_ENC=Corsair\x20
E: ID_MODEL=Voyager_Mini_3.0
E: ID_MODEL_ENC=Voyager\x20Mi...

[Rafael David Tinoco (rafaeldtinoco)]

@rharper: feel free to review the MR for Disco and Bionic (they're basically the same).

server-hwe team: feel free to test version from the PPA:
https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1833618

I think this is it, if no bad feedback, SRU is good to go in my POV.

[Rafael David Tinoco (rafaeldtinoco)]

NOTE for myself (for the future merge w/ upstream):

BTW, I got feedback from upstream developer saying that my logic of skipping VPD 0x83, if VPD 0x80 is not there - to avoid the big delay for the device addiction - can't be done. In his own words:

"""
SPC devices are a different can of worms, and I'd rather not touch them. And, incidentally, page 0x80 is considered legacy these days, and there are plenty of devices out there which support page 0x83 but not page 0x80. So the logic to skip page 0x83 if 0x80 is not present (or had errors) is flawed.
"""

So I have to check sysfs for vpd_pg80 and vpd_pg83 and dont do a sg_inq if files are not there (unsure if upstream does that). But I can't *not check* vpd_pg83 to reduce the delay.

[dann frazier (dannf)]

fyi, I tested a deployment w/ sg3-utils 1.42-2ubuntu2~19.04~1~ppa3 from your PPA and it was successful :)

[Timo Aaltonen (tjaalton)]

Hello Patricia, or anyone else affected,

Accepted sg3-utils into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sg3-utils/1.42-2ubuntu1.19.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Rafael David Tinoco (rafaeldtinoco)]

DISCO VERIFICATION:

A regular SCSI capable USB block device being attached:

Nov 04 11:26:13 lenovo kernel: usb 2-1: new SuperSpeed Gen 1 USB device number 7 using xhci_hcd
Nov 04 11:26:13 lenovo kernel: usb 2-1: New USB device found, idVendor=0bc2, idProduct=ab34, bcdDevice= 1.00
Nov 04 11:26:13 lenovo kernel: usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
Nov 04 11:26:13 lenovo kernel: usb 2-1: Product: Backup+ Desk
Nov 04 11:26:13 lenovo kernel: usb 2-1: Manufacturer: Seagate
Nov 04 11:26:13 lenovo kernel: usb 2-1: SerialNumber: NA7EMXLG
Nov 04 11:26:13 lenovo kernel: scsi host2: uas
Nov 04 11:26:13 lenovo kernel: scsi 2:0:0:0: Direct-Access Seagate Backup+ Desk 0406 PQ: 0 ANSI: 6
Nov 04 11:26:13 lenovo kernel: sd 2:0:0:0: Attached scsi generic sg1 type 0
Nov 04 11:26:13 lenovo kernel: sd 2:0:0:0: [sdb] Spinning up disk...
Nov 04 11:26:13 lenovo mtp-probe[1449005]: checking bus 2, device 7: "/sys/devices/pci0000:00/0000:00:14.0/usb2/2-1"
Nov 04 11:26:13 lenovo mtp-probe[1449005]: bus: 2, device: 7 was not an MTP device
Nov 04 11:26:13 lenovo mtp-probe[1449008]: checking bus 2, device 7: "/sys/devices/pci0000:00/0000:00:14.0/usb2/2-1"
Nov 04 11:26:13 lenovo mtp-probe[1449008]: bus: 2, device: 7 was not an MTP device

Nov 04 11:26:30 lenovo kernel: .................ready
Nov 04 11:26:30 lenovo kernel: sd 2:0:0:0: [sdb] 1220942645 4096-byte logical blocks: (5.00 TB/4.55 TiB)
Nov 04 11:26:30 lenovo kernel: sd 2:0:0:0: [sdb] 16384-byte physical blocks
Nov 04 11:26:30 lenovo kernel: sd 2:0:0:0: [sdb] Write Protect is off
Nov 04 11:26:30 lenovo kernel: sd 2:0:0:0: [sdb] Mode Sense: 4f 00 00 00
Nov 04 11:26:30 lenovo kernel: sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Nov 04 11:26:30 lenovo kernel: sd 2:0:0:0: [sdb] Optimal transfer size 268431360 bytes not a multiple of physical block size (16384 bytes)
Nov 04 11:26:30 lenovo kernel: sdb: sdb1
Nov 04 11:26:30 lenovo kernel: sd 2:0:0:0: [sdb] Attached SCSI disk

and its udev information (including good serial)

P: /devices/pci0000:00/0000:00:14.0/usb2/2-1/2-1:1.0/host2/target2:0:0/2:0:0:0/block/sdb
N: sdb
L: 0
S: disk/by-id/scsi-SSeagate_Backup+_Desk_NA7EMXLG
S: disk/by-id/wwn-0x5000000000000001
S: disk/by-path/pci-0000:00:14.0-usb-0:1:1.0-scsi-0:0:0:0
S: disk/by-id/scsi-35000000000000001
E: DEVPATH=/devices/pci0000:00/0000:00:14.0/usb2/2-1/2-1:1.0/host2/target2:0:0/2:0:0:0/block/sdb
E: SUBSYSTEM=block
E: DEVNAME=/dev/sdb
E: DEVTYPE=disk
E: MAJOR=8
E: MINOR=16
E: USEC_INITIALIZED=1133933104300
E: SCSI_TPGS=0
E: SCSI_TYPE=disk
E: SCSI_VENDOR=Seagate
E: SCSI_VENDOR_ENC=Seagate\x20
E: SCSI_MODEL=Backup+_Desk
E: SCSI_MODEL_ENC=Backup+\x20\x20Desk\x20\x20\x20
E: SCSI_REVISION=0406
E: ID_SCSI=1
E: ID_SCSI_SN=1
E: ID_SCSI_DI=1
E: ID_VENDOR=Seagate
E: ID_VENDOR_ENC=Seagate\x20
E: ID_MODEL=Backup+_Desk
E: ID_MODEL_ENC=Backup+\x20\x20Desk\x20\x20\x20
E: ID_REVISION=0406
E: ID_TYPE=disk
E: SCSI_IDENT_SERIAL=NA7EMXLG
E: SCSI_IDENT_LUN_NAA_REG=5000000000000001
E: ID_WWN=0x5000000000000001
E: ID_WWN_WITH_EXTENSION=0x5000000000000001
E: ID_BUS=scsi
E: ID_SERIAL=35000000000000001
E: ID_SERIAL_SHORT=5000000000000001
E: ID_PATH=pci-0000:00:1...

[Rafael David Tinoco (rafaeldtinoco)]

I still need the Bionic SRU to be sponsored!

[dann frazier (dannf)]

I hadn't noticed that bionic & disco were shipping the same of sg3-utils. Do you know why it is that we're not also seeing the MAAS deployment failure symptom on bionic?

[Rafael David Tinoco (rafaeldtinoco)]

I believe that happens because sg3-utils-udev started being deployed in cloud img from Disco and beyond. By not having the package installed by default, in Bionic, you won't face the issue. Nevertheless the issue is there after you manually install sg3-utils-udev, thus the need for both SRUs!

[dann frazier (dannf)]

Rafael - thanks for clarifying. If you don't mind posting a link to a source package for bionic (or attach a debdiff), I'd be happy to review it for sponsorship. I assume this is the bionic package in your ppa w/ the version adjusted, but I'd rather be 100% sure :)

[Ryan Harper (raharper)]

On Mon, Nov 4, 2019 at 9:41 AM Rafael David Tinoco <
<email address hidden>> wrote:

> I believe that happens because sg3-utils-udev started being deployed in
> cloud img from Disco and beyond. By not having the package installed by
>

Disco and newer include the multipath-tools package which brings in the
sg3-utils-udev package. The s390x support of multipath required us to
include the package in the images we deploy.

default, in Bionic, you won't face the issue. Nevertheless the issue is
> there after you manually install sg3-utils-udev, thus the need for both
> SRUs!
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1833618
>
> Title:
> MAAS can't deploy Ubuntu if ID_SERIAL of any block device is broken
> (USB pendrive in this case).
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/sg3-utils/+bug/1833618/+subscriptions
>

[Rafael David Tinoco (rafaeldtinoco)]

Hey Dann,

sure, its a merge request on the top of this bug, but here it is the direct url:

https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/sg3-utils/+git/sg3-utils/+ref/lp1833618-bionic

And yes, it is the same version as the one in the PPA.

Thx for reviewing it!

[dann frazier (dannf)]

On Mon, Nov 4, 2019 at 9:31 AM Rafael David Tinoco
<email address hidden> wrote:
>
> Hey Dann,
>
> sure, its a merge request on the top of this bug, but here it is the
> direct url:
>
> https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/sg3-utils/+git/sg3-utils/+ref/lp1833618-bionic
>
> And yes, it is the same version as the one in the PPA.
>
> Thx for reviewing it!

Hi Rafael,

Only change I'd ask for is the version, as it is currently > than the
disco-proposed version.

Seems like it should be 1.42-2ubuntu1.18.04.1?

  -dann

[Rafael David Tinoco (rafaeldtinoco)]

Thx Dann, not sure what I did there #). Will verify soon.

[Adam Conrad (adconrad)]

Hello Patricia, or anyone else affected,

Accepted sg3-utils into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sg3-utils/1.42-2ubuntu1.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Rafael David Tinoco (rafaeldtinoco)]

BIONIC VERIFICATION:

Nov 07 13:30:27 kbionic kernel: usb-storage 1-6:1.0: USB Mass Storage device detected
Nov 07 13:30:27 kbionic kernel: scsi host2: usb-storage 1-6:1.0
Nov 07 13:30:27 kbionic kernel: usbcore: registered new interface driver usb-storage
Nov 07 13:30:27 kbionic kernel: usbcore: registered new interface driver uas
Nov 07 13:30:27 kbionic mtp-probe[196132]: checking bus 1, device 5: "/sys/devices/pci0000:00/0000:00:14.0/usb1/1-6"
Nov 07 13:30:27 kbionic mtp-probe[196132]: bus: 1, device: 5 was not an MTP device
Nov 07 13:30:28 kbionic kernel: scsi 2:0:0:0: Direct-Access Generic Flash Disk 8.07 PQ: 0 ANSI: 4
Nov 07 13:30:28 kbionic kernel: sd 2:0:0:0: Attached scsi generic sg1 type 0
Nov 07 13:30:28 kbionic kernel: sd 2:0:0:0: [sdb] 15728640 512-byte logical blocks: (8.05 GB/7.50 GiB)
Nov 07 13:30:28 kbionic kernel: sd 2:0:0:0: [sdb] Write Protect is off
Nov 07 13:30:28 kbionic kernel: sd 2:0:0:0: [sdb] Mode Sense: 23 00 00 00
Nov 07 13:30:28 kbionic kernel: sd 2:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Nov 07 13:30:28 kbionic kernel: sdb: sdb1
Nov 07 13:30:28 kbionic kernel: sd 2:0:0:0: [sdb] Attached SCSI removable disk
Nov 07 13:30:28 kbionic systemd-udevd[196128]: Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg80 --raw' failed with exit code 15.
Nov 07 13:30:28 kbionic systemd-udevd[196128]: Process '/usr/bin/sg_inq --export --inhex=/sys/block/sdb/device/vpd_pg83 --raw' failed with exit code 15.
Nov 07 13:31:22 kbionic PackageKit[186101]: daemon quit
Nov 07 13:31:22 kbionic systemd[1]: packagekit.service: Main process exited, code=killed, status=15/TERM
Nov 07 13:31:22 kbionic systemd[1]: packagekit.service: Succeeded.

----

P: /devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6:1.0/host2/target2:0:0/2:0:0:0/block/sdb
N: sdb
L: 0
S: disk/by-id/usb-Generic_Flash_Disk_A3E8C929-0:0
S: disk/by-path/pci-0000:00:14.0-usb-0:6:1.0-scsi-0:0:0:0
E: DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6:1.0/host2/target2:0:0/2:0:0:0/block/sdb
E: SUBSYSTEM=block
E: DEVNAME=/dev/sdb
E: DEVTYPE=disk
E: MAJOR=8
E: MINOR=16
E: USEC_INITIALIZED=101644953827
E: SCSI_TPGS=0
E: SCSI_TYPE=disk
E: SCSI_VENDOR=Generic
E: SCSI_VENDOR_ENC=Generic\x20
E: SCSI_MODEL=Flash_Disk
E: SCSI_MODEL_ENC=Flash\x20Disk\x20\x20\x20\x20\x20\x20
E: SCSI_REVISION=8.07
E: ID_SCSI=1
E: ID_SCSI_SN=1
E: ID_SCSI_DI=1
E: ID_VENDOR=Generic
E: ID_VENDOR_ENC=Generic\x20
E: ID_MODEL=Flash_Disk
E: ID_MODEL_ENC=Flash\x20Disk\x20\x20\x20\x20\x20\x20
E: ID_REVISION=8.07
E: ID_TYPE=disk
E: ID_VENDOR_ID=058f
E: ID_MODEL_ID=6387
E: ID_SERIAL=Generic_Flash_Disk_A3E8C929-0:0
E: ID_SERIAL_SHORT=A3E8C929
E: ID_INSTANCE=0:0
E: ID_BUS=usb
E: ID_USB_INTERFACES=:080650:
E: ID_USB_INTERFACE_NUM=00
E: ID_USB_DRIVER=usb-storage
E: ID_PATH=pci-0000:00:14.0-usb-0:6:1.0-scsi-0:0:0:0
E: ID_PATH_TAG=pci-0000_00_14_0-usb-0_6_1_0-scsi-0_0_0_0
E: ID_PART_TABLE_UUID=01b60b77
E: ID_PART_TABLE_TYPE=dos
E: net.ifnames=0
E: DEVLINKS=/dev/disk/by-id/usb-Generic_Flash_Disk_A3E8C929-0:0 /dev/disk/by-path/pci-0000:00:14.0-usb-0:6:1.0-scsi-0:0:0:0
E: TAGS=:systemd:

USB serial is good and sg_inqXX failed as expected.

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for sg3-utils has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package sg3-utils - 1.42-2ubuntu1.18.04.1

---------------
sg3-utils (1.42-2ubuntu1.18.04.1) bionic; urgency=medium

  [ Rafael David Tinoco ]
  * d/p/55-scsi-sg3_id.rules-ID_SERIAL-fix.patch: ID_SERIAL fix for USB
    SPC-only devices in 55-scsi-sg3_id.rules (LP: #1833618)
  * d/p/new-location-for-major-and-minor.patch: Fix FTBS header issue

 -- dann frazier <email address hidden> Mon, 04 Nov 2019 12:48:30 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package sg3-utils - 1.42-2ubuntu1.19.04.1

---------------
sg3-utils (1.42-2ubuntu1.19.04.1) disco; urgency=medium

  * d/p/55-scsi-sg3_id.rules-ID_SERIAL-fix.patch: ID_SERIAL fix for USB
    SPC-only devices in 55-scsi-sg3_id.rules (LP: #1833618)
  * d/p/new-location-for-major-and-minor.patch: Fix FTBS header issue

 -- Rafael David Tinoco <email address hidden> Wed, 09 Oct 2019 06:13:54 +0000