Compound jobs are ignoring some items (noticed on disk tests)

[Jeff Lane ]

Compound jobs (Jobs that create other jobs) have been in use for some time, and they've been somewhat flaky. For a while now, there are cases where a machine with several disks will not see jobs created for each disk and this is something we need to resolve for 12.04 now that we're depending on checkbox to work for selftesting, Ubuntu Friendly and our own testing.

Here's the issue: on my laptop, I have an internal disk (sda), an external disk (sdb) and a usb stick (sdc). When checkbox runs, it sees jobs like this:

plugin: local
name: disk/benchmarks
requires: device.category == 'DISK'
_description: Benchmark for each disk
command:
 cat <<'EOF' | run_templates -t -s 'udev_resource | filter_templates -w "category=DISK"'
 plugin: shell
 name: disk/benchmark_`ls /sys$path/block`
 requires: device.path == "$path" and package.name == 'linux'
 user: root
 command: hdparm -tT /dev/`ls /sys$path/block` | sed 's/:.*= */ = /' | grep -v "^$"
 description: This test runs hdparm timing tests as a benchmark for $path
 EOF

which should create disk benchmark jobs for all three disks called:

benchmark_sda
benchmark_sdb
benchmark_sdc

However, it is only creating jobs for sdb and sdc. Looking at the log and it appears that sda is not behaving properly:

2012-02-02 11:55:48,496 DEBUG Calling ./plugins/message_info.py MessageInfo.report_messages([{'description': 'This test runs hdparm timing tests as a benchmark for /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2.3/2-1.2.3:1.0/host7/target7:0:0/7:0:0:0', 'plugin': 'shell', 'command': 'hdparm -tT /dev/sdb | sed \'s/:.*= */ = /\' | grep -v "^$"', 'user': 'root', 'requires': 'device.path == "/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2.3/2-1.2.3:1.0/host7/target7:0:0/7:0:0:0" and package.name == \'linux\'', 'name': 'disk/benchmark_sdb'}, {'description': 'This test runs hdparm timing tests as a benchmark for /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2.4/2-1.2.4:1.0/host9/target9:0:0/9:0:0:0', 'plugin': 'shell', 'command': 'hdparm -tT /dev/sdc | sed \'s/:.*= */ = /\' | grep -v "^$"', 'user': 'root', 'requires': 'device.path == "/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2.4/2-1.2.4:1.0/host9/target9:0:0/9:0:0:0" and package.name == \'linux\'', 'name': 'disk/benchmark_sdc'}, {'description': 'This test runs hdparm timing tests as a benchmark for /devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda', 'plugin': 'shell', 'command': 'hdparm -tT /dev/ | sed \'s/:.*= */ = /\' | grep -v "^$"', 'user': 'root', 'requires': 'device.path == "/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda" and package.name == \'linux\'', 'name': 'disk/benchmark_'}]) for report-messages with priority 0.

Notice that for sda, the 'name' item int he dictionary is 'disk/benchmark_' not 'disk/benchmark_sda'

[Jeff Lane  (bladernr)]

Here's a screenshot of checkbox run from trunk showing two disk tests that are only set up for sdb and sdc.

[Jeff Lane  (bladernr)]

here's checkbox log from an aborted run... I only ran checkbox up to the point where it shows the test list to verify that it was not showing sda tests.

[Jeff Lane  (bladernr)]

Additionally, I unplugged the two USB storage devices (sdb and sdc) and reran checkbox. In tis case, it also failed to run disk tests for sda, again showing the names as 'benchmark_' rather than 'benchmark_sda'. Here's the log from that run as well.

[Jeff Lane  (bladernr)]

And additionally, when this bug is met, the checkbox UI still shows items listed under Disk Tests, as though they were going to be run, though there really are no disk tests to run.

[Marc Tardif (cr3)]

Please show the output of running the command:

 cat <<'EOF' | run_templates -t -s 'udev_resource | filter_templates -w "category=DISK"'
 plugin: shell
 name: disk/benchmark_`ls /sys$path/block`
 requires: device.path == "$path" and package.name == 'linux'
 user: root
 command: hdparm -tT /dev/`ls /sys$path/block` | sed 's/:.*= */ = /' | grep -v "^$"
 description: This test runs hdparm timing tests as a benchmark for $path
 EOF

It shouldn't be very long, so please feel free to paste it directly as a comment.

[Jeff Lane  (bladernr)]

Interesting... that would be why....

bladernr@klaatu:~/development/checkbox$ cat <<'EOF' | PYTHONPATH=.:$PYTHONPATH scripts/run_templates -t -s 'scripts/udev_resource | scripts/filter_templates -w "category=DISK"'
plugin: shell
name: disk/benchmark_`ls /sys$path/block`
requires: device.path == "$path" and package.name == 'linux'
user: root
command: hdparm -tT /dev/`ls /sys$path/block` |sed 's/:.*= */ = /' | grep -v "^$"
description: This test runs hdparm timing tests as a benchmark for $path
EOF

ls: cannot access /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/block: No such file or directory
ls: cannot access /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/block: No such file or directory
plugin: shell
name: disk/benchmark_
requires: device.path == "/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda" and package.name == 'linux'
user: root
command: hdparm -tT /dev/ |sed 's/:.*= */ = /' | grep -v "^$"
description: This test runs hdparm timing tests as a benchmark for /devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda

[Jeff Lane  (bladernr)]

For comparison sake, here it is with the USB HDD and USB Stick plugged in:

plugin: shell
name: disk/benchmark_sdc
requires: device.path == "/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2.3/2-1.2.3:1.0/host11/target11:0:0/11:0:0:0" and package.name == 'linux'
user: root
command: hdparm -tT /dev/sdc |sed 's/:.*= */ = /' | grep -v "^$"
description: This test runs hdparm timing tests as a benchmark for /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2.3/2-1.2.3:1.0/host11/target11:0:0/11:0:0:0

plugin: shell
name: disk/benchmark_sdd
requires: device.path == "/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2.4/2-1.2.4:1.0/host12/target12:0:0/12:0:0:0" and package.name == 'linux'
user: root
command: hdparm -tT /dev/sdd |sed 's/:.*= */ = /' | grep -v "^$"
description: This test runs hdparm timing tests as a benchmark for /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2.4/2-1.2.4:1.0/host12/target12:0:0/12:0:0:0

ls: cannot access /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/block: No such file or directory
ls: cannot access /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/block: No such file or directory
plugin: shell
name: disk/benchmark_
requires: device.path == "/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda" and package.name == 'linux'
user: root
command: hdparm -tT /dev/ |sed 's/:.*= */ = /' | grep -v "^$"
description: This test runs hdparm timing tests as a benchmark for /devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda

[Marc Tardif (cr3)]

Please try the same commands with the branch attached to this bug report. Make sure to specify the PYTHONPATH accordingly:

bzr branch lp:~cr3/checkbox/925582
cd 925582
PYTHONPATH=. ./scripts/udev_resource ...

[Jeff Lane  (bladernr)]

That seems to have done it... here's confirmation both with and without the USB media plugged in:

bladernr@klaatu:~/development/925582$ cat <<'EOF' | PYTHONPATH=.:$PYTHONPATH scripts/run_templates -t -s 'scripts/udev_resource | scripts/filter_templates -w "category=DISK"'
> plugin: shell
> name: disk/benchmark_`ls /sys$path/block`
> requires: device.path == "$path" and package.name == 'linux'
> user: root
> command: hdparm -tT /dev/`ls /sys$path/block` |sed 's/:.*= */ = /' | grep -v "^$"
> description: This test runs hdparm timing tests as a benchmark for $path
> EOF
plugin: shell
name: disk/benchmark_sdc
requires: device.path == "/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2.3/2-1.2.3:1.0/host11/target11:0:0/11:0:0:0" and package.name == 'linux'
user: root
command: hdparm -tT /dev/sdc |sed 's/:.*= */ = /' | grep -v "^$"
description: This test runs hdparm timing tests as a benchmark for /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2.3/2-1.2.3:1.0/host11/target11:0:0/11:0:0:0

plugin: shell
name: disk/benchmark_sdd
requires: device.path == "/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2.4/2-1.2.4:1.0/host14/target14:0:0/14:0:0:0" and package.name == 'linux'
user: root
command: hdparm -tT /dev/sdd |sed 's/:.*= */ = /' | grep -v "^$"
description: This test runs hdparm timing tests as a benchmark for /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2.4/2-1.2.4:1.0/host14/target14:0:0/14:0:0:0

plugin: shell
name: disk/benchmark_sda
requires: device.path == "/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0" and package.name == 'linux'
user: root
command: hdparm -tT /dev/sda |sed 's/:.*= */ = /' | grep -v "^$"
description: This test runs hdparm timing tests as a benchmark for /devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0

bladernr@klaatu:~/development/925582$ cat <<'EOF' | PYTHONPATH=.:$PYTHONPATH scripts/run_templates -t -s 'scripts/udev_resource | scripts/filter_templates -w "category=DISK"'
plugin: shell
name: disk/benchmark_`ls /sys$path/block`
requires: device.path == "$path" and package.name == 'linux'
user: root
command: hdparm -tT /dev/`ls /sys$path/block` |sed 's/:.*= */ = /' | grep -v "^$"
description: This test runs hdparm timing tests as a benchmark for $path
EOF
plugin: shell
name: disk/benchmark_sda
requires: device.path == "/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0" and package.name == 'linux'
user: root
command: hdparm -tT /dev/sda |sed 's/:.*= */ = /' | grep -v "^$"
description: This test runs hdparm timing tests as a benchmark for /devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0

[Jeff Lane  (bladernr)]

also ran checkbox from that branch and verified that sda, sdb and sdc all show up with disk/benchmark_* jobs

[Jeff Lane  (bladernr)]

Actually, this only needed to be backported to 0.12.

0.11, 0.10 and 0.9 all seem to be unaffected. Linked a branch that backports the fix to 0.12