Some HIGH/CRITICAL failures in fwts related tests cannot be detected by checkbox

Bug #1252186 reported by Po-Hsu Lin on 2013-11-18
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Checkbox
Medium
Brendan Donegan
Firmware Test Suite
Critical
Colin Ian King

Bug Description

In this fwts report from 201308-14057 (12.04.3 + 3.8.0-33), there are some CRITICAL failures reported by fwts in method test, but not recognized by the test case parser.
In the final report, the fwts_test was marked as PASSED with only MEDIUM failures recorded.

CRITICAL failures: 14
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.PRID.P_D0._STA'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.PRID.P_D1._STA'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.SECD.S_D0._STA'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.SECD.S_D1._STA'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0._INI'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA._INI'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.PRID._PS0'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.PRID.P_D0._PS0'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.PRID.P_D1._PS0'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.SECD._PS0'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.SECD.S_D0._PS0'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.SECD.S_D1._PS0'.
  method: Detected error 'Type' when evaluating '\_SB_.PCI0.LPC0.ADP1._PCL'.
  method: Detected error 'Not exist' when evaluating '\_WAK'.

Po-Hsu Lin (cypressyew) wrote :
Daniel Manrique (roadmr) on 2013-11-18
tags: added: scripts
Ara Pulido (ara) on 2013-11-19
Changed in checkbox:
milestone: none → 2013-dec-06
Po-Hsu Lin (cypressyew) wrote :

A HIGH failure in fadt test cannot be correctly parsed as well

FAILED [HIGH] FADTPM1AInvalidWidth: Test 1, FADT pm1a register has invalid bit width of 0.

In fact, not all errors correctly parsed,
please find the attached fwts_result.log and the submission.xml

(I think this one is also related in fwts_test itself, so I didn't open a new bug.
Please inform me if you think it will be better to open a new one.)

Po-Hsu Lin (cypressyew) wrote :

According to this submission file, only one Low Failure was detected.
But there are many MEDIUM failures as well.

summary: - CRITICAL failures in fwts method test cannot be detected by checkbox
+ Some HIGH/CRITICAL failures in fwts related tests cannot be detected by
+ checkbox
description: updated
Changed in checkbox:
milestone: 2013-dec-06 → 2013-dec-20

I'll have a look at this. It will be easiest to work on if I can get access to this system.

Changed in checkbox:
status: New → In Progress
assignee: nobody → Brendan Donegan (brendan-donegan)
importance: Undecided → Medium
Daniel Manrique (roadmr) on 2013-12-20
Changed in checkbox:
milestone: 2013-dec-20 → 2014-01-17
Po-Hsu Lin (cypressyew) wrote :

Hello Brendand,
please contact us if you need to access this system
Thanks!

This is the current output:

ubuntu@201308-14057:~$ /usr/share/checkbox/scripts/fwts_test -l fwts_results.log

High Failures: 2
WARNING: The following test cases were reported as high
level failures by fwts. Please review the log at
fwts_results.log for more information.
 - klog
 - oops
Passed: 6
 - apicedge
 - version
 - dmar
 - bios_info
 - acpiinfo
 - cstates
Aborted Tests: 17
WARNING: The following test cases were aborted by fwts
Please review the log at fwts_results.log for more information.
 - hpet_check
 - fadt
 - aspm
 - mpcheck
 - ebda
 - msr
 - nx
 - wmi
 - virt
 - mcfg
 - uefirtvariable
 - mtrr
 - apicinstance
 - checksum
 - method
 - bios32
 - acpitables

It's curious there are so many aborted tests, but I can't seem to reproduce the initial issue.

That was without sudo - my bad. Here's the correct result:

ubuntu@201308-14057:~$ sudo /usr/share/checkbox/scripts/fwts_test -l fwts_results.log
[sudo] password for ubuntu:
High Failures: 2
WARNING: The following test cases were reported as high
level failures by fwts. Please review the log at
fwts_results.log for more information.
 - klog
 - oops
Passed: 18
 - apicedge
 - hpet_check
 - fadt
 - mpcheck
 - ebda
 - nx
 - wmi
 - version
 - virt
 - mcfg
 - mtrr
 - apicinstance
 - checksum
 - bios_info
 - bios32
 - acpitables
 - acpiinfo
 - cstates
Aborted Tests: 1
WARNING: The following test cases were aborted by fwts
Please review the log at fwts_results.log for more information.
 - uefirtvariable

It seems to be the case that the test is run but not reported on - unfortunately I lost access to the system in order to be able to figure out why. Once I get access again I can debug it.

Changed in checkbox:
milestone: 2014-01-17 → 2014-jan-31
Po-Hsu Lin (cypressyew) wrote :

Hello Brendan,

I found the poor connection might be caused by the IP address conflict between two systems, it should be fine now.

This seems to be a bug in FWTS. We use the --stdout-summary option of FWTS to get the overall status of each test - for the 'method' test on this system it's reported as SKIPPED, so we don't report on it. According to cking this is erroneous and should be FAILED in this instance. He will fix it and this bug will go away.

Changed in fwts:
status: New → In Progress
Colin Ian King (colin-king) wrote :

Fix sent to fwts-devel for review and inclusion at the next release https://lists.ubuntu.com/archives/fwts-devel/2014-January/004330.html

Changed in fwts:
importance: Undecided → Critical
assignee: nobody → Colin King (colin-king)
milestone: none → 14.02.00
Colin Ian King (colin-king) wrote :

A test version of fwts is available in my ppa, ppa:colin-king - this contains the fix. Is it possible for this to be tested?

Changed in checkbox:
status: In Progress → Fix Committed
Changed in fwts:
status: In Progress → Fix Committed
Colin Ian King (colin-king) wrote :

committed, commit e8a1258fc8a5af43964c9c4c424dc6c77d7b9708 and released in 14.01.01

Changed in fwts:
status: Fix Committed → Fix Released
Zygmunt Krynicki (zyga) on 2014-02-07
Changed in checkbox:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers