linux-armadaxp: 3.5.0-1610.14 -proposed tracker

[Brad Figg]

This bug is for tracking the 3.5.0-1610.14 upload package. This bug will contain status and testing results related to that upload.

For an explanation of the tasks and the associated workflow see: https://wiki.ubuntu.com/Kernel/kernel-sru-workflow
kernel-stable-Prepare-package-start:Tuesday, 26. February 2013 20:04 UTC
kernel-stable-master-bug:1133429
kernel-stable-Certification-testing-end:Tuesday, 26. February 2013 21:01 UTC
kernel-stable-Security-signoff-end:Tuesday, 26. February 2013 21:01 UTC
kernel-stable-Prepare-package-end:Wednesday, 27. February 2013 16:02 UTC
kernel-stable-Promote-to-proposed-start:Wednesday, 27. February 2013 16:02 UTC
kernel-stable-Promote-to-proposed-end:Wednesday, 27. February 2013 18:42 UTC
kernel-stable-Verification-testing-start:Wednesday, 27. February 2013 20:02 UTC
kernel-stable-phase:Testing
kernel-stable-phase-changed:Thursday, 28. February 2013 01:03 UTC
kernel-stable-Verification-testing-end:Thursday, 28. February 2013 01:03 UTC
kernel-stable-Regression-testing-start:Thursday, 28. February 2013 01:03 UTC
kernel-stable-Regression-testing-end:Monday, 18. March 2013 13:00 UTC

[Brad Figg (brad-figg)]

This tracking bug was opened to be worked from linux-3.5.0-26.40 update (bug 1133429)

[Brad Figg (brad-figg)]

All builds are complete, packages in this bug can be copied to -proposed.

[Adam Conrad (adconrad)]

Copied to -proposed and overrides adjusted.

[Ike Panhc (ikepanhc)]

Rebase only

[Paul Larson (pwlars)]

I hit something odd in the regression testing for this. The first run had a couple of failed tests then seemed that root was mounted readonly and the job got stuck. The link to the details is here:
https://jenkins.qa.ubuntu.com/job/sru_kernel-quantal-generic-armhf_omap4_armada1-serial/37/
I also saved off the syslog here: http://paste.ubuntu.com/5574592/ showing some several errors

I reran to see if it was reproducible, and the second run passed without any error:
https://jenkins.qa.ubuntu.com/job/sru_kernel-quantal-generic-armhf_omap4_armada1-serial/38/

[Ike Panhc (ikepanhc)]

Examining the log, looks like the error start with ata hard reset

Feb 27 22:53:04 armada1 kernel: [ 1162.849379] ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6
Feb 27 22:53:04 armada1 kernel: [ 1162.855846] ata1.00: edma_err_cause=02000084 pp_flags=00000003, dev error, EDMA self-disable
Feb 27 22:53:04 armada1 kernel: [ 1162.864342] ata1.00: failed command: READ FPDMA QUEUED
Feb 27 22:53:04 armada1 kernel: [ 1162.869527] ata1.00: cmd 60/08:00:98:22:34/00:00:11:00:00/40 tag 0 ncq 4096 in
Feb 27 22:53:04 armada1 kernel: [ 1162.869527] res 41/40:00:98:22:34/00:00:11:00:00/40 Emask 0x409 (media error) <F>
Feb 27 22:53:04 armada1 kernel: [ 1162.885061] ata1.00: status: { DRDY ERR }
Feb 27 22:53:04 armada1 kernel: [ 1162.889092] ata1.00: error: { UNC }
Feb 27 22:53:04 armada1 kernel: [ 1162.892623] ata1: hard resetting link
Feb 27 22:53:05 armada1 kernel: [ 1163.399289] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl F300)

[Ike Panhc (ikepanhc)]

@pwlars

I can not be sure its hardware or kernel problem. Could you run it several times and see if it happens again?

[Paul Larson (pwlars)]

I ran it 3 more times at the end of last week:
PASS: https://jenkins.qa.ubuntu.com/job/sru_kernel-quantal-generic-armhf_omap4_armada1-serial/39/
FAIL: https://jenkins.qa.ubuntu.com/job/sru_kernel-quantal-generic-armhf_omap4_armada1-serial/40/
PASS: https://jenkins.qa.ubuntu.com/job/sru_kernel-quantal-generic-armhf_omap4_armada1-serial/41/

The one that failed looked similar in that there were messages about / being mounted read-only, but I do not have the full syslog unfortunately. In the one that failed, it seemed to fail after booting the new kernel and while installing some deps for the kernel tests to run, rather than later in the process like the time before.

[Ike Panhc (ikepanhc)]

@pwlars,

Could you also try 3.5.0-1609.13 so that we will know this is a regression or not. In the mean time, I will make several testing kernel and try to bisect from patches of rebasing.

[Ike Panhc (ikepanhc)]

and I am also running bonnie as sata disk stress testing on armadaxp node with this kernel, see if I can hit the link reset.

[Ike Panhc (ikepanhc)]

The kernel for bisect is put at http://kernel.ubuntu.com/~ikepanhc/armadaxp

[Paul Larson (pwlars)]

I think I have a way of trying these out now, and also looking in to the possibility that it could just be an issue with the drive. I'll see what I can figure out.

[Paul Larson (pwlars)]

Quick update: I tried a few of the bisect kernels and didn't get it to reproduce so far, but on one of the reboots, fsck complained about a lot of errors. So I'm going to shift to make sure we don't have a bad drive. I have a new drive attached now and I'm trying again with the kernel for this sru. I have 3 consecutive good runs so far and will keep running over the weekend

[Paul Larson (pwlars)]

Apologies for the delay, but there was a problem we were seeing in multiple runs. After further testing it looks like it was just a bad drive though. Since swapping out the drive, I've run the tests 10 times in a row with no problems at all.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-armadaxp - 3.5.0-1611.17

---------------
linux-armadaxp (3.5.0-1611.17) quantal-proposed; urgency=low

  [ Ike Panhc ]

  * Release Tracking Bug
    - LP: #1160181
  * Rebase onto Ubuntu-3.5.0-27.46

  [ Ubuntu: 3.5.0-27.46 ]

  * Release Tracking Bug
    - LP: #1159991
  * Start New Release
  * crypto: user - fix info leaks in report API
    - LP: #1156790, #1156795, #1156799
    - CVE-2013-2546
  * brcmsmac: fix mismatch in number of custom regulatory rules
    - LP: #1156769
  * SAUCE: PCI: define macro for marvell vendor ID
    - LP: #1159863
  * SAUCE: PCI: fix system hang issue of Marvell SATA host controller
    - LP: #1159863

  [ Ubuntu: 3.5.0-27.45 ]

  * no change

  [ Ubuntu: 3.5.0-27.44 ]

  * no change
 -- Ike Panhc <email address hidden> Tue, 26 Mar 2013 16:59:14 +0800