blktap-dkms version in 12.04.2 is not compatible with the 12.04.2 kernel

[Jeff Lane ]

[IMPACT]

Makes the package unusable for 12.04.2 LTS users unless they downgrade to a 3.2.x kernel

[TEST CASE]

Install Ubuntu 12.04.2 LTS and attempt to install blktap-dkms package.

[REGRESSION POTENTIAL]

Could make the package unusable for 12.04.x users who remain on 3.2.x kernel. That needs testing.

[OTHER INFO]

Here is the bug showing the debdiff and tested/Fixed in a later Ubuntu release: https://bugs.launchpad.net/xcp/+bug/1028135

This is on a system with a fresh install of 12.04.2 server. After install, I installed the Xen and XAPI bits. the blktap module fails to load on 3.5 kernels due to broken symbol errors:

Trying to start a newly created vm and I get the following with 12.04.2:
bladernr@xenserver:~$ xe vm-start uuid=fb22747d-a25a-4f6f-b33a-9ef14783b927
There was an SR backend failure.
status: non-zero exit
stdout:
stderr: Traceback (most recent call last):
  File "/usr/lib/xcp/sm/FileSR", line 814, in <module>
    SRCommand.run(FileSR, DRIVER_INFO)
  File "/usr/lib/xcp/sm/SRCommand.py", line 263, in run
    ret = cmd.run(sr)
  File "/usr/lib/xcp/sm/SRCommand.py", line 94, in run
    return self._run_locked(sr)
  File "/usr/lib/xcp/sm/SRCommand.py", line 131, in _run_locked
    return self._run(sr, target)
  File "/usr/lib/xcp/sm/SRCommand.py", line 196, in _run
    writable, caching_params)
  File "/usr/lib/xcp/sm/blktap2.py", line 1305, in activate
    writable, caching_params):
  File "/usr/lib/xcp/sm/blktap2.py", line 46, in wrapper
    ret = op(self, *args)
  File "/usr/lib/xcp/sm/blktap2.py", line 1337, in _activate_locked
    dev_path = self._activate(sr_uuid, vdi_uuid, writable, caching_params)
  File "/usr/lib/xcp/sm/blktap2.py", line 1357, in _activate
    dev_path = self._tap_activate(phy_path, vdi_type, sr_uuid, writable)
  File "/usr/lib/xcp/sm/blktap2.py", line 1149, in _tap_activate
    tapdisk = Tapdisk.find_by_path(phy_path)
  File "/usr/lib/xcp/sm/blktap2.py", line 616, in find_by_path
    return cls.find(path=path)
  File "/usr/lib/xcp/sm/blktap2.py", line 604, in find
    found = list(cls.list(**args))
  File "/usr/lib/xcp/sm/blktap2.py", line 579, in list
    for row in TapCtl.list(**args):
  File "/usr/lib/xcp/sm/blktap2.py", line 81, in loop
    return f(*__t, **__d)
  File "/usr/lib/xcp/sm/blktap2.py", line 293, in list
    return list(cls.__list(**args))
  File "/usr/lib/xcp/sm/blktap2.py", line 262, in __list
    raise Exception, "blktap kernel module not installed: try 'modprobe blktap'"
Exception: blktap kernel module not installed: try 'modprobe blktap'
If I try manually loading blktap, I get this error:
bladernr@xenserver:~$ sudo modprobe blktap
FATAL: Error inserting blktap (/lib/modules/3.5.0-23-generic/updates/dkms/blktap.ko): Unknown symbol in module, or unknown parameter (see dmesg)

And looking at dmesg as suggested:
[ 4148.840713] blktap: Unknown symbol do_munmap (err 0)
[ 4148.840735] blktap: Unknown symbol do_mmap_pgoff (err 0)
and I found this that seems to indicate that the Precise version of blktap is incompatible with kernels > 3.3
http://lists.xen.org/archives/html/xen-api/2012-11/msg00107.html

Checking it out, this is what is installed on a fresh 12.04.2 server install:
bladernr@xenserver:~$ apt-cache policy blktap-dkms
blktap-dkms:
  Installed: 2.0.91-1
  Candidate: 2.0.91-1
  Version table:
 *** 2.0.91-1 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise/universe amd64 Packages
        100 /var/lib/dpkg/status

Using clues here:
http://blog.sleeplessbeastie.eu/2012/10/08/ubuntu-precise-install-youtube-dl-package-using-quantal-repo/

I installed blktap-dkms from the Quantal repo and this time it loaded without errors.
bladernr@xenserver:~$ sudo modprobe blktap
bladernr@xenserver:~$ sudo apt-cache policy blktap-dkms
blktap-dkms:
  Installed: 2.0.93-0ubuntu1
  Candidate: 2.0.93-0ubuntu1
  Package pin: 2.0.93-0ubuntu1
  Version table:
 *** 2.0.93-0ubuntu1 990
        500 http://us.archive.ubuntu.com/ubuntu/ quantal/universe amd64 Packages
        100 /var/lib/dpkg/status
     2.0.91-1 990
        990 http://us.archive.ubuntu.com/ubuntu/ precise/universe amd64 Packages

After doing this, as mentioned, the blktap module loads without issue and I am able to kick off that VM with no further tracebacks.

Thus, the older version of blktap is not compatible with 3.5 kernels, so we need to backport 2.0.93-0ubuntu1 to precise for quantal kernel usage...

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: blktap-dkms 2.0.93-0ubuntu1
ProcVersionSignature: Ubuntu 3.5.0-23.35~precise1-generic 3.5.7.2
Uname: Linux 3.5.0-23-generic x86_64
ApportVersion: 2.0.1-0ubuntu17.1
Architecture: amd64
Date: Tue Mar 19 16:55:04 2013
InstallationMedia: Ubuntu-Server 12.04.2 LTS "Precise Pangolin" - Release amd64 (20130214)
MarkForUpload: True
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: blktap-dkms
UpgradeStatus: No upgrade log present (probably fresh install)

[Daniel Letzeisen (dtl131)]

Confirmed. Starting SRU:

[IMPACT]

Makes the package unusable for 12.04.2 LTS users unless they downgrade to a 3.2.x kernel

[TEST CASE]

Install Ubuntu 12.04.2 LTS and attempt to install blktap-dkms package.

[REGRESSION POTENTIAL]

Could make the package unusable for 12.04.x users who remain on 3.2.x kernel. That needs testing.

[OTHER INFO]

Here is the bug showing the debdiff and tested/Fixed in a later Ubuntu release: https://bugs.launchpad.net/xcp/+bug/1028135

[Dmitry Shachnev (mitya57)]

For those who want to test the new version (which is in the SRU queue now), the package is available in ppa:mitya57/test2.

[Brian Murray (brian-murray)]

Hello Jeff, or anyone else affected,

Accepted blktap-dkms into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/blktap-dkms/2.0.91-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Eythor Kristleifsson (eythork)]

I recently installed 12.04.2 and am running the 3.5.0-23 kernel. I ran into the exact same problems mentioned above with the blktap module so I tested the precise-proposed package. After upgrading the package I was able to load the blktap module without problems. When starting a HVM virtual machine with xe start there is however a problem booting from a cdrom iso. The quemu bootload claims gives the error

CDROM boot error code 0003
Boot from CD-rom failed: could not read the boot disk

Syslog has errors regarding blktap:

[75925.627105] blktap_device_make_request: 732 callbacks suppressed
[75925.627114] block tdb: make request: -22, failing
[75925.627123] end_request: I/O error, dev tdb, sector 0
[75925.629387] Buffer I/O error on device tdb, logical block 0
[75925.654192] block tda: make request: -22, failing
[75925.654205] end_request: I/O error, dev tda, sector 64
[75925.656909] Buffer I/O error on device tda, logical block 16
[75925.658163] Buffer I/O error on device tda, logical block 17
[75925.662738] block tda: make request: -22, failing
[75925.662746] end_request: I/O error, dev tda, sector 64
[75925.665825] Buffer I/O error on device tda, logical block 16
[75925.666700] Buffer I/O error on device tda, logical block 17
[75925.672574] block tda: make request: -22, failing
[75925.672582] end_request: I/O error, dev tda, sector 64
[75925.676078] Buffer I/O error on device tda, logical block 16

I'm guessing this is related to the blktab-dkms package

Anyone else seeing similar issues using blktap-dkms/precise-proposed ?

[Henrik Welk (henrik-welk)]

While setting up a new Xen box yesterday, I can totally reproduce the behaviour reported by Eythor.
Xen doesnt boot from virtual CD drive with error code 0003 and dmesg is full of blktap messages.

Anyone around who has seen this before or might know if this is related to the latest blktap-dkms changes?

[Henrik Welk (henrik-welk)]

Additionally I figured out, that installing the version from the Quantal Repo as Jeff suggested fixes the problem, so it seems to be a bug in blktap-dkms.

[Dmitry Shachnev (mitya57)]

Can anybody please test version 2.0.91-1ubuntu0.2~ppa1 that I have just uploaded to ppa:mitya57/test2 (will be available in 5–6 hours).

This is second attempt to cherry-pick the needed bits from 2.0.93. If that doesn't help, we'll have to backport the whole 2.0.93 release to precise…

[Matt (hookenz)]

I tried it but I still saw I/O errors on the console.

Now the server seems to be stuck in maintenance mode. If I delete the crash dumps and reboot, the server starts back up in maintenance mode. It's a never ending loop.

I tried all the solutions suggested here: http://www.crucial.com.au/blog/2011/03/31/xenserver-server-won%E2%80%99t-exist-maintenance-mode-cannot-forward-messages-because-the-server-cannot-be-contacted/

But it's still stuck in maintenance mode after installing your blktap patch. It's a cleanly installed server.

[John Ho (john-ph-ho)]

I tried using 2.0.93-0ubuntu4 (blktap-dkms/saucy) and it's working so far, no I/O errors and I'm able to boot from cdrom iso.

[Martin Pitt (pitti)]

The version of blktap-dkms in the proposed pocket of Precise that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.