IPsec problems: setkey error "invalid keymsg length", racoon spins in busy loop

[Matt Mullins]

Binary package hint: linux-image-2.6.27-9-generic

When I upgraded from hardy to intrepid, my IPsec configuration stopped working. The first symptom I noticed was upon boot, racoon was spinning itself in a busy loop, sucking up all of one CPU. After tinkering with setkey trying to edit the running IPsec configuration, I found that setkey would give me the error "invalid keymsg length".

I found http://patchwork.ozlabs.org/patch/6754/ which shows a patch that fixes the "invalid keymsg length" problem. This has been applied upstream, and should be included in 2.6.28. This corresponds to commit 920da6923cf03c8a78fbaffa408f8ab37f6abfc1 in Linus's tree.

I built a kernel from the git tree at kernel.ubuntu.org with this fix, and both setkey and racoon work as intended. I will build a kernel from the same tree without that patch later today to verify that it is indeed the culprit.

[Leann Ogasawara (leannogasawara)]

Hi Matt,

Thanks for the report and especially the reference to the upstream git commit id. The patch you've referenced is already included in the upcoming Jaunty kernel. I've gone ahead and opened an Intrepid nomination for this to be considered for a Stable Release Update for the Intrepid kernel. Thanks.

ogasawara@yoji:~/ubuntu-jaunty$ git log 920da6923cf03c8a78fbaffa408f8ab37f6abfc1
commit 920da6923cf03c8a78fbaffa408f8ab37f6abfc1
Author: Alexey Dobriyan <email address hidden>
Date: Fri Oct 31 16:41:26 2008 -0700

    key: fix setkey(8) policy set breakage

[Andy Whitcroft (apw)]

@Matt -- it seems that this fix has been applied to the Intrepid kernel via the stable 2.6.27.10 updates. The current -proposed kernel should already include this fix. Could you test this kernel and report back here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed.

[Matt Mullins (mokomull)]

It does appear that the kernel in -proposed resolves this issue.
--
Matt Mullins

[Andy Whitcroft (apw)]

Moving this Fix Committed as the fix is in -proposed.

[Alex Valavanis (valavanisalex)]

Intrepid Ibex reached end-of-life on 30 April 2010 so I am closing the report. The bug has been fixed in newer releases of Ubuntu.