Assertion failed at socket.c:629

[Jarrett Miller]

Binary package hint: openvpn

Description: Ubuntu 10.04.1 LTS
Release: 10.04
openvpn:
  Installed: 2.1.0-1ubuntu1.1
  Candidate: 2.1.0-1ubuntu1.1
  Version table:
 *** 2.1.0-1ubuntu1.1 0
        500 http://us.archive.ubuntu.com/ubuntu/ lucid-updates/main Packages
        100 /var/lib/dpkg/status
     2.1.0-1ubuntu1 0
        500 http://us.archive.ubuntu.com/ubuntu/ lucid/main Packages

What I expected:
I have open vpn configured for bridged networking off a single port via xinetd over TCP.
That is to say each incoming connection on port 22 (I need to use this port). spawns a
new instance of /usr/sbin/openvpn with the "inet nowait" option.

What happens instead:
xinetd properly spawns openvpn but then openvpn aborts with: "Assertion failed at socket.c:629"
the following is what shows up in daemon.log:
OpenVPN 2.1.0 x86_64-pc-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [MH] [PF_INET6] [eurephia] built on Jul 20 2010
IMPORTANT: OpenVPN's default port number is now 1194, based on an official port number assignment by IANA. OpenVPN 2.0-beta16 and earlier used 5000 as the default port.
NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Diffie-Hellman initialized with 2048 bit key
/usr/bin/openssl-vulnkey -q -b 2048 -m <modulus omitted>
Control Channel Authentication: using '/etc/openvpn/ta.key' as a OpenVPN static key file
Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
LZO compression initialized
Control Channel MTU parms [ L:1592 D:168 EF:68 EB:0 ET:0 EL:0 ]
TUN/TAP device tap0 opened
TUN/TAP TX queue length set to 100
/etc/openvpn/per-user-up.sh tap0 1500 1592 init
Data Channel MTU parms [ L:1592 D:1450 EF:60 EB:135 ET:32 EL:0 AF:3/1 ]
Local Options hash (VER=V4): '3d4363c6'
Expected Remote Options hash (VER=V4): '29f6c8b2'
GID set to nogroup
UID set to nobody
Assertion failed at socket.c:629
------------------------------------------------------------------
This problem does not exist upstream. To fix the problem I simply compiled and installed openvpn-2.1.3 from source and installed it to /usr/local. I then updated my xinetd service file to point to /usr/local/sbin/openvpn and the problem went away.
------------------------------------------------------------------

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: openvpn 2.1.0-1ubuntu1.1 [modified: usr/sbin/openvpn]
Uname: Linux 2.6.35.4-quick x86_64
Architecture: amd64
Date: Wed Sep 1 06:27:14 2010
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: openvpn

[Thierry Carrez (ttx)]

This is probably an issue with the debian ipv6 support patch (jjo-ipv6-support.patch).
Might be the same as http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=574164

[Thierry Carrez (ttx)]

Could you attach your openvpn configuration ? In particular, I'm interested in seeing if you use "mode p2p".

[Jarrett Miller (spook)]

Not using p2p mode.
Here is my server config file
--------------------------------------------
script-security 3
inetd nowait
proto tcp-server
tls-server
dev tap
up "/etc/openvpn/per-user-up.sh"
down "/etc/openvpn/per-user-down.sh"

ca /etc/openvpn/ca.crt
cert /etc/openvpn/server.crt
key /etc/openvpn/server.key # This file should be kept secret
dh /etc/openvpn/dh2048.pem

keepalive 10 120

tls-auth /etc/openvpn/ta.key 0 # This file is secret

cipher AES-128-CBC # AES

comp-lzo

user nobody
group nogroup

verb 4

[Thierry Carrez (ttx)]

There is no specific "mode" set (and "server" is not specified), so it uses p2p mode by default. I'll have a deeper look into this.

[Jarrett Miller (spook)]

Here are my related config files. Hopefully these will help as well.

per-user-up.sh
------------------------
#!/bin/bash

#################################
# Set up Ethernet bridge on Linux
# Requires: bridge-utils
#################################

# Define Bridge Interface
br="bridge0"

#NOTE $1 contains the tap interface name

brctl addif $br $1
ifconfig $1 mtu $2 promisc up

--------------------------
per-user-down.sh
--------------------------
#!/bin/bash

#################################
# Tear down Ethernet bridge on Linux
# Requires: bridge-utils
#################################

# Define Bridge Interface
br="bridge0"

# NOTE #1 contains the tap devce name

brctl delif $br $1
ifconfig $1 down
openvpn --rmtun --dev $1

-----------------------------
/etc/xinet.d/openvpn
----------------------------
service echo
{
 disable = no
 type = UNLISTED
 port = 1194
 socket_type = stream
 protocol = tcp
 user = root
 wait = no
 server = /usr/sbin/openvpn
 server_args = --config /etc/openvpn/tcp-tap.conf.backup
}
-----------------------------
The funny name for the server_args value is to prevent the SYSVINIT scripts from trying to start a server that should only be started by xinet.d.

I have a bridge created in my /etc/networking/interfaces file called bridge0

[Thierry Carrez (ttx)]

The IPv6 patch has a case where it would hit that ASSERT if ! (host && addr) -- while the upstream code would just pass without triggering any error. I'll prepare a package for you to test in a few, maybe it's just that corner case.

[Jarrett Miller (spook)]

Ok I would be happy to test. Not sure if its an issue but that system is now running Maverick and no longer runs Lucid. So any test won't be apples to apples.

[Thierry Carrez (ttx)]

I suppose you reproduce the issue in maverick, so I'll build packages for Maverick.
Package will build @ https://launchpad.net/~ttx/+archive/ppa

[Thierry Carrez (ttx)]

Arh, this needs a bit more work -- more tomorrow :)

[Jarrett Miller (spook)]

No worries. I cobbled together a Lucid system to test this so no need to worry about Maverick anymore.
I just wanted to thank you for the prompt attention you have shown this bug.

Are you a Canonical employee? If so is there anyway I can provide feedback to let your employer know how pleased I am with the service you have provided? I like to make sure hard work is rewarded whenever I can.

Cheers

[Thierry Carrez (ttx)]

Test packages have been uploaded to my PPA (for Lucid and Maverick) at [1]... They will build in the next hours. If those fix it, then it's an easy fix and I can push it to lucid and maverick. If not, then I'm a bit more clueless as to what happens here...

Yes, I'm a Canonical employee, but I'm helping you as an Ubuntu Server core developer... My pleasure to help, and your thanks are enough as a reward :)

[Thierry Carrez (ttx)]

[1] https://launchpad.net/~ttx/+archive/ppa

[Jarrett Miller (spook)]

I can't test them today due to work conflicts. With any luck I will get them tested on Friday.

Cheers

[Jarrett Miller (spook)]

I tested the AMD64 build on Lucid today. It fixes the socket assertion failure but still features a regression relative to upstream.

With the build from you PPA and the config file I provided I get the following in syslog:
TLS: Initial packet from [AF_INET]192.5.38.152:49308, sid=713f6ed1 b48b32c4
Authenticate/Decrypt packet error: packet HMAC authentication failed
TLS Error: incoming packet authentication failed from [AF_INET]192.5.38.152:49308
Fatal TLS error (check_tls_errors_co), restarting
OpenVPN started by inetd/xinetd cannot restart... Exiting.

-----------------------------------------------------------------
To fix it I had to comment out the "tls-auth" lines in my client and server configuration files.
For example I commented this out in my server configuration file:
# tls-auth /etc/openvpn/ta.key 0 # This file is secret

After doing that it works like upstream, except I am not getting the PSK protection of the TLS negotiation. So its still not working right but at least it no longer produces the socket assertion message.

[Thierry Carrez (ttx)]

Hmm, so the problem still lies in the ipv6 support patch from debian, but is not fixed by fixing the host && addr code flow. Probably your use case (usage through xinetd) hits code that triggered the assertion (host or addr is null) but will also trigger further failure...

[Thierry Carrez (ttx)]

I think that's the same issue as the debian bug mentioned above, and is probably an incompatibility between using inetd to spawn openvpn and the ipv6 patch from Debian.

In openvpn 2.0, "Server Mode" can be used to serve multiple clients from a single port (with scalability in mind, so all client connections are routed through a single tun or tap interface). Have you considered switching your inetd-based setup to that model ?

[Jarrett Miller (spook)]

Cant't do it. I need multiple clients through one port and each needs their own tap device. We use broadcast/multi-cast and stuff like wireshark. So the scalable tun approach is not for us. Basically everyone needs their own unique MAC address.

[Jarrett Miller (spook)]

UPDATE: I did some more testing on this. I found out that the TLS HMAC errors were the fault of an update to Viscosity (Mac OSX OpenVPN client). The program was incorrectly passing '0' for the direction of the "ta.key" file on the clients.

So the good news is that the package in your PPA fixes this issue completely. The TLS HMAC issue was unrelated.

[Thierry Carrez (ttx)]

Ah! That's good news indeed. I'll push the fix in SRU once the Maverick release heat calms down.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openvpn - 2.1.3-1ubuntu2

---------------
openvpn (2.1.3-1ubuntu2) natty; urgency=low

  * Fix jjo-ipv6-support.patch to avoid assertion failure at socket.c:629 in
    corner cases where ! host && addr (LP: #627973)
 -- Thierry Carrez (ttx) <email address hidden> Wed, 20 Oct 2010 16:22:25 +0200