socat bug with SSL "file transfers"

[Sean Reifschneider]

I'm running ganeti 3.0.1 on Ubuntu 20.04, and running into issues with moving instances between host nodes. Under the hood, this use socat to copy the disk data from one node to another. In 20.04, the copy seems to copy nearly all the data, but then at the very end it fails with:

    Wed Jul 14 19:57:00 2021 - WARNING: import 'import-disk0-2021-07-14_19_52_25-8ockgvr6' on gnt1.example.com failed: Exited with status 1
    Wed Jul 14 19:57:00 2021 disk/0 failed to receive data: Exited with status 1 (recent output: dd: warning: partial read (65494 bytes); suggest iflag=fullblock\nsocat: E SSL_read():Connection reset by peer\n0+980142 records in\n0+980142 records out\n21976203264 bytes (22 GB, 20 GiB) copied, 273.039 s, 80.5 MB/s)

The copy seems to be almost 2MB short, expected size is 21,978,152,960 the copy above reports it copied 21,976,203,264

I've tried to isolate this down to a reproducable test case that does not require ganeti, but I don't seem to have the socat chops to make it happen. I'll list what I have so far down below.

I believe this is an issue with socat version 1.7.3.3-2 because:

- If I install socat_1.7.3.2-2ubuntu2_amd64.deb from 18.04 or socat_1.7.4.1-3ubuntu1_amd64.deb from 21.04 (just download those packages and "dpkg -i", no dependency issues are reported), my VM copy will succeed.
- On the socat website ( http://www.dest-unreach.org/socat/ ) it notes that "Socat version 1.7.4.1 fixes [...] file transfer with OpenSSL.". That seems like exactly what ganeti is doing, SSL bulk transfers.

The actual commands being run by ganeti are:

    bash -o errexit -o pipefail -c { echo -E -n M=b0f141f7085de052088d687fc70a027e53928be8 &&{ LC_ALL=C dd bs=1048576 <&0 2>&6 & pid=${!}; echo $pid >&8; wait $pid; } } | /usr/bin/socat -ls -d -d -b1048576 -u stdin OPENSSL:10.1.1.1:38219,connect-timeout=20,retry=10,intervall=1,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=1,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/var/lib/ganeti/server.pem,cert=/var/lib/ganeti/server.pem,cafile=/var/run/ganeti/import-export/export-disk0-2021-07-14_19_52_30-ehngj3q8/ca,pf=ipv4,openssl-commonname=ganeti.example.com 2>&4

and:

    bash -o errexit -o pipefail -c /usr/bin/socat -ls -d -d -b1048576 -u OPENSSL-LISTEN:0,reuseaddr,forever,intervall=0.01,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=1,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/var/lib/ganeti/server.pem,cert=/var/lib/ganeti/server.pem,cafile=/var/run/ganeti/import-export/import-disk0-2021-07-14_19_52_25-8ockgvr6/ca,pf=ipv4 stdout 2>&4 | { { read -n 42 magic && if test "$magic" != M=b0f141f7085de052088d687fc70a027e53928be8; then echo 'Magic value mismatch' >&2; exit 1;fi; } && { LC_ALL=C dd bs=1048576 <&0 2>&6 & pid=${!}; echo $pid >&8; wait $pid; } }

I tried to simplify this down to a pair of socat commands like this:

    /usr/bin/socat -ls -d -d -b1048576 -u OPENSSL-LISTEN:12345,reuseaddr,forever,intervall=0.01,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=0,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/etc/ssl/private/ssl-cert-snakeoil.key,cert=/etc/ssl/certs/ssl-cert-snakeoil.pem,pf=ipv4 stdout | sha1sum &
    dd if=/usr/bin/perl bs=1048576 | /usr/bin/socat -ls -d -d -b1048576 -u stdin OPENSSL:127.0.0.1:12345,connect-timeout=20,retry=10,intervall=1,keepalive,keepidle=60,keepintvl=10,keepcnt=5,verify=1,cipher=HIGH:-DES:-3DES:-EXPORT:-DH,compress=none,key=/etc/ssl/private/ssl-cert-snakeoil.key,cert=//etc/ssl/certs/ssl-cert-snakeoil.pem

But this fails with: "ioctl(6, IOCTL_VM_SOCKETS_GET_LOCAL_CID, ...): Inappropriate ioctl for device" and my google searches didn't turn up what might be the problem.

I'm going to be just installing the 21.04 socat package on my ganeti+20.04 systems, as so far in my testing that seems to be working fine and will prevent me from having to pin the 18.04 packages.

[Brian Candler (b-candler)]

In the socat git repository, if you diff tag 1.7.4.0 to 1.7.4.1, you see this:

+Corrections:
...
+ Under certain conditions OpenSSL stream connections, in particular bulk
+ data transfer in unidirectional mode, failed during transfer or near
+ its with Connection reset by peer on receiver side.
+ This happened with Socat versions 1.7.3.3 to 1.7.4.0. Reasons were
+ lazy SSL shutdown handling on the sender side in combination with
+ SSL_MODE_AUTO_RETRY turned off.
+ Fix: After SSH_shutdown but before socket shutdown call SSL_read()
+ Test: OPENSSL_STREAM_TO_SERVER
+ Fixes Red Hat issue 1870279.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Danny Howard (dannyman)]

I am hitting this, too, also when using ganeti.

There's a workaround mentioned at https://groups.google.com/g/ganeti/c/BV8GvyN93w0 to pull the older socat package from Ubuntu 18.04 and install it via dpkg.

Workaround:

wget https://launchpad.net/ubuntu/+archive/primary/+files/socat_1.7.3.2-2ubuntu2_amd64.deb
sudo dpkg -i socat_1.7.3.2-2ubuntu2_amd64.deb

I tried pulling from jammy but could not meet the dependencies.