OpenSSH should support VIA PadLock

[Pascal de Bruijn]

VIA PadLock is a hardware cryptography engine for AES and SHA1/256.

OpenSSH should support PadLock. Upstream OpenSSH versions do support padlock, and a working patch exists in openssh-portable bug #1437 (see remote bug watches) as of 4.9p1. The patch applies and works fine in 4.7p1.

A small bugfix patch to the version of OpenSSL in Ubuntu is also required for this to work. The bugfix is included in OpenSSL upstream 0.9.8h. It applies fine and works fine on 0.9.8g.

Initial work on PadLock support was done some time ago:
http://www.logix.cz/michal/devel/padlock/

[Colin Watson (cjwatson)]

With OpenSSH >= 4.4, we just need to configure --with-ssl-engine.

[Colin Watson (cjwatson)]

openssh (1:4.6p1-1) unstable; urgency=low

  * New upstream release (closes: #395507, #397961, #420035). Important
    changes not previously backported to 4.3p2:
    - 4.4/4.4p1 (http://www.openssh.org/txt/release-4.4):
      + On portable OpenSSH, fix a GSSAPI authentication abort that could be
        used to determine the validity of usernames on some platforms.
      + Implemented conditional configuration in sshd_config(5) using the
        "Match" directive. This allows some configuration options to be
        selectively overridden if specific criteria (based on user, group,
        hostname and/or address) are met. So far a useful subset of
        post-authentication options are supported and more are expected to
        be added in future releases.
      + Add support for Diffie-Hellman group exchange key agreement with a
        final hash of SHA256.
      + Added a "ForceCommand" directive to sshd_config(5). Similar to the
        command="..." option accepted in ~/.ssh/authorized_keys, this forces
        the execution of the specified command regardless of what the user
        requested. This is very useful in conjunction with the new "Match"
        option.
      + Add a "PermitOpen" directive to sshd_config(5). This mirrors the
        permitopen="..." authorized_keys option, allowing fine-grained
        control over the port-forwardings that a user is allowed to
        establish.
      + Add optional logging of transactions to sftp-server(8).
      + ssh(1) will now record port numbers for hosts stored in
        ~/.ssh/known_hosts when a non-standard port has been requested
        (closes: #50612).
      + Add an "ExitOnForwardFailure" option to cause ssh(1) to exit (with a
        non-zero exit code) when requested port forwardings could not be
        established.
      + Extend sshd_config(5) "SubSystem" declarations to allow the
        specification of command-line arguments.
      + Replacement of all integer overflow susceptible invocations of
        malloc(3) and realloc(3) with overflow-checking equivalents.
      + Many manpage fixes and improvements.
      + Add optional support for OpenSSL hardware accelerators (engines),
        enabled using the --with-ssl-engine configure option.
      + Tokens in configuration files may be double-quoted in order to
        contain spaces (closes: #319639).
      + Move a debug() call out of a SIGCHLD handler, fixing a hang when the
        session exits very quickly (closes: #307890).
      + Fix some incorrect buffer allocation calculations (closes: #410599).
      + ssh-add doesn't ask for a passphrase if key file permissions are too
        liberal (closes: #103677).
      + Likewise, ssh doesn't ask either (closes: #99675).
    - 4.6/4.6p1 (http://www.openssh.org/txt/release-4.6):
      + sshd now allows the enabling and disabling of authentication methods
        on a per user, group, host and network basis via the Match directive
        in sshd_config.
      + Fixed an inconsistent check for a terminal when displaying scp
        progress meter (closes: #257524).
      + Fix "hang on exit" when background proces...

[John Steele Scott (toojays)]

Unfortunately this doesn't actually work.

I run the following test:

scp -c aes128-cbc 100meg.test localhost:/dev/null

And the throughput is the same with openssh 4.6 as it was with openssh 4.3. (6.0 MB/s on my EN12000E board).

[John Steele Scott (toojays)]

The original implementation of --with-ssl-engine doesn't seem to me to do anything at all, because it relies on an ugly re'#define which is not included anywhere it's needed. Even with that included in the appropriate spots, the padlock engine is still not being used.

I've attached a "works for me" kind of patch. It adds the necessary includes so that --with-ssl-engine actually does something, and it also explicitly sets the padlock engine as the preferred cipher provider. Now I can get throughput of 13.6MB/s when performing the above scp test.

[Ian Lister (ubuntu-bugs-lister)]

FWIW I've reported this (with a patch) upstream:

    https://bugzilla.mindrot.org/show_bug.cgi?id=1437

BTW John, did you actually find it necessary to explicitly set the padlock engine as the default? For me it has the expected speedup without, as long as the engine is loaded and registered.

[John Steele Scott (toojays)]

I've just revisited this after upgrading my system to hardy. I still need to apply my patch to openssl version 4.7 from hardy, including the call to ENGINE_set_default_ciphers(ENGINE_by_id("padlock")). Without that call, I was still only getting about 5.6 MB/s.

Ian, are you passing any command line flags, or have you changed a configuration file to tell OpenSSH to use padlock? Did you build against a patch openssl or something?

[Ian Lister (ubuntu-bugs-lister)]

From the OpenSSL engine documentation at http://www.openssl.org/docs/crypto/engine.html :

> Automatically using builtin ENGINE implementations
>
> Here we'll assume we want to load and register all ENGINE implementations
> bundled with OpenSSL, such that for any cryptographic algorithm required by
> OpenSSL - if there is an ENGINE that implements it and can be initialise, it
> should be used. The following code illustrates how this can work;
>
> /* Load all bundled ENGINEs into memory and make them visible */
> ENGINE_load_builtin_engines();
> /* Register all of them for every algorithm they collectively implement */
> ENGINE_register_all_complete();
>
> That's all that's required. Eg. the next time OpenSSL tries to set up an RSA
> key, any bundled ENGINEs that implement RSA_METHOD will be passed to
> ENGINE_init() and if any of those succeed, that ENGINE will be set as the
> default for RSA use from then on.

From the above, as long as the Padlock engine loads successfully it should be used where possible. The output of "openssl engine" should indicate whether the engine is able to load successfully, and calling ENGINE_set_default_ciphers() shouldn't affect what's loaded or not loaded.

At the time I posted my previous comment I was testing with OpenSSH 4.7p1 on FreeBSD 6.2 and its system OpenSSL 0.9.7e-p1. Since then I've updated the machine to FreeBSD 7.0 (with its system OpenSSL 0.9.8e). Like you, I do *not* now see any acceleration unless I add a call to ENGINE_set_default_ciphers(). Further, a call to ENGINE_get_cipher_engine(NID_aes_128_ecb) immediately prior to calling ENGINE_set_default_ciphers() returns NULL, but a call immediately afterwards returns non-NULL.

I thought I previously didn't need to call ENGINE_set_default_ciphers() at all (and that I was seeing acceleration without any configuration or other code changes), but it's possible I confused myself and only ever tested a version with an explicit ENGINE_set_default_ciphers() call. It's also possible that the behaviour changed between versions. Either way the current behaviour we're both seeing doesn't match the documentation above, which would be a bug in OpenSSL. I'll have a further dig when I get a chance (unless you or somebody else beats me to it).

[Ian Lister (ubuntu-bugs-lister)]

At first glance it looks like the OpenSSL engine code has broken caching logic (in crypto/engine/eng_table.c) in all versions of 0.9.8, but is reversed and correct in all versions of 0.9.7. I haven't tested it, but I suspect the attached patch will fix the issue.

[Ian Lister (ubuntu-bugs-lister)]

I've now tested the above patch and it seems to do the trick, so I've sent it in upstream to the OpenSSL RT system.

[Craig Ringer (ringerc)]

That patch applies to openssl as shipped in Hardy, but doesn't appear to have any effect. After patching openssl, rebuilding the packages and installing them, `openssl engine padlock' reports:

(padlock) VIA PadLock (no-RNG, ACE)

on my C3 thin clients. That should get me at least accelerated aes-128, but yet:

openssl speed aes-128-cbc -engine padlock

loads the padlock engine successfully but does NOT appear to be using its crypto facilities. Performance remains miserable, around 10MB/s of aes-128-cbc throughput. SSH gets ~5 MB/s throughput, which seems reasonable given the other overheads it faces.

If I build openssl-0.9.8h upstream and test with that I also see no performance change.

The processor DOES report ACE support. CPUinfo:

flags : fpu vme de pse tsc msr cx8 sep mtrr pge cmov pat mmx fxsr sse up rng rng_en ace ace_en

There's no change when the padlock-aes module is loaded (but it shouldn't be needed for openssl/openssh anyway, as padlock is done in userspace with CPU instruction extensions).

It's like OpenSSL is silently falling back to software crypto at some level. I haven't dug into it deeply yet, but I thought it important to mention that the proposed fix does NOT appear to work on my hardware.

This support is *REALLY* important for use of C3/C7 machines as LTSP thin clients, because currently the X server and network (via ssh) fight for CPU, severely limiting performance. Using the hardware crypto should massively reduce SSH's CPU demands and dramatically boost performance.

Full CPUInfo:

processor : 0
vendor_id : CentaurHauls
cpu family : 6
model : 9
model name : VIA Nehemiah
stepping : 8
cpu MHz : 666.577
cache size : 64 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr cx8 sep mtrr pge cmov pat mmx fxsr sse up rng rng_en ace ace_en
bogomips : 1334.91
clflush size : 32

[Craig Ringer (ringerc)]

Confirmed: openssl is using x86 crypto even with "-engine padlock", both in latest upstream and in the current hardy packages.

If I interrupt a debug build of openssl while running "openssl speed aes-128-cbc -engnie padlock" on a C3 gdb generally reports that it's been interrupted in:

_x86_AES_encrypt () at ax86-elf.s:55

The engine loads fine, and "openssl engine padlock" confirms it's detecting hardware support, but for some reason it's not actually using it.

[Craig Ringer (ringerc)]

** These bugs are fixed upstream in OpenSSH 4.9 and OpenSSL 0.9.8h **

You can apply the fix to OpenSSH 4.7 from Ubuntu just fine: https://bugzilla.mindrot.org/attachment.cgi?id=1458 . It applies cleanly except for two rejects at points where the changes have already been applied, so the rejects can be safely ignored. With the patch I get 100Mbit wire speed with the aes128-cbc cipher. You will also need to apply Ian Lister's OpenSSL patch above.

PLEASE merge both these patches (the openssl cache logic fix and the openssh engine init fix) for the next hardy update.

I can confirm that with both patches OpenSSH performs vastly better and with much lower CPU use.

As for why the tests I was doing weren't working:

It's necessary to specify "-evp aes-128-cbc" instead of just "aes-128-cbc" to get an engine to work; just passing "-engine padlock" is insufficient. The "engine" argument requests loading of a given engine, but doesn't tell "openssl speed" to use the engine system; it still calls the AES code directly. -evp tells openssl speed to use the engine system, but doesn't say anything about which engine.

/usr/bin/openssl speed -evp aes-128-cbc -engine padlock:
aes-128-cbc 30934.33k 102451.76k 251594.56k 391449.69k 468731.35k

/usr/bin/openssl speed aes-128-cbc -engine padlock:
aes-128 cbc 6827.95k 9055.61k 9926.85k 10172.77k 10244.14k

[Craig Ringer (ringerc)]

Quick instructions on rebuilding openssh and openssl to include the fix, for those not used to patching Debian packages:

<pre>
mkdir wrk
cd wrk
sudo apt-get install build-essential fakeroot wget
sudo apt-get build-dep openssl openssh
apt-get source openssl openssh
cd openssl-0.9.8g
wget --quiet -O - http://launchpadlibrarian.net/13798833/bug119295.patch | patch -p1
fakeroot debian/rules binary
cd ../openssh-4.7p1
wget --quiet --no-check-certificate -O - https://bugzilla.mindrot.org/attachment.cgi?id=1458 | patch -p0 -N
fakeroot debian/rules binary
</pre>

... then install the generated debs, or at least openssh-client_4.7p1-8ubuntu1.2_i386.deb and libssl0.9.8_0.9.8g-4ubuntu3.1_i386.deb . You'll find them in the `wrk' directory.

If you're doing this for LTSP thin clients remember to install the debs in the /opt/ltsp/i386 chroot then re-run ltsp-build-image .

[Colin Watson (cjwatson)]

See also bug 119294 for the OpenSSL side of things.

[Colin Watson (cjwatson)]

Fixed via OpenSSH 4.9p1. I'll be aiming to get 5.1p1 into Intrepid.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssh - 1:5.1p1-1ubuntu1

---------------
openssh (1:5.1p1-1ubuntu1) intrepid; urgency=low

  * Resynchronise with Debian. Remaining changes:
    - Add support for registering ConsoleKit sessions on login.
    - Drop openssh-blacklist and openssh-blacklist-extra to Suggests; they
      take up a lot of CD space, and I suspect that rolling them out in
      security updates has covered most affected systems now.

openssh (1:5.1p1-1) unstable; urgency=low

  * New upstream release (closes: #474301). Important changes not previously
    backported to 4.7p1:
    - 4.9/4.9p1 (http://www.openssh.com/txt/release-4.9):
      + Added chroot(2) support for sshd(8), controlled by a new option
        "ChrootDirectory" (closes: #139047, LP: #24777).
      + Linked sftp-server(8) into sshd(8). The internal sftp server is used
        when the command "internal-sftp" is specified in a Subsystem or
        ForceCommand declaration. When used with ChrootDirectory, the
        internal sftp server requires no special configuration of files
        inside the chroot environment.
      + Added a protocol extension method "<email address hidden>" for
        sftp-server(8) to perform POSIX atomic rename() operations; sftp(1)
        prefers this if available (closes: #308561).
      + Removed the fixed limit of 100 file handles in sftp-server(8).
      + ssh(8) will now skip generation of SSH protocol 1 ephemeral server
        keys when in inetd mode and protocol 2 connections are negotiated.
        This speeds up protocol 2 connections to inetd-mode servers that
        also allow Protocol 1.
      + Accept the PermitRootLogin directive in a sshd_config(5) Match
        block. Allows for, e.g. permitting root only from the local network.
      + Reworked sftp(1) argument splitting and escaping to be more
        internally consistent (i.e. between sftp commands) and more
        consistent with sh(1). Please note that this will change the
        interpretation of some quoted strings, especially those with
        embedded backslash escape sequences.
      + Support "Banner=none" in sshd_config(5) to disable sending of a
        pre-login banner (e.g. in a Match block).
      + ssh(1) ProxyCommands are now executed with $SHELL rather than
        /bin/sh.
      + ssh(1)'s ConnectTimeout option is now applied to both the TCP
        connection and the SSH banner exchange (previously it just covered
        the TCP connection). This allows callers of ssh(1) to better detect
        and deal with stuck servers that accept a TCP connection but don't
        progress the protocol, and also makes ConnectTimeout useful for
        connections via a ProxyCommand.
      + scp(1) incorrectly reported "stalled" on slow copies (closes:
        #140828).
      + scp(1) date underflow for timestamps before epoch.
      + ssh(1) used the obsolete SIG DNS RRtype for host keys in DNS,
        instead of the current standard RRSIG.
      + Correctly drain ACKs when a sftp(1) upload write fails midway,
        avoids a fatal() exit from what should be a recoverable condition.
      + Fixed ssh-keygen(1) selective host key hashing (i.e. "ssh...

[flamacue (lessonno25)]

Launchpad Janitor wrote on 2008-07-25:

"This bug was fixed in the package openssh - 1:5.1p1-1ubuntu1"

Will this fix be put into Ubuntu 8.04 (Hardy)? I have a low power machine that I think I would like to keep running Hardy, since it's an LTS release, but I don't want to do without this feature.

Thanks.

[flamacue (lessonno25)]

Addendum: I would be happy enough if it was Hardy backports, vs. the distro proper.