Enabling FIPS breaks password hashing

[Dominik Zäuner]

We are running x11vnc 0.9.16-8 on a FIPS enabled Ubuntu 22.04 with libvncserver1 0.9.13+dfsg-3build2 and libssl3 3.0.2-0ubuntu1.12+Fips1.

With the fips=1 kernel parameter enabling FIPS, it seems that the password hashing is broken and only a clear text password is written:

$ cat /proc/sys/crypto/fips_enabled
1
$ x11vnc -storepasswd Abc /tmp/.testpw && cat /tmp/.testpw
stored passwd in file: /tmp/.testpw
Abc

Any connection attempt fails with a 'password check failed!' error.

Running x11vnc with

sudo /usr/bin/x11vnc -auth guess -forever -localhost -loop -noxdamage -repeat -rfbauth /root/.vncpasswd -rfbport 5900 -shared

logs the following:

Got connection from client 127.0.0.1
  0 other clients
Normal socket connection
check_access: client 127.0.0.1 matches host 127.0.0.1
incr accepted_client=1 for 127.0.0.1:54968 sock=10
Client Protocol Version 3.8
Protocol version sent 3.8, using 3.8
rfbProcessClientSecurityType: executing handler for type 2
Couldn't read password file: /root/.vncpasswd
rfbAuthProcessClientMessage: password check failed
rfbClientSendString("password check failed!")
client_count: 0
Client 127.0.0.1 gone

By turning off FIPS with fips=0 in the kernel, it works as expected:

$ cat /proc/sys/crypto/fips_enabled
0
$ x11vnc -storepasswd Abc /tmp/.testpw && cat /tmp/.testpw
stored passwd in file: /tmp/.testpw
�97l܊

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.