Cannot connect anymore to Azure Database for MySQL with 8.0.27-0ubuntu0.20.04

[Olivier Le Pogam]

Hello,

When I upgrade my Ubuntu 20.04 VM with with mysql-8.0 version 8.0.27-0ubuntu0.20.04, the connection to any of my Azure Database for MySQL is frozen (no timeout, but no response) using the mysql CLI.

Downgrading to 8.0.19 fixes the issue.

Of course, there is no network related issue (since 8.0.19 works fine).

Are the security enhancements the cause of this issue ? Sounds like the authentication negociation is failing and is stuck somewhere ...

Weird thing is the "no response" behavior, instead of having an error returned.

Best regards

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Richard (f-richaad-h)]

I've come across this too, and it's causing our Azure DevOps pipelines to fail as they are set to use ubuntu-latest. Trying to look into this locally stracing the pid shows that nothing is happening, it's completely dead. The server version at Azure is 8.0.15.

[Olivier Le Pogam (lepogam)]

The Azure Database for MySQL Servers we can't connect to are :
MySQL version : 5.7
Performance configuration : General Purpose, 2 vCore(s), 100 GB
With TLS 1.2 enforced

[Olivier Le Pogam (lepogam)]

Any help we could provide to go ahead ?

[Utkarsh Gupta (utkarsh)]

Hi Olivier,

Thanks for filing the bug. Can you please mention the steps for reproducing? This way we can easily reproduce and see where did things go wrong.

Thank you!

[Olivier Le Pogam (lepogam)]

Hi Utkarsh !

Quite simple, but unfortunately an Azure Environment is required to reproduce it ...
- A simple VM with Ubuntu 20.04
- An instance of Azure Database for MySQL (which is actually a PaaS MySQL) : https://azure.microsoft.com/en-us/pricing/details/mysql/server/

Latest mysql-8.0 libraries

Then simply try to connect with "mysql" command line to the Azure Database for MySQL => you'll be stuck.

If you don't have such en environment, I can open a free trial one, create a VM & an Azure Database for MySQL Service, and we can work on it together.

Best regards

[Utkarsh Gupta (utkarsh)]

Hi Olivier,

Thank you, just to let you know, we're investigating this and will get back as soon as we can. \o/

[Entertainment Media Group AG (emgag)]

Upstream bug: https://bugs.mysql.com/bug.php?id=105288

This regression seems to be independent of the OS and introduced in 8.0.27, versions <8.0.27 are unaffected (downgrading to 8.0.26-0ubuntu0.20.04.2 fixed it for me).

[navjeet (navjeetc)]

How do downgrade to version 8.0.26-0ubuntu0.20.04.2 of mysql-client.
apt-install for this version on Ubuntu 20.04.3 LTS returns package not found.

sudo apt-get install mysql-client=8.0.26-0ubuntu0.20.04.2
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Version '8.0.26-0ubuntu0.20.04.2' for 'mysql-client' was not found

[Entertainment Media Group AG (emgag)]

You can download the specific version from launchpad: https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+build/21811814 (I don't think there's an APT repository you can use for this, but I might be wrong)

What I ended up doing after the quick'n dirty downgrade workaround though is to switch to the mariadb client instead, which is in the repo and can connect to Azure MySQL without any issues.

[Christian Ehrhardt  (paelzer)]

Hi,
this is a really sad case, but as Ubuntu I'm not seeing what else we could do here than leaving things as is. The delivery of security fixes is important to the majority of users.

In this particular case in the referenced upstream bug and further links from there [2] indicate that Microsoft <-> Mysql have given up to implement something on the Azure DB side to work with the security-fixed version of mysql. But on the other hand we can not really revert that change to provide mysql re-opening known security issues.

It seems in [2] that Microsoft has settled on "then use Flexible Server instead of Single Server". I have to admit I do not know what all that implied (more cost, data migration, ... it does not seem to be as drop-in replacement to me :-/) but right now as sad as it is this seems to be the only way out of this for affected users.

I see that @emgag continues to use the older builds, while that works it means you lack CVE fixes and with every new upload there will be more that you miss. So please consider using that only as fallback until you have made a transition to the "Flexible Server" or other alternatives.

If there is a way to retain the CVE fix, but a follow up change allows to use the "Azure DB for Mysql - Single server" again please speak up and let us know. But unless such a thing exists this has to be "incomplete" as no one can really act on it for an Ubuntu upload to fix it.

[1]: https://bugs.mysql.com/bug.php?id=105288
[2]: https://docs.microsoft.com/en-us/azure/mysql/concepts-compatibility#mysql-drivers

[Entertainment Media Group AG (emgag)]

Just to clarify: We didn't continue to use the older build, this was just a workaround to fix an immediate downtime after the upgrade. What worked for us long term is to use the mariadb client instead, which is unaffected and still works with Single Server Azure MySQL. This might not be an option for everyone though - it's disappointing that neither MySQL nor Microsoft are willing to address this issue.