[Jammy] NetworkManager-openconnect 1.2.6 not compatible with openconnect 8.20

[Ernst Persson]

This bug only affects the specific combination of network-manager-openconnect and openconnect that ended up in Jammy.

openconnect 8.20 breaks compatibility with NetworkManager-openconnect 8.20:

"As of openconnect 8.20, INTERNAL_IPx_NETMASK can be set to 0.0.0.0 and
/0 and this causes network manager to fail with a bad IP configuration.
This happens because 0.0.0.0/0 is set as a split route, but rewritten to
be used as netmask instead.
If we detect this we force a /32 or /128 (IPv6) netmask prefix and avoid
setting the CONFIG_NEVER_DEFAULT options."

This commit was reverted because the upstream devs intention is to always be backwards compatible. Later the feature was implemented again in another way.

So the best way forward for Jammy is to revert the openconnect commit.

Working on making an SRU from this...

[Impact]
Users with a common GlobalProtect serverside configuration will not be able to connect.

This is caused by an backwards incompatible change in openconnect between openconnect and network-manager-openconnect, that adds the ability for NetworkManager to override the server-provided configuration for split VPN.

The debdiff fixes it by reverting this change.

[Test Plan]
A GlobalProtect server is needed to test it, so perhaps we can collect reports from affected users.

This follows upstream fixes only.

[Where problems could occur]
Other packages in the Ubuntu archive can depend on the feature, potentially causing regressions, or have other regressions due to the change. However, this is extremely unlikely as the feature was introduced in the Ubuntu archive on 21 February 2022, that is only 3 days before Debian Import Freeze for Ubuntu 22.04 (24 February 2022).

It is also possible that third-party software (outside of the Ubuntu archive) depends on the feature. However, the feature only affects GlobalProtect VPNs with the common server-side configuration mentioned above, where NetworkManager-openconnect is unusable due to the feature, and was reverted shortly thereafter in release 9.01, released on 29 April. Therefore, it is unlikely that such software is using the feature, even if the software in question is a workaround for this bug.

It is also possible that users have configured their systems in a way that depends on this feature. However, this is extremely unlikely.

Several users have commented on this bug that they can connect to GlobalProtect VPNs with this common server-side configuration using openconnect directly, and they are not relying on the feature. The only way that I see that a workaround could use the feature is to detect if the feature is present, to check if the workaround is needed.

[Other Info]
There is no Debian release with this combination of versions so we can't import the fix from there.

[Ernst Persson (ernstp)]

Upstream has identified the bug and solved it in 1.2.9+:

https://gitlab.gnome.org/GNOME/NetworkManager-openconnect/-/merge_requests/31

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[dwmw2 (dwmw2)]

We considered this a regression in OpenConnect and it is fixed in the 9.01 release.

We also made NetworkManager more resilient but don't wait for that.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[boris.zubanov (boris.zubanov-redhat-bugs)]

Description of problem:

Using native Gnome network vpn connection applet, attempt to establish vpn connection with PaloAlto gateway via GlogalProtect protocol fails with "invalid IP4 config received: no valid IP address/prefix" error. At same time CLI openconnect client does that flawlessly as well as native PA's GlobalProtect client.

Version-Release number of selected component (if applicable):

How reproducible:

Just try to establish vpn connection.

Steps to Reproduce:
1. Open Gnome Settings applet
2. Go to Network category
3. Add VPN connection with + button
4. Fill in gateway FQDN
5. Click Save
6. Turn on just created vpn connection
7. Enter creds

Actual results:

Gnome's notification "Connection failed" appears, no vpn connection established.

Expected results:

VPN connection's up and running

Additional info:

journalctl part of failure:

Jun 07 15:02:18 [HOSTNAME] NetworkManager[12054]: Configured as 100.64.98.207, with SSL disconnected and ESP established
Jun 07 15:02:18 [HOSTNAME] NetworkManager[12054]: Session authentication will expire at Tue Jun 14 15:02:16 2022
Jun 07 15:02:18 [HOSTNAME] openconnect[12054]: SIOCSIFMTU: Operation not permitted
Jun 07 15:02:19 [HOSTNAME] NetworkManager[1200]: <warn> [1654603339.0172] vpn[0x55cf89824350,664d128e-3224-4fd9-aa24-afe6036c010d,"[VPN]",if:6,dev:4:(vpn0)]: invalid IP4 config received: no valid IP address/prefix
Jun 07 15:02:19 [HOSTNAME] NetworkManager[1200]: <warn> [1654603339.0173] vpn[0x55cf89824350,664d128e-3224-4fd9-aa24-afe6036c010d,"[VPN]",if:6,dev:4:(vpn0)]: did not receive valid IP config information
Jun 07 15:02:19 [HOSTNAME] openconnect[12054]: Failed to spawn script '/usr/libexec/nm-openconnect-service-openconnect-helper' for connect: Interrupted system call
Jun 07 15:02:19 [HOSTNAME] openconnect[12054]: POST https://vpn-by.epam.com/ssl-vpn/logout.esp

Somewhere I've read that it can't digest some routing table entries being applied to the system during vpn connection and it seems to be true because it works to my other gates which have a bit simpler routing. But nevertheless.

[Ernst Persson (ernstp)]

I created a fix here:

https://launchpad.net/~ernstp/+archive/ubuntu/ppa/+packages?field.name_filter=&field.status_filter=published&field.series_filter=jammy

[Ernst Persson (ernstp)]

(new version with fixed lp: # changelog.)

[Christian Reis (kiko)]

Thanks Ernst -- let me test that.

(Just a nit, one thing to keep in mind though is that the version number you used for the package matches the version in the 22.04 LTS release; I think best practice for PPAs (it's been a while) is that you should change the suffix to something unique. But regardless thanks so much for the build)

[cedric.bellegarde (cedric.bellegarde-redhat-bugs)]

Fixed by https://gitlab.gnome.org/GNOME/NetworkManager-openconnect/-/issues/63

Should be awesome to have a fix for F36

[Christian Reis (kiko)]

It worked fine. As I use split networking, there's one thing I had to do which was to mark the "never-default" checkbox in the NetworkManager config, but as I'm upgrading from 20.04 LTS there was perhaps a behavior change that I hadn't noticed. The rest, split DNS etc works as expected. Thanks again!

[Luís Infante da Câmara (luis220413)]

This is the .diff.gz converted to .debdiff format and with the changelog fixed.

[Luís Infante da Câmara (luis220413)]

I just uploaded a patched version with an additional changelog entry to my PPA: https://launchpad.net/~luis220413/+archive/ubuntu/test-builds. Do not add this PPA to your system. Instead, download the package from the entry for the package under the "View package details" menu and install it with
  sudo apt install ./<relative path to .deb>
or
  sudo apt install /<absolute path to .deb>

[Łukasz Zemczak (sil2100)]

I have reviewed the debdiff, the proposed changes are sane and I was able to find the original change in the upstream repository. I'll sponsor it to the Unapproved queue, but before this can be reviewed and accepted as an SRU, me (and possibly other SRU members) would like to know:

 * What is the regression potential of this change? The [Where problems could occur] section needs to be filled in with regression analysis.
 * Related to the point above, since this fix basically reverts a bugfix, how big of an impact will this have on the users? Was the bug a very frequently encountered one? Seems more like an edge-case though? I'd like this info included in the bug description as well.

Thanks.

[Ernst Persson (ernstp)]

* since this fix basically reverts a bugfix, how big of an impact will this have on the users?

It's not really a bugfix that's being reverted, more of an optional feature. The patch enabled NetworkManager to override the server provided configuration for "split" VPN. But since it also is incompatible with the current version of NetworkManager-openconnect it wasn't a useful feature.

For those using GlobalProtect VPN which this relates to, this revert helps a 100% occurrence issue with a very common configuration.

[Luís Infante da Câmara (luis220413)]

The package was sponsored into the Unapproved queue on Tuesday (UTC), but the queue is generally processed from oldest to newest, there are 31 earlier items in the queue and the queue is soft-frozen for the release of 22.04.1, that is expected to occur on August 11.

[Chris Halse Rogers (raof)]

I think what I'm missing in reviewing this is some high-level description of what the patch does, and how it relates to the interaction of openconnect & network-manager-openconnect.

In the SRU team we get asked to review packages from all over the archive; the uploader presumably has a significantly deeper understanding of the package and how it fits into its immediate dependencies than we do. We might be missing system-level context that seems obvious to the uploader, so it's often useful for them to provide such context.

From the diff, it seems that the changes are in how translates the netmask option in the "config xml" into VPN config. But I don't know where the "config XML" comes from (is it NetworkManager? Is it sent over the wire from the VPN server? Is it read from an openconnect-specific config file, not set by anything else?).

From an SRU perspective, a significant concern of ours is “will this disrupt anyone's working system”. NetworkManager is not the only way to configure openconnect, right? How likely is it that a non-NetworkManager configuration could be using this feature? Since the bug seems to entirely break common NetworkManager/openconnect configurations, users are likely to have tried work-arounds. *Are* there any work-arounds for this bug, and how would they be affected by this revert? These are the sorts of questions we want answered in the [Where problems could occur] section.

[Ernst Persson (ernstp)]

You can read more about the developers reasoning for the revert in openconnect here:
https://gitlab.gnome.org/GNOME/NetworkManager-openconnect/-/merge_requests/36

"Handle default routes in split excludes
We attempted to 'fix' OpenConnect not to send these and to set the netmask
on the interface to 0 instead, but that caused compatibility problems which
we had to work around in commit 84e279cb ("src/helper: handle openconnect
8.20 netmask values.")
We want to revert that from the OpenConnect side as it's a regression, so
let's find a better way to achieve the original objective. Scan the split
includes to see if they include a default route. If they do, drop it from
the list we pass to NM explicitly, but don't set the never-default flag.
That should allow NM to honour the 'Use only for resources on this
connection' setting while still doing the right thing in other cases."

[Chris Halse Rogers (raof)]

Pointing to the upstream discussion is helpful, but upstream's priorities and reasoning doesn't necessarily apply to us. A fix can be absolutely correct in the upstream context but unacceptable as an SRU.

Here, we know this changes the interface openconnect presents to the rest of the system. This change is clearly fine for NetworkManager, but we want to know what *other* things this interface change might affect. Does connman interface with openconnect, and if it does, how will this affect VPNs set up with connman? Users can invoke openconnect directly, right? How will this change affect that? Might this break scripts users have set up to work around this bug? What would that look like?

This is the sort of analysis we want in the [Where problems could occur] section.

[Ernst Persson (ernstp)]

I'm still thinking about something to write regarding standalone openconnect and connman+openconnect...

However we can say this commit was very short lived and it was a bit of bad luck that it snuck into Ubuntu 22.04 LTS. It only exists in 8.20 and was reverted in the next release.

Maybe upstream developer @dwmw2 can explain more himself?

Here's more notes on upstreams thoughts:
https://gitlab.gnome.org/GNOME/NetworkManager-openconnect/-/issues/63#note_1437353

"No problem. Apologies for the regression. I think this is the best way forward — we revert the offending change in OpenConnect, and make NetworkManager-openconnect recognise the default route in the split includes and handle it accordingly. If we're going to have to update NM-openconnect anyway, we shouldn't have broken the compatibility in the first place. So this is what I'd like to merge into both projects:"

[Luís Infante da Câmara (luis220413)]

I have filed a nearly-empty main inclusion report (MIR) for these two packages (bug #1986592), referencing this bug report.

[Chris Halse Rogers (raof)]

The commit may have been short-lived upstream, and it is bad luck that 22.04 happened to pick it up, but that means that it's now the stable behaviour of the 22.04 release. Which doesn't make it *impossible* to do the revert, but it *does* mean we need to know the possible ramifications of the behaviour change on end-users who may¹ have setups which depend on this behaviour.

¹: Working out what “may” means here is basically the question I'm asking. ☺

[Steve Langasek (vorlon)]

Marking incomplete based on last comment from Chris.

[Luís Infante da Câmara (luis220413)]

I filed bug #1987707 to get OpenConnect support preinstalled in Ubuntu Desktop (and all other Ubuntu flavors; Kubuntu and Ubuntu Studio already have it).

[Gundy (nsgundy)]

Just to chime in here as a user who is affected by this bug:

Me (and some of my colleagues using Ubuntu) have upgraded from 21.10 to 22.04 shortly after its release. Right after upgrading, we noticed that we could no longer connect to the company VPN as we did before (using openconnect with a GlobalProtect VPN).

Workarounds we are using since:
1. Reverting to and pinning of libopenconnect5=8.10-2build1 openconnect=8.10-2build1 network-manager-openconnect=1.2.6-1 network-manager-openconnect-gnome=1.2.6-1 from Impish
2. Using the offical GlobalProtect VPN client
3. Using TeamViewer to work on a PC that sits inside of the company network

Hoping this upstream fix will make it into stable soon.

[Ernst Persson (ernstp)]

@nsgundy Can you test that the https://launchpad.net/~luis220413/+archive/ubuntu/test-builds/+build/24235590 build solves the issue for you also, and doesn't have any regressions?

[Luís Infante da Câmara (luis220413)]

My MIRs for network-manager-openconnect and its dependencies are ready.

[Luís Infante da Câmara (luis220413)]

For the .dsc file and for the changes file for a clean build of the new version, Lintian produces the following output:

W: openconnect-dbgsym: elf-error In program headers: Unable to find program interpreter name [usr/lib/debug/.build-id/7d/a995cf6bde8d118080489aebfebeeb66a280f5.debug]
N: 1 hint overridden (1 error); 0 unused overrides

The elf-error warning is due to bug #1977883.

[Robie Basak (racb)]

> The patch enabled NetworkManager to override the server provided configuration for "split" VPN. But since it also is incompatible with the current version of NetworkManager-openconnect it wasn't a useful feature.

Like Chris, I'm concerned about what happens to users who may be relying on this feature, for example by configuring it directly, without Network Manager. Is it possible that there are users in this situation? What will happen to them if this revert lands in Jammy?

I think this update remains blocked until someone performs this analysis.

[gerben (gerben-redhat-bugs-1)]

The above mentioned GitLab issue also mentions a 4 month old fix by that was tested by the reporter. I ran into this exact issue today so I compiled new packages based on that fix and I can confirm the VPN now works. Please build new packages for Fedora containing this fix.

[raphael.kubo.da.costa (raphael.kubo.da.costa-redhat-bugs)]

I ran into this problem today on a machine running Ubuntu, but another one running Fedora 36 and connecting to the same server worked just fine. I don't see any recent updates to either openconnect or NetworkManager-openconnect though, so this is at least confusing...

[Raphael Kubo da Costa (rakuco-intc)]

> @nsgundy Can you test that the https://launchpad.net/~luis220413/+archive/ubuntu/test-builds/+build/24235590 build solves the issue for you also, and doesn't have any regressions?

My company is trying out GlobalProtect, and I've run into this bug while testing things on Ubuntu 22.04. I can confirm that the builds from https://launchpad.net/~luis220413/+archive/ubuntu/test-builds/+build/24235590 solve the issue here and I was able to connect to the GlobalProtect gateway via Network Manager.

For the record, only invoking openconnect directly was working before, and it continues to with this patch.

[Mike Sollanych (msollanych)]

Connecting to a Cisco Firepower router with the stock Openconnect in Jammy via Gnome, I get a "timeout was reached"; openconnect at the commandline works fine, but nmcli or the Gnome graphical connector fail. This worked fine on Ubuntu 18.

The attached build at https://launchpad.net/~luis220413/+archive/ubuntu/test-builds/+build/24235590 fixes this problem for me.

[Steve Langasek (vorlon)]

An upload of openconnect to jammy-proposed has been rejected from the upload queue for the following reason: "unanswered questions about regression potential for > 2 months".

[projects.rg (projects.rg-redhat-bugs)]

Can you reproduce with F37 as well? My setup allows a connection to Cisco AnyConnect with SSO and in usage of OTP.

[Joshua Cornutt (oioooioi)]

#11 fixed the issue on Ubuntu 22.04.1 (bumping libopenconnect5 and openconnect to version 8.20-1ubuntu0.1~ppa1 over 8.20-1 from the Ubuntu repos). Without that fix, or a downgrade, OpenConnect was not usable within NetworkManager for any of my VPN endpoints.

[proussel (proussel-redhat-bugs)]

Still not working in F37.
fortinet VPN : works fine using openconnect command, fails with NetworkManager.
Same error : invalid IP4 config received: no valid IP address/prefix

(openconnect-9.01-3.fc37, NetworkManager-openconnect-1.2.8-3.fc37)

[tinotom (tinotom)]

Hi everybody,

can anybody please provide a status update on this bug ?
I have been following it since the beginning but also with 22.04.2 it still seem to be there.
Thank you very much,

[bcotton (bcotton-redhat-bugs)]

This message is a reminder that Fedora Linux 36 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 36 on 2023-05-16.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '36'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version'
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora Linux 36 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

[gerben (gerben-redhat-bugs-1)]

The just released NetworkManager-openconnect-1.2.10-1.fc38 fixed it for me. No more patching needed. Thank you to everyone involved.

[lsmid (lsmid-redhat-bugs)]

Fedora Linux 36 entered end-of-life (EOL) status on 2023-05-16.

Fedora Linux 36 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "openconnect_8.20-1_8.20-1ubuntu1.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Gundy (nsgundy)]

Sorry to have not responded when I was asked to, somehow missed that. When Ubuntu 22.10 got released, I upgraded to that and it's versions for openconnect and with that the problem was solved for me, no regressions. I have since moved to 23.04 and all is still good.

[Steve Langasek (vorlon)]

An SRU of this bug was rejected from the queue due to the lack of analysis here of the potential for regression. I am unsubscribing the ubuntu-sponsors team since there is at the moment nothing to sponsor, until those questions are addressed.

When the bug description has been updated with this information, please subscribe the ubuntu-sponsors team again.

[Stephen Carter (kennocha)]

I can report that this is still a problem on Ubuntu 23.04, and im a little bit confused on how that is possible.

Presently, you cannot use openconnect through networkmanager, due to the no valid prefix.

libopenconnect5/lunar,now 9.01-3 amd64 [installed,automatic]
network-manager-openconnect-gnome/lunar,now 1.2.8-3 amd64 [installed]
network-manager-openconnect/lunar,now 1.2.8-3 amd64 [installed]
openconnect/lunar,now 9.01-3 amd64 [installed]

I was able to grab some more recent 1.2.10-1 that made it so connections would work, but it broke other things.

So as of present, Ubuntu 23.04 fully up to date, is unable to use openconnect to establish VPN connections for me through the GUI due to this bug.

[Lukas Märdian (slyon)]

I see that the [Where problems could occur] section was updated with a regression analysis.
And I agree with comment #13 about the debdiff being reasonable.

So I'm sponsoring it into the Jammy queue again, so the SRU team can have another look into it.

PS: I also ran the "update-maintainer" script on the source packge before uploading. Unsubscribing ~ubuntu-sponsors.

[Luís Infante da Câmara (luis220413)]

@kennocha Is this a problem in Ubuntu 23.10? If yes, this is a blocker for the SRU, because the SRU policy (https://wiki.ubuntu.com/StableReleaseUpdates) requires that a bug to be fixed via an SRU be fixed in the development release first.

[Ernst Persson (ernstp)]

"This bug only affects the specific combination of network-manager-openconnect and openconnect that ended up in Jammy."

@luis220413 It was solved straight away in Kinetic 22.10 when it got openconnect 9.01.

@kennocha I would guess you have another problem, feel free to open a new bug report.

[Robie Basak (racb)]

> I see that the [Where problems could occur] section was updated with a regression analysis.
And I agree with comment #13 about the debdiff being reasonable.

> So I'm sponsoring it into the Jammy queue again, so the SRU team can have another look into it.

That section plainly does not address the questions raised above in the bug comments by Chris and by myself. For example, it does not consider scenarios where users might have configured direct use of functionality that you now propose to remove.

[Steve Langasek (vorlon)]

I am once again marking this bug incomplete.

It was marked 'confirmed' again after a set of changes to the bug description. But still, NOWHERE IN THE BUG HAS THE QUESTION BEEN ANSWERED:

Could a user, who has configured openconnect directly rather than using NetworkManager-openconnect, have a configuration which relies on the current (incorrect) behavior, and that configuration will then be broken as a result of this SRU?

That question needs a clear yes or no answer. If the answer is yes, then there either needs to be mitigation of this as part of a patch, or a rationale for why it's somehow acceptable to break the user's VPN configuration on upgrade (such as: that config wouldn't actually work and would be buggy in other ways).

It has wasted a lot of SRU Team time trying to get an answer to this question. If I find that this bug has been set back to 'confirmed' again without this question being clearly answered, I am going to reject the SRU.

[Luís Infante da Câmara (luis220413)]

@vorlon The answer to your question is: Yes, but it is extremely unlikely for a user to have such a configuration.

[Robie Basak (racb)]

> If the answer is yes, then there either needs to be mitigation of this as part of a patch, or a rationale for why it's somehow acceptable to break the user's VPN configuration on upgrade (such as: that config wouldn't actually work and would be buggy in other ways).

Sorry, "unlikely" is insufficient as an excuse to break users in a stable update, so I'm rejecting this upload. Perhaps a suitable mitigation is still possible, but those interested in driving this bug seem disinterested in that. Realistically it seems likely that this bug will therefore be fixed, so I'm setting this bug as Won't Fix to reflect this.

If somebody prepares a suitable patch and an Ubuntu developer considers this case suitably mitigated, then you can re-upload.

[Ernst Persson (ernstp)]

We are very interested in driving this, that's why I was very quick to file this only a few days after the Ubuntu 22.04 release.
However it's very hard to _guarantee_ that _nobody_ will be affected.

But the setup in Ubuntu 22.04 is clearly a mistake, confirmed by the upstream developers. At least we have a new LTS release around the corner now...