Ubuntu 23.10 cloud images unexpected UDP listening port 5353

[Philip Roche]

[Impact]

In the latest Ubuntu 23.10 cloud images we are seeing unexpected UDP listening port 5353.

By default and by policy, aside from port 22 there should be no other open ports on Ubuntu cloud images. Listening port 5353 is a regression.

[Test Plan]

Check that port 5353 is not open, and in particular that systemd-resolved is not listening on 5353. This is what it looks like when systemd-resolved *is* listening on 5353:

```
$ ss --listening --no-header --tcp --udp --numeric
udp UNCONN 0 0 127.0.0.54:53 0.0.0.0:*
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:*
udp UNCONN 0 0 10.154.0.17%ens4:68 0.0.0.0:*
udp UNCONN 0 0 127.0.0.1:323 0.0.0.0:*
udp UNCONN 0 0 0.0.0.0:5353 0.0.0.0:*
udp UNCONN 0 0 [::1]:323 [::]:*
udp UNCONN 0 0 [::]:5353 [::]:*
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.54:53 0.0.0.0:*
tcp LISTEN 0 4096 *:22 *:*
```

```
$ sudo lsof -i -n -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 153u IPv6 17848 0t0 TCP *:22 (LISTEN)
systemd-r 321 systemd-resolve 11u IPv4 16159 0t0 UDP *:5353
systemd-r 321 systemd-resolve 12u IPv6 16161 0t0 UDP *:5353
systemd-r 321 systemd-resolve 15u IPv4 16164 0t0 UDP 127.0.0.53:53
systemd-r 321 systemd-resolve 16u IPv4 16165 0t0 TCP 127.0.0.53:53 (LISTEN)
systemd-r 321 systemd-resolve 17u IPv4 16166 0t0 UDP 127.0.0.54:53
systemd-r 321 systemd-resolve 18u IPv4 16167 0t0 TCP 127.0.0.54:53 (LISTEN)
systemd-n 431 systemd-network 18u IPv4 17227 0t0 UDP 10.154.0.17:68
google_os 566 root 3u IPv4 18555 0t0 TCP 10.154.0.17:60818->169.254.169.254:80 (ESTABLISHED)
google_gu 739 root 13u IPv4 19822 0t0 TCP 10.154.0.17:35516->169.254.169.254:80 (ESTABLISHED)
sshd 747 root 3u IPv6 17848 0t0 TCP *:22 (LISTEN)
chronyd 1720 _chrony 5u IPv4 21448 0t0 UDP 127.0.0.1:323
chronyd 1720 _chrony 6u IPv6 21449 0t0 UDP [::1]:323
sshd 1761 root 4u IPv6 22688 0t0 TCP 10.154.0.17:22->185.202.17.195:45142 (ESTABLISHED)
sshd 1882 ubuntu 4u IPv6 22688 0t0 TCP 10.154.0.17:22->185.202.17.195:45142 (ESTABLISHED)

```

[Where problems could occur]

This patch reverts a change that enables MulticastDNS=resolve by default in systemd. Mantic is the first release where this is done, so it should not break existing users. If a user does want this behavior back, all they need to do is override the default /etc/systemd/resolved.conf.

[Original Description]

In the latest Ubuntu 23.10 cloud images we are seeing unexpected UDP listening port 5353.

By default and by policy, aside from port 22 there should be no other open ports on Ubuntu cloud images. Listening port 5353 is a regression.

Ubuntu 23.10 debug

```
$ ss --listening --no-header --tcp --udp --numeric
udp UNCONN 0 0 127.0.0.54:53 0.0.0.0:*
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:*
udp UNCONN 0 0 10.154.0.17%ens4:68 0.0.0.0:*
udp UNCONN 0 0 127.0.0.1:323 0.0.0.0:*
udp UNCONN 0 0 0.0.0.0:5353 0.0.0.0:*
udp UNCONN 0 0 [::1]:323 [::]:*
udp UNCONN 0 0 [::]:5353 [::]:*
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.54:53 0.0.0.0:*
tcp LISTEN 0 4096 *:22 *:*
```

This shows port 5353 open.

To find out what is listening on this port:

```
$ sudo lsof -i -n -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 153u IPv6 17848 0t0 TCP *:22 (LISTEN)
systemd-r 321 systemd-resolve 11u IPv4 16159 0t0 UDP *:5353
systemd-r 321 systemd-resolve 12u IPv6 16161 0t0 UDP *:5353
systemd-r 321 systemd-resolve 15u IPv4 16164 0t0 UDP 127.0.0.53:53
systemd-r 321 systemd-resolve 16u IPv4 16165 0t0 TCP 127.0.0.53:53 (LISTEN)
systemd-r 321 systemd-resolve 17u IPv4 16166 0t0 UDP 127.0.0.54:53
systemd-r 321 systemd-resolve 18u IPv4 16167 0t0 TCP 127.0.0.54:53 (LISTEN)
systemd-n 431 systemd-network 18u IPv4 17227 0t0 UDP 10.154.0.17:68
google_os 566 root 3u IPv4 18555 0t0 TCP 10.154.0.17:60818->169.254.169.254:80 (ESTABLISHED)
google_gu 739 root 13u IPv4 19822 0t0 TCP 10.154.0.17:35516->169.254.169.254:80 (ESTABLISHED)
sshd 747 root 3u IPv6 17848 0t0 TCP *:22 (LISTEN)
chronyd 1720 _chrony 5u IPv4 21448 0t0 UDP 127.0.0.1:323
chronyd 1720 _chrony 6u IPv6 21449 0t0 UDP [::1]:323
sshd 1761 root 4u IPv6 22688 0t0 TCP 10.154.0.17:22->185.202.17.195:45142 (ESTABLISHED)
sshd 1882 ubuntu 4u IPv6 22688 0t0 TCP 10.154.0.17:22->185.202.17.195:45142 (ESTABLISHED)

```

Shows that it is systemd-resolved that is listening and from https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html

> The systemd-resolved service listens on the following IP ports:

> Port 5353 on all local addresses, both IPv4 and IPv6 (0.0.0.0 and ::0), for MulticastDNS on UDP. Note that even though the socket is bound to all local interfaces via the selected "wildcard" IP addresses, the incoming datagrams are filtered by the network interface they are coming in on, and separate MulticastDNS link-local scopes are maintained for each, taking into consideration whether MulticastDNS is enabled for the interface or not.

So listening on port 5353 is expected for systemd-resolved and MulticastDNS but we do not expect this to be enabled by default on cloud images.

```
$ dpkg -l systemd
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-==============-============-=================================
ii systemd 253.5-1ubuntu6 amd64 system and service manager
```

Comparing the open ports on an Ubuntu 22.04 multipass VM

```
$ ss --listening --no-header --tcp --udp --numeric
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:*
udp UNCONN 0 0 10.212.201.146%ens3:68 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
tcp LISTEN 0 128 [::]:22 [::]:*
```

```
$ dpkg -l systemd
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-==================-============-=================================
ii systemd 249.11-0ubuntu3.10 amd64 system and service manager
```

[Philip Roche (philroche)]

This is related to https://bugs.launchpad.net/cloud-images/+bug/2032933 too

[Philip Roche (philroche)]

I do not believe this is a 23.10 release blocker

[Philip Roche (philroche)]

This change was introduced in 253.5-1ubuntu1 of systemd

https://launchpad.net/ubuntu/+source/systemd/253.5-1ubuntu1

[Philip Roche (philroche)]

This can be overridden for cloud images

```
mkdir --parents --verbose /etc/systemd/resolved.conf.d
cat > "/etc/systemd/resolved.conf.d/50-cloudimg.conf" <<EOF
[Resolve]
MulticastDNS=false
EOF
```

results in no longer listening on port 5353

```
$ sudo ss --listening --no-header --tcp --udp --numeric
udp UNCONN 0 0 127.0.0.54:53 0.0.0.0:*
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:*
udp UNCONN 0 0 10.0.2.15%enp0s2:68 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.54:53 0.0.0.0:*
tcp LISTEN 0 128 127.0.0.1:6010 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
tcp LISTEN 0 4096 *:22 *:*
tcp LISTEN 0 128 [::1]:6010 [::]:*
```

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andreas Hasenack (ahasenack)]

Hi Philip,

I have some questions here:

a) You state that some policy says that no ports other than 22 should be open, which policy is that? Does it apply only to cloud images, or is it an Ubuntu policy in general?

b) This is in mantic release at the moment, and switching that option back to "no" could regress users that were relying on this default. What exactly are we losing when we disable this service in this SRU? I checked the original commit[1] but it does not have a bug number linked to it with more details about what was the reasoning to enable this option in the first place.

c) If this is only about cloud images, is the workaround in comment #4 something that could be added to the cloud image build process, or we really want to avoid that?

d) Are there specific security concerns with keeping this service enabled? I presume these were considered when the option was set to "resolve" in that commit[1].

1. https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b308303f34484b293920473e5c4e0395142e4bcc

[Steve Langasek (vorlon)]

"no open ports" is the long standing policy for all Ubuntu and is not just for cloud images. (Even port 22 is not supposed to be open on bare metal server by default, only as opt in.)

I don't have a link handy at the moment to documentation of this policy but it IS the policy.

[Steve Langasek (vorlon)]

The effect of opening a port was a not-understood consequence of changing the default.

[Philip Roche (philroche)]

> a) You state that some policy says that no ports other than 22 should be open, which policy is that? Does it apply only to cloud images, or is it an Ubuntu policy in general?

I will try find the referenced policy.

> b) This is in mantic release at the moment, and switching that option back to "no" could regress users that were relying on this default. What exactly are we losing when we disable this service in this SRU?

This was added in version 253.5-1ubuntu1 [1] of systemd on 11 Jul 2023 in the devel release. It was not an intentional change to open port 5353.

I am not entirely sure on what we lose but based on the systemd-resolved docs [2] we lose ability to resolve .local domains

> This resolver has a notion of the special ".local" domain used for MulticastDNS

> c) If this is only about cloud images, is the workaround in comment #4 something that could be added to the cloud image build process, or we really want to avoid that?

CPC are primarily concerned about cloud images but enabling a new open port was an unintended consequence of the change and I understand not one that is desired.

> d) Are there specific security concerns with keeping this service enabled?

Yes. Google/GCE specifically have flagged this as an issue and a regression to have more than port 22 open.

[1] https://launchpad.net/ubuntu/+source/systemd/253.5-1ubuntu1
[2] https://www.freedesktop.org/software/systemd/man/latest/systemd-resolved.service.html

[Philip Roche (philroche)]

> a) You state that some policy says that no ports other than 22 should be open, which policy is that? Does it apply only to cloud images, or is it an Ubuntu policy in general

This policy is detailed @ https://wiki.ubuntu.com/Security/Features#ports

> Default installations of Ubuntu must have no listening network services after initial install. Exceptions to this rule on desktop systems include network infrastructure services such as a DHCP client and mDNS (Avahi/ZeroConf, see [ZeroConfPolicySpec](https://wiki.ubuntu.com/ZeroConfPolicySpec) for implementation details and justification). For Ubuntu in the cloud, exceptions include network infrastructure services for the cloud and OpenSSH running with client public key and port access configured by the cloud provider. When installing Ubuntu Server, the administrator can, of course, select specific services to install beyond the defaults (e.g. Apache).

> Testing for this can be done with netstat -an --inet | grep LISTEN | grep -v 127.0.0.1: on a fresh install.

[Andreas Hasenack (ahasenack)]

Thanks for the replies.

I note that the resolved defaults are also baked in in the default config file /etc/systemd/resolved.conf, which on mantic currently states "#MulticastDNS=resolve". With the new build, that shall change to "=no". Unless users changed that file, this update should NOT result in a dpkg conf prompt. Ok.

[Andreas Hasenack (ahasenack)]

Hello Philip, or anyone else affected,

Accepted systemd into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/253.5-1ubuntu6.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (253.5-1ubuntu6.1) for mantic have finished running.
The following regressions have been reported in tests triggered by the package:

conntrack-tools/unknown (s390x)
cryptsetup/unknown (s390x)
cups/2.4.6-0ubuntu3 (s390x)
dropbear/2022.83-2 (arm64)
haproxy/unknown (s390x)
mosquitto/2.0.18-1 (s390x)
mutter/45.0-3ubuntu3 (s390x)
network-manager/1.44.2-1ubuntu1.2 (arm64)
puppet-agent/7.23.0-1 (s390x)
samba/2:4.18.6+dfsg-1ubuntu2.1 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/mantic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 253.5-1ubuntu7

---------------
systemd (253.5-1ubuntu7) noble; urgency=medium

  * Revert "debian/rules: set MulticastDNS=resolve by default" (LP: #2038894)
    File: debian/rules
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3d753238699f54e8c2892d8107136d49f09e44b6
  * debian/gbp.conf: update for noble
    File: debian/gbp.conf
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=a65decb3a73592af8b952b159cfb453e9c0babd5

 -- Nick Rosbrook <email address hidden> Thu, 26 Oct 2023 09:51:33 -0400

[Nick Rosbrook (enr0n)]

The autopkgtest failures were resolved with retries.

[Nick Rosbrook (enr0n)]

I have verified the fix using systemd-resolved 253.5-1ubuntu6.1 from mantic-proposed:

root@mantic:~# apt policy systemd-resolved
systemd-resolved:
  Installed: 253.5-1ubuntu6.1
  Candidate: 253.5-1ubuntu6.1
  Version table:
 *** 253.5-1ubuntu6.1 500
        500 http://security.ubuntu.com/ubuntu mantic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     253.5-1ubuntu6 500
        500 http://archive.ubuntu.com/ubuntu mantic/main amd64 Packages
root@mantic:~# ss --listening --no-header --tcp --udp --numeric
udp UNCONN 0 0 127.0.0.54:53 0.0.0.0:*
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:*
udp UNCONN 0 0 10.19.111.15%eth0:68 0.0.0.0:*
udp UNCONN 0 0 [fe80::216:3eff:feb4:d412]%eth0:546 [::]:*
tcp LISTEN 0 4096 127.0.0.54:53 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
root@mantic:~# lsof -i -n -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-n 844 systemd-network 17u IPv4 1737561 0t0 UDP 10.19.111.15:68
systemd-n 844 systemd-network 20u IPv6 1738516 0t0 UDP [fe80::216:3eff:feb4:d412]:546
systemd-r 1363 systemd-resolve 13u IPv4 1743909 0t0 UDP 127.0.0.53:53
systemd-r 1363 systemd-resolve 14u IPv4 1743910 0t0 TCP 127.0.0.53:53 (LISTEN)
systemd-r 1363 systemd-resolve 15u IPv4 1743911 0t0 UDP 127.0.0.54:53
systemd-r 1363 systemd-resolve 16u IPv4 1743912 0t0 TCP 127.0.0.54:53 (LISTEN)
root@mantic:~# resolvectl
Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 30 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
       DNS Servers: 10.19.111.1 fe80::216:3eff:fe07:85b6
        DNS Domain: lxd

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 253.5-1ubuntu6.1

---------------
systemd (253.5-1ubuntu6.1) mantic; urgency=medium

  * Revert "debian/rules: set MulticastDNS=resolve by default" (LP: #2038894)
    File: debian/rules
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3d753238699f54e8c2892d8107136d49f09e44b6

 -- Nick Rosbrook <email address hidden> Thu, 26 Oct 2023 09:55:41 -0400

[Timo Aaltonen (tjaalton)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.