rpcbind grabs ports used by other daemons such as cupsd

[Alvin]

[impact]

rpcbind binds to a 'random' reserved port at startup, which can conflict with the reserved port number for other applications that actually 'own' the reserved port number. One example is cups, which uses the reserved port 631.

This prevents the actual 'owner' of the reserved port from starting, since it can't bind to its reserved port.

Additionally, this can raise alarms from security monitoring software that does not expect programs to be listening on random reserved ports.

[test case]

start rpcbind and check which ports it is listening on, e.g.:

$ sudo netstat --inet -p -l | grep rpcbind | grep -v sunrpc
udp 0 0 0.0.0.0:614 0.0.0.0:* 4678/rpcbind

each time rpcbind is restarted, it will be listening to a different 'random' port.

[regression potential]

this adds a way to disable rpcbind from listening to the 'random' port. any regression would likely prevent rpcbind from starting, or may cause problems with the interaction between rpcinfo and rpcbind, as rpcinfo may use the random reserved port in some cases, as detailed in the Debian bug.

[scope]

This is needed only for Bionic and earlier.

In Focal and later, and in Debian, rpcbind defaults to not opening the random reserved port. The admin can use the -r parameter to cause rpcbind to restore the old behavior of opening the random reserved port.

[other info]

Note that the -r parameter is a Debian addition, and the upstream rpcbind has disabled the random port functionality at build time; there is no runtime parameter to allow the admin to choose the behavior.

Also, as discussed in the Debian bug, disabling this rpcbind 'feature' is known to cause problems for the rpcinfo program, which is why Debian introduced the -r parameter. So, when this -r parameter is backported to Bionic and earlier, we must retain the default behavior for those releases, which is for rpcbind to open the random reserved port.

Thus, the patch for this will first backport the upstream patch that adds functionality to be able to disable the 'remote calls' function, and also backports the debian patch to change that from a compile-time to run-time option. Then, another patch is added, which changes the default back to the behavior of x/b, which is for remote calls to be enabled by default,
and also adds a check for the existence of an environment variable "RPCBIND_RMTCALL_DEFAULT_DISABLED" which, if defined (to anything), will change the default to disabled.

This allows 1) retaining the existing default behavior of rpcbind in x and b, while also 2) providing a mechanism to change that default for anyone who does *not* want remote calls to be enabled, and 3) allowing the mechanism to change the default to remain in place after an upgrade to Focal. Using the environment variable allows anyone to disable the remote calls in x and/or b, and then upgrade to Focal without breaking rpcbind or needing to remove the env var. After the upgrade to Focal, the environment variable (defined in /etc/default/rpcbind and/or /etc/rpcbind.conf) will simply be ignored without any change needed to the rpcbind package in Focal or later.

[original description]

Binary package hint: cups

cups 1.3.9-2ubuntu4
From /var/log/cups/error_log:
cups: unable to bind socket for address 127.0.0.1:631 - Address already in use.

Nothing actually looks wrong. 127.0.0.1:631 is only in use by cupsd when started.

[sbonnevi (sbonnevi-redhat-bugs)]

Description of problem:

On boot, ypbind occasionally grabs port 631/udp, blocking CUPS from binding to
the port. This is a glibc problem because ypbind is a RPC service that has its
port assigned dynamically through bindresvport().

The code in libc/sunrpc/bindrsvprt.c shows the port number is assigned purely
based on the PID of the ypbind process, something like

  port = (PID % 424) + 600

The PID seems to vary slightly from reboot to reboot, but generally is in the
870s on the machine in question, resulting in ports assigned in the vicinity of
630. CUPS (actually, IPP) has ports 631/tcp and 631/udp reserved. NIS starts
first, so it wins, and since CUPS has a reserved Well Known Port, it can't
relocate and loses.

Version-Release number of selected component (if applicable):
glibc-2.3.2-27.9

How reproducible:
Depends entirely on the PID handed to ypbind on boot, and the exact set of
services configured affects this.

Suggested Fix:
The glibc algorithm already blacklists all reserved ports below 600, presumably
to avoid this exact problem. Consider altering the code to blacklist 5 to 8
additional ports in the 600-1023 range that are or may be in common use:

  631 (IPP == CUPS)
  636 (LDAPS)
  749 (Kerberos V kadmin)
  873 (rsyncd)
  992-995 (SSL-enabled telnet, IMAP, IRC, and POP3)

The ports lost could be recovered, if desired, by allowing ports in the 590-600
range to be assigned by bindrsvprt().

[jakub (jakub-redhat-bugs-1)]

According to Ulrich Drepper, daemons requiring specific ports in the 600-1023
range need to be started before any daemons using bindresvport.

[twaugh (twaugh-redhat-bugs)]

I think that portmap ought to be tought how to avoid a list of ports.
Otherwise, once I've stopped cupsd (for whatever reason) and restarted ypbind, I
can no longer be sure that cupsd will ever start again.

[twaugh (twaugh-redhat-bugs)]

(see above comment)

[sbonnevi (sbonnevi-redhat-bugs)]

If you accept Ulrich's argument that this is a service bug, then this is NOT
just a CUPS bug. It's also a bug against xinetd-2.3.12-0.3E, openldap-2.0.27-9,
and krb5-server-1.2.7-15. ALL of the services in the distribution that I know
about that use Well Known Ports in the 600-1023 range start up AFTER ypbind, not
before!

[twaugh (twaugh-redhat-bugs)]

This problem should be solved in portmap. Here is a proof-of-concept, which I
think should be merged into portmap: http://cyberelk.net/tim/portreserve. In
summary: portmap should read a directory of configuration files, one per daemon,
each containing the name of the service that is out of bounds.

(I know that bindresvport is the culprit, but what other users of that are there
really?)

[sbonnevi (sbonnevi-redhat-bugs)]

I'm moving this to RHEL 3 because we're still seeing this in operation
on servers with that version installed.

I don't know what else uses bindresvport, if anything, I just know
that if you have ypserv and CUPS running, there's a pretty decent
chance of a collision on boot which breaks CUPS.

[jss (jss-redhat-bugs)]

Can someone enlighten me how portreserve is not a race condition waiting to happen? As far as I can tell, you get portreserve to release the port before starting the service. How do you make sure nothing else gets in there first before the portreserve command and the service-starting command? I suspect it's unlikely that something could slip in between these commands, but is this good programming?

[twaugh (twaugh-redhat-bugs)]

(In reply to comment #75)
> Can someone enlighten me how portreserve is not a race condition waiting to
> happen? As far as I can tell, you get portreserve to release the port before
> starting the service. How do you make sure nothing else gets in there first
> before the portreserve command and the service-starting command? I suspect it's
> unlikely that something could slip in between these commands, but is this good
> programming?

Without support in glibc/portmap there is no generic way around this.

[Pascal De Vuyst (pascal-devuyst)]

Please attach your /etc/cups/cupsd.conf

[Andrea76 (andrea76)]

I have same problem on Ubuntu 8.10, too. Here is my cupsd.conf

[Alex Maguran (amaguran)]

Same problem here on Ubuntu 8.10. I think that this problem was caused by an update of cups (this week). Please note that I have intrepid-proposed repository enabled. I have attached ny cupsd.conf as well. Please let me know if you need more info.

[Alex Maguran (amaguran)]

I have cups 1.3.9-2ubuntu5 (intrepid-proposed)

[Andrea76 (andrea76)]

Any fix? This is a really stupid bug. I cannot print with evince.

[Till Kamppeter (till-kamppeter)]

Alex, cups 1.3.9-2ubuntu5 is completel;y broken, please update to cups 1.3.9-2ubuntu6 (if not yet in -updates from -proposed).

[Till Kamppeter (till-kamppeter)]

If you reboot your machine, does it really not start up CUPS?

Before rebooting, please edit /etc/cups/cupsd.conf, so that it has a line

LogLevel debug

and no other "LogLevel" line.

After rebooting check whether CUPS is running by executing the command "lpstat -p". If CUIPS is not running, post you /var/log/cups/error_log file.

[svsig (svsig)]

I have the same problem with Ubuntu Server 8.10. I have attached my cupsd.conf.

[Till Kamppeter (till-kamppeter)]

Can you post the output of

sudo lsof -i :631

Is there anything else than cupsd listed? Can you kill all the listed processes, check whether no new processes come up and then start CUPS with

sudo /etc/init.d/cups start

Does CUPS start up, are you able to print?

Does CUPS start up again when you reboot? If not, do

sudo lsof -i :631

and post the output here.

[Till Kamppeter (till-kamppeter)]

If you do not have lsof on your system, do

sudo apt-get install lsof

[Till Kamppeter (till-kamppeter)]

Trapanator, if you cannot print from evince but from any other program, then your problem is not this bug. In this bug users are complaining that CUPS does not start because something is blocking the CUPS port 631. In this case printing will not work from any application.

[Andrea76 (andrea76)]

here is my lsof:

trap@dell:~$ sudo lsof -i :631

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
cupsd 4589 root 3u IPv4 42507 TCP localhost:ipp (LISTEN)

and now?

[Alvin (alvind)]

alvin@xanthippe:~$ sudo lsof -i :631
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
cupsd 5115 root 3u IPv4 27358 TCP localhost:ipp (LISTEN)

Cups actually works. The message is just plain wrong. If cups is not started, the port is not in use.

[Till Kamppeter (till-kamppeter)]

So Alvin and Trapanator, you should be able to print. If not, your problem is something else.

[Alvin (alvind)]

Like I reported originally, cupsd is started like it should be. Still, this error message appears and we can't have false errors now, can we?

(I do have trouble printing, but not with every printer, so this is not related)

[Alex Maguran (amaguran)]

The problem has been fixed with the 1.3.9-2ubuntu6 update. Thanks.

[svsig (svsig)]

I have upgraded to 1.3.9-2ubuntu6 on my server, but I still have this problem.

When I run the lsof -i :631 command I get no output at all. Can it be that my problem is a different one?

[Till Kamppeter (till-kamppeter)]

svsig, can you check whether CUPS is running ("lpstat -r", "ps auxwww | grep cupsd")?

Did you call the lsof command with the "sudo" in the beginning?

sudo lsof -i :631

Please edit /etc/cups/cupsd.conf, so that it has a line

LogLevel debug

and no other "LogLevel" line.

Then do

sudo /etc/init.d/cups stop
sudo /etc/init.d/cups start

Then post your /var/log/cups/error_log file here.

[svsig (svsig)]

When I execute "lpstat -r" I get the response:
"scheduler is running"

When I execute "ps auxwww | grep cupsd" I get the response:
"root 4189 0.0 0.1 6408 2340 ? Ss 10:05 0:00 /usr/sbin/cupsd"

I am logged on as root, so I didn't use sudo here. I tried the same logged on with my ordinary account just for the record, but I got the same result here using sudo.

My error_log file is quite long now, so I removed most of it leaving only the output from the last restart process. There seems to be one error message here:
"Unable to bind socket for address 10.0.0.145:631 - Cannot assign requested address."

The IP address 10.0.0.145 is the client machine that I have been trying to access the CUPS web interface from. I changed the cupsd.conf file according to the how-to I was following. This file I have attached in a previous comment.

I notice the file says printers=0. I suppose this is the number of printers attached. I have a HP LaserJet 1100 attached to my server.

[Till Kamppeter (till-kamppeter)]

svsig, what exactly is not working for you? Can you print from the server? Can you print from the client? Can you ping the client from the server? What is the output of "nmap 10.0.0.145" (nmap package). What is the output of ifconfig on your server? What is the output of ifconfig on your client? Why did you comment out "Listen 127.0.0.1:631"? Can you Reactivate this line?

[svsig (svsig)]

My problem is that I can't access the CUPS web interface on the server from a client. The server seems to be OK otherwise. I have installed LAMP server, MediaWiki and Tomcat amongst other things, and these are working fine. But when I try to access the CUPS web interface I get the error message:
Failed to Connect
Firefox can't establish a connection to the server at 10.0.0.201:631.

Both machines are connected to the network and seems to be working fine otherwise.

Here is the output from the "nmap 10.0.0.145" command:
Starting Nmap 4.62 ( http://nmap.org ) at 2009-01-11 13:18 CET
Interesting ports on sveinung1 (10.0.0.145):
Not shown: 1712 closed ports
PORT STATE SERVICE
80/tcp open http
111/tcp open rpcbind
443/tcp open https
MAC Address: 00:1E:8C:0A:F8:E6 (Asustek Computer)

Nmap done: 1 IP address (1 host up) scanned in 0.172 seconds

Here is the output of ifconfig on the server:
eth0 Link encap:Ethernet HWaddr 00:1f:d0:0a:c7:34
          inet addr:10.0.0.201 Bcast:255.255.255.255 Mask:255.255.255.0
          inet6 addr: fe80::21f:d0ff:fe0a:c734/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:8024 errors:0 dropped:2235526125 overruns:0 frame:0
          TX packets:5209 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:611143 (611.1 KB) TX bytes:755856 (755.8 KB)
          Interrupt:221 Base address:0x6000

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:1827 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1827 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:156063 (156.0 KB) TX bytes:156063 (156.0 KB)

Here is the output of ifconfig on the client:
eth0 Link encap:Ethernet HWaddr 00:1e:8c:0a:f8:e6
          inet addr:10.0.0.145 Bcast:255.255.255.255 Mask:255.255.255.0
          inet6 addr: fe80::21e:8cff:fe0a:f8e6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:24817 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21543 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5125350 (5.1 MB) TX bytes:2181559 (2.1 MB)
          Interrupt:18

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:73 errors:0 dropped:0 overruns:0 frame:0
          TX packets:73 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:7278 (7.2 KB) TX bytes:7278 (7.2 KB)

I have tried to reactivate the Listen localhost:631 line, but the error is the same. I have attached the new output from error_log.

[Till Kamppeter (till-kamppeter)]

Do you have some firewall on the server or on the client? Perhaps this blocks the CUPS communication.

Is the complete web interface inaccessible for you, or only the admin parts, or even only doing changes (like server settings, adding/modifying print queues)?

[svsig (svsig)]

I have no firewalls installed, at least not that I know of. And as I can reach web pages both in apache through port 80 and in Tomcat through port 8080 I should be able to reach CUPS through port 631 as far as I can see. The whole web interface is inaccessible, I can't reach it at all. When I try I get the error message:

Firefox can't establish a connection to the server at 10.0.0.201:631

[Andrea76 (andrea76)]

I upgraded to 1.3.9-2ubuntu6, but I cannot print on my PC. The printer is connected through to my PC.

Note that I have problem only with Evince, but with OpenOffice I can print.

[svsig (svsig)]

I think I solved the problem now. I also placed a question on ubuntuforums.org, and I got a tip here. What I did was replace the lines

Listen localhost:631
Listen 10.0.0.145:631

with the line

Listen 0.0.0.0:631

in the cupsd.conf file.

Now I can access the CUPS web interface on my server from a client machine in the network.

[jay (jay-redhat-bugs-1)]

have any of the portreserve changes in Fedora 10 made it into the RHEL5.3 release? or would this wait until RHEL6?

[twaugh (twaugh-redhat-bugs)]

It's not in RHEL-5.3.

[Alvin (alvind)]

Well, to solve this, /etc/cups/cupsd.conf has to be changed:
- Listen localhost:631
+ Listen 127.0.0.1:631

/etc/cups/cupsd.conf.default should be adapted too.

(Actually, I don't know why localhost wouldn't be ok.)

[Brett Alton (brett-alton-deactivatedaccount)]

This is still happening to me on Ubuntu 8.10 Server, cups installed after-the-fact.

Everything is up-to-date and I can't find a working fix from the above posts.

[Brett Alton (brett-alton-deactivatedaccount)]

Running,

$ sudo tasksel install print-server

doesn't seem to help either.

[Brett Alton (brett-alton-deactivatedaccount)]

And sorry to triple post, but my error log is filling up like this:

$ tail /var/log/cups/error_log
E [15/Feb/2009:13:05:47 -0500] Unable to bind socket for address 127.0.0.1:631 - Address already in use.

[Brett Alton (brett-alton-deactivatedaccount)]

And to quadruple post (you guys are going to love me), accessing http://192.168.1.102:631 still does not work after:

$ sudo cp -p /etc/cups/cupsd.conf.default /etc/cups/cupsd.conf
$ sudo tasksel install print-server
$ sudo /etc/init.d/cups start
 * Starting Common Unix Printing System: cupsd [ OK ]
$ sudo lsof -i :631
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
cupsd 5932 root 2u IPv4 1647526 TCP localhost:ipp (LISTEN)
$ sudo kill -9 5932
$ sudo /etc/init.d/cups start
 * Starting Common Unix Printing System: cupsd [ OK ]
$ sudo lsof -i :631
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
cupsd 5950 root 2u IPv4 1648352 TCP localhost:ipp (LISTEN)
$ sudo tail -n 1 /var/log/cups/error_log
E [15/Feb/2009:13:07:31 -0500] Unable to bind socket for address 127.0.0.1:631 - Address already in use.

[bugzilla (bugzilla-redhat-bugs)]

Confirming 5.3 has issues. After updating and rebooting 105 clients with a recent kernel patch + others, one had this issue logged in /var/log/cups/error_log:

E [25/Jun/2009:21:11:11 -0400] Unable to bind broadcast socket - Address already in use.

Cups didn't get broadcasted print queues until after restarting cups.

The curious thing is that I hard-bind rpc services; status, nlockmgr, ypbind are listing on *my* chosen ports, non of the conflicting with cups's 631.

We've also had past (but none this time) random issues with rsyncd not being able to bind to its listening port, I'm extrapolating the root cause is the same.

[wtogami (wtogami-redhat-bugs)]

I'm adding the portreserve hack to spamassassin-3.3.0 for RHEL-6. But this is not a complete solution. Arbitrary other services can still cause this failure.

[twaugh (twaugh-redhat-bugs)]

Warren: how so?

The only way I know of that it can fail is after the protected service is stopped. There's no way to close that hold without kernel support.

[bugs (bugs-redhat-bugs)]

Some ethernet NICs (broadcom, intel gigabit) have something called
ASF enabled. This causes the NIC to gobble stuff sent to port 623
and port 664! So there are some other ports that we need to exclude.

RPC mounts avoid these ports by using sunrpc.min_resvport which is
set to 665 these days. So the code for mounts won't use ports below
665.

Maybe bindrsvprt.c could be modified to also use sunrpc.min_resvport
and sunrpc.max_resvport.

In the 623/664 cases there is no collision with another daemon. The
affected NICs lose the packets.

ypmatch and ypcat will hang with do_ypcall: clnt_call: RPC: Timed out
when they pick 623 or 664 on the affected hardware.

sshd (particularly when AllowGroups is used) will also hang on the
affected hardware when it selects 623 or 664 to do an NIS lookup.

A kludge is to use xinetd to grab the 623/664 ports for a workaround.

[Colin.Simpson (colin.simpson-redhat-bugs)]

Just been hit with this again:

# netstat -anp | grep 631
tcp 0 0 0.0.0.0:631 0.0.0.0:* LISTEN 25591/cupsd
udp 0 0 0.0.0.0:631 0.0.0.0:* 24623/rpc.statd

Whilst portreserve looks like a fix, from my understanding is it not a bit of a hack around this. Shouldn't portmaper not just be given a list of ports to never use from a file that gets appended to as services are added from packages? Or am I missing something?

[twaugh (twaugh-redhat-bugs)]

See comment #16.

[Colin.Simpson (colin.simpson-redhat-bugs)]

comment #16 doesn't answer why we need portreserve, it just says that there are no free unallocated low ports in the official allocation list. That's true, but shouldn't portmapper have a notion of services THIS system has or needs reserved low and not allocate them.

Or is there just no easy way of allocating a low port in a system call that can exclude ports we want to hold back? Sounds like a new call is called for eventually?

[releng-rhel (releng-rhel-redhat-bugs)]

Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

[pm-rhel (pm-rhel-redhat-bugs)]

This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.

[bugzilla (bugzilla-redhat-bugs)]

(In reply to comment #91)
> Red Hat Enterprise Linux 6.0 is now available and should resolve
> the problem described in this bug report.

It won't since there is no portrelease support in the 6.x ypbind scripts.

[orion (orion-redhat-bugs-1)]

It's not ypbind that needs to use portreserve/release, it's everything else that uses ports in the rpc port range.

[jeremy (jeremy-redhat-bugs-1)]

Dovecot is missing support portreserve/release (at least in Scientific Linux 6.1).

[bugzilla (bugzilla-redhat-bugs)]

(In reply to comment #103)
> It's not ypbind that needs to use portreserve/release, it's everything else
> that uses ports in the rpc port range.

What I discovered via portreserve(1) and testing is that when portreserve starts, it listens on all the reserved ports that have been configured. Just before the real daemon starts, it needs to run portrelease so the *real* daemon can listen on it. Look at:

grep portrelease /etc/init.d/*
cat /etc/portreserve/cups

In the case of ypbind, which can listen either on a random port (unfixable) or a fixed port (via setting OTHER_YPBIND_OPTS). I hard-code to port 900 by setting

/etc/sysconfig/network:OTHER_YPBIND_OPTS="-p 900"

To fix ypbind, I see two potential ways - (1) it's init script needs to parse out the -p parameter and if you get something, call portrelease on that port. Something like:

ypport=`echo $OTHER_YPBIND_OPTS | sed 's/.*-p[[:space:]]*\(\<.*\>\).*/\1/'`
[ -n "$ypport" -a -x /sbin/portrelease ] && /sbin/portrelease $ypport &>/dev/null || :

Or (2), since the user needs to reserve a port by dropping a file into /etc/portreserve/ anyway, force them to use the service named 'ypbind' and just portrelease that:
[ -x /sbin/portrelease ] && /sbin/portrelease ypbind &>/dev/null || :

Hopefully I'll get time to test this today.

Should I open a new bug on RHEL6 for this issue? I searched and could not find one...

[ksrot (ksrot-redhat-bugs)]

I have changed the component to "distribution". Please file a new bug for each affected component and use this one as a tracker.

rsync bug filed as Bug 786076.

[dyocum (dyocum-redhat-bugs)]

This bug exists in RHELv5.x, too. I noticed it when trying to use heartbeat (port 694).

Can we get portreserve backported to RHELv5.x?

[Yves Dorfsman (dorfsmay)]

The problem is that other daemon grab this port before cupsd gets a chance, typcally rpc. See:
https://bugzilla.redhat.com/show_bug.cgi?id=103401

Work around:
lsof -i :631
stop or kill the process, restart cups

long term solution:
I guess distros should ensure processes use different port number.

Note:
The same problem happens on a workstation that try to browse for remote printers (cupsctl --remote-printers), you don't get an error, but "lpstat -a" doesn't show remote printers.

[ovasik (ovasik-redhat-bugs)]

Just having portreserve in RHEL-5.X will not solve the issue, you need to update all the packages which already use it in RHEL-6 to achieve same results. And this is IMHO very unlikely.

[dyocum (dyocum-redhat-bugs)]

I wasn't suggesting updating all the RHEL5 packages that would need portreserve, but having the package itself as part of the distro (instead of having to go to some third party repo, which is what I ended up doing) would go a long way to allow a sysadmin solve the problem, IMO.

Even if it made it into EPEL for RHEL5 I think I'd be happy...

[hamiller (hamiller-redhat-bugs)]

I attached this bug to case 00721174, as the new RHS product can (and will) assign a port for every storage brick. In huge systems we have seen this consume all available ports under 1024.

[ejtr (ejtr-redhat-bugs)]

Having just been caught out by this bug in CentOS6 with heartbeat, ( UDP Port 694 ), and looking at the AFS /IPMI Port 623/664 bug mentioned in previous comments, it doesn't seem unreasonable to me to make portmap/rpcbind honour sunrpc.min_resvport/sunrpc.max_resvport as per comment 86.

I presume rpc.mountd honours these proc settings, and the original setting of 600 minimum was modified specifically so as to avoid IPMI hardware clashes. If that's the case, then it would seem rpcbind is STILL vulnerable to a potential clash with IPMI hardware as long as a suitably configured portreserve is not running, even if no actual "real" daemons use those ports. Since there's willingness to avoid NFS mounts clashing, why not "protect" rpcbind with the same tweak?

If this were done, then sysadmins who only have a few clashing services in the 6xx range, but only a modicum of simultaneous RPC services/ports, could avoid the whole mess by simply raising sunrpc.min_resvport to 700, for instance.

In more complex cases, I admit that it might well be necessary to employ portreserve, but the /proc setting might well be sufficient in many cases.

[lhecking (lhecking-redhat-bugs)]

I would just like to add that I have this problem on CentOS6 with ypserv *every* time I reboot a particular server - 6.0 through 6.6. And occasionally on CentOS 5.11 with ypbind/cups.

[erco (erco-redhat-bugs)]

I haven't seen this solution mentioned in this thread:
/proc/sys/net/ipv4/ip_local_reserved_ports

Supposedly port numbers added to this file will be skipped over by the
glibc functions that auto-assign reserved ports as used by tools like
nfs/nis/etc.

Some interesting details here:
http://www.mjmwired.net/kernel/Documentation/networking/ip-sysctl.txt#716
https://lkml.org/lkml/2012/3/10/187

[D J Gardner (djgardner)]

Re: #32
  Cups and other processes expect to be able to assign their configured ports, But rpcbind / portmap etc. assign ports randomly.
e.g.: http://www.linuxproblem.org/art_13.html

This bug has just hit me, with a deadline to print and no way to reboot the computer without losing work.
It took a lot of debugging before I stumbled upon the article above, and I've been using linux since it came on two floppies. The problem has been noted in redhat, suse, even BSD.

I propose a solution: rpcbind should NOT assign random ports, it should assign random *unallocated* ports (i.e. it should
make a list of things listed in /etc/services and avoid those ports.)

[areichow (areichow-redhat-bugs)]

Any chance we'll see a fix? Having this issue on RHEL 6.7 with rpc.statd. No port collisions, but in our environment we are required to have specific ports/port ranges defined for a particular service. There doesn't seem to be anyway to force rpc.statd to use a particular port rather than follow the pid % 424 + 600 formula.

[rkudyba (rkudyba-redhat-bugs)]

Seeing this with Dovecot on Fedora 28:
Jul 24 11:12:24 ourdomain dovecot[1838]: Error: service(imap-login): listen(::, 993) failed: Address already in use
Jul 24 11:12:24 ourdomain dovecot[1838]: master: Error: service(pop3-login): listen(*, 995) failed: Address already in use
Jul 24 11:12:24 ourdomain dovecot[1838]: master: Error: service(pop3-login): listen(::, 995) failed: Address already in use
Jul 24 11:12:24 ourdomain dovecot[1838]: Fatal: Failed to start listeners
Jul 24 11:12:24 ourdomain dovecot[1838]: master: Error: service(imap-login): listen(*, 993) failed: Address already in use
Jul 24 11:12:24 ourdomain dovecot[1838]: master: Error: service(imap-login): listen(::, 993) failed: Address already in use
Jul 24 11:12:24 ourdomain dovecot[1838]: master: Fatal: Failed to start listeners
Jul 24 11:12:24 ourdomain systemd[1]: dovecot.service: Control process exited, code=exited status=89
Jul 24 11:12:24 ourdomain systemd[1]: dovecot.service: Failed with result 'exit-code'.
Jul 24 11:12:24 ourdomain systemd[1]: Failed to start Dovecot IMAP/POP3 email server.

netstat -lnp | grep 993
unix 2 [ ACC ] STREAM LISTENING 39993 1075/httpd /run/httpd/cgisock.972

[fweimer (fweimer-redhat-bugs)]

(In reply to RobbieTheK from comment #137)
> Seeing this with Dovecot on Fedora 28:
> Jul 24 11:12:24 ourdomain dovecot[1838]: Error: service(imap-login):
> listen(::, 993) failed: Address already in use

> netstat -lnp | grep 993
> unix 2 [ ACC ] STREAM LISTENING 39993 1075/httpd
> /run/httpd/cgisock.972

That is something else, the grep doesn't show a TCP socket on port 993.

[rkudyba (rkudyba-redhat-bugs)]

This is still happening after a reboot, Fedora 28,

Nov 8 12:21:41 kopernik dovecot[1386]: Error: service(pop3-login): listen(*, 995) failed: Address already in use
Nov 8 12:21:41 kopernik dovecot[1386]: Error: service(pop3-login): listen(::, 995) failed: Address already in use
Nov 8 12:21:41 kopernik dovecot[1386]: Error: service(imap-login): listen(*, 993) failed: Address already in use
Nov 8 12:21:41 kopernik dovecot[1386]: Error: service(imap-login): listen(::, 993) failed: Address already in use

After restarting dovecot it works:
netstat -lnp | grep 993
tcp 0 0 0.0.0.0:993 0.0.0.0:* LISTEN 4622/dovecot
tcp6 0 0 :::993 :::* LISTEN 4622/dovecot

[fweimer (fweimer-redhat-bugs)]

We are tracking this upstream. It is unclear whether bindresvport will remain as a public API in glibc. So far, it has not been subject to the Sun RPC transition in glibc, although libtirpc implements the symbol as well. It may therefore be possible to turn the glibc symbol in to a compat symbol, effectively removing it from the glibc API. In this case, it is unlikely that upstream will implement a denylist, similar to what is available in libtirpc today.

Applications built on Red Hat Enterprise Linux 8 can already use the libtirpc denylist functionality for bindresvport by linking against libtirpc.

[fweimer (fweimer-redhat-bugs)]

I filed bug 1854147 against libtirpc with the denylist additions suggested in comment 0.

[Timo Aaltonen (tjaalton)]

Hello Alvin, or anyone else affected,

Accepted rpcbind into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/rpcbind/0.2.3-0.6ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dan Streetman (ddstreet)]

bionic verification:

ubuntu@lp304393-b:~$ dpkg -l|grep rpcbind
ii rpcbind 0.2.3-0.6 amd64 converts RPC program numbers into universal addresses
ubuntu@lp304393-b:~$ sudo netstat --inet -p -l | grep rpcbind | grep -v sunrpc
udp 0 0 0.0.0.0:788 0.0.0.0:* 13332/rpcbind

after upgrade, same default behavior:

ubuntu@lp304393-b:~$ dpkg -l|grep rpcbind
ii rpcbind 0.2.3-0.6ubuntu0.18.04.1 amd64 converts RPC program numbers into universal addresses
ubuntu@lp304393-b:~$ sudo systemctl restart rpcbind
ubuntu@lp304393-b:~$ sudo netstat --inet -p -l | grep rpcbind | grep -v sunrpc
udp 0 0 0.0.0.0:707 0.0.0.0:* 14099/rpcbind

with parameter set, random port is disabled:

ubuntu@lp304393-b:~$ cat /etc/default/rpcbind
RPCBIND_RMTCALL_DEFAULT_DISABLED=1

ubuntu@lp304393-b:~$ sudo systemctl restart rpcbind
ubuntu@lp304393-b:~$ sudo netstat --inet -p -l | grep rpcbind | grep -v sunrpc
ubuntu@lp304393-b:~$

with param commented, default behavior restored:

ubuntu@lp304393-b:~$ cat /etc/default/rpcbind
#RPCBIND_RMTCALL_DEFAULT_DISABLED=1

ubuntu@lp304393-b:~$ sudo systemctl restart rpcbind
ubuntu@lp304393-b:~$ sudo netstat --inet -p -l | grep rpcbind | grep -v sunrpc
udp 0 0 0.0.0.0:877 0.0.0.0:* 14269/rpcbind

[Robie Basak (racb)]

The Ubuntu Xenial SRU looks fine in principle but is currently blocked until the previous SRU is released. I've not reviewed this in detail yet.

[Launchpad Janitor (janitor)]

This bug was fixed in the package rpcbind - 0.2.3-0.6ubuntu0.18.04.1

---------------
rpcbind (0.2.3-0.6ubuntu0.18.04.1) bionic; urgency=medium

  * d/rules: force dh_autoreconf to fix FTBFS (LP: #1885389)
  * d/p/lp304393/0001-rpcbind-Disable-remote-calls-by-default.patch,
    d/p/lp304393/0002-rmt-calls.patch,
    d/p/lp304393/0003-change-rmtcalls-default-to-enabled.patch:
    - Allow disabling rmtcalls port binding (LP: #304393)

 -- Dan Streetman <email address hidden> Tue, 04 Aug 2020 17:17:50 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for rpcbind has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Łukasz Zemczak (sil2100)]

Hello Alvin, or anyone else affected,

Accepted rpcbind into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/rpcbind/0.2.3-0.2ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dan Streetman (ddstreet)]

ubuntu@lp304393-x:~$ dpkg -l|grep rpcbind
ii rpcbind 0.2.3-0.2ubuntu0.1 amd64 converts RPC program numbers into universal addresses
ubuntu@lp304393-x:~$ sudo netstat --inet -p -l | grep rpcbind | grep -v sunrpc
udp 0 0 *:927 *:* 13471/rpcbind

same default behavior after upgrade:

ubuntu@lp304393-x:~$ dpkg -l|grep rpcbind
ii rpcbind 0.2.3-0.2ubuntu0.16.04.1 amd64 converts RPC program numbers into universal addresses
ubuntu@lp304393-x:~$ sudo systemctl restart rpcbind
ubuntu@lp304393-x:~$ sudo netstat --inet -p -l | grep rpcbind | grep -v sunrpc
udp 0 0 *:625 *:* 14441/rpcbind

with parameter added, random port disabled:

ubuntu@lp304393-x:~$ cat /etc/default/rpcbind
RPCBIND_RMTCALL_DEFAULT_DISABLED=1

ubuntu@lp304393-x:~$ sudo systemctl restart rpcbind
ubuntu@lp304393-x:~$ sudo netstat --inet -p -l | grep rpcbind | grep -v sunrpc
ubuntu@lp304393-x:~$

with parameter commented, default behavior restored:

ubuntu@lp304393-x:~$ cat /etc/default/rpcbind
#RPCBIND_RMTCALL_DEFAULT_DISABLED=1

ubuntu@lp304393-x:~$ sudo systemctl restart rpcbind
ubuntu@lp304393-x:~$ sudo netstat --inet -p -l | grep rpcbind | grep -v sunrpc
udp 0 0 *:660 *:* 14476/rpcbind

[Dan Streetman (ddstreet)]

note there is only one xenial reverse-dep autopkgtest failure, for nfs-utils on s390x, and that test has always failed and can be ignored.

[Launchpad Janitor (janitor)]

This bug was fixed in the package rpcbind - 0.2.3-0.2ubuntu0.16.04.1

---------------
rpcbind (0.2.3-0.2ubuntu0.16.04.1) xenial; urgency=medium

  * d/rules, d/control:
    - Add dh-autoreconf build-dep and force dh_autoreconf
      to fix FTBFS (LP: #1885389)
  * d/p/lp304393/0001-rpcbind-Disable-remote-calls-by-default.patch,
    d/p/lp304393/0002-rmt-calls.patch,
    d/p/lp304393/0003-change-rmtcalls-default-to-enabled.patch:
    - Allow disabling rmtcalls port binding (LP: #304393)

 -- Dan Streetman <email address hidden> Mon, 10 Aug 2020 11:05:43 -0400