Accessing http://localhost gives a 403 Forbidden when using apache "allow localhost" directive

[papukaija]

Binary package hint: apache2

For some reason accessing http://localhost gives a 403 Forbidden error page. Even a non-existing page gives the forbidden page (403 Forbidden: You don't have permission to access /aaa on this server).
After reloading apache (sudo service apache2 restart), apache loads correctly the requested page. The /var/www directory is mounted to a separate partition.

ProblemType: Bug
Architecture: i386
Dependencies: apache2-mpm-worker None [modified: /var/lib/dpkg/info/apache2-mpm-worker.list]
DistroRelease: Ubuntu 9.04
Package: apache2 2.2.11-2ubuntu2
PackageArchitecture: all
ProcEnviron:
 LANG=fi_FI.UTF-8
 SHELL=/bin/bash
SourcePackage: apache2
Uname: Linux 2.6.28-11-generic i686

[papukaija (papukaija)]

Here is the output from apache's error log:

[Fri May 01 22:09:58 2009] [notice] Apache/2.2.11 (Ubuntu) PHP/5.2.6-3ubuntu4.1 with Suhosin-Patch configured -- resuming normal operations
[Fri May 01 22:26:57 2009] [error] [client ::1] client denied by server configuration: /var/www/
[Fri May 01 22:33:19 2009] [error] [client ::1] client denied by server configuration: /var/www/aaa
[Fri May 01 22:35:20 2009] [notice] caught SIGTERM, shutting down
[Fri May 01 22:35:21 2009] [notice] Apache/2.2.11 (Ubuntu) PHP/5.2.6-3ubuntu4.1 with Suhosin-Patch configured -- resuming normal operations
[Fri May 01 22:35:25 2009] [error] [client 127.0.0.1] File does not exist: /var/www/aaa

I'm also attaching /etc/apache2/sites-avaible/default file. I haven't modified the apache2.conf file.

[Andreas Olsson (andol)]

Let me see if I understand this correctly..

You installed Apache2, and when you directly afterwards try to visit http://localhost you are greeted by the 403? Later restarting apache2 solves the problem?

I did a fresh install of apache2 and http://localhost/ immediately gave me a "It works!".

How did you go about installing apache2? Can you manage to reproduce this error?

[Martin Zuther (mzuther)]

Hi!

It seems that I'm also affected by this bug. Here's how to reproduce this problem:

1. do a fresh install of apache2 in Ubuntu Jaunty
2. change line 13 in /etc/apache2/sites-available/default from "allow from all" to "allow from localhost"
3. reboot (this is important!)
4. open http://localhost/ in the browser of your choice. You'll get the error message "Forbidden. You don't have permission to access / on this server", whereas the address http://127.0.0.1/ will greet you with the correct "It works!"

"sudo /etc/init.d/apache2 restart" will solve the problem until the next boot, and changing line 13 back to "allow from all" will work as it should.

Maybe apache is started before the network is correctly initialised and therefore doesn't know "localhost"?

Martin

[Jonathan Marsden (jmarsden)]

DIAGNOSIS:

Based on the [client ::1] in the log entries, I strongly suspect an IPv6 related issue here.

In Jaunty, /etc/hosts has a line

::1 localhost ip6-localhost ip6-loopback

where in Intrepid, this was

::1 ip6-localhost ip6-loopback

I *strongly* suspect that "allow from localhost" in Apache maps to an IPv4 "allow from 127.0.0.1", but in Jaunty, the incoming request is actually coming in from IPv6 address ::1, which therefore does not match the allow rule.

EVIDENCE:

(A) Once you edit things to use "allow localhost" and then reboot, browsing to http://[::1]/. fails but browsing to http://[127.0.0.1]/ works.

(B) Editing the line of .etrc/hosts that starts with ::1 to remove the "localhost" name and then restarting your browser allows browsing to http://localhost/ to work once more (because now localhost resolves to 127.0.0.1).

SOLUTION:

The 'fix' is now trivially obvious. Edit the line of /etc/apache2/sites-available/default to say "allow localhost" as before, and then add a new line immediately after it that says "allow ::1" . Now both IPv4 locahost and IPv6 localhost clients are permitted the rules the sysadmin has manually specified, and now browsing from Firefox on the local machine works as expected.

QUESTION: Is this really a bug? Or is it simply a change of overall system behaviour as the world becomes more IPv6 conscious, that those who like editing Apache config files by hand "should" know about? [I suspect the latter, but I'm not at all dogmatic about that opinion :) ]

In case it matters, my tests were all done in a Jaunty i386 virtual machine using virtualbox-ose on a Jaunty amd86 host system, and with Firefox as my "browser of choice".

Jonathan

[Andreas Olsson (andol)]

@Martin: Actually, failed to reproduce using your recipe. Sounds like your last suspision might be right. Are you using NetworkManager? In system och session mode? I'm not. Will have to get back to you when I have a Jaunty system using NetworkManager, which I can reboot and test.

@Jonathan: Actually, I'm pretty sure the ::1 entry in /etc/hosts isn't something which was changed in Jaunty. It happens to differ based on if you used Ubiquity or the Debian Installer. Compare to bug #285132. At least I think so, not sure if I really have done any Ubiquity installs of Jaunty yet. I happen to

Except that I do agree about your analysis about it being related to ipv6 and lookups. Just not sure if adding an Allow from ::1 will help. From what I take it the default Apache2 doesn't listen to ipv6 connections at all.

[Andreas Olsson (andol)]

@papukaija: Just for confirmation; are your problems depending on "Allow from localhost" or do they also occur using "Allow from all". How does your /etc/hosts look like? Using the NetworkManager?

[Martin Zuther (mzuther)]

Hi!

Damn, you're fast... :)

I tried Jonathan's suggestions (A) and (B) as listed under EVIDENCE, and each one resolved the issue.

Still, I think there's more to it than "IPv6 awareness" alone. Otherwise, running "sudo /etc/init.d/apache2 restart" wouldn't change anything, would it? By the way, "sudo /etc/init.d/apache2 reload" does NOT help at all, suggesting that this problem might lie outside of Apache's config files.

@Andreas: I'm using the Gnome NetworkManager applet, so it's session based. Is that what you meant?

BTW, I always reinstall my complete system using the graphical installer and then restore (and of course thoroughly adapt) my important settings using a config file backup software I wrote. When diffing the old and new "/etc/hosts" files, the "localhost" entry in the ::1 line of "/etc/hosts" was clearly new, but I kept the addition.

Martin

[Jonathan Marsden (jmarsden)]

@Andreas: When troubleshooting Ubuntu, please do not "think so", please test instead :)

All my Intrepid Desktop VMs have "::1 ip6-localhost ip6-loopback", and all my Jaunty VMs have "::1 localhost ip6-localhost ip6-loopback" in /etc/hosts.

Further, both on Intrepid and on Jaunty, Apache2 *does* listen on IPv6 addresses by default, as a quick

  sudo netstat -ntlp |grep ::80

will show. Again: please do not "take it", please test instead :)

As a further test, I just now:

(1) Created two new VMs, one booted/installed from a ubuntu-8.10-desktop-i386.iso image and one from a ubuntu-9.04-desktop-i386.iso image. Other than their hostnames (itest and jtest respectively) I was careful to perform the installs in exactly the same way. Result: itest has "::1 ip6-localhost ip6-loopback", and jtest has "::1 localhost ip6-localhost ip6-loopback" in /etc/hosts. I conclude that this *is* in fact something that changed in Jaunty.

(2) I then installed apache2 (sudo apt-get install apache2 --no-install-recommends) on each VM and restarted it.
At this point both VMs can browse to http://localhost/ and also http://[::1]/ successsfully.

(3) I then edited /etc/apache2/sites-available/default on both machines, editing only the line 13 from "allow from all" into "allow from localhost", and then rebooted both VMs. At this point the Intrepid machine can still browse localhost, but the Jaunty machine cannot. At this point both machines can browse http://127.0.0.1, and neither machine can browse http://[::1]/ .

(4) On jtest (the Jaunty machine), I then edited /etc/apache2/sites-available/default to add an additional line "allow from ::1" just after the "allow from localhost" line, and rebooted. Now http://localhost and http://[::1] are both browseable on this machine.

I trust this is sufficient testing to support my earlier diagnosis of this issue. I can provide tcpdump packet captures to demonstrate that in Jaunty, browsing http://localhost is generating IPv6 traffic, but in Intrepid browsing http://localhost uses IPv4, if we really need to do that to demonstrate that this really is the difference here :)

Jonathan

[Andreas Olsson (andol)]

@Jonathan: You are right, I should test more and speculate less. I guess I got a bit carried away. Thanks for pointing it out.

Regarding the part of listening on ipv6 I actually did some testing, on a server install of 9.04. This is the result I got.

andreas@pc13267v1:~$ sudo netstat -ntlp | grep 80
[sudo] password for andreas:
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2041/apache2

Now today, testing it on a Desktop install, I get the following interesting results

* Doing a reboot *
andreas@pc13267v5:~$ sudo netstat -ntlp | grep 80
[sudo] password for andreas:
tcp6 0 0 :::80 :::* LISTEN 2438/apache2
andreas@pc13267v5:~$ sudo /etc/init.d/apache2 restart
 * Restarting web server apache2 ... waiting [ OK ]
andreas@pc13267v5:~$ sudo netstat -ntlp | grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2814/apache2

(Note that Apache is switching from ipv6 to ipv4 as it is manually restarted)

Removing NetworkManager from my Desktop install, and instead configure eth0 (using dhcp) from /etc/network/interfaces makes my desktop install behave like my server install.

* Doing a reboot *
andreas@pc13267v5:~$ sudo netstat -tlpn | grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2545/apache2

[papukaija (papukaija)]

@Andreas: Before editing /etc/apache2/sites-avaible/default the apache was working normally (but I didn't check it after restart). I'm using NetworkManager. I will now test the "Allow from all"-option.

As Martin said, http://127.0.0.1 is working.

[papukaija (papukaija)]

With "Allow from all"-option, http://[::1]/ , http://127.0.0.1/ and http://localhost/ are all working. So this is a problem with "Allow from localhost" or a specified ip address.

[Martin Zuther (mzuther)]

I observe exactly the same behaviour as Andreas did on his desktop install, right down to everything being fine when I use /etc/network/interfaces instead of NetworkManager.

[Jonathan Marsden (jmarsden)]

OK. The underlying issue here now seems to be:

On Ubuntu 9.04 Desktop, Network Manager causes a network configuration that leads Apache to start up listening on IPv6, but to restart listening on IPv4. Specifying network configuration using /etc/network/interfaces does not cause this behaviour, and works as expected (apache2 uses IPv4 consistently).

Therefore, this is really a Network Manager bug, not an apache2 bug -- should we change it to be against network-manager?

This also implies that doing "allow from ::1" as well as "allow from localhost" is a only workaround, not really addressing the issue itself.

[Andreas Olsson (andol)]

@Jonathan: Yes, I agree that this is clearly related to the NetworkManager. That said I'm not convinced that is an actual bug in Network Manager. For example; compare to sshd, which doesn't have any trouble with Network Manager and its ipv6 behavior.

Anyway, I'm changing the Status to Confirmed. While we have yet to triage the actual cause we have definitely confirmed the issue.

[papukaija (papukaija)]

Has someone tried to reproduce this bug in Karmic Alpha2 ?

[papukaija (papukaija)]

Any news with alpha6 ?

[Károly Oláh (aiden-soravis)]

The bug still exists in Ubuntu Lucid Lynx (10.04), with the exast same symptoms amd solutions. At least in my case.
kernel: 2.6.32-23-generic
apache: 2.2.14-5ubuntu8, mpm-prefork
network-manager: 0.8-0ubuntu3

[Lele Long (schemacs)]

Still affected me on 10.04
nm:0.8-0ubuntu3
uname :Linux 2.6.32-24-generic #38-Ubuntu SMP Mon Jul 5 09:22:14 UTC 2010 i686 GNU/Linux
apache:2.2.14-5ubuntu8
I have encountered so many problems with the Network-Manager,Will Ubuntu switch to wicd at any chance?

[Martin Kucej (mkucej)]

This bug still persists in 10.04.2. I can't be that hard to fix guys. Can it?

[papukaija (papukaija)]

@Martin Kucej: It is impossible to fix this bug until its root cause is known, see comments 13 and 14 for details. Please be patient.