Accept-Language header contains incorrect value

[Phill]

Both my OS and Firefox preferences has en-GB and en as the language, however, it's sending "en-US, en" in the "accept-language" request header. This was verified using Bugzilla after finding the problem. Dates on some sites are a particular problem given that for many dates there is no way to tell the difference between en-GB and en-US, you just read the date wrong.

Steps to reproduce:
1) Start Firefox
2) In Edit -> Preferences - check the lanugage settings are "en-gb" and "en".
3) Start Firebug.
4) Access www.google.co.uk
5) Check the "Net" tab in firebug.
6) Expand the first (or any line).
7) Scroll to the "Request Headers".
8) Observe "Accept-language" contains "en-us,en;q=0.5".

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: firefox 7.0.1+build1+nobinonly-0ubuntu0.11.04.1
ProcVersionSignature: Ubuntu 2.6.38-11.50-generic 2.6.38.8
Uname: Linux 2.6.38-11-generic x86_64
AddonCompatCheckDisabled: False
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.23.
Architecture: amd64
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: ALC662 rev1 Analog [ALC662 rev1 Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: phill 1675 F.... pulseaudio
BuildID: 20110928224508
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xf6800000 irq 43'
   Mixer name : 'Intel IbexPeak HDMI'
   Components : 'HDA:10ec0662,14621033,00100101 HDA:80862804,14621033,00100000'
   Controls : 20
   Simple ctrls : 11
Channel: release
Date: Tue Oct 4 19:03:40 2011
EcryptfsInUse: Yes
ForcedLayersAccel: False
IfupdownConfig:
 auto lo
 iface lo inet loopback
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64 (20110427.1)
IpRoute:
 192.168.0.0/24 dev wlan0 proto kernel scope link src 192.168.0.2 metric 2
 169.254.0.0/16 dev wlan0 scope link metric 1000
 default via 192.168.0.1 dev wlan0 proto static
ProcEnviron:
 LANGUAGE=en_GB:en
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
Profiles: Profile0 (Default) - LastVersion=7.0.1/20110928224508 (Running)
RfKill:
 0: phy0: Wireless LAN
  Soft blocked: no
  Hard blocked: no
RunningIncompatibleAddons: False
SourcePackage: firefox
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 07/21/2010
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: E1681IG6 VER.109
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: To be filled by O.E.M.
dmi.board.vendor: Micro-Star International
dmi.board.version: Ver.001
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 9
dmi.chassis.vendor: Micro-Star International
dmi.chassis.version: Ver.001
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrE1681IG6VER.109:bd07/21/2010:svnMicro-StarInternational:pnCalpellaplatform:pvrVer.001:rvnMicro-StarInternational:rnTobefilledbyO.E.M.:rvrVer.001:cvnMicro-StarInternational:ct9:cvrVer.001:
dmi.product.name: Calpella platform
dmi.product.version: Ver.001
dmi.sys.vendor: Micro-Star International

[Chris Coulson (chrisccoulson)]

Thanks, but that works fine here. What is intl.accept_languages set to in about:config, and what does the Status column say?

[Phill (phill.l)]

Thanks for looking at it. My "intl.accept_languages" contains status: "detault", type: "string" and value: "en-gb, en". I have two computers exhibiting the same problem both running Ubuntu 11.04 64-bit.

[Chris Coulson (chrisccoulson)]

Still not confirming, even with your steps. The Accept-Language header is taken directly from intl.accept_languages, so I'm not sure how you end up with a different value

[Chris Coulson (chrisccoulson)]

This is caused by the JSONovich addon, which just seems to override all the standard headers. In future, please try disabling your addons before reporting a bug, as it can save a lot of time

[Phill (phill.l)]

Sorry, should have tried that. Thanks for your help.

[Chris Coulson (chrisccoulson)]

It's probably worth contacting the developer of that addon, or leaving a comment on addons.mozilla.org.

[Phill (phill.l)]

It appears to be something about the handling of the default language, manually set languages are fine with that plug-in. It would appear that without the GB language pack, Firefox defaults to en-US (ignoring the language set in the OS/desktop), which is not what I'd expect, but never mind. The plug-in developer and I have failed to locate the source code for that language pack and I was hoping you'd be able to point me in the right direction.

[Chris Coulson (chrisccoulson)]

The way this works is that intl.accept_languages is set to a localized value which comes from the language pack for your selected locale (and the selected locale comes from the environment), Defaulting to en-us when you uninstall the GB language pack is the expected behaviour regardless of your environment, as you have nothing providing a correctly localized value for intl.accept_languages anymore.

[Phill (phill.l)]

No idea's on where we can get hold of the source code then?

[Chris Coulson (chrisccoulson)]

apt-get source firefox ?

[Chris Coulson (chrisccoulson)]

I figured this out in the end. The problem is that the startup() entry point for bootstrapped extensions gets called by the addon manager before the chrome manifests for other extensions (including language packs) have been loaded. Because this extension does something with starts nsIHttpHandler when it loads, nsIHttpHandler initializes the Accept-Language value before the chrome for the en-GB language pack has been loaded (so it ends up with the fallback value for en-US, with that being the only provider registered at that point)

[Chris Coulson (chrisccoulson)]

Ok, I'm not sure the bug title is entirely appropriate, and I'm not sure whether the addon manager is the appropriate place for this bug.

This was initially reported at https://launchpad.net/bugs/867753

This started off when the reporter stated that the Accept-Language header in all of their outgoing http requests was set to the wrong value ("en-us,en;q=0.5"), despite the fact that intl.accept_languages was set to the correct value according to about:config ("en-gb, en"). This is set to a complex value, with the real value coming from the reporters en-GB language pack addon.

We initially suspected that the bug was due to the JSONovich addon, as the problem was resolved after disabling that addon. However, further investigation has shown that JSONovich doesn't appear to be doing anything wrong.

Note, than JSONovich is a bootstrapped extension.

What appears to be happening is that something in it's startup() entry point triggers the initialization of the nsHttpHandler service, and nsHttpHandler reads and processes the value of intl.accept_languages at this point. However, when this happens, no extension chrome has been loaded - so the value ends up being provided by the built-in en-US provider instead.

So, it seems that it currently isn't possible to reliably read complex prefs from anything which runs inside the startup() function.

[Chris Coulson (chrisccoulson)]

Hmmm, so actually, I guess that this is where the problem occurs:

http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsXREDirProvider.cpp#715

[Dtownsend (dtownsend)]

The startup method is called early in startup, before chrome for non-restartless add-ons is registered. That is the problem here.

There are really only two solutions. One is to move the startup call to later in startup, the other is to have JSONovich fixed so it doesn't initialize nsHttpHandler until later in startup. The former has the potential to break existing add-ons as well as potentially make it harder to do certain tasks so I'd tend to side with just notifying the JSONovich author and leaving this as-is.

I would note that while startup() is called early it is still much later than non-restartless add-ons used to be able to be called, they could hit exactly the same problem if they weren't careful about delaying things they did to later in startup.

[Will Elwood (w.elwood08)]

I have just committed a fix to JSONovich for this, I was unintentionally calling Services.io.newURI('http://...') at startup due to a silly mistake decrementing a counter too early.

Apologies for the noise.

[Bmcbride (bmcbride)]

Thanks for the quick response!

[Sven Jörns (proofy)]

It was possible for me to fix this problem with this addOn
https://addons.mozilla.org/de/firefox/addon/quick-accept-language-switc/?src=api
Changing Accept-Language to any value, restart and set to default in about:config.