User-Agent string results in poor UX on web

[dobey]

The User-Agent string for the browser is similar enough to the Android browser User-Agent, that it creates a poor experience when browsing the web on an Ubuntu phone. An inordinately large number of web sites persistently advertise to "install our app" instead of providing the best web experience; an app which cannot be installed.

This is exacerbated by the pervasiveness of webapps on Ubuntu phone, which simply embed the mobile web site with webapp-container, and still result in seeing such advertisements, despite the fact that the "app" on Ubuntu is already installed, and the Android apps being advertised are simply not installable.

Having all the big web sites telling users of Ubuntu that they should be using Android instead, is not very good for the user experience at all.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Olivier Tilloy (osomon)]

Thanks for the report Rodney. Getting the default UA string right is a very tricky thing to achieve. We’ve tried several approaches in the past, and there’s no perfect solution. Removing the android token from the default UA usually gets us a very poor UX on most websites and webapps as we are served either desktop content, or WAP pages 'optimized' for the mobile web of the 1990s.

As unfortunate as this situation is indeed, it’s a reality that the vast majority of web content producers assume mobile == (Android || iOS).

Note that we have an override mechanism that allows us to select a different UA string on the fly when talking to certain websites. We can’t abuse this mechanism and ship thousands of overrides though, as it would slow it down noticeably, so we need a default UA that works well enough "out of the box".

Also note that webapps can override their UA too, so each webapp author can pick a suitable UA for the content they target.

[Olivier Tilloy (osomon)]

I’m marking this bug as invalid, not because it’s not an issue indeed, but because it will be better addressed as separate issues for specific websites/webapps.

The issues related to including an "android" token in the default UA have been discussed extensively already, it’s a tradeoff we agreed to make, we’re now left with improving the situation for specific use cases where possible.

@daker: would you mind filing a separate bug for twitter?

[Matthew Paul Thomas (mpt)]

The Windows Phone UA string does not mention Android. Does that cause IE Mobile to be served desktop or WAP pages on most Web sites? If not, why not?

[Olivier Tilloy (osomon)]

I don’t have a Windows phone to test, but it would be interesting to know how they deal with that indeed. Do you happen to know which UA string they send?

[Olivier Tilloy (osomon)]

Note that our default UA contains "Ubuntu 14.04 like Android 4.4", so we’re not actually pretending to be Android, just hinting that we have similar capabilities, which leaves the door open to content producers to do a proper detection.

For comparison, Firefox OS chose a deliberately short and clean default UA string (one that doesn’t pretend to be who it isn’t), and they’re having to do a lot of (largely volunteer) work to try and evangelise major websites to recognize their UA, while at the same time making heavy use of the override mechanism to send a fake UA (that most of the time contains an Android token anyway).

[dobey (dobey)]

Somehow "well enough" and "everyone telling all our users they should just use Android instead" don't seem to equate with me.

Also, this bug is hardly invalid. You can say it won't be fixed, but that is very different from invalid. If what you are saying is that it won't ever be fixed, then change the status to won't fix, please.

The list of strings being in the browser to serve up different User-Agent values for different sites, is a very bad "solution" IMO. You say it shouldn't grow too big, but if we add another site (or several sites including subdomains/etc) every time someone complains about this issue on a new web site, it's going to grow.

Also, how does this play into the converged future? Sure, on the phone, I might want to see the mobile version of the site instead, but when running the browser on my workstation, surely I should be seeing the full sites, which means dropping the "Android" from the User-Agent.

And finally, sites really should not be using the User-Agent to decide what data to send in response. Mozilla might have an evangelist team to help with their situation, but if they're evangelizing the adoption of support for their User-Agent, they are not evangelizing a "free and open web," but rather many different webs. Maybe we could work with Mozilla though, to get their evangelist team to promote web sites which do not use User-Agent to decide how to deliver a site to the user, and not have "mobile" sites, but use responsive design (or whatever the buzzword of the week is for making web sites that just work), for web sites.

[Olivier Tilloy (osomon)]

> Also, how does this play into the converged future?

The default UA depends on the form factor. The "like Android" token is present only on mobile devices. This is not dynamic yet, but will be eventually.

> And finally, sites really should not be using the User-Agent to
> decide what data to send in response.

But they do (an overwhelming majority of websites and webapps out there anyway). While I agree with your statement, it’s only wishful thinking and won’t help solve the problem with a finite amount of time and resources.

Note that Mozilla is advocating for detection of the "mobile" keyword only to identify mobile devices. While not ideal, this is already a huge step forward compared to the current situation. But in practice it is a very tedious and time consuming process, with limited results.

The approach we’re taking with webbrowser-app in Ubuntu is a pragmatic one given the constraints and user expectations.
It certainly isn’t the cleanest solution, and it does have a number of drawbacks. Realistic suggestions to improve the situation are very welcome.

I’m fine with leaving this bug open (we can actually use it as a reference to explain the rationale behind the current approach), although I hope you understand that given how complex the situation is there’s very little chance it will ever be "fixed".

[Olivier Tilloy (osomon)]

Here’s an interesting read from the mobile IE development team: http://blogs.msdn.com/b/ie/archive/2014/07/31/the-mobile-web-should-just-work-for-everyone.aspx.

In essence, they had to modify their UA string to get a number of sites to render properly in mobile IE. Unfortunately the article doesn’t go into details as to whether they only changed the default UA or if they use a per-site override mechanism, or a combination of both (like we do).

[Shuduo Sang (sangshuduo)]

I have a webapp need customize UA for getting right behavior of website. Right now it recogizes the browser is mobile version and refuse to serve.

[Olivier Tilloy (osomon)]

Shuduo Sang, do I understand it correctly that your webapp refuses to serve its content because it recognizes the browser as a mobile one? Is this on desktop or on a mobile device (if so which one)?
It would help if you could provide the URL of the app.

Note that there is a mechanism to override the default UA string for a given webapp: you can set it with the "user-agent-override" key in the webapp’s manifest file.

[Shuduo Sang (sangshuduo)]

Olivier, the url is https://wx.qq.com. the web requests a desktop version browser. could you pls point me where i can find the document or example how to overide UA in manifest file? thanks!

[Olivier Tilloy (osomon)]

In your app’s manifest.json, just add the following key-value pair:

    "user-agent-override": "<overridden UA string>"

where <overridden UA string> is a suitable UA string. From what I understand, you’ll want to provide a UA string of a desktop browser (try e.g. desktop chromium’s UA string).

[Shuduo Sang (sangshuduo)]

I have tried to override UA with "user-agent-override": "Mozilla/5.0 custom-user-agent-string (unlike AnyOtheBrowser)"
in the webapp-properties.json file as David Barth suggests. First problem is 'click build .' command do not package webapp-properties.json into click package as result. Then I manually copy it into the place of installed application directory of /opt. But the UA string I get from http://myhttp.info is still original mobile version.

[Olivier Tilloy (osomon)]

I’m changing the status of the bug to confirmed/triaged to acknoledge the issue, and to serve as a reference for similar future reports. Note however that as discussed at length already, there’s no good fix for this.

[Matthew Paul Thomas (mpt)]

In 2012, Mozilla developers explained why they chose not to include "Android" in their mobile UA string. <https://wiki.mozilla.org/B2G/User_Agent> Summary: Sites that mistakenly serve desktop-formatted pages are less likely to result in support calls, and are easier to address with evangelism, than sites that reasonably promote Android apps to a browser that claims to be running on Android. And that evangelizing sites to "Send your mobile version to any browser with 'Mobi' in its UA string" helps not just Firefox but every other mobile browser (for example, Ubuntu's).

If bug 1466427 has not been fixed by the time this bug is resolved, it may save time to fix them together.

[Olivier Tilloy (osomon)]

Thanks for the reminder mpt.

For the record, our initial approach a couple of years ago was to strictly follow Mozilla’s recommendations. However, we faced two severe issues that made us change our approach:

 - A vast majority of popular websites were serving incorrect content. Sometimes, it was even worse than just getting a desktop version, the "mobi" token would get us WAP-oriented content, or plainly broken content.

 - Evangelizing takes time and energy, and we realistically don’t have the resources to do that properly. Even just promoting and coordinating a 100% community effort would require more resources than we have at our disposal.

[Rüdiger Kupper (ruediger.kupper)]

Confirming the problem exists, and is imminent. Acknowledging it is hard to solve. However, it must be worked upon. UX depends on it.

[Matthew Exon (ubuntubugs-mexon)]

Thanks for the discussion of this issue. I came here from Bug #1506836, learned something, and now I've just finished fixing the code on my own site to handle "mobi" and "like android" tokens correctly :-) I've also submitted a feature request to LastPass.

To broaden the discussion slightly, could anyone comment on the likelihood of any of the following things being implemented? They would all provide a (perhaps unfriendly) workaround in the situation where a particular website refuses to play ball but a user has a desperate need for it to work correctly:

* Provide a button or something in the browser to explicitly switch UA string (like Sogou does to handle Chinese websites that require IE6 brokenness)
* Support an "about:debug" page or similar that allows a custom UA string to be typed in
* Allow the user to edit the override list on their phone (even just a text file editable in the terminal would do)
* Support plugins in the browser that would allow third-party developers to fix the problem for you

[Olivier Tilloy (osomon)]

> now I've just finished fixing the code on my own site to handle "mobi"
> and "like android" tokens correctly

Glad that this discussion helped. However the real positive take-away should be: don’t use the UA string to infer device capabilities at all. There are nowadays enough web APIs to properly detect device capabilities, and on the other hand, if done properly responsive design should ensure that a website renders well on all form factors without the need to discriminate user agents based on their identifiers, only on viewport size.

> * Provide a button or something in the browser to explicitly switch UA
> string (like Sogou does to handle Chinese websites that require IE6
> brokenness)
> * Support an "about:debug" page or similar that allows a custom UA
> string to be typed in
> * Allow the user to edit the override list on their phone (even just a
> text file editable in the terminal would do)

Any of the above three ideas would be doable, although none is currently on the roadmap. Contributions are very welcome, though!

> * Support plugins in the browser that would allow third-party
> developers to fix the problem for you

Not a goal for webbrowser-app, very unlikely to happen ever (or at least in the foreseeable future).

[flohack (flori-bin)]

Alas, how can it be that not even photos.google.com works with the default UA string from Ubuntu Touch? If I remove MobileSafari part it´s ok.

What kind of weird detection are they doing?