Fake useragent string causing 406 Errors on Apache Servers with default configuration

[Stephen Lyons]

From user Sanaki
On forums {http://forums.mudlet.org/viewtopic.php?f=9&t=4594}:
> downloadFile error 406 issue
>
> Sat Sep 06, 2014 5:14 am
> When attempting to download an image using downloadFile from my personal server, it's throwing an error 406 message at me ("Not Acceptable", for those unfamiliar). Fetching the same file from a browser works fine. I grabbed the communication chain with wireshark to see what's going awry, but I can't see anything obviously wrong there. Theoretically error 406 should mean the accept header sent by Mudlet is wrong. Granted, I'm years out of practice when it comes to dealing with apache, TCP, PHP, etc, but even still I'm wondering if someone has insight. I noticed Akaya had the same issue here, which was left unanswered.

> EDIT: I removed a bit of information here because I've identified the issue, but have no simple solution. This is caused by apache Mod_Security rule 900095, "Bad UA :: Fake Mozilla Agent". This will be an issue on any server using the boilerplate Mod_Security rules, and that includes irreparably most shared hosting providers. Mudlet identifies itself with the full UA string "Mozilla/5.0". Updating this to a more accurate UA would appear to fix the issue, though I'm unsure what that would be. Is this something that can't be adjusted for some reason?

======
> ... this is http and it's a response to Mudlet using a fake UserAgent string to identify itself. Technically the 406 isn't an error, it's a perfectly rational response to an unknown browser that's blatantly lying about what it is.
======
> I seem to have found the solution, though I have yet to test it. In the source, in TLuaInterpreter.cpp, I believe inserting this line after line 8082 would solve our issue completely:

>> request.setRawHeader("User-Agent", "Mudlet 2.1");

> Now, that UA string can be changed to anything really, as long as it's something that -isn't- "Mozilla/5.0", which is what it defaults to. Realistically, if there's a variable accessible from there that contains the current Mudlet version or such, that would probably be best, but if so I have no idea where to find it. Or it could just be "Mudlet".

[Stephen Lyons (slysven)]

We could produce the modification to do this quite easily to comply with RFC2616 {http://www.ietf.org/rfc/rfc2616.txt} but the questions are:
* should Mudlet's UA string begin with "Mozilla/5.0" - I think it has to, but it must include additional stuff to make it clear that it is NOT the famous browser but something with some behaviour in common (which I think is a comment in '(' ')''s such as "(Mudlet/3.0.0)" ) on the other hand we may wish to limit the amount of version information for "User Agent Sniffing" to, e.g. no more than major and minor numbers to prevent too much user information from being revealed, so what do we choose?
* the second non-comment component seems to be the rendering platform - I'd need to do more research to pronounce on this but I suspect it may be something like "WebKit/?.?" so what is the right thing?
* do we need to provide the user with a way to change this string if they so desire, and know what they are doing?

[Vadim Peretokin (vperetokin)]

The default comes from Qt... that function just needs to download a file, not render a webpage, so I think we just should do research on what would be a good user agent string these days and set it to that. I don't think we need to provide a function to have the user change it, given how this has come up once in six years of Mudlet and no circumstances have really changed to necessitate the user agent changing often.

[Stephen Lyons (slysven)]

So either "Mudlet/<major version number>.<minor version number>" or "Mozilla/5.0 (Mudlet/<major version number>.<minor version number>)" with <major version number> and <minor version number> parsed from the APP_VERSION #define. Then hope that we don't break it for other users who have gotten away with "Mozilla/5.0" so far.

[Colin L Crowley (sanaki)]

As long as the user agent string is almost anything but "Mozilla/5.0", it'll be fixed. It's running into a default apache Mod_Security rule that looks for that exact string. There isn't really any conceivable reason changing it should mess up current scripts. If there is somehow concern about that anyway, or concern about UA string choice changing the format of the files people fetch, making the UA changeable from the lua side is an option as well.

[Stephen Lyons (slysven)]

Ah, right, well I'll code in "Mudlet/<major>.<minor>" then.

[Vadim Peretokin (vperetokin)]

Sounds OK. Should test it with Dropbox that we can still download from
there, that is one of the more popular locations for hosting stuff that
Mudlet scripts download.

On Wed, Sep 10, 2014 at 4:00 AM, Stephen Lyons <email address hidden>
wrote:

> Ah, right, well I'll code in "Mudlet/<major>.<minor>" then.
>
> ** Changed in: mudlet
> Assignee: (unassigned) => Stephen Lyons (slysven)
>
> ** Changed in: mudlet
> Importance: Undecided => Low
>
> --
> You received this bug notification because you are a member of Mudlet
> Makers, which is subscribed to Mudlet.
> https://bugs.launchpad.net/bugs/1366781
>
> Title:
> Fake useragent string causing 406 Errors on Apache Servers with
> default configuration
>
> Status in Mudlet the MUD client:
> New
>
> Bug description:
> From user Sanaki
> On forums {http://forums.mudlet.org/viewtopic.php?f=9&t=4594}:
> > downloadFile error 406 issue
> >
> > Sat Sep 06, 2014 5:14 am
> > When attempting to download an image using downloadFile from my
> personal server, it's throwing an error 406 message at me ("Not
> Acceptable", for those unfamiliar). Fetching the same file from a browser
> works fine. I grabbed the communication chain with wireshark to see what's
> going awry, but I can't see anything obviously wrong there. Theoretically
> error 406 should mean the accept header sent by Mudlet is wrong. Granted,
> I'm years out of practice when it comes to dealing with apache, TCP, PHP,
> etc, but even still I'm wondering if someone has insight. I noticed Akaya
> had the same issue here, which was left unanswered.
>
> > EDIT: I removed a bit of information here because I've identified
> the issue, but have no simple solution. This is caused by apache
> Mod_Security rule 900095, "Bad UA :: Fake Mozilla Agent". This will be
> an issue on any server using the boilerplate Mod_Security rules, and
> that includes irreparably most shared hosting providers. Mudlet
> identifies itself with the full UA string "Mozilla/5.0". Updating this
> to a more accurate UA would appear to fix the issue, though I'm unsure
> what that would be. Is this something that can't be adjusted for some
> reason?
>
> ======
> > ... this is http and it's a response to Mudlet using a fake UserAgent
> string to identify itself. Technically the 406 isn't an error, it's a
> perfectly rational response to an unknown browser that's blatantly lying
> about what it is.
> ======
> > I seem to have found the solution, though I have yet to test it. In
> the source, in TLuaInterpreter.cpp, I believe inserting this line after
> line 8082 would solve our issue completely:
>
> >> request.setRawHeader("User-Agent", "Mudlet 2.1");
>
> > Now, that UA string can be changed to anything really, as long as
> it's something that -isn't- "Mozilla/5.0", which is what it defaults
> to. Realistically, if there's a variable accessible from there that
> contains the current Mudlet version or such, that would probably be
> best, but if so I have no idea where to find it. Or it could just be
> "Mudlet".
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mudlet/+bug/1366781/+subscr...

[Colin L Crowley (sanaki)]

I realize now I only referenced a line number earlier, which is subject to change (currently appears to be 8593). The line the user agent definition needs to be added after is:

QNetworkRequest request = QNetworkRequest( QUrl( _url ) );

I have no familiarity with QT myself and limited familiarity with Mudlet's source, though I could give it a glance if it would help speed up implementation of a fix. I realize this is a low priority in the grand scheme of things, but for me personally it's rather significantly high, so I'm happy to help with it where I can. Without delving into the source a bit I'm not sure what the syntax to include the version numbers would be, but the basic line needed would be:

request.setRawHeader("User-Agent", "Mudlet/3.0");

[Vadim Peretokin (vperetokin)]

It's not something we'll have in 3.0, as that is frozen - but the release after that would be able to include it.

[Stephen Lyons (slysven)]

This issue should be addressed as part of another bug fix, in Pull Requests https://github.com/Mudlet/Mudlet/pull/323 (development) AND https://github.com/Mudlet/Mudlet/pull/324 (release_30) branches.

[Vadim Peretokin (vperetokin)]

Migrating issues to Github, please follow the new discussion here: https://github.com/Mudlet/Mudlet/issues/570

This issue needs to be closed and there is no appropriate status, so will set it to "Opinion" just for migration purposes.

[chawila chawila (chawila)]

So how can i solve that problem in QT/Qml?.
i want to access the image from the server, but i get (Not acceptable).

[Stephen Lyons (slysven)]

Please go to the migrated issue on Github https://github.com/Mudlet/Mudlet/issues/570 - although we have fixed the original issue so I do not quite get what your issue is!