please stop smashing valid UTF-8 to ASCII in .changes files

Bug #33137 reported by Colin Watson
4
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Medium
Julian Edwards

Bug Description

Launchpad smashes valid UTF-8 in .changes files to ASCII when sending them out to the announcement list. The result of this is often sheer gibberish. For example:

https://lists.ubuntu.com/archives/dapper-changes/2006-February/006449.html

The string rendered there as "KAazeaka" was in fact "Қазақ"; the mangling made it incomprehensible.

An example from today:

https://lists.ubuntu.com/archives/dapper-changes/2006-February/006981.html

Instead of the original:

    - Use × rather than x in progress bar

... we got:

    - Use x rather than x in progress bar

... which made the changelog entry completely meaningless and meant I had to go and fetch the .diff.gz to figure out what it was talking about.

There's no good reason to smash valid UTF-8 to ASCII in .changes files when sending it out to the announcement list; katie didn't do it and I don't think Launchpad should either. Aside from the semantic problems above, it also breaks GPG signatures, which is problematic for Ubuntu developers when we want to figure out from the announcement list who signed a given upload (who in the case of sponsorship might not be mentioned in the Maintainer or Changed-By fields). There *is* a case for checking for invalid UTF-8 (most commonly, ISO-8859-1), although mangling that would also break GPG signatures. If it's only required to make the Content-Type in outgoing mail true, then I honestly suggest just leaving the data alone; if it's required for Launchpad's database or something, then perhaps you could only mangle the data if it's not already valid UTF-8.

Revision history for this message
Celso Providelo (cprov) wrote :

Ok, soon we should get people involved with the lp standard mail dispatcher and sort out this issue. The changefile content is preserved in librarian.

Changed in launchpad-upload-and-queue:
assignee: nobody → cprov
status: Unconfirmed → Confirmed
Revision history for this message
Celso Providelo (cprov) wrote :

We have improved mail dispatcher recently, let's bump priority to investigate it ASAP.

Changed in qprocd:
importance: Medium → High
Celso Providelo (cprov)
Changed in soyuz:
assignee: cprov → julian-edwards
Revision history for this message
Christian Reis (kiko) wrote :

Will be an optional for 1.1.7.

Changed in soyuz:
importance: High → Medium
Revision history for this message
Celso Providelo (cprov) wrote :

definitely, 1.1.8

Revision history for this message
Julian Edwards (julian-edwards) wrote :

We can also take this opportunity to migrate to the standard LP get_email_template() stuff instead of importing the local template_messages file.

Changed in soyuz:
status: Confirmed → In Progress
Revision history for this message
Julian Edwards (julian-edwards) wrote :

RF 4623.

Changed in soyuz:
status: In Progress → Fix Committed
Changed in soyuz:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.