Unicode errors with PostgreSQL and RawStr fields

[Aurélien Bompard]

When using PostgreSQL as a database, it's impossible to set the moderator_password field of a mailing list, which is of type RawStr.
To be precise, it's possible to set it, but getting it back from the database after a commit will cause a TypeError with the message:
"Expected str, found <type 'unicode'>" as if PostgreSQL always returned unicode on TEXT fields and Storm did not convert them to bytestrings. I get no errors on an SQLite database.

I don't know if the problem comes from Mailman or Storm. I have written a python script to reproduce it, see the attached file. There are comments, it should be easily readable.

[Aurélien Bompard (abompard)]

I found why! In "mailman/src/mailman/database/schema/postgres.sql", the moderator_password field is of type "TEXT" instead of being "BYTEA". As a result, the psycopg2 driver converts it to an unicode string.
This is coherent with the Storm manual which states that RawStr types should be set as BYTEA types in PostgreSQL:
https://storm.canonical.com/Manual#Table_of_properties_vs._python_vs._database_types

I believe a schema migration script will be needed.

[Aurélien Bompard (abompard)]

Here's the schema migration script, it's pretty simple.
I'm converting existing data to ASCII and not UTF-8 because we don't expect non-ascii chars to be in the moderator_password column (it should contain the result of the config.password_context.encrypt() function, which is ASCII-only.

[Aurélien Bompard (abompard)]

This bug is no longer valid since Storm has been replaced by SQLAlchemy.