api response for database error includes traceback

[Michael Nelson]

This should be returning an oops ID in the response content.

We do have oopses for other errors, so need to check what's special here that causes the traceback to be included:

https://pastebin.canonical.com/109234/

[Fabián Ezequiel Gallina (fgallina)]

After a search on the matter I think this becomes really two issues:

  1. is the database error itself.
  2. the oopses machinery failing to handle this.

For #1 I think it's related to upgrading from Postgres 8.x to 9.1.

I was confirmed we are using 9.1.9-0ubuntu12.04 and from wiki and logs I understand RnR started its life in Natty which had 8.4.

Now on Postgres release notes (http://www.postgresql.org/docs/9.1/static/release-9-1-2.html) there's this excerpt:

    Make contrib/citext's upgrade script fix collations of citext
    columns and indexes (Tom Lane)

    Existing citext columns and indexes aren't correctly marked as
    being of a collatable data type during pg_upgrade from a pre-9.1
    server, or when a pre-9.1 dump containing the citext type is
    loaded into a 9.1 server. That leads to operations on these
    columns failing with errors such as "could not determine which
    collation to use for string comparison". This change allows them
    to be fixed by the same script that upgrades the citext module
    into a proper 9.1 extension during CREATE EXTENSION citext FROM
    unpackaged.

    If you have a previously-upgraded database that is suffering from
    this problem, and you already ran the CREATE EXTENSION command,
    you can manually run (as superuser) the UPDATE commands found at
    the end of SHAREDIR/extension/citext--unpackaged--1.0.sql. (Run
    pg_config --sharedir if you're uncertain where SHAREDIR is.) There
    is no harm in doing this again if unsure.

So I think that running this may fix the database error itself.

Now WRT to the oops, I wasn't able to replicate it, probably code
changed since then and this is not happening anymore, I'll try harder
but I think it's important to ensure the COLLATE error won't happen
again.

[Michael Nelson (michael.nelson)]

Sheesh - we hit a similar issue on staging and fixed the citext issue there, and specifically asked how we can make sure the same fix is applied on prod when the prod db was upgraded, as it wasn't at that point.

Not sure if it's identical, but here's the RT with the details https://rt.admin.canonical.com/Ticket/Display.html?id=69883