[MIR] postgresql-14

[Christian Ehrhardt ]

TL;DR:
We are in the process of clarifying if the dependencies really will be ready to move to PG-14.
If yes => promote PG-14, demote/remove PG-13
If no => remove PG-14

--- Details ---

postgresql-14 now shows up in Jammy component mismatches.
https://people.canonical.com/~ubuntu-archive/component-mismatches-proposed.html

That is due to postgresql-common being auto-synced and changing the default to PG14.

Conceptually this is the same as
- https://bugs.launchpad.net/ubuntu/+source/postgresql-13/+bug/1902059
- https://bugs.launchpad.net/ubuntu/+source/postgresql-12/+bug/1851396
and all the others before it.

It seems we can go PG-13 in 21.04 as it looks rather good on the main PG-13 transition. Yet there are still a bunch of blockers left:
https://release.debian.org/transitions/html/postgresql-14.html

I guess those dependencies will need some time and then 4-8 weeks to get everything in shape.
Until then this will probably clutter update-excuses, therefore I'm tagging this as such to be known.

But for expectation, last time a few took quite a while and some even got removed for a short while.
Open issues are tracked here (thanks to Devrim / Myon for the great work)
https://wiki.postgresql.org/wiki/PostgreSQL_14_Extension_Bugs

Hopefully then eventually (probably ~January) we re-evaluate this and remove PG-13.

Due to the above this is filed as MIR bug so that one looking at mismatches can find it, but not meant to be promoted yet until the to-dos mentioned are resolved - therefore keeping it "invalid" for now..

[Christian Ehrhardt  (paelzer)]

I think we can do the MIR processing on this now, even though the transition itself might still take a while for the last dependencies to be ready. Therefore setting state back to NEW.

I'd assume that this is a "yes, just as before but please ensure that PG-13 is removed before end of cycle". But then I do not want to sign-off my own MIRs and need to wait for a second pair of eyes.

[Lukas Märdian (slyon)]

As in the past, we do not really need a MIR re-review for a version bump like this, so this is just a formalism:

MIR team ACK

But please add a bug-subscriber (=ubuntu-server) for the new postgresql-14 package and let's make sure to demote the old postgresql-13 package before the end of the Jammy cycle.

[Christian Ehrhardt  (paelzer)]

Yes, demoting PG-13 is part of the plan to promote this.

Ubuntu-Server is now subscribed to postgresql-14.

[Christian Ehrhardt  (paelzer)]

Some smaller fixes related got uploaded to Ubuntu recently:
- https://launchpad.net/ubuntu/+source/postgis/3.1.4+dfsg-3ubuntu1
- https://bugs.launchpad.net/ubuntu/+source/postgresql-14/+bug/1953128

But all of them still wait on the overall transition to happen.

As usual we waited for the transition in Debian to complete to then follow accordingly.
This now happened, [1] is over and it is time to resolve whatever is missing on our side to let the same happen.

postgresql-14 | 14.1-1 | testing | source, amd64, arm64, armel, armhf, i386, mips64el, mipsel, ppc64el, s390x
postgresql-14 | 14.1-1 | unstable | source, armel, armhf, s390x
postgresql-14 | 14.1-3 | unstable | source, amd64, arm64, i386, mips64el, mipsel, ppc64el
 postgresql-14 | 14.1-1ubuntu1 | jammy-proposed/universe | source, amd64, arm64, armhf, ppc64el, riscv64, s390x

Note: the armel/armhf issue is bug 1953128 and thereby not holding us back.

[1]: https://release.debian.org/transitions/html/postgresql-14.html

[Christian Ehrhardt  (paelzer)]

Current Autopkgtest-Blockers:

Note: all of those that remain already had "just reruns" from vorlon and ginggs on their +1 duty, to each of them there must be more :-/

Postgreql-14:
  - dovecot
    Affects all architectures.
    This wasn't really a PG-14 issue. But with the new dovecot 2.3.16 + openssl 3
    as it is in Jammy now it should work better. Tests retriggered with those versions.
  => This was resolved on retry as expected

postgresql-common:
- pgloader/3.6.2-1build1
  Affects only arm64/amd64
  No new version in Debian that we miss to sync
  Seems reproducible on arm64/amd64 on autopkgtest.u.c at least before the major release
  of packages with the wopenssl transition tonight.
  I've re-ran a few tests variations
    #1 as-is
    #2 with common+PG-14
    #3 all-proposed
    #4 started a local repro for debugging.
  It happens in all of them, so debugging on the local repro is the next step.

- sqlalchemy-i18n
  Affects all architectures.
  Similar post setup connection issues as pgloader is showing.
  Re-tested on autopkgtets.u.c with the same set oif conditions as pg-load and started
  a local repro.
  The local repro passed just fine, but not on the retries on
  the real infrastructure.
  For this one we seem to need a PPA with sqlalchemy-i18n tests + debugging
  code to iterate on it until we find the root cause

Note: I haven't started yet to check reverse dependencies, but it is known that multicorn will not get PG-14 compatible, so it is likely to be removed.
The Debian removal was in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1000589

I add tasks for those two packages that need test fixed here to have everyone find this bug more easily. And if we eventually need an upload we can use the bug as reference in the changelog.

[Christian Ehrhardt  (paelzer)]

PGLOADER

The pgloader issue stays reproducible but unclear.
I double checked if this was a regression in release, but without PG-14/PG-common from proposed it continues to work fine.

Debian seems to be affected by the same (at least for subtest ssl)
https://ci.debian.net/data/autopkgtest/unstable/amd64/p/pgloader/17321761/log.gz

But there it was essentially avoided via migration-reference/0 resetting the expectation.
See:
https://ci.debian.net/packages/p/pgloader/unstable/amd64/

I'm not sure this was correct, in Debian at the last time of a pass it was PG-13.
Then something unrelated broke the ssl test and then migration-reference/0 made it reset expectations and thereby accept PG-14.

As outlined above, it seems to be (at least on Ubuntu) a real regression and a recheck with a migration-reference/0 on the real infrastructure confirmed this. We can't get out the same way :-)

SQLALCHEMY-I18N

The story here is similar, this also works fine with the older version and seems to be a real regression. A no-change migration-reference run confirmed that even on that infrastructure it works fine with the non-proposed set of packages.
Since this one can not be reproduced locally (so far) the path still is debugging in a PPA. But currently there is an issue in LP so that builds do not get uploaded stalling this debugging.

[Christian Ehrhardt  (paelzer)]

Bonus-pain on pgloader - pgloader wasn't updated for ages:
https://launchpad.net/ubuntu/+source/pgloader/3.6.2-1

And the new rebuild (openssl 3) fails on armhf:
https://launchpad.net/ubuntu/+source/pgloader/3.6.2-1build1

[Christian Ehrhardt  (paelzer)]

After talking with the maintainer and further summarizing the situation I really think pgloader should be removed for now and come back into jammy later.
I filed bug 1953486 for that removal - removing the pgloader task here as it now has an own bug.

[Christian Ehrhardt  (paelzer)]

FYI I created a tracker for postgresql-14
https://people.canonical.com/~ubuntu-archive/transitions/html/postgresql-14.html

[Christian Ehrhardt  (paelzer)]

Also filed a removal bug against the known-bad multicorn at
https://bugs.launchpad.net/debian/+source/postgresql-multicorn/+bug/1953515

[Christian Ehrhardt  (paelzer)]

Since the MIR Acks is here let us mark this committed to show up correctly

[Christian Ehrhardt  (paelzer)]

Status update:
- Both removals got resolved

- sqlalchemy-i18n still has issues, but seems to be a flaky heisenbug (any debug I add makes it work fine)

- next to resolve is glom, it still has a dependency to postgresql-13 but it
  FTBFS right now.
  In Debian [1] it worked to pick up PG-14, both Distros had just rebuilds [2]
  for quite some time
  In sbuild I see crashes like:

PASS: tests/translations_po/test_document_import_po
./test-driver: line 112: 2368870 Aborted (core dumped) "$@" >> "$log_file" 2>&1
FAIL: tests/python/test_python_execute_func

After that the build hangs, Debian also had some fails in the most recent build, but no segfaults AFAICS. I need to check if a build on LP is any different and started a PPA for that.

[1]: https://buildd.debian.org/status/fetch.php?pkg=glom&arch=amd64&ver=1.30.4-6.1%2Bb4&stamp=1635888689&raw=1
[2]: https://launchpadlibrarian.net/510985351/buildlog_ubuntu-hirsute-amd64.glom_1.30.4-6.1build3_BUILDING.txt.gz

[Christian Ehrhardt  (paelzer)]

sqlalchemy was a case of re-debugging, I'll skip all the details, it essentially is the same as:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=997086
That works for us as well, but the version was an FTBFS - on all PPA and local tests that I had for the tests I used 1.0.1 already - so it should now build fine.
I restarted the build of the version in proposed.
If it succeeds we can re-trigger the tests and it should complete then.

[Christian Ehrhardt  (paelzer)]

sqlalchemy worked out as expected, some tests are still in the queue but others completed.

[Christian Ehrhardt  (paelzer)]

All other entries on the transition tracker that are left are false positives AFAICS.

Thereby all that is left I'm aware of is glom, which I found to be an FTBFS (independent to postgresql-14). The work on glom can continue after postgresql-14 migrates as we do not immediately remove postgresql-13.
Worst case in a week I'll file a bug to remove glom once I have a compelling reason out of trying to fix it.

But all of the above being said, postgresql-14 is IMHO ready now.
MIR ready, Dependencies ready, autopkgtests ready \o/
I'd ask the Ubuntu Archive admins to please promote postgresql-14 now.

Once it migrated it will clear many things in excuses, then afterwards we can have a look if we can remove postgresql-13 without further fallout (after glom is resolved).

[Steve Langasek (vorlon)]

Override component to main
postgresql-14 14.1-1ubuntu1 in jammy: universe/misc -> main
libecpg-compat3 14.1-1ubuntu1 in jammy amd64: main/libs/optional/100% -> main
libecpg-compat3 14.1-1ubuntu1 in jammy arm64: main/libs/optional/100% -> main
libecpg-compat3 14.1-1ubuntu1 in jammy armhf: main/libs/optional/100% -> main
libecpg-compat3 14.1-1ubuntu1 in jammy ppc64el: main/libs/optional/100% -> main
libecpg-compat3 14.1-1ubuntu1 in jammy riscv64: main/libs/optional/100% -> main
libecpg-compat3 14.1-1ubuntu1 in jammy s390x: main/libs/optional/100% -> main
libecpg-dev 14.1-1ubuntu1 in jammy amd64: main/libdevel/optional/100% -> main
libecpg-dev 14.1-1ubuntu1 in jammy arm64: main/libdevel/optional/100% -> main
libecpg-dev 14.1-1ubuntu1 in jammy armhf: main/libdevel/optional/100% -> main
libecpg-dev 14.1-1ubuntu1 in jammy ppc64el: main/libdevel/optional/100% -> main
libecpg-dev 14.1-1ubuntu1 in jammy riscv64: main/libdevel/optional/100% -> main
libecpg-dev 14.1-1ubuntu1 in jammy s390x: main/libdevel/optional/100% -> main
libecpg6 14.1-1ubuntu1 in jammy amd64: main/libs/optional/100% -> main
libecpg6 14.1-1ubuntu1 in jammy arm64: main/libs/optional/100% -> main
libecpg6 14.1-1ubuntu1 in jammy armhf: main/libs/optional/100% -> main
libecpg6 14.1-1ubuntu1 in jammy ppc64el: main/libs/optional/100% -> main
libecpg6 14.1-1ubuntu1 in jammy riscv64: main/libs/optional/100% -> main
libecpg6 14.1-1ubuntu1 in jammy s390x: main/libs/optional/100% -> main
libpgtypes3 14.1-1ubuntu1 in jammy amd64: main/libs/optional/100% -> main
libpgtypes3 14.1-1ubuntu1 in jammy arm64: main/libs/optional/100% -> main
libpgtypes3 14.1-1ubuntu1 in jammy armhf: main/libs/optional/100% -> main
libpgtypes3 14.1-1ubuntu1 in jammy ppc64el: main/libs/optional/100% -> main
libpgtypes3 14.1-1ubuntu1 in jammy riscv64: main/libs/optional/100% -> main
libpgtypes3 14.1-1ubuntu1 in jammy s390x: main/libs/optional/100% -> main
libpq-dev 14.1-1ubuntu1 in jammy amd64: main/libdevel/optional/100% -> main
libpq-dev 14.1-1ubuntu1 in jammy arm64: main/libdevel/optional/100% -> main
libpq-dev 14.1-1ubuntu1 in jammy armhf: main/libdevel/optional/100% -> main
libpq-dev 14.1-1ubuntu1 in jammy ppc64el: main/libdevel/optional/100% -> main
libpq-dev 14.1-1ubuntu1 in jammy riscv64: main/libdevel/optional/100% -> main
libpq-dev 14.1-1ubuntu1 in jammy s390x: main/libdevel/optional/100% -> main
libpq5 14.1-1ubuntu1 in jammy amd64: main/libs/optional/100% -> main
libpq5 14.1-1ubuntu1 in jammy arm64: main/libs/optional/100% -> main
libpq5 14.1-1ubuntu1 in jammy armhf: main/libs/optional/100% -> main
libpq5 14.1-1ubuntu1 in jammy ppc64el: main/libs/optional/100% -> main
libpq5 14.1-1ubuntu1 in jammy riscv64: main/libs/optional/100% -> main
libpq5 14.1-1ubuntu1 in jammy s390x: main/libs/optional/100% -> main
postgresql-14 14.1-1ubuntu1 in jammy amd64: universe/database/optional/100% -> main
postgresql-14 14.1-1ubuntu1 in jammy arm64: universe/database/optional/100% -> main
postgresql-14 14.1-1ubuntu1 in jammy armhf: universe/database/optional/100% -> main
postgresql-14 14.1-1ubuntu1 in jammy ppc64el: universe/database/optional/100% -> main
postgresql-14 14.1-1ubuntu1 in ja...

[Christian Ehrhardt  (paelzer)]

Thank you,
it seems its llvm-11 dependency (incompatible with 13 on s390x, being worked on in Debian) still prevents migration to -release. I'll have a more detailed look tomorrow.

[Christian Ehrhardt  (paelzer)]

python-sqlalchemy-utils was not shown before, but now is a test fail for postgresql-common

About the dependency to llvm-11-dev, clang-11.
Two things to know here, first of all that will sooner or later be llvm-13 due to [1].
But right now there is an issue around s390x that blocks this:
  +ERROR: failed to JIT module: Added modules have incompatible data layouts: E-m:e-i1:8:16-i8:8:16-i64:64-f128:64-a:8:16-n32:64 (module) vs E-m:e-i1:8:16-i8:8:16-i64:64-f128:64-v128:64-a:8:16-n32:64 (jit)

Both are not important right now, the problem is that the dependency is from postgresqls -dev package which is not supposed to be in main. Even the older postgresql-server-dev-13 depends on it, but postgresql-server-dev-13 is and postgresql-server-dev-14 should not be in main.
That is just an artifact from promoting everything in comment #16.
Nothing changed in seeds or dependencies - these are properly listed in [2].

Binary only movements to universe (ubuntu-server)
postgresql-server-dev-14 postgresql-14

I'll ping an AA for the demotion of postgresql-server-dev-14 and will check why/where the test fail of python-sqlalchemy-utils comes from.

[1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1000915
[2]: https://people.canonical.com/~ubuntu-archive/component-mismatches-proposed.html

[Christian Ehrhardt  (paelzer)]

Thanks to RAOF the postgresql-server-dev-14 -> llvm issue is resolved.

python-sqlalchemy-utils was a test on an old version 0.36.8-2ubuntu2 failing when openssl3 migrated. There now is a merge of the new version 0.37.8-1ubuntu1 with new tests.
Migration reference runs with that new version worked fine, I'm retesting postgresql with that new version locally and on LP.
The new version passes fine with the right triggers to the new postgresql and should unblock on the next migration run.
The x86 run is the only one missing and in the autopkgtest queue which is rather slow atm - estimating 10h.

pgsql-ogr-fdw also had a test fail, but someone already submitted a migration-reference run and that failed as well independent to PG-14 - so that should be fine as well.

... waiting for tests and proposed migration to run showing me new blockers

[Christian Ehrhardt  (paelzer)]

python-sqlalchemy-utils fully resolved now

Now excuses lists the amd64 variant of pgsql-ogr-fdw as failing (it didn't before)
https://autopkgtest.ubuntu.com/packages/p/pgsql-ogr-fdw/jammy/amd64
Here (as on the other architectures) a migration-reference run is running, assuming that fails like the others it should be good, otherwise it needs further investigation.

[Christian Ehrhardt  (paelzer)]

pgsql-ogr-fdw@amd64 migration-reference/0 (was auto-scheduled it seems) is bad as expected.
That should (tm) have been the last blocker.

Maybe I need to check installability in update_output after the next run, I didn't do so so far ...

[Christian Ehrhardt  (paelzer)]

 postgresql-14 | 14.1-1ubuntu1 | jammy | source, amd64, arm64, armhf, ppc64el, riscv64, s390x

This is complete.

The cleanup will be getting glom to build as then we can remove src:postgresql-13

I also removed the no more needed transition tracker.