Upgrade script failed: 'su: user postgres does not exist'

[Dennis Dmitriev]

Steps to reproduce:
    1. Deploy Fuel master node from ISO 5.1.1 build
    2. Upload and unpack *.tar.lrz file from http://jenkins-product.srt.mirantis.net:8080/job/6.0.iso/46/ to /var folder on the fuel master node.
    3. Run upgrade.sh script to perform upgrade from 5.1.1 to 6.0

Expected result:
    - upgrade script is completed successfully.

Actual result:
    - upgrade script is failed during start the container 'fuel-core-6.0-postgres' with the following error and performs rollback to 5.1.1 :
=============================
2014-10-27 07:08:16 DEBUG 5467 (utils) Stdout and stderr of command "lxc-attach --name 3a9ecf21f95f780050c15e2905ee7d682def3355b5d1a2a8427ffa6e3c0c55af -- su postgres -c "psql -f /tmp/upgrade/pg_dump_all.sql postgres"":
2014-10-27 07:08:16 DEBUG 5467 (utils) su: user postgres does not exist
=============================

Stdout of upgrade.sh:
=================
http://paste.openstack.org/show/125190/

Fuel master node version:
=====================
api: '1.0'
astute_sha: f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13
auth_required: true
build_id: 2014-10-20_00-37-45
build_number: '34'
feature_groups:
- mirantis
fuellib_sha: 4f8414a08316a0c569bf74752b801be77169a9c5
fuelmain_sha: 7bac3edb9760449ccd2c43d9078a6150c0685590
nailgun_sha: b9792cb5bbecddfa9c5c3afb4d0f961a2a2776a7
ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
production: docker
release: 5.1.1
release_versions:
  2014.1.1-5.1.1:
    VERSION:
      api: '1.0'
      astute_sha: f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13
      build_id: 2014-10-20_00-37-45
      build_number: '34'
      feature_groups:
      - mirantis
      fuellib_sha: 4f8414a08316a0c569bf74752b801be77169a9c5
      fuelmain_sha: 7bac3edb9760449ccd2c43d9078a6150c0685590
      nailgun_sha: b9792cb5bbecddfa9c5c3afb4d0f961a2a2776a7
      ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
      production: docker
      release: 5.1.1

[Dennis Dmitriev (ddmitriev)]

Looks like new 'fuel/postgres_6.0' docker container actually is not contained postgresql-server yet.

I've added "ps auxf" to the command line with "psql" and here what we have: the container installation process is ongoing at the moment (see "/usr/bin/python /usr/bin/yum -d 0 -e 0 -y install postgresql-server" at the end of log):

=============================================================================
2014-10-28 07:06:00 DEBUG 8013 (utils) Execute command "lxc-attach --name 0aed43347a113681a3abfacd6a74c2e2c7d41ecd7e997dd24e59ed243559bba2 -- ps auxf; psql -f /tmp/upgrade/pg_dump_all.sql postgres"
2014-10-28 07:06:00 DEBUG 8013 (utils) Stdout and stderr of command "lxc-attach --name 0aed43347a113681a3abfacd6a74c2e2c7d41ecd7e997dd24e59ed243559bba2 -- ps auxf; psql -f /tmp/upgrade/pg_dump_all.sql postgres":
2014-10-28 07:06:00 DEBUG 8013 (utils) USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
2014-10-28 07:06:00 DEBUG 8013 (utils) root 10 0.0 0.0 13368 1028 ? R+ 07:05 0:00 ps auxf
2014-10-28 07:06:00 DEBUG 8013 (utils) root 1 1.0 0.0 11300 1328 ? S 07:05 0:00 /bin/bash /usr/local/bin/start.sh
2014-10-28 07:06:00 DEBUG 8013 (utils) root 9 0.0 0.1 70660 3168 ? R 07:05 0:00 rpm --rebuilddb
2014-10-28 07:06:00 DEBUG 8013 (utils) /bin/sh: psql: command not found
2014-10-28 07:06:04 DEBUG 8013 (utils) Execute command "lxc-attach --name 0aed43347a113681a3abfacd6a74c2e2c7d41ecd7e997dd24e59ed243559bba2 -- ps auxf; psql -f /tmp/upgrade/pg_dump_all.sql postgres"
2014-10-28 07:06:04 DEBUG 8013 (utils) Stdout and stderr of command "lxc-attach --name 0aed43347a113681a3abfacd6a74c2e2c7d41ecd7e997dd24e59ed243559bba2 -- ps auxf; psql -f /tmp/upgrade/pg_dump_all.sql postgres":
2014-10-28 07:06:11 DEBUG 8013 (utils) USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
2014-10-28 07:06:11 DEBUG 8013 (utils) root 175 0.0 0.0 13360 1024 ? R+ 07:06 0:00 ps auxf
2014-10-28 07:06:11 DEBUG 8013 (utils) root 1 0.0 0.0 11300 1328 ? S 07:05 0:00 /bin/bash /usr/local/bin/start.sh
2014-10-28 07:06:11 DEBUG 8013 (utils) root 11 19.1 2.4 227360 49900 ? Dl 07:06 0:01 /usr/bin/ruby /usr/bin/puppet apply -v /etc/puppet/modules/nailgun/examples/postgres-only.pp
2014-10-28 07:06:11 DEBUG 8013 (utils) /bin/sh: psql: command not found
2014-10-28 07:06:16 DEBUG 8013 (utils) Execute command "lxc-attach --name 0aed43347a113681a3abfacd6a74c2e2c7d41ecd7e997dd24e59ed243559bba2 -- ps auxf; psql -f /tmp/upgrade/pg_dump_all.sql postgres"
2014-10-28 07:06:16 DEBUG 8013 (utils) Stdout and stderr of command "lxc-attach --name 0aed43347a113681a3abfacd6a74c2e2c7d41ecd7e997dd24e59ed243559bba2 -- ps auxf; psql -f /tmp/upgrade/pg_dump_all.sql postgres":
2014-10-28 07:06:27 DEBUG 8013 (utils) USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
2014-10-28 07:06:27 DEBUG 8013 (utils) root 340 0.0 0.0 13364 1032 ? R+ 07:06 0:00 ps auxf
2014-10-28 07:06:27 DEBUG 8013 (utils) root 1 0.0 0.0 11300 1328 ? S 07:05 0:00 /bin/bash /usr/local/bin/start.sh
2014-10-28 ...

[Dennis Dmitriev (ddmitriev)]

Confirmed on the ISO http://jenkins-product.srt.mirantis.net:8080/view/6.0/job/6.0.iso/65/ which contains all the fixes from the bug https://bugs.launchpad.net/fuel/+bug/1386118

Please take a look at the log in the https://bugs.launchpad.net/fuel/+bug/1386069/comments/1 . Postgres was not installed when container postgres.tar was built, but it tries to install right after container 'fuel-core-6.0-postgres' started. Looks like some check at the building time is working incorrectly.

[Matthew Mosesohn (raytrac3r)]

You need to wait for puppet to run. I opted to not run puppet during container build because our puppet manifests can't set initial values for all the DBs and then create the users with randomized passwords later. This is not a change from 5.1 at all.

[Ihor Kalnytskyi (ikalnytskyi)]

https://review.openstack.org/#/c/133170/
https://review.openstack.org/#/c/133171/
https://review.openstack.org/#/c/133190/

Fix Commited

[Andrey Sledzinskiy (asledzinskiy)]

verified on {

    "build_id": "2014-12-17_03-38-09",
    "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4",
    "build_number": "50",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "dc621c96e30f0dd284998e975f3b800afe9fc082",
    "production": "docker",
    "fuelmain_sha": "fb89876812320464ebad60e7ca34b433d6b1d815",
    "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91",
    "feature_groups": [
        "mirantis"
    ],
    "release": "6.0",
    "release_versions": {
        "2014.1.3-5.1.1": {
            "VERSION": {
                "build_id": "2014-12-03_01-07-36",
                "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
                "build_number": "48",
                "api": "1.0",
                "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d",
                "production": "docker",
                "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404",
                "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "5.1.1",
                "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"
            }
        },
        "2014.2-6.0": {
            "VERSION": {
                "build_id": "2014-12-17_03-38-09",
                "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4",
                "build_number": "50",
                "api": "1.0",
                "nailgun_sha": "dc621c96e30f0dd284998e975f3b800afe9fc082",
                "production": "docker",
                "fuelmain_sha": "fb89876812320464ebad60e7ca34b433d6b1d815",
                "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "6.0",
                "fuellib_sha": "484f53f586c084cc38c88615b0594d9c48882c5a"
            }
        }
    },
    "fuellib_sha": "484f53f586c084cc38c88615b0594d9c48882c5a"

}