binary/name gets confused under upgrades of osapi_compute and metadata when using wsgi files

[Ebbex]

Before an upgrade, we have these type of entries in the db.

MariaDB [nova]> SELECT id, host, `binary`, deleted, version FROM services;
+----+--------------+--------------------+---------+---------+
| id | host | binary | deleted | version |
+----+--------------+--------------------+---------+---------+
| 5 | r1-n-os-api | nova-osapi_compute | 0 | 16 |
| 21 | r1-n-m-api | nova-metadata | 0 | 16 |

The wsgi files we run basically boil down to something like

  NAME=metadata
  return wsgi_app.init_application(NAME)

In the wsgi_app.py we see this function

  service_ref = objects.Service.get_by_host_and_binary(ctxt, host, name)

Which results in a really big query, which again comes down to

  SELECT host, `binary` FROM services
    WHERE host = 'r1-n-m-api' AND `binary` == 'metadata'

No results. service_ref is set to None. Carry on.

  if service_ref:
    #Nope.
  else:
    try:
      ...
      service_obj.host = host
      service_obj.binary = 'nova-%s' % name
      service_obj.create()

Which results in a INSERT statement something like this;

  INSERT INTO services(host, `binary`, report_count, disabled, deleted, version)
    VALUES ('r1-n-m-api', 'nova-metadata', 0, 0, 0, 22)

  ERROR 1062 (23000): Duplicate entry 'r1-n-m-api-nova-metadata-0' for key 'uniq_services0host0binary0deleted'

[Sean Dague (sdague)]

What sequence of events got you here? Is this changing the deployment type post upgrade as well? It would be good to fully qualify what failed here, and why we didn't catch it, and if we need to put something into nova-manage to help.

[Sean Dague (sdague)]

Patch in progress - https://review.openstack.org/#/c/501359/

[Ebbex (eb4x)]

We're running nova-os-compute-api (nova/api/openstack/compute/wsgi.py), nova-metadata-api (nova/api/metadata/wsgi.py) and nova-placement-api (nova/api/openstack/placement/wsgi.py) under uwsgi. Each in separate containers, under venvs.

[Ebbex (eb4x)]

If you look at the code in the patch, or rather the code before the patch, it tries to update one service-binary 'osapi_compute'. If it doesn't find it, it tries to insert a *different* service-binary 'nova-osapi_compute'.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/501359
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0b4a021e4224981e83dca67e7519458f9939f3cd
Submitter: Zuul
Branch: master

commit 0b4a021e4224981e83dca67e7519458f9939f3cd
Author: Erik Berg <email address hidden>
Date: Wed Sep 6 18:38:29 2017 +0000

    Fix binary name

    Before an upgrade, we have these type of entries in the db.

    MariaDB [nova]> SELECT id, host, `binary`, deleted, version FROM services;
    +----+--------------+--------------------+---------+---------+
    | id | host | binary | deleted | version |
    +----+--------------+--------------------+---------+---------+
    | 5 | r1-n-os-api | nova-osapi_compute | 0 | 16 |
    | 21 | r1-n-m-api | nova-metadata | 0 | 16 |

    The wsgi files we run basically boil down to something like

      NAME=metadata
      return wsgi_app.init_application(NAME)

    In the wsgi_app.py we see this function

      service_ref = objects.Service.get_by_host_and_binary(ctxt, host, name)

    Which results in a really big query, which again comes down to

      SELECT host, `binary` FROM services
        WHERE host = 'r1-n-m-api' AND `binary` == 'metadata'

    No results. service_ref is set to None. Carry on.

      if service_ref:
        #Nope.
      else:
        try:
          ...
          service_obj.host = host
          service_obj.binary = 'nova-%s' % name
          service_obj.create()

    Which results in a INSERT statement something like this;

      INSERT INTO services(host, `binary`, report_count, disabled, deleted, version)
        VALUES ('r1-n-m-api', 'nova-metadata', 0, 0, 0, 22)

      ERROR 1062 (23000): Duplicate entry 'r1-n-m-api-nova-metadata-0' for key 'uniq_services0host0binary0deleted'

    So the first suggested fix is to prepend 'nova-' to the name, and make both
    queries ask for 'nova-metadata'. There's also a check that it doesn't start
    with 'nova-', incase someone decides to prepend 'nova-' to the NAME= in the
    wsgi-file. Which migth be a litte overkill, but just a safeguard none the less.

    Change-Id: I58cf9a0115a98c78e5d2fb57c41c13ba6fac0fad
    Closes-bug: 1715463

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/524531

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/524531
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2d79baf6a3ce4529ba61a5efded7f043d89825a2
Submitter: Zuul
Branch: stable/pike

commit 2d79baf6a3ce4529ba61a5efded7f043d89825a2
Author: Erik Berg <email address hidden>
Date: Wed Sep 6 18:38:29 2017 +0000

    Fix binary name

    Before an upgrade, we have these type of entries in the db.

    MariaDB [nova]> SELECT id, host, `binary`, deleted, version FROM services;
    +----+--------------+--------------------+---------+---------+
    | id | host | binary | deleted | version |
    +----+--------------+--------------------+---------+---------+
    | 5 | r1-n-os-api | nova-osapi_compute | 0 | 16 |
    | 21 | r1-n-m-api | nova-metadata | 0 | 16 |

    The wsgi files we run basically boil down to something like

      NAME=metadata
      return wsgi_app.init_application(NAME)

    In the wsgi_app.py we see this function

      service_ref = objects.Service.get_by_host_and_binary(ctxt, host, name)

    Which results in a really big query, which again comes down to

      SELECT host, `binary` FROM services
        WHERE host = 'r1-n-m-api' AND `binary` == 'metadata'

    No results. service_ref is set to None. Carry on.

      if service_ref:
        #Nope.
      else:
        try:
          ...
          service_obj.host = host
          service_obj.binary = 'nova-%s' % name
          service_obj.create()

    Which results in a INSERT statement something like this;

      INSERT INTO services(host, `binary`, report_count, disabled, deleted, version)
        VALUES ('r1-n-m-api', 'nova-metadata', 0, 0, 0, 22)

      ERROR 1062 (23000): Duplicate entry 'r1-n-m-api-nova-metadata-0' for key 'uniq_services0host0binary0deleted'

    So the first suggested fix is to prepend 'nova-' to the name, and make both
    queries ask for 'nova-metadata'. There's also a check that it doesn't start
    with 'nova-', incase someone decides to prepend 'nova-' to the NAME= in the
    wsgi-file. Which migth be a litte overkill, but just a safeguard none the less.

    Change-Id: I58cf9a0115a98c78e5d2fb57c41c13ba6fac0fad
    Closes-bug: 1715463
    (cherry picked from commit 0b4a021e4224981e83dca67e7519458f9939f3cd)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.4 release.