synchronous_reader: Either the code or documentation is wrong

[Arun S A G]

Line https://github.com/openstack/oslo.db/blob/0260f0e/oslo_db/sqlalchemy/enginefacade.py#L268 reads

""" :param synchronous_reader: whether or not to assume a "reader" context
         needs to guarantee it can read data committed by a "writer" assuming
         replication lag is present; defaults to True. When False, a
         @reader context works the same as @async_reader and will select
         the "slave" database if present.
"""

According to code
https://github.com/openstack/oslo.db/blob/master/oslo_db/sqlalchemy/enginefacade.py#L396

When synchronous_reader or _ASYNC_READER set to True it uses the reader, but the documentation describes opposite.

Please clarify how this is supposed to work and how an operator is supposed to set synchronous_reader to False from configuration (nova doesn't seem to have anything related to synchronous_reader)

[Mike Bayer (zzzeek)]

code is wrong. All of the intent for synchronous_reader throughout enginefacade as well as in the test code in test_enginefacade needs to have the role of synchronous_reader reversed. it is written as though it means to say "asynchronous_reader", which might have been a better name for this field.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/539046

[Arun S A G (sagarun)]

Hello,

I tested this patch in our staging environment, i can confirm that i no longer see race condition between FixedIP.associate and FixedIPList.get_by_instance_uuid

This is how i tested the patch,

1. Disabled semi-synchronous replication on the read-only slave

mysql> SHOW STATUS LIKE 'rpl_semi_sync_slave%';
+----------------------------+-------+
| Variable_name | Value |
+----------------------------+-------+
| Rpl_semi_sync_slave_status | ON |
+----------------------------+-------+
1 row in set (0.00 sec)

mysql> SET GLOBAL rpl_semi_sync_slave_enabled=0;
Query OK, 0 rows affected (0.00 sec)

mysql> STOP SLAVE IO_THREAD;
Query OK, 0 rows affected (0.04 sec)

mysql> START SLAVE IO_THREAD;
Query OK, 0 rows affected (0.00 sec)

mysql> SHOW STATUS LIKE 'rpl_semi_sync_slave%';
+----------------------------+-------+
| Variable_name | Value |
+----------------------------+-------+
| Rpl_semi_sync_slave_status | OFF |
+----------------------------+-------+
1 row in set (0.00 sec)

Ran 50 VM create very quickly using following script

(nova) saga@api1[openstack_stage]:~$ cat create.sh
for i in {1..50}
do
    openstack server create --flavor small --image ylinux-6.8.5_217 saga6-$i &
done

All the VMs created came back to 'ACTIVE' state without any errors. The same test cases used to cause VMs to be stuck in 'BUILD' state or just Error out because of DB race , without the patch that no longer happens.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/539046
Committed: https://git.openstack.org/cgit/openstack/oslo.db/commit/?id=5ca329884d492d78e623e2313142238eb256f1a6
Submitter: Zuul
Branch: master

commit 5ca329884d492d78e623e2313142238eb256f1a6
Author: Mike Bayer <email address hidden>
Date: Mon Jan 29 18:25:19 2018 -0500

    Reverse role of synchronous_reader

    Repaired the "synchronous_reader" modifier of enginefacade so that it
    refers to the "writer" engine when set to True, thereby allowing
    "synchronous" behavior with the writer. When set to False, this is
    "asynchronous", so this should be associated with the async engines.
    The flag had the reverse behavior previously.

    Change-Id: Id7fea7562ba90eb710176d497af103303f230531
    Closes-bug: #1746116

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/543498

[Arun S A G (sagarun)]

Any chance we can backport them to other stable branches? pike and ocata?

[Mike Bayer (zzzeek)]

I'd vote as far back as ocata, yes.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.db 4.34.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/543498
Committed: https://git.openstack.org/cgit/openstack/oslo.db/commit/?id=268edcb4e32856ce28db2bc53a9f73e79f9c2daa
Submitter: Zuul
Branch: stable/queens

commit 268edcb4e32856ce28db2bc53a9f73e79f9c2daa
Author: Mike Bayer <email address hidden>
Date: Mon Jan 29 18:25:19 2018 -0500

    Reverse role of synchronous_reader

    Repaired the "synchronous_reader" modifier of enginefacade so that it
    refers to the "writer" engine when set to True, thereby allowing
    "synchronous" behavior with the writer. When set to False, this is
    "asynchronous", so this should be associated with the async engines.
    The flag had the reverse behavior previously.

    Change-Id: Id7fea7562ba90eb710176d497af103303f230531
    Closes-bug: #1746116
    (cherry picked from commit 5ca329884d492d78e623e2313142238eb256f1a6)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.db 4.33.1 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/570479

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/570479
Committed: https://git.openstack.org/cgit/openstack/oslo.db/commit/?id=a46f09ffc59ba69c5b63b2fb61b319267d3e7273
Submitter: Zuul
Branch: stable/pike

commit a46f09ffc59ba69c5b63b2fb61b319267d3e7273
Author: Mike Bayer <email address hidden>
Date: Mon Jan 29 18:25:19 2018 -0500

    Reverse role of synchronous_reader

    Repaired the "synchronous_reader" modifier of enginefacade so that it
    refers to the "writer" engine when set to True, thereby allowing
    "synchronous" behavior with the writer. When set to False, this is
    "asynchronous", so this should be associated with the async engines.
    The flag had the reverse behavior previously.

    Change-Id: Id7fea7562ba90eb710176d497af103303f230531
    Closes-bug: #1746116
    (cherry picked from commit 5ca329884d492d78e623e2313142238eb256f1a6)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/609161

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/609161
Committed: https://git.openstack.org/cgit/openstack/oslo.db/commit/?id=f5b6707621e425cf824e2909ddfd9270f76de002
Submitter: Zuul
Branch: stable/ocata

commit f5b6707621e425cf824e2909ddfd9270f76de002
Author: Mike Bayer <email address hidden>
Date: Mon Jan 29 18:25:19 2018 -0500

    Reverse role of synchronous_reader

    Repaired the "synchronous_reader" modifier of enginefacade so that it
    refers to the "writer" engine when set to True, thereby allowing
    "synchronous" behavior with the writer. When set to False, this is
    "asynchronous", so this should be associated with the async engines.
    The flag had the reverse behavior previously.

    Change-Id: Id7fea7562ba90eb710176d497af103303f230531
    Closes-bug: #1746116

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.db 4.25.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.db ocata-eol release.