The test_create_update_get_service test is failing due to the service update taking too long to propagate

[Castulo J. Martinez]

The test case tempest.api.identity.admin.v3.test_services.ServicesTestJSON.test_create_update_get_service thta is part of the smoke tes suite is failing every time becuase the service that its updating is taking 10 minutes to reflect the changes.

These are the steps to reproduce (steps followed by the Tempest test):

1) Creates a new service
2) Updates the description of the service
3) Gets the service and verifies it has the newly updated data

Expected Results;
 - When getting the service, it should show the changes done through the service update.

Actual Results:
 - The service does not reflect the updated data until 10 minutes have passed, causing the test to fail everytime.

If I debug the test and go step by step this is what I see:

1) Creates a new service

The service gets created correctly as can be seen with keystone service-list (service name: tempest-service-195308057):

root@node-1:~# keystone service-list
+----------------------------------+---------------------------+-------------------------+----------------------------------+
| id | name | type | description |
+----------------------------------+---------------------------+-------------------------+----------------------------------+
| d2715775fcaf4b469e0aab5551e660c1 | cinder | volume | Cinder Service |
| 112e5f470a25469ca8a0ca464c9f69cc | cinderv2 | volumev2 | Cinder Service v2 |
| f5294f0e114b46b3ba93168baba21da9 | glance | image | Openstack Image Service |
| b3916c4b179647b996c3b0a8102265cc | heat | orchestration | Openstack Orchestration Service |
| 20e515e720324fd6810cb5ee7855357c | heat-cfn | cloudformation | Openstack Cloudformation Service |
| 662022a782fb405989718162b9cacf15 | keystone | identity | OpenStack Identity Service |
| 7e87fe59df2d494786f6bccbf8a49b1d | neutron | network | Neutron Networking Service |
| 92ad86293090446c8ded9268591bc07a | nova | compute | Openstack Compute Service |
| 5c8b6b3527ac41cdaa2c6f786f22c918 | nova_ec2 | ec2 | EC2 Service |
| 5a0ddb40b82e445a8e4a4e2bf2c314f4 | tempest-service-195308057 | tempest-type-1204186955 | tempest-description-1852291595 |
+----------------------------------+---------------------------+-------------------------+----------------------------------+

2) Updates the description of the service

After the service update is executed, if "keystone service-list" is used, it shows the updated service as expected, but if "keystone service-get" is used, the service shows its old description:

root@node-1:~# keystone service-list
+----------------------------------+---------------------------+-------------------------+----------------------------------+
| id | name | type | description |
+----------------------------------+---------------------------+-------------------------+----------------------------------+
| d2715775fcaf4b469e0aab5551e660c1 | cinder | volume | Cinder Service |
| 112e5f470a25469ca8a0ca464c9f69cc | cinderv2 | volumev2 | Cinder Service v2 |
| f5294f0e114b46b3ba93168baba21da9 | glance | image | Openstack Image Service |
| b3916c4b179647b996c3b0a8102265cc | heat | orchestration | Openstack Orchestration Service |
| 20e515e720324fd6810cb5ee7855357c | heat-cfn | cloudformation | Openstack Cloudformation Service |
| 662022a782fb405989718162b9cacf15 | keystone | identity | OpenStack Identity Service |
| 7e87fe59df2d494786f6bccbf8a49b1d | neutron | network | Neutron Networking Service |
| 92ad86293090446c8ded9268591bc07a | nova | compute | Openstack Compute Service |
| 5c8b6b3527ac41cdaa2c6f786f22c918 | nova_ec2 | ec2 | EC2 Service |
| 5a0ddb40b82e445a8e4a4e2bf2c314f4 | tempest-service-195308057 | tempest-type-1204186955 | tempest-desc2-177519707 |
+----------------------------------+---------------------------+-------------------------+----------------------------------+
root@node-1:~# keystone service-get 5a0ddb40b82e445a8e4a4e2bf2c314f4
+-------------+----------------------------------+
| Property | Value |
+-------------+----------------------------------+
| description | tempest-description-1852291595 |
| enabled | True |
| id | 5a0ddb40b82e445a8e4a4e2bf2c314f4 |
| name | tempest-service-195308057 |
| type | tempest-type-1204186955 |
+-------------+----------------------------------+

3) Gets the service and verifies it has the newly updated data

After 10 minutes, if I do the "keystone service-get"again, it now shows the correct data. And if I unpause the test then, it passes.
root@node-1:~# keystone service-get 5a0ddb40b82e445a8e4a4e2bf2c314f4
+-------------+----------------------------------+
| Property | Value |
+-------------+----------------------------------+
| description | tempest-desc2-177519707 |
| enabled | True |
| id | 5a0ddb40b82e445a8e4a4e2bf2c314f4 |
| name | tempest-service-195308057 |
| type | tempest-type-1204186955 |
+-------------+----------------------------------+

I repeated this process several times, it behaved the same way every time.
I tried this in two deployments: one OpenStack deployed by Fuel in Virtualbox and one OpenStack deployed by Fuel in real physical servers.

[Castulo J. Martinez (castulo-martinez)]

It seems to me that a timeout should be added to this test, so the service get can be attempted within the timeout, if the service changes (is updated) within the timeout, the test should pass if not the test should fail.
As it is, is giving false negative, the service is actually getting updated, it just takes long to have the changes reflected for some reason (10 minutes). The only problem with this is that given this test is part of the smoke test, if a timeout is added, it is also going to exted the time the smoke test takes to finish.

[Matthew Treinish (treinish)]

So to be honest while we can add a timeout option to this test, I think there is a more basic problem with your OpenStack deployments. The test is dealing with strictly database operations, taking 10mins to update a row (or maybe a couple of rows) is incredibly slow, like when we run this test in the gate, which is far from an ideal scenario, being an all in one node running devstack in a vm with 8gb of ram on a public cloud, this test has an average execute time of: 0.435142 secs (over 134554 runs since Nov.) I also made a graph: http://i.imgur.com/Pkl0tpA.png (the data up until mid-march only had sec. resolution) but you can see even the extreme outliers in that case are ~20 sec.

Instead of trying to make the test tolerant of this extreme edge case to make your environment pass, I'd recommend looking at fixing your environment to avoid this kind of issue. (which I suspect is however fuel is deploying things since you tried different environments)

[Castulo J. Martinez (castulo-martinez)]

Thanks for looking into this Matthew, your response makes sense, I thought it was more like a general problem, but based on your response you are right, it may be bound to the Fuel delpoyments. I'll try to narrow the problem in these Fuel deployed OpenStacks and if find anything will report it as bug in the fuel project instead.