performance regression from mitaka to ocata

[Jacopo Secchiero]

With rally tests i have noticed a generic performance drop in production on neutron from 50% to 10% in all components after migration to Ocata from Mitaka and Newton.
I'm able to reproduce the problem with isolated VM using ubuntu 16.04 packages with related repo and use create port as a test because seems the more relevant. Here below the times of 40 serial create port using curl direct to neutron:

mitaka 0m 21s
newton 0m 31s
ocata 0m 50s
rocky 0m 37s

I have done more tests with the next releases with devstack to check if the problem was solved, but unfortunately seems not.

I have also done a bit of profiling and seems also that the orm produce different behavior on ocata and have some call that on mitaka are never done:

ORMAdapter.traverse
ORMAdapter.replace
ORMAdapter._corresponding_column
ORMAdapter._locate_col

[Pawel Suder (pasuder)]

Hello, thank you for your commitment. Unfortunately I am not sure if that could be handled easily. Neutron server/agent is expanding from release to release. Performance might change between versions. I would leave that issue as open question. Maybe someone else will be able to share his insights related to performance here. Cheers!

[Brian Haley (brian-haley)]

Do you see the same behavior on master or stable/rocky? It could be different there especially given Ocata is quite old, plus any fix would need to be targeted at master first anyways.

[Jacopo Secchiero (jsecchiero)]

I have tested it with devstack from ocata to rocky and there are no differences. Unfortunately i was not able to install mitaka with devstack.
For completeness i'll do the same test next week with rocky (ubuntu packages and same specs).

[Jacopo Secchiero (jsecchiero)]

i have done the tests also on rocky and are better then ocata. First post updated with the new test.