unit tests failing in some environments

[Thomas Goirand]

Hi,

Over here:
https://bugs.debian.org/908862

Santiago Vila reported that unit tests are failing in his environment. As I am not sure what's going on, and can't work more on the problem, I'm opening this upstream, in the hope you guys can be of some help. At first, it probably looks more like a bug in oslo.db or in SQLAlchemy to me, with tables not being created before the unit tests start or something.

Any clue what's going on here?

Cheers,

Thomas Goirand (zigo)

[Brian Haley (brian-haley)]

Can't say I've seen errors like that from a unit test before.

Is it possible to bisect this to a neutron change?

[Slawek Kaplonski (slaweq)]

Can You provide us exact way how to reproduce it locally maybe? I didn't saw issue like that in U/S CI recently. Is it happening all the time? I tried locally and I didn't had such problems on latest commit from stable/train

[Santiago Vila (sanvila)]

Hi. Original reporter here. I added some info in the Debian BTS:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=908862#99

To summarize: It seems to be random, but the probability of failure is low on multi-core machines and very high in single-CPU machines.

So: Please try building with prefix "taskset -c 0".

Thanks.

[Lajos Katona (lajos-katona)]

Hi, I am in Shanghai with limited access to any test environment.
Actually I am not sure what kind of test VM-s zuul uses for unit tests,but I suppose more vCPUs are used as we run tests paralelly.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Thomas:

According to the log reported in the description, the backend used is slqlite
```
oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) no such table: ml2_geneve_allocations [SQL: 'SELECT ml2_geneve_allocations.geneve_vni AS ml2_geneve_allocations_geneve_vni, ml2_geneve_allocations.allocated AS ml2_geneve_allocations_allocated \nFROM ml2_geneve_allocations'] (Background on this error at: http://sqlalche.me/e/e3q8)
```

Is that correct? In upstream CI we are testing PostgreSQL and MySQL, and with [1] MariaDB. But we don't test sqlite and there is no official support for a DB backend used for testing, not for production.

Can you swap to another backend?

Regards.

[1] https://review.opendev.org/#/c/681202/

[Thomas Goirand (thomas-goirand)]

Hi,

No, we cannot easily switch to something else than SQLite, in the context of running unit tests when building a Debian package. It's been the view of the whole project that using SQLite is ok when doing unit tests by the way, so I think it'd be worth having a look. I do well understand and agree that SQLite is not well suited for production, and that's not what I'm doing.

[Slawek Kaplonski (slaweq)]

@Rodolfo, IIRC we are using something like "in memory" sqlite db in unit tests and it is like that even in u/s CI jobs.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Bug closed due to lack of activity, please feel free to reopen if needed.