Horizon is very slow in the first hour after the 6:35 morning log reload

[Andrei Tira]

Horizon is very slow in the morning after the 6:35 reload for the log rotation.

We have a deployment with juju and MAAS of OpenStack Train with 3 units of horizon and we noticed that in the first hour after the apache log rotates it is very slow, this means:
Instances tab 41 sec (normally around 10 sec)
Networks tab 27 sec (normally around 4 sec)
Routers tab 23 sec (normally around 3 sec)
Volumes tab 33 sec (normally around 9 sec)

We did some profiling with OSprofiler on the horizon pages and saw that the differences are at these functions:

Instances tab:
openstack_dashboard.api.nova.server_list_paged horizon
openstack_dashboard.api.nova.extension_supported horizon

The rest of the tabs:
openstack_dashboard.api.nova.extension_supported horizon

Initially, we thought that this is from the number of workers and we changed them from 56 to 8 but this did not change anything. In the logs, there are no error messages or anything that would help us debug this issue.

We don't know exactly from where these stems but we suspect that it has something to do with the WSGI module, as after accessing horizon several times all threads are initialized and the loading times go down.