[swarm] Some threads are aborted by timeout

[Ilya Bumarskov]

Following threads are aborted by timeout:

* https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.services_reconfiguration_thread_2/194/
* https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.services_ha.ceilometer/194/
* https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.plugins.detach-rabbitmq-ci-group/14/
* https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.ha_neutron_destructive/195/
* https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.thread_7/193/
* https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.thread_3/

Log analysis shows that deployment time was increased for all deployment operations up to 20-40 minutes comparing with 9.x deployments. Dev team, please check why it takes much time

Example from astute.log:
9.2: 2017-03-01 08:04:39 INFO [1008] Deployment summary: time was spent 00:37:42
10.0: 2017-03-01 06:57:26 INFO [17502] Deployment summary: time was spent 00:55:40

[Georgy Kibardin (gkibardin)]

I was comparing 10 and 9.x bvt. It turn out that horizon task takes 3 times longer in 10.0 (~100 vs 300 sec.)
One of the reasons:
2017-03-03 06:40:00 +0000 Puppet (debug): Executing '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o APT::Get::AllowUnauthenticated=1 install openstack-dashboard-apache'
2017-03-03 06:42:53 +0000 /Stage[main]/Horizon/Package[horizon]/ensure (notice): ensure changed 'purged' to 'present'

[Georgy Kibardin (gkibardin)]

Aha, the reason is a lot of dependencies:

Start-Date: 2017-03-03 06:40:03
Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o APT::Get::AllowUnauthenticated=1 install openstack-dashboard-apache
Install: python-xstatic-jquery:amd64 (1.7.2.0+dfsg1-3, automatic), python-xstatic-d3:amd64 (3.4.11-3, automatic), libjs-angular-gettext:amd64 (2.1.0-2, automatic), libjs-rickshaw:amd64 (1.5.1.dfsg-1, automatic), python-xstatic-jquery.tablesorter:amd64 (2.14.5.1-2, automatic), python-django-pyscss:amd64 (2.0.2-4, automatic), fonts-font-awesome:amd64 (4.5.0~dfsg-1, automatic), python-xstatic-angular-lrdragndrop:amd64 (1.0.2.2-1, automatic), python-xstatic-spin:amd64 (1.2.8.0+dfsg1-1, automatic), libjs-jquery-tablesorter:amd64 (2.14.5-0u~u16.04+mos1, automatic), python-pyscss:amd64 (1.3.4-5, automatic), libjs-twitter-bootstrap-datepicker:amd64 (1.3.1+dfsg1-1, automatic), python-xstatic-angular-bootstrap:amd64 (0.11.0.3-1, automatic), python-xstatic-angular-schema-form:amd64 (0.8.13.0-1~u16.04+mos1, automatic), libjs-magic-search:amd64 (0.2.5-1, automatic), fonts-roboto-fontface:amd64 (0.4.3-1, automatic), python-django-appconf:amd64 (1.0.1-4, automatic), python-xstatic-font-awesome:amd64 (4.1.0-1, automatic), libjs-twitter-bootstrap:amd64 (2.0.2+dfsg-9, automatic), python-pathlib:amd64 (1.0.1-2, automatic), libjs-term.js:amd64 (0.0.4-1, automatic), python-django-common:amd64 (1.8.7-2~u16.04+mos1, automatic), libjs-angular-file-upload:amd64 (12.0.4+dfsg1-2~u16.04+mos1, automatic), libjs-jquery.quicksearch:amd64 (2.0.4-1, automatic), python-xstatic:amd64 (1.0.0-4, automatic), python-xstatic-magic-search:amd64 (0.2.5.1-1, automatic), libjs-jquery-cookie:amd64 (10-2ubuntu2, automatic), python-xstatic-angular:amd64 (1.3.7.0-2, automatic), libjs-bootswatch:amd64 (3.3.5+2+dfsg1-1~u14.04+mos2, automatic), libjs-jsencrypt:amd64 (2.0.0+dfsg1-1, automatic), python-django:amd64 (1.8.7-2~u16.04+mos1, automatic), python-xstatic-term.js:amd64 (0.0.4.2-2, automatic), python-xstatic-jsencrypt:amd64 (2.0.0.2-1, automatic), python-django-openstack-auth:amd64 (2.4.2-1~u16.04+mos0, automatic), python-xstatic-roboto-fontface:amd64 (0.4.3.2-1, automatic), python-django-compressor:amd64 (2.0-1ubuntu1, automatic), python-xstatic-angular-fileupload:amd64 (12.0.4.0+dfsg1-1~u16.04+mos1, automatic), python-xstatic-smart-table:amd64 (1.4.5.3-1, automatic), python-xstatic-jquery.quicksearch:amd64 (2.0.4.1-1, automatic), libjs-d3:amd64 (3.5.14-1, automatic), libjs-spin.js:amd64 (1.2.8+dfsg2-1, automatic), python-csscompressor:amd64 (0.9.4-2, automatic), libjs-jquery-ui:amd64 (1.10.1+dfsg-1, automatic), python-xstatic-mdi:amd64 (1.4.57.0-1~u16.04+mos1, automatic), python-xstatic-jquery-ui:amd64 (1.10.1.1+debian+dfsg1-5, automatic), libjs-angularjs-smart-table:amd64 (1.4.5-1, automatic), python-xstatic-rickshaw:amd64 (1.5.0.2-2, automatic), openstack-dashboard:amd64 (3:10.0.2-1~u16.04+mos11, automatic), python-xstatic-bootstrap-scss:amd64 (3.2.0.1-1, automatic), python-xstatic-bootswatch:amd64 (3.3.5.2-2, automatic), python-xstatic-bootstrap-datepicker:amd64 (0.0.0.1-3, automatic), python-xstatic-tv4:amd64 (1.2.7.0-1~u16.04+mos0, automatic), python-xstatic-angular-...

[Georgy Kibardin (gkibardin)]

Hm... it seems to be the case anyway, on 9.x we have the same long list:

Start-Date: 2017-03-03 07:52:55
Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o APT::Get::AllowUnauthenticated=1 install openstack-dashboard-apache
Install: python-rcssmin:amd64 (1.0.6-1~u14.04+mos1, automatic), openstack-dashboard:amd64 (9.1.1-3~u14.04+mos33, automatic), python-xstatic:amd64 (1.0.0-4~u14.04+mos1, automatic), horizon-vendor-theme:amd64 (1.0.0~mos8-4~u14.04+mos3, automatic), libjs-angular-gettext:amd64 (2.1.0-2~u14.04+mos1, automatic), python-pint:amd64 (0.6-1~u14.04+mos1, automatic), python-django-common:amd64 (1.8.7-2~u14.04+mos1, automatic), libjs-jquery-metadata:amd64 (8-2, automatic), python-xstatic-smart-table:amd64 (1.4.5.3-1~u14.04+mos1, automatic), libjs-angularjs:amd64 (1.3.17-1~u14.04+mos1, automatic), python-xstatic-spin:amd64 (1.2.8.0+dfsg1-1~u14.04+mos1, automatic), libjs-jquery-cookie:amd64 (8-2, automatic), python-xstatic-angular:amd64 (1.3.7.0-2~u14.04+mos1, automatic), python-xstatic-bootstrap-scss:amd64 (3.2.0.1-1~u14.04+mos1, automatic), python-xstatic-jasmine:amd64 (2.1.2.0-1~u14.04+mos1, automatic), python-xstatic-hogan:amd64 (2.0.0.2-1~u14.04+mos3, automatic), python-django-openstack-auth:amd64 (2.2.1-2~u14.04+mos9, automatic), python-xstatic-rickshaw:amd64 (1.5.0.2-2~u14.04+mos1, automatic), python-pyscss:amd64 (1.3.4-4~u14.04+mos1, automatic), python-django-appconf:amd64 (1.0.1-5~u14.04+mos2, automatic), python-xstatic-jquery.quicksearch:amd64 (2.0.4.1-1~u14.04+mos3, automatic), libjs-twitter-bootstrap:amd64 (2.0.2+dfsg-4, automatic), libjs-term.js:amd64 (0.0.4-1~u14.04+mos1, automatic), libjs-jquery.quicksearch:amd64 (2.0.4-1~u14.04+mos3, automatic), python-xstatic-term.js:amd64 (0.0.4.2-2~u14.04+mos1, automatic), fonts-roboto-fontface:amd64 (0.4.3-1~u14.04+mos1, automatic), python-rjsmin:amd64 (1.0.12+dfsg1-2~u14.04+mos1, automatic), python-xstatic-jquery.tablesorter:amd64 (2.14.5.1-1~u14.04+mos1, automatic), libjs-twitter-bootstrap-datepicker:amd64 (1.3.1+dfsg1-1~u14.04+mos1, automatic), python-xstatic-jquery:amd64 (1.10.2.1-1~u14.04+mos2, automatic), python-django-overextends:amd64 (0.4.0-1~u14.04+mos1, automatic), libjs-jquery-ui:amd64 (1.10.1+dfsg-1, automatic), libjs-spin.js:amd64 (1.2.8+dfsg2-1~u14.04+mos3, automatic), libjs-rickshaw:amd64 (1.5.1.dfsg-1~u14.04+mos1, automatic), libjs-angularjs-smart-table:amd64 (1.4.5-1~u14.04+mos1, automatic), libjs-magic-search:amd64 (0.2.5-1~u14.04+mos2, automatic), libjs-lrdragndrop:amd64 (1.0.2-2~u14.04+mos1, automatic), python-xstatic-roboto-fontface:amd64 (0.4.3.2-1~u14.04+mos1, automatic), libjs-bootswatch:amd64 (3.3.5+2+dfsg1-1~u14.04+mos2, automatic), python-ceilometerclient:amd64 (2.4.0-3~u14.04+mos1, automatic), python-django:amd64 (1.8.7-2~u14.04+mos1, automatic), python-django-horizon:amd64 (9.1.1-3~u14.04+mos33, automatic), python-xstatic-jquery-ui:amd64 (1.10.4.1~u14.04+mos2, automatic), python-pathlib:amd64 (1.0.1-2~u14.04+mos1, automatic), python-django-compressor:amd64 (2.0-1~u14.04+mos1, automatic), python-xstatic-mdi:amd64 (1.1.70.1-1~u14.04+mos2, automatic), libjs-d3:amd64 (3.5.6-1~u14.04+mos1, automatic), python-django-babel:amd64 (0.4.0-1...

[Georgy Kibardin (gkibardin)]

The above seems not to be the case - rather a local anomaly.

[Georgy Kibardin (gkibardin)]

It seems that for some reason in newton amount of external program calls by puppet increased drastically. For instance updatedb.pp manifest processing led to 130 execs in mitaka and 439 in newton. It is hard to say what exacts new calls has been introduced in newton. However it is clear that some calls are repeated several times, such as "ifconfig <iface>" (on the same iface).

[Georgy Kibardin (gkibardin)]

In newton there are more calls related to networking. In mitaka only "ifconfig" is used to query interface information, while in newton there are also "ip" and "ethtool". There are also new calls of "blkid" in newton. These additional calls might not be a problem if only there weren't so much duplicates among them. Each blkid call is performed twice on the same device, there are up to 6(!) calls to ifconfig with the same interface. There are 5 "cat /proc/mounts" calls in a row! etc.

[Georgy Kibardin (gkibardin)]

The reason seems to be the new facter 2.4.6 (in mitaka we had 1.7.5). So, it looks, we cannot fix this.