HTTP Error 504: Gateway Time-out during deploy 100 nodes cluster via API

[Sergey Galkin]

api: '1.0'
astute_sha: 97eea90efe0a1f17b4934919d6e459d270c10372
auth_required: true
build_id: 2014-10-22_20-38-28
build_number: '31'
feature_groups:
- mirantis
fuellib_sha: d4f7d9b97ecfed33bc7ee4f628dd9e8ed3651a05
fuelmain_sha: dab17913263bbea7e9a3b55de8a0f3af5ac0e3e2
nailgun_sha: 172c84c728256a09918ef41103e6b7c37cfe87d8
ostf_sha: de177931b53fbe9655502b73d03910b8118e25f1
production: docker
release: '6.0'

During deploy 100 nodes cluster through API API returns ' HTTP Error 504: Gateway Time-out'
Logs with requests attached
Snapshot attached to https://bugs.launchpad.net/fuel/+bug/1384536

[Sergey Galkin (sgalkin)]

Logs in a more convenient form

[Tomasz 'Zen' Napierala (tzn)]

This is addressed as part of general optimizations. We will connect it to the bug, as it was worked on as part of blueprint
https://review.openstack.org/#/c/128194/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/128194
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=564e51033759db248617c8219417cde7364daf87
Submitter: Jenkins
Branch: master

commit 564e51033759db248617c8219417cde7364daf87
Author: Kamil Sambor <email address hidden>
Date: Fri Oct 10 11:49:22 2014 +0200

    Speed up assign_ips and assign_admin_ips methods

    assign_ips assign_admin_ips methods are the bottlenecks during deployment,
    changes in this methods reduce execution time near 10 times

    * added and correct tests
    * added ignored_keys param to diff method to ignore some keys during
    comparing two dictionaries, this is good when we are not sure or we are
    not able to predict values in dictionary
    * improved assign_ips and assign_admin_ips methods

    Change-Id: I6b60ff722211ddb634c791fbb700a2e46c7143bc
    Closes-Bug: #1384623
    Blueprint: 100-nodes-support

[Sergey Galkin (sgalkin)]

Reproduced on
api: '1.0'
astute_sha: c72dac7b31646fbedbfc56a2a87676c6d5713fcf
auth_required: true
build_id: 2014-11-02_21-27-58
build_number: '69'
feature_groups:
- mirantis
fuellib_sha: 45ad9b42666d7e3e14ab9af2911808e6c8806842
fuelmain_sha: ac3ba5f5c6073b7776ec69fc3cb4dd3c56df36c5
nailgun_sha: 35946b1f225c984f11915ba8e985584160f0b129
ostf_sha: 9c6fadca272427bb933bc459e14bb1bad7f614aa
production: docker
release: '6.0'

[Sergey Galkin (sgalkin)]

Reproduced on
api: '1.0'
astute_sha: f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13
auth_required: true
build_id: 2014-11-04_21-17-27
build_number: '17'
feature_groups:
- mirantis
fuellib_sha: 42c25fc764dd931615ae767a3b91574ed1c95f16
fuelmain_sha: 16e6cfd1b4775cccfd9a6f0d66d3826d41d3214e
nailgun_sha: 4aa4fb5f89c02aaee5bec1aa4677b48598ffe482
ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
production: docker
release: 5.1.1

[Kamil Sambor (ksambor)]

Hi Sergey, could you attach new snapshot from both cases?

[Sergey Galkin (sgalkin)]

Hi Kamil, unfortunately no but I have reproduce on

api: '1.0'
astute_sha: cff84956fbe8ac8f95020aa04d326230cae92158
auth_required: true
build_id: 2014-11-06_21-29-20
build_number: '82'
feature_groups:
- mirantis
fuellib_sha: ef9f6407d0d555346ac34b9f29ee004083f8e6d2
fuelmain_sha: 0d3f16254760999f90564c384f34fda8d6c82f51
nailgun_sha: 172c1acadabf59b9434718d9a1c27a3a4db5f3cc
ostf_sha: 9c6fadca272427bb933bc459e14bb1bad7f614aa
production: docker
release: '6.0'

with snapshot

[Kamil Sambor (ksambor)]

Thank you Sergey

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/135323

[Kamil Sambor (ksambor)]

Today fix should be ready for review.

[Mike Scherbakov (mihgen)]

Kamil,
any update here?

[Kamil Sambor (ksambor)]

This fix is waiting for merge

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/135323
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=a08434e5fb667e2320c2890dc68f4d965adc8f42
Submitter: Jenkins
Branch: master

commit a08434e5fb667e2320c2890dc68f4d965adc8f42
Author: Kamil Sambor <email address hidden>
Date: Thu Nov 13 12:40:09 2014 +0100

    Speed up ClusterChangesHandler

    * added performance test
    * improved assign_vip
    * improved geting controllers in CheckBeforeDeployment
    * improved checking untagged networks
    * changed name of property default_group on to default_group_id
    * correct tests and add new

    Change-Id: Iba10fdae455c3f19a71cbcdc88f8cd370ffd1993
    Closes-Bug: #1384623

[Sergey Galkin (sgalkin)]

Reproduced on
astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
auth_required: true
build_id: 2014-12-04_18-22-07
build_number: '39'
feature_groups:
- mirantis
fuellib_sha: 605ac6deffd601c1da809eb114cd315f670029d8
fuelmain_sha: 7ab330b4958ab20955372e85de05e8732e8f6df2
nailgun_sha: 0df508c3247a06f8a636ded8fb04733d5aed7732
ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
production: docker

[Kamil Sambor (ksambor)]

Without rewriting part of the code as a background tasks repairing this bug is almost impossible. But this kind of changes was rejected for this release and postponed to next release in review: https://review.openstack.org/#/c/135323/

[Kamil Sambor (ksambor)]

It's now after HCF so I move this bug to 6.1

[Dima Shulyak (dshulyak)]

Maybe it should be closed if there is such point in blueprint?

[Sergey Galkin (sgalkin)]

Reproduced on
astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
auth_required: true
build_id: 2015-02-07_20-50-01
build_number: '76'
feature_groups:
- mirantis
fuellib_sha: 64f3ebe9fcbd18bf6c80a948e06061783a090347
fuelmain_sha: c799e3a6d88289e58db764a6be7910aab7da3149
nailgun_sha: 2ef819732a3ee7acf7b610e7d1c1a6da0434c1a0
ostf_sha: 3b57985d4d2155510894a1f6d03b478b201f7780
production: docker
release: 6.0.1

with 100 nodes

[Kamil Sambor (ksambor)]

This is part of bp https://review.openstack.org/#/c/146900/

[Dina Belova (dbelova)]

Did not face this issue on #233 ISO for 6.1. Seems to be fixed.