[SystemTests] AttributeError: 'NoneType' object has no attribute 'group'

[Vasyl Saienko]

System test: backup_restore_master_base
http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/7.0.custom_system_test/764/console
has failed with error:

2015-09-16 22:22:59,158 - INFO decorators.py:46 -- Saving logs to "/home/jenkins/workspace/7.0.custom_system_test/logs/fail_error_neutron_tun_scalability-fuel-snapshot-2015-09-16_22-18-58.tar.xz" file
2015-09-16 22:26:36,876 - INFO decorators.py:295 -- <<<<<****************************************************************************************************>>>>>
2015-09-16 22:26:36,876 - INFO decorators.py:296 -- Failed in method 'neutron_tun_scalability'. Make snapshot: error_neutron_tun_scalability
2015-09-16 22:26:36,876 - INFO decorators.py:306 -- You could revert and ssh to master node: [source /home/jenkins/venv-nailgun-tests-2.9/bin/activate; dos.py revert-resume 7.0.custom_system_test.764.2015-09-16_17-09-47 --snapshot-name error_neutron_tun_scalability && ssh root@10.109.0.2]
2015-09-16 22:26:36,876 - INFO decorators.py:308 -- <<<<<****************************************************************************************************>>>>>
ERROR

======================================================================
ERROR: Check HA mode on scalability
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/7.0.custom_system_test/fuelweb_test/helpers/decorators.py", line 80, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.custom_system_test/fuelweb_test/tests/test_neutron_tun.py", line 351, in neutron_tun_scalability
    _check_swift(primary_node)
  File "/home/jenkins/workspace/7.0.custom_system_test/fuelweb_test/tests/test_neutron_tun.py", line 252, in _check_swift
    checkers.check_swift_ring(remote)
  File "/home/jenkins/workspace/7.0.custom_system_test/fuelweb_test/helpers/checkers.py", line 840, in check_swift_ring
    balance = re.search('(\d+.\d+) balance', res).group(1)
AttributeError: 'NoneType' object has no attribute 'group'

----------------------------------------------------------------------

[Vasyl Saienko (vsaienko)]

From the logs we are trying to check ring status on node: 10.109.0.12

https://paste.mirantis.net/show/1134/

but this node is discovered slave:

[root@nailgun ~]# fuel node
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|----------|---------------------|---------|-------------|-------------------|------------|---------------|--------|---------
2 | ready | slave-04_controller | 1 | 10.109.0.6 | 64:0b:fe:51:e3:a0 | controller | | True | 1
6 | ready | slave-06_compute | 1 | 10.109.0.8 | 64:05:cb:80:27:96 | compute | | True | 1
7 | discover | Untitled (5d:af) | None | 10.109.0.9 | 64:32:ef:9f:5d:af | | | True | None
4 | ready | slave-03_controller | 1 | 10.109.0.5 | 64:3b:a1:71:75:43 | controller | | True | 1
9 | discover | Untitled (11:d1) | None | 10.109.0.10 | 64:d7:7b:48:11:d1 | | | True | None
11 | discover | Untitled (e3:95) | None | 10.109.0.12 | 64:ac:f0:5c:e3:95 | | | True | None
10 | discover | Untitled (18:4c) | None | 10.109.0.7 | 64:c7:ad:9a:18:4c | | | True | None
8 | discover | Untitled (13:64) | None | 10.109.0.11 | 64:f0:65:c7:13:64 | | | True | None
1 | ready | slave-02_controller | 1 | 10.109.0.4 | 64:4d:2e:4b:5d:14 | controller | | True | 1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/224576

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/224587

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/224576
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=06f1f2faca0e40d31dda405f65d069f4a0389675
Submitter: Jenkins
Branch: master

commit 06f1f2faca0e40d31dda405f65d069f4a0389675
Author: asledzinskiy <email address hidden>
Date: Thu Sep 17 14:11:02 2015 +0300

    Fix neutron_tun_scalability test

    - In scalability test we add/delete nodes so
    we need to get primary controller again after
    every re-deploy

    Change-Id: If0b15c2b778faf7c5527a3e31e1b99105ab54128
    Closes-Bug: #1496709

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/225207

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/225207
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=4b319c928a3e26be696ff9adb7c60ec80e88b214
Submitter: Jenkins
Branch: stable/7.0

commit 4b319c928a3e26be696ff9adb7c60ec80e88b214
Author: asledzinskiy <email address hidden>
Date: Thu Sep 17 14:11:02 2015 +0300

    Fix neutron_tun_scalability test

    - In scalability test we add/delete nodes so
    we need to get primary controller again after
    every re-deploy

    Change-Id: If0b15c2b778faf7c5527a3e31e1b99105ab54128
    Closes-Bug: #1496709
    (cherry picked from commit 06f1f2faca0e40d31dda405f65d069f4a0389675)

[Timur Nurlygayanov (tnurlygayanov)]

On custom ISO with the fixes: http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_7.0_iso/1225/

Steps:
1. Deploy cluster with 3 controllers, 1 compute node, Ceph OSD nodes on controllers, Neutron VLANs. Other configuration by default.
2. Run Smoke and sanity OSTF tests - it will pass.
3. Run the following commands on master node:
puppet apply -vd /etc/puppet/modules/osnailyfacter/modular/apache/apache.pp; puppet apply -vd /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp

Observed Result:
Info: /Stage[main]/Apache/Concat::Fragment[Apache ports header]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_Apache ports header]: Scheduling refresh of Exec[concat_/etc/httpd/conf/ports.conf]
Debug: /Stage[main]/Apache/Concat::Fragment[Apache ports header]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_Apache ports header]: The container Concat::Fragment[Apache ports header] will propagate my refresh event
Debug: Concat::Fragment[Apache ports header]: The container Class[Apache] will propagate my refresh event
Notice: /Stage[main]/Osnailyfacter::Apache/Osnailyfacter::Apache::Apache_port[80]/Apache::Listen[80]/Concat::Fragment[Listen 80]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_Listen 80]/ensure: defined content as '{md5}d5fcefc335117f400d451de47efeca87'
Info: /Stage[main]/Osnailyfacter::Apache/Osnailyfacter::Apache::Apache_port[80]/Apache::Listen[80]/Concat::Fragment[Listen 80]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_Listen 80]: Scheduling refresh of Exec[concat_/etc/httpd/conf/ports.conf]
Debug: /Stage[main]/Osnailyfacter::Apache/Osnailyfacter::Apache::Apache_port[80]/Apache::Listen[80]/Concat::Fragment[Listen 80]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_Listen 80]: The container Concat::Fragment[Listen 80] will propagate my refresh event
Notice: /Stage[main]/Apache/Concat[/etc/httpd/conf/ports.conf]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments.concat]/ensure: created
Debug: /Stage[main]/Apache/Concat[/etc/httpd/conf/ports.conf]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments.concat]: The container Concat[/etc/httpd/conf/ports.conf] will propagate my refresh event
Notice: /Stage[main]/Osnailyfacter::Apache/Osnailyfacter::Apache::Apache_port[80]/Apache::Namevirtualhost[*:80]/Concat::Fragment[NameVirtualHost *:80]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_NameVirtualHost *_80]/ensure: defined content as '{md5}a6144f5815141f7fa494ea04970d6b11'
Info: /Stage[main]/Osnailyfacter::Apache/Osnailyfacter::Apache::Apache_port[80]/Apache::Namevirtualhost[*:80]/Concat::Fragment[NameVirtualHost *:80]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_NameVirtualHost *_80]: Scheduling refresh of Exec[concat_/etc/httpd/conf/ports.conf]
Debug: /Stage[main]/Osnailyfacter::Apache/Osnailyfacter::Apache::Apache_port[80]/Apache::Namevirtualhost[*:80]/Concat::Fragment[NameVirtualHost *:80]/File[/var/lib/puppet/concat/_etc_httpd_conf_ports.conf/fragments/10_NameVirtualHost *_80]: The container Concat::Fragment[NameVirtualHost *:80] will propaga...

[Timur Nurlygayanov (tnurlygayanov)]

root@node-1:~# ps ax | grep apache
 8452 pts/23 R+ 0:00 grep --color=auto apache
23900 ? Ss 0:01 /usr/sbin/apache2 -k start
28157 ? Sl 0:01 /usr/sbin/apache2 -k start
28158 ? Sl 0:01 /usr/sbin/apache2 -k start
28159 ? Sl 0:16 /usr/sbin/apache2 -k start
28160 ? Sl 0:15 /usr/sbin/apache2 -k start

[Nastya Urlapova (aurlapova)]

Timur, I think your comments were not related to system tests issue.

[Tatyanka (tatyana-leontovich)]

verified on latest custom tests runs

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/224587
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=368bfc5db1a915addcb5b8d018a23ff79438368f
Submitter: Jenkins
Branch: master

commit 368bfc5db1a915addcb5b8d018a23ff79438368f
Author: Dmitry Tyzhnenko <email address hidden>
Date: Thu Sep 17 14:39:25 2015 +0300

    Fix fetch ctrl in neutron_tun_scalability test

    Use other slave after remove primary controller

    Change-Id: I4ae72b6b3fbabcace7787704fb0952dbd86e3270
    Closes-bug: #1496709