amulet _get_proc_start_time has a race which causes service restart checks to fail
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Charm Helpers |
Fix Released
|
Medium
|
Ryan Beisner | ||
cinder (Juju Charms Collection) |
Fix Released
|
Medium
|
Ryan Beisner | ||
glance (Juju Charms Collection) |
Invalid
|
Medium
|
Ryan Beisner | ||
heat (Juju Charms Collection) |
Invalid
|
Medium
|
Ryan Beisner | ||
neutron-api (Juju Charms Collection) |
Fix Released
|
Medium
|
Ryan Beisner | ||
neutron-gateway (Juju Charms Collection) |
Fix Released
|
Medium
|
Ryan Beisner | ||
neutron-openvswitch (Juju Charms Collection) |
Fix Released
|
Medium
|
Ryan Beisner | ||
openstack-dashboard (Juju Charms Collection) |
Fix Released
|
Medium
|
Ryan Beisner |
Bug Description
Although most frequently observed in the openstack-
The test is to make a charm config change, then inspect the unit for process start time, and compare that start time with file modification time of the relevant config file. In some cases, that arbitrary sleep results in the _get_proc_
2015-07-13 14:57:39,934 test_900_
2015-07-13 14:57:40,527 test_900_
2015-07-13 14:59:29,632 _get_proc_
2015-07-13 14:59:29,632 _get_proc_
Traceback (most recent call last):
File "tests/
deployment.
File "/home/
getattr(self, test)()
File "/home/
pgrep_
File "/home/
if (self._
File "/home/
return self._get_
File "/home/
return sentry_
File "/usr/lib/
return self._fs_data(path)
File "/usr/lib/
return self._run_
File "/usr/lib/
raise IOError(output)
IOError: Traceback (most recent call last):
File "/tmp/amulet/
s = os.stat(
OSError: [Errno 2] No such file or directory: '/proc/1242'
ERROR subprocess encountered error code 1
real 17m21.555s
Related branches
- Billy Olsen: Pending requested
- OpenStack Charmers: Pending requested
- James Page: Pending requested
- Corey Bryant: Pending requested
-
Diff: 965 lines (+362/-232)8 files modifiedMakefile (+11/-12)
tests/00-setup (+7/-3)
tests/020-basic-trusty-liberty (+11/-0)
tests/021-basic-wily-liberty (+9/-0)
tests/README (+33/-0)
tests/basic_deployment.py (+171/-164)
tests/charmhelpers/contrib/amulet/utils.py (+101/-53)
tests/tests.yaml (+19/-0)
- Billy Olsen: Approve
- Adam Collard (community): Abstain
- charmers: Pending requested
-
Diff: 241 lines (+101/-53)1 file modifiedcharmhelpers/contrib/amulet/utils.py (+101/-53)
- Billy Olsen: Pending requested
- OpenStack Charmers: Pending requested
-
Diff: 699 lines (+287/-180)7 files modifiedMakefile (+10/-10)
metadata.yaml (+3/-1)
tests/00-setup (+7/-3)
tests/README (+10/-0)
tests/basic_deployment.py (+136/-113)
tests/charmhelpers/contrib/amulet/utils.py (+101/-53)
tests/tests.yaml (+20/-0)
- Billy Olsen: Pending requested
- OpenStack Charmers: Pending requested
-
Diff: 335 lines (+118/-65)5 files modifiedMakefile (+1/-1)
tests/00-setup (+3/-1)
tests/basic_deployment.py (+11/-9)
tests/charmhelpers/contrib/amulet/utils.py (+101/-53)
tests/tests.yaml (+2/-1)
- Chris Glass (community): Approve
-
Diff: 122 lines (+47/-16)1 file modifiedcharmhelpers/contrib/amulet/utils.py (+47/-16)
- Liam Young (community): Approve
-
Diff: 2164 lines (+1242/-341)11 files modifiedMakefile (+1/-1)
tests/00-setup (+7/-3)
tests/020-basic-trusty-liberty (+11/-0)
tests/021-basic-wily-liberty (+9/-0)
tests/README (+10/-0)
tests/basic_deployment.py (+514/-264)
tests/charmhelpers/contrib/amulet/deployment.py (+4/-2)
tests/charmhelpers/contrib/amulet/utils.py (+284/-62)
tests/charmhelpers/contrib/openstack/amulet/deployment.py (+23/-9)
tests/charmhelpers/contrib/openstack/amulet/utils.py (+359/-0)
tests/tests.yaml (+20/-0)
- David Ames (community): Approve
-
Diff: 71 lines (+17/-13)1 file modifiedtests/basic_deployment.py (+17/-13)
- David Ames (community): Approve
-
Diff: 190 lines (+76/-25)1 file modifiedtests/basic_deployment.py (+76/-25)
- OpenStack Charmers: Pending requested
-
Diff: 75 lines (+17/-10)1 file modifiedtests/basic_deployment.py (+17/-10)
no longer affects: | neutron-gateway (Ubuntu) |
Changed in neutron-api (Juju Charms Collection): | |
importance: | Undecided → Medium |
Changed in neutron-gateway (Juju Charms Collection): | |
importance: | Undecided → Medium |
Changed in openstack-dashboard (Juju Charms Collection): | |
importance: | Undecided → Medium |
status: | New → In Progress |
Changed in neutron-gateway (Juju Charms Collection): | |
status: | New → In Progress |
Changed in neutron-api (Juju Charms Collection): | |
status: | New → In Progress |
Changed in openstack-dashboard (Juju Charms Collection): | |
milestone: | none → 15.10 |
Changed in neutron-gateway (Juju Charms Collection): | |
milestone: | none → 15.10 |
Changed in neutron-api (Juju Charms Collection): | |
milestone: | none → 15.10 |
assignee: | nobody → Ryan Beisner (1chb1n) |
Changed in neutron-gateway (Juju Charms Collection): | |
assignee: | nobody → Ryan Beisner (1chb1n) |
Changed in openstack-dashboard (Juju Charms Collection): | |
assignee: | nobody → Ryan Beisner (1chb1n) |
Changed in neutron-api (Juju Charms Collection): | |
status: | In Progress → Fix Committed |
Changed in charm-helpers: | |
status: | New → Fix Released |
Changed in openstack-dashboard (Juju Charms Collection): | |
status: | In Progress → Fix Committed |
Changed in neutron-gateway (Juju Charms Collection): | |
status: | In Progress → Fix Committed |
Changed in neutron-api (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in neutron-gateway (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in openstack-dashboard (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in neutron-openvswitch (Ubuntu): | |
assignee: | nobody → Ryan Beisner (1chb1n) |
Changed in heat (Ubuntu): | |
assignee: | nobody → Ryan Beisner (1chb1n) |
Changed in glance (Ubuntu): | |
assignee: | nobody → Ryan Beisner (1chb1n) |
Changed in cinder (Juju Charms Collection): | |
assignee: | nobody → Ryan Beisner (1chb1n) |
status: | New → In Progress |
Changed in glance (Ubuntu): | |
status: | New → In Progress |
Changed in heat (Ubuntu): | |
status: | New → In Progress |
Changed in neutron-openvswitch (Ubuntu): | |
status: | New → In Progress |
Changed in glance (Ubuntu): | |
status: | In Progress → Confirmed |
Changed in heat (Ubuntu): | |
status: | In Progress → Confirmed |
Changed in cinder (Juju Charms Collection): | |
milestone: | none → 16.01 |
no longer affects: | neutron-openvswitch (Ubuntu) |
no longer affects: | heat (Ubuntu) |
no longer affects: | glance (Ubuntu) |
Changed in heat (Juju Charms Collection): | |
status: | New → Confirmed |
milestone: | none → 16.01 |
Changed in glance (Juju Charms Collection): | |
status: | New → Confirmed |
milestone: | none → 16.01 |
Changed in neutron-openvswitch (Juju Charms Collection): | |
status: | New → In Progress |
milestone: | none → 16.01 |
assignee: | nobody → Ryan Beisner (1chb1n) |
Changed in cinder (Juju Charms Collection): | |
status: | In Progress → Fix Committed |
Changed in neutron-openvswitch (Juju Charms Collection): | |
status: | In Progress → Fix Committed |
Changed in heat (Juju Charms Collection): | |
assignee: | nobody → Ryan Beisner (1chb1n) |
importance: | Undecided → Medium |
Changed in heat (Juju Charms Collection): | |
milestone: | 16.01 → 16.04 |
Changed in cinder (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in neutron-openvswitch (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in glance (Juju Charms Collection): | |
milestone: | 16.01 → 16.04 |
Changed in heat (Juju Charms Collection): | |
milestone: | 16.04 → 16.07 |
Changed in glance (Juju Charms Collection): | |
milestone: | 16.04 → 16.07 |
Changed in heat (Juju Charms Collection): | |
milestone: | 16.07 → 16.10 |
Changed in glance (Juju Charms Collection): | |
milestone: | 16.07 → 16.10 |
Changed in heat (Juju Charms Collection): | |
milestone: | 16.10 → 17.01 |
Changed in neutron-openvswitch (Juju Charms Collection): | |
importance: | Undecided → Medium |
Changed in glance (Juju Charms Collection): | |
importance: | Undecided → Medium |
Changed in charm-helpers: | |
importance: | Undecided → Medium |
Changed in cinder (Juju Charms Collection): | |
importance: | Undecided → Medium |
Changed in charm-helpers: | |
assignee: | nobody → Ryan Beisner (1chb1n) |
Changed in glance (Juju Charms Collection): | |
assignee: | nobody → Ryan Beisner (1chb1n) |
Changed in charm-glance: | |
assignee: | nobody → Ryan Beisner (1chb1n) |
importance: | Undecided → Medium |
status: | New → Confirmed |
Changed in glance (Juju Charms Collection): | |
status: | Confirmed → Invalid |
Changed in charm-heat: | |
assignee: | nobody → Ryan Beisner (1chb1n) |
importance: | Undecided → Medium |
status: | New → Confirmed |
Changed in heat (Juju Charms Collection): | |
status: | Confirmed → Invalid |
no longer affects: | charm-heat |
no longer affects: | charm-glance |
Switching from service_restarted to validate_ service_ config_ changed, the race in this bug still exists, as the process may not be started at the moment that it is checked. I think it needs a try/retry loop with a timeout threshold.
2015-08-24 16:27:24,641 test_900_ restart_ on_config_ change DEBUG: Making config change on neutron-api... logger: Making config change on neutron-api... restart_ on_config_ change DEBUG: Checking that service restarted: neutron-server logger: Checking that service restarted: neutron-server service_ config_ changed DEBUG: Checking neutron-server restarted since 1440433643.0 logger: Checking neutron-server restarted since 1440433643.0 restarted_ since DEBUG: Checking neutron-server restarted since 1440433643.0 logger: Checking neutron-server restarted since 1440433643.0 start_time DEBUG: CMDout: (u'15786', 0) logger: CMDout: (u'15786', 0) start_time DEBUG: Pid for neutron-server 15786 logger: Pid for neutron-server 15786 019-basic- vivid-kilo" , line 9, in <module> run_tests( ) ubuntu/ bzr/next/ neutron- api/tests/ charmhelpers/ contrib/ amulet/ deployment. py", line 93, in run_tests ubuntu/ bzr/next/ neutron- api/tests/ basic_deploymen t.py", line 548, in test_900_ restart_ on_config_ change time=sleep_ time): ubuntu/ bzr/next/ neutron- api/tests/ charmhelpers/ contrib/ amulet/ utils.py" , line 411, in validate_ service_ config_ changed count=retry_ count) ubuntu/ bzr/next/ neutron- api/tests/ charmhelpers/ contrib/ amulet/ utils.py" , line 327, in service_ restarted_ since ubuntu/ bzr/next/ neutron- api/tests/ charmhelpers/ contrib/ amulet/ utils.py" , line 289, in _get_proc_ start_time dir_mtime( sentry_ unit, proc_dir) ubuntu/ bzr/next/ neutron- api/tests/ charmhelpers/ contrib/ amulet/ utils.py" , line 270, in _get_dir_mtime unit.directory_ stat(directory) ['mtime' ] python2. 7/dist- packages/ amulet/ sentry. py", line 88, in directory_stat python2. 7/dist- packages/ amulet/ sentry. py", line 75, in _fs_data unit_script( "filesystem_ data.py {}".format(path)) python2. 7/dist- packages/ amulet/ sentry. py", line 126, in _run_unit_script filesystem_ data.py" , line 7, in <module> sys.argv[ 1])
DEBUG:amulet-
2015-08-24 16:27:26,794 test_900_
DEBUG:amulet-
2015-08-24 16:27:26,796 validate_
DEBUG:amulet-
2015-08-24 16:28:26,856 service_
DEBUG:amulet-
2015-08-24 16:29:15,571 _get_proc_
DEBUG:amulet-
2015-08-24 16:29:15,571 _get_proc_
DEBUG:amulet-
Traceback (most recent call last):
File "tests/
deployment.
File "/home/
getattr(self, test)()
File "/home/
sleep_
File "/home/
retry_
File "/home/
pgrep_full)
File "/home/
return self._get_
File "/home/
return sentry_
File "/usr/lib/
return self._fs_data(path)
File "/usr/lib/
return self._run_
File "/usr/lib/
raise IOError(output)
IOError: Traceback (most recent call last):
File "/tmp/amulet/
s = os.stat(
OSError: [Errno 2] No such file or directory: '/proc/15786'
ERROR subprocess encountered error code 1