OpenStack Heat

update_and_save raises NoneType exception

Bug #1274664 reported by Steven Dake on 2014-01-30

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Heat	Fix Released	Critical	Jason Dunsmore	OpenStack Heat 2014.1 "icehouse"

Bug Description

Fungi reported in the unit tests:

I seem to recall we had this problem at one point and it was fixed, but fungi is indicating it is happening 50-75% of the time in the unit tests.

2014-01-30 18:45:23.560 | Started by user anonymous
2014-01-30 18:45:23.562 | [EnvInject] - Loading node environment variables.
2014-01-30 18:45:26.835 | Building remotely on bare-precise-rax-iad-1252791 in workspace /home/jenkins/workspace/gate-heat-python27
2014-01-30 18:45:41.523 | [gate-heat-python27] $ /bin/bash -xe /tmp/hudson4037635517247339399.sh
2014-01-30 18:45:42.281 | + /usr/local/jenkins/slave_scripts/gerrit-git-prep.sh https://review.openstack.org http://zuul.openstack.org git://git.openstack.org
2014-01-30 18:45:42.289 | Triggered by: https://review.openstack.org/70045
2014-01-30 18:45:42.289 | + [[ ! -e .git ]]
2014-01-30 18:45:42.289 | + ls -a
2014-01-30 18:45:42.293 | .
2014-01-30 18:45:42.293 | ..
2014-01-30 18:45:42.294 | + rm -fr '.[^.]*' '*'
2014-01-30 18:45:42.297 | + '[' -d /opt/git/openstack/heat/.git ']'
2014-01-30 18:45:42.321 | + git clone file:///opt/git/openstack/heat .
2014-01-30 18:45:42.328 | Cloning into '.'...
2014-01-30 18:45:44.301 | + git remote set-url origin git://git.openstack.org/openstack/heat
2014-01-30 18:45:44.305 | + git remote update
2014-01-30 18:45:44.311 | Fetching origin
2014-01-30 18:45:45.642 | From git://git.openstack.org/openstack/heat
2014-01-30 18:45:45.642 | ae02ce4..0fe6748 master -> origin/master
2014-01-30 18:45:45.642 | * [new branch] stable/grizzly -> origin/stable/grizzly
2014-01-30 18:45:45.642 | * [new branch] stable/havana -> origin/stable/havana
2014-01-30 18:45:45.645 | + git reset --hard
2014-01-30 18:45:45.777 | HEAD is now at ae02ce4 Merge "fix some flaws in heat documents"
2014-01-30 18:45:45.777 | + git clean -x -f -d -q
2014-01-30 18:45:45.786 | + '[' -z '' ']'
2014-01-30 18:45:45.787 | + git fetch http://zuul.openstack.org/p/openstack/heat refs/zuul/master/Z4b3b11b8734842aabbe0a8750ede244b
2014-01-30 18:45:47.619 | From http://zuul.openstack.org/p/openstack/heat
2014-01-30 18:45:47.619 | * branch refs/zuul/master/Z4b3b11b8734842aabbe0a8750ede244b -> FETCH_HEAD
2014-01-30 18:45:47.624 | + git checkout FETCH_HEAD
2014-01-30 18:45:47.644 | Note: checking out 'FETCH_HEAD'.
2014-01-30 18:45:47.644 |
2014-01-30 18:45:47.644 | You are in 'detached HEAD' state. You can look around, make experimental
2014-01-30 18:45:47.644 | changes and commit them, and you can discard any commits you make in this
2014-01-30 18:45:47.644 | state without impacting any branches by performing another checkout.
2014-01-30 18:45:47.644 |
2014-01-30 18:45:47.644 | If you want to create a new branch to retain commits you create, you may
2014-01-30 18:45:47.644 | do so (now or later) by using -b with the checkout command again. Example:
2014-01-30 18:45:47.644 |
2014-01-30 18:45:47.644 | git checkout -b new_branch_name
2014-01-30 18:45:47.644 |
2014-01-30 18:45:47.645 | HEAD is now at 91f5cec... serialize non-string nova metadata
2014-01-30 18:45:47.645 | + git reset --hard FETCH_HEAD
2014-01-30 18:45:47.667 | HEAD is now at 91f5cec serialize non-string nova metadata
2014-01-30 18:45:47.667 | + git clean -x -f -d -q
2014-01-30 18:45:47.676 | + '[' -f .gitmodules ']'
2014-01-30 18:45:48.167 | [gate-heat-python27] $ /bin/bash -xe /tmp/hudson6820519068134555845.sh
2014-01-30 18:45:48.180 | + /usr/local/jenkins/slave_scripts/run-unittests.sh 27 openstack heat
2014-01-30 18:45:48.186 | + version=27
2014-01-30 18:45:48.186 | + org=openstack
2014-01-30 18:45:48.186 | + project=heat
2014-01-30 18:45:48.186 | + source /usr/local/jenkins/slave_scripts/functions.sh
2014-01-30 18:45:48.187 | + check_variable_version_org_project 27 openstack heat /usr/local/jenkins/slave_scripts/run-unittests.sh
2014-01-30 18:45:48.187 | + version=27
2014-01-30 18:45:48.187 | + org=openstack
2014-01-30 18:45:48.187 | + project=heat
2014-01-30 18:45:48.188 | + filename=/usr/local/jenkins/slave_scripts/run-unittests.sh
2014-01-30 18:45:48.188 | + [[ -z 27 ]]
2014-01-30 18:45:48.188 | + [[ -z openstack ]]
2014-01-30 18:45:48.188 | + [[ -z heat ]]
2014-01-30 18:45:48.188 | + venv=py27
2014-01-30 18:45:48.188 | + export NOSE_WITH_XUNIT=1
2014-01-30 18:45:48.188 | + NOSE_WITH_XUNIT=1
2014-01-30 18:45:48.206 | + export NOSE_WITH_HTML_OUTPUT=1
2014-01-30 18:45:48.206 | + NOSE_WITH_HTML_OUTPUT=1
2014-01-30 18:45:48.206 | + export NOSE_HTML_OUT_FILE=nose_results.html
2014-01-30 18:45:48.206 | + NOSE_HTML_OUT_FILE=nose_results.html
2014-01-30 18:45:48.207 | ++ /bin/mktemp -d
2014-01-30 18:45:48.207 | + export TMPDIR=/tmp/tmp.vK8628ftia
2014-01-30 18:45:48.207 | + TMPDIR=/tmp/tmp.vK8628ftia
2014-01-30 18:45:48.207 | + trap 'rm -rf /tmp/tmp.vK8628ftia' EXIT
2014-01-30 18:45:48.207 | + /usr/local/jenkins/slave_scripts/jenkins-oom-grep.sh pre
2014-01-30 18:45:48.214 | + sudo /usr/local/jenkins/slave_scripts/jenkins-sudo-grep.sh pre
2014-01-30 18:45:48.423 | + source /usr/local/jenkins/slave_scripts/select-mirror.sh openstack heat
2014-01-30 18:45:48.423 | ++ org=openstack
2014-01-30 18:45:48.424 | ++ project=heat
2014-01-30 18:45:48.424 | ++ source /usr/local/jenkins/slave_scripts/functions.sh
2014-01-30 18:45:48.424 | ++ check_variable_org_project openstack heat /usr/local/jenkins/slave_scripts/run-unittests.sh
2014-01-30 18:45:48.424 | ++ org=openstack
2014-01-30 18:45:48.424 | ++ project=heat
2014-01-30 18:45:48.424 | ++ filename=/usr/local/jenkins/slave_scripts/run-unittests.sh
2014-01-30 18:45:48.425 | ++ [[ -z openstack ]]
2014-01-30 18:45:48.425 | ++ [[ -z heat ]]
2014-01-30 18:45:48.425 | ++ rm -f /home/jenkins/.pydistutils.cfg
2014-01-30 18:45:48.426 | ++ mkdir -p /home/jenkins/.pip
2014-01-30 18:45:48.429 | ++ rm -f /home/jenkins/.pip/pip.conf
2014-01-30 18:45:48.432 | ++ cat
2014-01-30 18:45:48.436 | ++ grep -x openstack/heat /opt/requirements/projects.txt
2014-01-30 18:45:48.463 | openstack/heat
2014-01-30 18:45:48.463 | ++ export TOX_INDEX_URL=http://pypi.openstack.org/openstack
2014-01-30 18:45:48.463 | ++ TOX_INDEX_URL=http://pypi.openstack.org/openstack
2014-01-30 18:45:48.463 | ++ cat
2014-01-30 18:45:48.463 | ++ cat
2014-01-30 18:45:48.464 | + tox -epy27
2014-01-30 18:45:48.634 | py27 create: /home/jenkins/workspace/gate-heat-python27/.tox/py27
2014-01-30 18:45:49.822 | py27 installdeps: -r/home/jenkins/workspace/gate-heat-python27/requirements.txt, -r/home/jenkins/workspace/gate-heat-python27/test-requirements.txt
2014-01-30 18:48:21.493 | py27 develop-inst: /home/jenkins/workspace/gate-heat-python27
2014-01-30 18:48:23.181 | py27 runtests: commands[0] | python setup.py testr --slowest --testr-args=
2014-01-30 18:48:23.609 | running testr
2014-01-30 18:48:27.142 | INFO [migrate.versioning.api] 14 -> 15...
2014-01-30 18:48:27.165 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.169 | INFO [migrate.versioning.api] 15 -> 16...
2014-01-30 18:48:27.199 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.204 | INFO [migrate.versioning.api] 16 -> 17...
2014-01-30 18:48:27.303 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.305 | INFO [migrate.versioning.api] 17 -> 18...
2014-01-30 18:48:27.330 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.342 | INFO [migrate.versioning.api] 18 -> 19...
2014-01-30 18:48:27.342 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.353 | INFO [migrate.versioning.api] 19 -> 20...
2014-01-30 18:48:27.382 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.382 | INFO [migrate.versioning.api] 20 -> 21...
2014-01-30 18:48:27.384 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.421 | INFO [migrate.versioning.api] 21 -> 22...
2014-01-30 18:48:27.421 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.423 | INFO [migrate.versioning.api] 22 -> 23...
2014-01-30 18:48:27.426 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.434 | INFO [migrate.versioning.api] 23 -> 24...
2014-01-30 18:48:27.437 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.440 | INFO [migrate.versioning.api] 24 -> 25...
2014-01-30 18:48:27.444 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.447 | INFO [migrate.versioning.api] 25 -> 26...
2014-01-30 18:48:27.452 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.461 | INFO [migrate.versioning.api] 26 -> 27...
2014-01-30 18:48:27.461 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.461 | INFO [migrate.versioning.api] 27 -> 28...
2014-01-30 18:48:27.463 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.466 | INFO [migrate.versioning.api] 28 -> 29...
2014-01-30 18:48:27.474 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.477 | INFO [migrate.versioning.api] 29 -> 30...
2014-01-30 18:48:27.491 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.507 | INFO [migrate.versioning.api] 30 -> 31...
2014-01-30 18:48:27.514 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.591 | INFO [migrate.versioning.api] 31 -> 32...
2014-01-30 18:48:27.606 | INFO [migrate.versioning.api] done
2014-01-30 18:48:27.611 | INFO [migrate.versioning.api] 32 -> 33...
2014-01-30 18:48:27.630 | INFO [migrate.versioning.api] done
2014-01-30 18:48:28.617 | Traceback (most recent call last):
2014-01-30 18:48:28.617 | File "/home/jenkins/workspace/gate-heat-python27/.tox/py27/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in fire_timers
2014-01-30 18:48:28.617 | timer()
2014-01-30 18:48:28.618 | File "/home/jenkins/workspace/gate-heat-python27/.tox/py27/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 56, in __call__
2014-01-30 18:48:28.618 | cb(*args, **kw)
2014-01-30 18:48:28.656 | File "/home/jenkins/workspace/gate-heat-python27/.tox/py27/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2014-01-30 18:48:28.657 | result = function(*args, **kwargs)
2014-01-30 18:48:28.657 | File "heat/engine/service.py", line 363, in _stack_create
2014-01-30 18:48:28.657 | stack.create()
2014-01-30 18:48:28.658 | File "heat/engine/parser.py", line 410, in create
2014-01-30 18:48:28.658 | creator(timeout=self.timeout_secs())
2014-01-30 18:48:28.658 | File "heat/engine/scheduler.py", line 141, in __call__
2014-01-30 18:48:28.658 | self.start(timeout=timeout)
2014-01-30 18:48:28.659 | File "heat/engine/scheduler.py", line 162, in start
2014-01-30 18:48:28.659 | self.step()
2014-01-30 18:48:28.659 | File "heat/engine/scheduler.py", line 190, in step
2014-01-30 18:48:28.659 | next(self._runner)
2014-01-30 18:48:28.659 | File "heat/engine/scheduler.py", line 251, in wrapper
2014-01-30 18:48:28.660 | subtask = next(parent)
2014-01-30 18:48:28.660 | File "heat/engine/parser.py", line 426, in stack_task
2014-01-30 18:48:28.660 | 'Stack %s started' % action)
2014-01-30 18:48:28.660 | File "heat/engine/parser.py", line 377, in state_set
2014-01-30 18:48:28.661 | stack.update_and_save({'action': action,
2014-01-30 18:48:28.661 | AttributeError: 'NoneType' object has no attribute 'update_and_save'
2014-01-30 18:48:28.661 | Traceback (most recent call last):
2014-01-30 18:48:28.661 | File "/home/jenkins/workspace/gate-heat-python27/.tox/py27/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in fire_timers
2014-01-30 18:48:28.662 | timer()
2014-01-30 18:48:28.662 | File "/home/jenkins/workspace/gate-heat-python27/.tox/py27/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 56, in __call__
2014-01-30 18:48:28.662 | cb(*args, **kw)
2014-01-30 18:48:28.663 | File "/home/jenkins/workspace/gate-heat-python27/.tox/py27/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2014-01-30 18:48:28.663 | result = function(*args, **kwargs)
2014-01-30 18:48:28.663 | File "heat/engine/parser.py", line 603, in delete
2014-01-30 18:48:28.664 | self.state_set(action, self.IN_PROGRESS, 'Stack %s started' % action)
2014-01-30 18:48:28.664 | File "heat/engine/parser.py", line 377, in state_set
2014-01-30 18:48:28.664 | stack.update_and_save({'action': action,
2014-01-30 18:48:28.664 | AttributeError: 'NoneType' object has no attribute 'update_and_save'
2014-01-30 18:48:35.343 | INFO [heat.engine.resource] Validating RandomString "secret3"
2014-01-30 18:48:35.344 | INFO [heat.engine.resource] Validating RandomString "secret1"
2014-01-30 18:48:35.344 | INFO [heat.engine.resource] Validating RandomString "secret2"
2014-01-30 18:48:35.354 | INFO [heat.engine.resource] creating RandomString "secret3" Stack "test_stack" [ac435a1f-0be6-45c6-a50d-0950dc106641]
2014-01-30 18:48:35.383 | INFO [heat.engine.resource] creating RandomString "secret1" Stack "test_stack" [ac435a1f-0be6-45c6-a50d-0950dc106641]
2014-01-30 18:48:35.387 | INFO [heat.engine.resource] creating RandomString "secret2" Stack "test_stack" [ac435a1f-0be6-45c6-a50d-0950dc106641]
2014-01-30 18:48:49.660 | running=
2014-01-30 18:48:49.660 | ${PYTHON:-python} -m subunit.run discover -s . --list
2014-01-30 18:48:49.660 | ${PYTHON:-python} -m subunit.run discover -s contrib --list
2014-01-30 18:48:49.660 | running=
2014-01-30 18:48:49.660 | ${PYTHON:-python} -m subunit.run discover -s . --load-list /tmp/tmp.vK8628ftia/tmp9O0sbj
2014-01-30 18:48:49.660 | ${PYTHON:-python} -m subunit.run discover -s contrib --load-list /tmp/tmp.vK8628ftia/tmp9O0sbj
2014-01-30 18:48:49.661 | running=
2014-01-30 18:48:49.661 | ${PYTHON:-python} -m subunit.run discover -s . --load-list /tmp/tmp.vK8628ftia/tmp2K_sMm
2014-01-30 18:48:49.661 | ${PYTHON:-python} -m subunit.run discover -s contrib --load-list /tmp/tmp.vK8628ftia/tmp2K_sMm
2014-01-30 18:48:49.661 | running=
2014-01-30 18:48:49.661 | ${PYTHON:-python} -m subunit.run discover -s . --load-list /tmp/tmp.vK8628ftia/tmplDYcRT
2014-01-30 18:48:49.661 | ${PYTHON:-python} -m subunit.run discover -s contrib --load-list /tmp/tmp.vK8628ftia/tmplDYcRT
2014-01-30 18:48:49.661 | running=
2014-01-30 18:48:49.661 | ${PYTHON:-python} -m subunit.run discover -s . --load-list /tmp/tmp.vK8628ftia/tmpjQPyRd
2014-01-30 18:48:49.661 | ${PYTHON:-python} -m subunit.run discover -s contrib --load-list /tmp/tmp.vK8628ftia/tmpjQPyRd
2014-01-30 18:48:49.661 | running=
2014-01-30 18:48:49.661 | ${PYTHON:-python} -m subunit.run discover -s . --load-list /tmp/tmp.vK8628ftia/tmpu4cHVP
2014-01-30 18:48:49.661 | ${PYTHON:-python} -m subunit.run discover -s contrib --load-list /tmp/tmp.vK8628ftia/tmpu4cHVP
2014-01-30 18:48:49.662 | running=
2014-01-30 18:48:49.662 | ${PYTHON:-python} -m subunit.run discover -s . --load-list /tmp/tmp.vK8628ftia/tmpRSFl_w
2014-01-30 18:48:49.662 | ${PYTHON:-python} -m subunit.run discover -s contrib --load-list /tmp/tmp.vK8628ftia/tmpRSFl_w
2014-01-30 18:48:49.662 | running=
2014-01-30 18:48:49.662 | ${PYTHON:-python} -m subunit.run discover -s . --load-list /tmp/tmp.vK8628ftia/tmpVTOjau
2014-01-30 18:48:49.662 | ${PYTHON:-python} -m subunit.run discover -s contrib --load-list /tmp/tmp.vK8628ftia/tmpVTOjau
2014-01-30 18:48:49.662 | running=
2014-01-30 18:48:49.662 | ${PYTHON:-python} -m subunit.run discover -s . --load-list /tmp/tmp.vK8628ftia/tmp38uxPs
2014-01-30 18:48:49.662 | ${PYTHON:-python} -m subunit.run discover -s contrib --load-list /tmp/tmp.vK8628ftia/tmp38uxPs
2014-01-30 18:48:49.662 | Ran 2152 tests in 19.275s
2014-01-30 18:48:49.663 | PASSED (id=0)
2014-01-30 18:48:49.663 | Slowest Tests
2014-01-30 18:48:49.663 | Test id Runtime (s)
2014-01-30 18:48:49.663 | --------------------------------------------------------------------------------------------------------------------------- -----------
2014-01-30 18:48:49.663 | heat.tests.test_autoscaling_update_policy.AutoScalingGroupTest.test_autoscaling_group_update

Jason Dunsmore (jasondunsmore) on 2014-01-30

Changed in heat:
assignee:	nobody → Jason Dunsmore (jasondunsmore)

Revision history for this message

Clint Byrum (clint-fewbar) wrote on 2014-01-30:

Marking as Critical, if it is in fact happening often in the unit tests it will be a problem for the check queue and the gate.

summary:	- update_and_save raises NonType exception + update_and_save raises NoneType exception
Changed in heat:
status:	New → Triaged
importance:	Undecided → Critical

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-30: Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/70209

Changed in heat:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-31: Fix merged to heat (master)

Reviewed: https://review.openstack.org/70209
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=efe3bf1c2252dd35fb6fff2be80f33abb227a51f
Submitter: Jenkins
Branch: master

commit efe3bf1c2252dd35fb6fff2be80f33abb227a51f
Author: Jason Dunsmore <email address hidden>
Date: Thu Jan 30 13:32:30 2014 -0600

Only update_and_save if the stack exists

The following error can happen during a stack-delete:

AttributeError: 'NoneType' object has no attribute 'update_and_save'

Only update_and_save the stack if it is not None.

Change-Id: I8c27ed6a461c699eba7da5bef8e879f1c0bdaf6b
Closes-Bug: #1274664

Changed in heat:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2014-03-05

Changed in heat:
milestone:	none → icehouse-3
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-04-17

Changed in heat:
milestone:	icehouse-3 → 2014.1

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.