InvocationError while running Gerrit Jenkins

[Amit Kumar Das]

Patch reviews fail at gate-cinder-python26 & gate-cinder-python27 with below error :

2014-06-26 10:14:08.255 | check_uptodate.sh: Please run /home/jenkins/workspace/gate-cinder-python27/tools/config/generate_sample.sh.
2014-06-26 10:14:08.255 | ERROR: InvocationError: '/home/jenkins/workspace/gate-cinder-python27/tools/config/check_uptodate.sh'
2014-06-26 10:14:08.256 | ___________________________________ summary ____________________________________
2014-06-26 10:14:08.256 | ERROR: py27: commands failed
2014-06-26 10:14:08.266 | + result=1
2014-06-26 10:14:08.266 | + echo 'Begin pip freeze output from test virtualenv:'
2014-06-26 10:14:08.266 | Begin pip freeze output from test virtualenv:
2014-06-26 10:14:08.266 | + echo ======================================================================
2014-06-26 10:14:08.267 | ======================================================================
2014-06-26 10:14:08.267 | + .tox/py27/bin/pip freeze
2014-06-26 10:14:08.639 | Babel==1.3
2014-06-26 10:14:08.639 | Jinja2==2.7.3
2014-06-26 10:14:08.639 | MarkupSafe==0.23
2014-06-26 10:14:08.639 | MySQL-python==1.2.5
2014-06-26 10:14:08.639 | Paste==1.7.5.1
2014-06-26 10:14:08.639 | PasteDeploy==1.5.2
2014-06-26 10:14:08.639 | PyYAML==3.11
2014-06-26 10:14:08.640 | Pygments==1.6
2014-06-26 10:14:08.640 | Routes==2.0
2014-06-26 10:14:08.640 | SQLAlchemy==0.9.6
2014-06-26 10:14:08.640 | Sphinx==1.2.2
2014-06-26 10:14:08.640 | Tempita==0.5.2
2014-06-26 10:14:08.640 | WebOb==1.4
2014-06-26 10:14:08.640 | amqp==1.4.5
2014-06-26 10:14:08.641 | anyjson==0.3.3
2014-06-26 10:14:08.641 | argparse==1.2.1
2014-06-26 10:14:08.641 | cffi==0.8.2
2014-06-26 10:14:08.641 | -e git://git.openstack.org/openstack/cinder@6408a2a59f9c7611b9db56b8c162622a3d3bede5#egg=cinder-dev
2014-06-26 10:14:08.641 | coverage==3.7.1
2014-06-26 10:14:08.641 | cryptography==0.4
2014-06-26 10:14:08.641 | decorator==3.4.0
2014-06-26 10:14:08.642 | discover==0.4.0
2014-06-26 10:14:08.642 | docutils==0.10
2014-06-26 10:14:08.642 | ecdsa==0.11
2014-06-26 10:14:08.642 | eventlet==0.14.0
2014-06-26 10:14:08.642 | extras==0.0.3
2014-06-26 10:14:08.642 | fixtures==0.3.14
2014-06-26 10:14:08.642 | flake8==2.0
2014-06-26 10:14:08.643 | futures==2.1.6
2014-06-26 10:14:08.643 | greenlet==0.4.2
2014-06-26 10:14:08.643 | hacking==0.8.1
2014-06-26 10:14:08.643 | iso8601==0.1.10
2014-06-26 10:14:08.643 | jsonpatch==1.6
2014-06-26 10:14:08.643 | jsonpointer==1.3
2014-06-26 10:14:08.643 | jsonschema==2.3.0
2014-06-26 10:14:08.644 | kombu==3.0.20
2014-06-26 10:14:08.644 | lxml==3.3.5
2014-06-26 10:14:08.644 | mccabe==0.2.1
2014-06-26 10:14:08.644 | mock==1.0.1
2014-06-26 10:14:08.644 | mox==0.5.3
2014-06-26 10:14:08.644 | netaddr==0.7.11
2014-06-26 10:14:08.645 | networkx==1.9
2014-06-26 10:14:08.645 | oslo.config==1.3.0
2014-06-26 10:14:08.645 | oslo.messaging==1.3.0
2014-06-26 10:14:08.645 | oslo.rootwrap==1.2.0
2014-06-26 10:14:08.645 | oslosphinx==2.1.0
2014-06-26 10:14:08.645 | paramiko==1.14.0
2014-06-26 10:14:08.645 | pbr==0.8.2
2014-06-26 10:14:08.646 | pep8==1.4.5
2014-06-26 10:14:08.646 | prettytable==0.7.2
2014-06-26 10:14:08.646 | psycopg2==2.5.3
2014-06-26 10:14:08.646 | pyOpenSSL==0.14
2014-06-26 10:14:08.646 | pycparser==2.10
2014-06-26 10:14:08.646 | pycrypto==2.6.1
2014-06-26 10:14:08.646 | pyflakes==0.7.3
2014-06-26 10:14:08.647 | python-glanceclient==0.12.0
2014-06-26 10:14:08.647 | python-keystoneclient==0.9.0
2014-06-26 10:14:08.647 | python-mimeparse==0.1.4
2014-06-26 10:14:08.647 | python-novaclient==2.17.0
2014-06-26 10:14:08.647 | python-subunit==0.0.18
2014-06-26 10:14:08.647 | python-swiftclient==2.1.0
2014-06-26 10:14:08.647 | pytz==2014.4
2014-06-26 10:14:08.648 | repoze.lru==0.6
2014-06-26 10:14:08.648 | requests==2.3.0
2014-06-26 10:14:08.648 | rtslib-fb==2.1.47
2014-06-26 10:14:08.648 | simplejson==3.5.3
2014-06-26 10:14:08.648 | six==1.7.2
2014-06-26 10:14:08.648 | sqlalchemy-migrate==0.9.1
2014-06-26 10:14:08.648 | stevedore==0.15
2014-06-26 10:14:08.649 | suds==0.4
2014-06-26 10:14:08.649 | taskflow==0.3.21
2014-06-26 10:14:08.649 | testrepository==0.0.18
2014-06-26 10:14:08.649 | testscenarios==0.4
2014-06-26 10:14:08.649 | testtools==0.9.35
2014-06-26 10:14:08.649 | warlock==1.1.0
2014-06-26 10:14:08.649 | wsgiref==0.1.2
2014-06-26 10:14:08.676 | + echo ======================================================================
2014-06-26 10:14:08.676 | ======================================================================
2014-06-26 10:14:08.677 | + '[' -d .testrepository ']'
2014-06-26 10:14:09.042 | + '[' -f .testrepository/0.2 ']'
2014-06-26 10:14:09.042 | + '[' -f .testrepository/0 ']'
2014-06-26 10:14:09.042 | + .tox/py27/bin/subunit-1to2
2014-06-26 10:14:11.145 | + .tox/py27/bin/python /usr/local/jenkins/slave_scripts/subunit2html.py ./subunit_log.txt testr_results.html
2014-06-26 10:14:13.356 | ++ du -k ./subunit_log.txt
2014-06-26 10:14:13.357 | ++ awk '{print $1}'
2014-06-26 10:14:13.381 | + SUBUNIT_SIZE=2244
2014-06-26 10:14:13.381 | + gzip -9 ./subunit_log.txt
2014-06-26 10:14:13.447 | + gzip -9 ./testr_results.html
2014-06-26 10:14:13.488 | + export PYTHON=.tox/py27/bin/python
2014-06-26 10:14:13.488 | + PYTHON=.tox/py27/bin/python
2014-06-26 10:14:13.489 | + [[ 2244 -gt 50000 ]]
2014-06-26 10:14:13.489 | ++ .tox/py27/bin/testr last
2014-06-26 10:14:13.490 | ++ sed -ne 's/Ran \([0-9]\+\).*tests in.*/\1/p'
2014-06-26 10:14:15.415 | + rancount=3064
2014-06-26 10:14:15.416 | + '[' -z 3064 ']'
2014-06-26 10:14:15.416 | + '[' 3064 -eq 0 ']'
2014-06-26 10:14:15.416 | + sudo /usr/local/jenkins/slave_scripts/jenkins-sudo-grep.sh post
2014-06-26 10:14:15.418 | sudo: unable to resolve host bare-precise-hpcloud-b1-583839
2014-06-26 10:14:15.515 | + sudoresult=0
2014-06-26 10:14:15.516 | + '[' 0 -ne 0 ']'
2014-06-26 10:14:15.516 | + /usr/local/jenkins/slave_scripts/jenkins-oom-grep.sh post
2014-06-26 10:14:15.523 | + oomresult=0
2014-06-26 10:14:15.523 | + '[' 0 -ne 0 ']'
2014-06-26 10:14:15.524 | ++ find . -name nose_results.html
2014-06-26 10:14:15.544 | + htmlreport=
2014-06-26 10:14:15.544 | + '[' -f '' ']'
2014-06-26 10:14:15.545 | + echo
2014-06-26 10:14:15.545 |
2014-06-26 10:14:15.545 | + echo 'WARNING: Unable to find nose_results.html to confirm results!'
2014-06-26 10:14:15.545 | WARNING: Unable to find nose_results.html to confirm results!
2014-06-26 10:14:15.545 | + echo
2014-06-26 10:14:15.545 |
2014-06-26 10:14:15.546 | + exit 1
2014-06-26 10:14:15.546 | + rm -rf /tmp/tmp.Q4GnFerx0N
2014-06-26 10:14:15.751 | Build step 'Execute shell' marked build as failure
2014-06-26 10:14:17.190 | [SCP] Connecting to static.openstack.org
2014-06-26 10:14:21.511 | [SCP] No file(s) found: **/*nose_results.html
2014-06-26 10:14:23.176 | [SCP] ‘**/*nose_results.html’ doesn’t match anything: ‘**’ exists but not ‘**/*nose_results.html’
2014-06-26 10:14:23.646 | [SCP] Trying to create /srv/static/logs/11/102511/2/check/gate-cinder-python27
2014-06-26 10:14:23.660 | [SCP] Trying to create /srv/static/logs/11/102511/2/check/gate-cinder-python27/3fc3b72
2014-06-26 10:14:23.694 | [SCP] uploading file: '/srv/static/logs/11/102511/2/check/gate-cinder-python27/3fc3b72/testr_results.html.gz'
2014-06-26 10:14:24.053 | [SCP] No file(s) found: .testrepository/tmp*
2014-06-26 10:14:24.361 | [SCP] ‘.testrepository/tmp*’ doesn’t match anything: ‘.testrepository’ exists but not ‘.testrepository/tmp*’
2014-06-26 10:14:24.765 | [SCP] uploading file: '/srv/static/logs/11/102511/2/check/gate-cinder-python27/3fc3b72/subunit_log.txt.gz'
2014-06-26 10:14:25.389 | [SCP] Copying console log.
2014-06-26 10:14:26.211 | Finished: FAILURE

[Parashuram Hallur (hallur-p-t)]

I'm also seeing this issue from patchsets https://review.openstack.org/#/c/104639/12.

I have generated the cinder.conf.sample using the script ../tools/config/generate_sample.sh
and ran the ./tools/config/check_uptodate.sh

Here is the status of the run on my machine
---------
stack@lglw9118:/home/stack/cinder-repo/cinder$ source .venv/bin/activate
(.venv)stack@lglw9118:/home/stack/cinder-repo/cinder$ ./tools/config/generate_sample.sh
No handlers could be found for logger "cinder.volume.drivers.san.hp.hp_lefthand_rest_proxy"
(.venv)stack@lglw9118:/home/stack/cinder-repo/cinder$ ./tools/config/check_uptodate.sh
No handlers could be found for logger "cinder.volume.drivers.san.hp.hp_lefthand_rest_proxy"
(.venv)stack@lglw9118:/home/stack/cinder-repo/cinder$
---------

2014-07-09 06:01:29.596 | check_uptodate.sh: cinder.conf.sample is not up to date.
2014-07-09 06:01:29.596 | check_uptodate.sh: Please run /home/jenkins/workspace/gate-cinder-python27/tools/config/generate_sample.sh.
2014-07-09 06:01:29.600 | ERROR: InvocationError: '/home/jenkins/workspace/gate-cinder-python27/tools/config/check_uptodate.sh'
2014-07-09 06:01:29.600 | ___________________________________ summary ____________________________________
2014-07-09 06:01:29.600 | ERROR: py27: commands failed
2014-07-09 06:01:29.605 | + result=1
2014-07-09 06:01:29.605 | + echo 'Begin pip freeze output from test virtualenv:'
2014-07-09 06:01:29.605 | Begin pip freeze output from test virtualenv:
2014-07-09 06:01:29.605 | + echo ======================================================================
2014-07-09 06:01:29.605 | ======================================================================
2014-07-09 06:01:29.605 | + .tox/py27/bin/pip freeze
2014-07-09 06:01:29.925 | Babel==1.3
2014-07-09 06:01:29.925 | Jinja2==2.7.3
2014-07-09 06:01:29.926 | MarkupSafe==0.23
2014-07-09 06:01:29.926 | MySQL-python==1.2.5
2014-07-09 06:01:29.926 | Paste==1.7.5.1
2014-07-09 06:01:29.926 | PasteDeploy==1.5.2
2014-07-09 06:01:29.926 | PyYAML==3.11
2014-07-09 06:01:29.926 | Pygments==1.6
2014-07-09 06:01:29.926 | Routes==1.13
2014-07-09 06:01:29.926 | SQLAlchemy==0.9.6
2014-07-09 06:01:29.926 | Sphinx==1.2.2
2014-07-09 06:01:29.926 | Tempita==0.5.2
2014-07-09 06:01:29.926 | WebOb==1.4
2014-07-09 06:01:29.927 | amqp==1.4.5
2014-07-09 06:01:29.927 | anyjson==0.3.3
2014-07-09 06:01:29.927 | argparse==1.2.1
2014-07-09 06:01:29.927 | cffi==0.8.6
2014-07-09 06:01:29.927 | -e git://git.openstack.org/openstack/cinder@bbf843a7872d0850f42f0d7aa247b01bf449ef33#egg=cinder-dev
2014-07-09 06:01:29.927 | coverage==3.7.1
2014-07-09 06:01:29.927 | cryptography==0.5.1
2014-07-09 06:01:29.927 | decorator==3.4.0
2014-07-09 06:01:29.927 | discover==0.4.0
2014-07-09 06:01:29.927 | docutils==0.11
2014-07-09 06:01:29.927 | ecdsa==0.11
2014-07-09 06:01:29.927 | eventlet==0.15.0
2014-07-09 06:01:29.928 | extras==0.0.3
2014-07-09 06:01:29.928 | fixtures==0.3.14
2014-07-09 06:01:29.928 | flake8==2.1.0
2014-07-09 06:01:29.928 | futures==2.1.6
2014-07-09 06:01:29.928 | greenlet==0.4.2
2014-07-09 06:01:29.928 | hacking==0.9.2
2014-07-09 06:01:29.928 | iso8601==0.1.10
2014-07-09 06:01:29.928 | jsonpatch==1.7
2014-07-09 06:01:29.928 | js...

[Amit Kumar Das (amit-das-8)]

This issue was resolved once i generated the cinder.conf.sample file & git commited the diff.

[Parashuram Hallur (hallur-p-t)]

Hi Amit:
Could you mention the steps you followed

I did the following
1. Git checkout master
2. git pull origin master
3. git checkout <feature branch> and rebase with master
4. Generated the file
5. git add cinder.conf.sample
6. git commit --amend
7. git review -v

However, I still see the issue.

[Amit Kumar Das (amit-das-8)]

Hi,

When you generate the file the git status should show that the sample conf file has been modified.

When you do a git diff, it should show the relevant additions or deletions that you have made to your configuration options py file.

This diff should be uploaded along with your original commit or this might be a new commit if configuration was the only change.

Step 5 is not required.

[Amit Kumar Das (amit-das-8)]

"When you do a git diff, it should show the relevant additions or deletions that you have made to your configuration options py file."

The changes/diff in the sample conf file will be based on the changes you have made to your configuration py file.

[Parashuram Hallur (hallur-p-t)]

Thanks Amit for the response/suggestion. My patchset includes the changes from the configuration py file. I did ensure that the diff showed up correctly. However these tests are still failing. Not sure why

[Parashuram Hallur (hallur-p-t)]

And step 5 in the above list is always required, otherwise the change will stay uncommitted in the staging area and it does not allow to do the "git review -v"

[Amit Kumar Das (amit-das-8)]

Ah yes. Step 5 is required. I had confused it for new file added to git branch.

[Amit Kumar Das (amit-das-8)]

Hi Parashuram,

Have posted review comment on the patch. Kindly check them.

They seem to be related to this issue of yours.

"
 @@ -19,7 +19,7 @@
2014-07-09 06:01:14.333 | #allowed_rpc_exception_modules=oslo.messaging.exceptions,nova.exception,cinder.exception,exceptions
2014-07-09 06:01:14.336 |
2014-07-09 06:01:14.337 | # Qpid broker hostname. (string value)
2014-07-09 06:01:14.337 | -#qpid_hostname=localhost
2014-07-09 06:01:14.337 | +#qpid_hostname=cinder
2014-07-09 06:01:14.337 |
2014-07-09 06:01:14.338 | # Qpid broker port. (integer value)
2014-07-09 06:01:14.338 | #qpid_port=5672
2014-07-09 06:01:14.338 | @@ -76,7 +76,7 @@
2014-07-09 06:01:14.339 |
2014-07-09 06:01:14.339 | # The RabbitMQ broker address where a single node is used.
2014-07-09 06:01:14.339 | # (string value)
2014-07-09 06:01:14.340 | -#rabbit_host=localhost
2014-07-09 06:01:14.340 | +#rabbit_host=cinder
2014-07-09 06:01:14.340 |
2014-07-09 06:01:14.340 | # The RabbitMQ broker port where a single node is used.
2014-07-09 06:01:14.341 | # (integer value)
"

[Parashuram Hallur (hallur-p-t)]

Thanks Amit, I got it working. The problem was one of the empty line towards the end of the file was deleted.

Also the entries mentioned in the last comment of yours are introduced by ./generate_sample.sh script, I removed those changes manually. Now these tests are passing.

[Gloria Gu (gloria-gu)]

lots of cinder build failed on the same spot for gate-cinder-python27 again.