Cobbler token timeout causes RedHat deployment failure

[Nikolay Fedotov]

ISO: {"release": "3.2.1", "nailgun_sha": "523b673edadb41f076284f2ca389b6724c64693c", "ostf_sha": "ea44d89ebe4e4fc7e24f64a3bdc6f9be6bcea9f6", "astute_sha": "df6ddea3abc93fbe1cab9b4534d4d5e9508c95d6", "fuellib_sha": "555081c25d85250c04e568f2287768387614ed5d"}

*steps*
- configure redhat release
- create environment. Multi-node, VLANManager
- add nodes: 1 controller, 1 compute
- deploy changes

*result*
Orchestrator error in /var/log/naily/naily.log
2013-11-28 09:49:37 ERR
[9281] Error running provisioning: <class 'cobbler.cexceptions.CX'>:'invalid token: zxasGA5+Mn9gJyN9ttEVLU7zyrZkxcJwFA==', trace: ["/opt/rbenv/versions/1.9.3-p392/lib/ruby/1.9.1/xmlrpc/client.rb:414:in `call'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/astute-0.0.2/lib/astute/cobbler.rb:105:in `sync'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/astute-0.0.2/lib/astute/orchestrator.rb:68:in `provision'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/dispatcher.rb:80:in `provision'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:103:in `dispatch_message'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:70:in `block in dispatch'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:68:in `each'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:68:in `each_with_index'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:68:in `dispatch'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:38:in `block (2 levels) in server_loop'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:49:in `block (2 levels) in consume_one'"]

In cobbler.log:
Mon Dec 2 05:46:46 2013 - INFO | REMOTE invalid token; user(???)
Mon Dec 2 05:46:46 2013 - INFO | Exception occured: <class 'cobbler.cexceptions.CX'>
Mon Dec 2 05:46:46 2013 - INFO | Exception value: 'invalid token: Qur3qoOFzrKbbSBQCo4sxxRWMf5iMZi+5Q=='
Mon Dec 2 05:46:46 2013 - INFO | Exception Info:
  File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 1970, in _dispatch
    return method_handle(*params)
   File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 815, in modify_item
    self.check_access(token, "modify_%s"%what, obj, attribute)
   File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 1733, in check_access
    user = self.get_user_from_token(token)
   File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 396, in get_user_from_token
    raise CX("invalid token: %s" % token)

Mon Dec 2 05:46:46 2013 - INFO | REMOTE sync; user(???)
Mon Dec 2 05:46:46 2013 - INFO | REMOTE invalid token; user(???)
Mon Dec 2 05:46:46 2013 - INFO | Exception occured: <class 'cobbler.cexceptions.CX'>
Mon Dec 2 05:46:46 2013 - INFO | Exception value: 'invalid token: Qur3qoOFzrKbbSBQCo4sxxRWMf5iMZi+5Q=='
Mon Dec 2 05:46:46 2013 - INFO | Exception Info:
  File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 1970, in _dispatch
    return method_handle(*params)
   File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 1801, in sync
    self.check_access(token,"sync")
   File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 1733, in check_access
    user = self.get_user_from_token(token)
   File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 396, in get_user_from_token
    raise CX("invalid token: %s" % token)

[Matthew Mosesohn (raytrac3r)]

Nikolay, could you tell me which 3.2.1 ISO from Jenkins you used so I can try to reproduce this?

[Nikolay Fedotov (nfedotov)]

ISO: fuel-3.2.1-73-2013-11-28_06-53-30.iso

[Matthew Mosesohn (raytrac3r)]

Nikolay,

I downloaded this ISO and it just worked. I was able to download RHOS from RHSM and install 2 nodes in non-HA mode. I passed all health check tests (except external Internet, which is fine).

[Nikolay Fedotov (nfedotov)]

Can reproduce it. I tried to install 1 node (1 controller) without computes. It is a bit strange case but anyway could take a look.

[Matthew Mosesohn (raytrac3r)]

Clarification: This problem occurs when you make snapshots and then resume them the next day. The default Cobbler timeout is 60 minutes and is hardcoded.

What we need to do in naily is request a new auth token if "invalid token" error is received.

[Matthew Mosesohn (raytrac3r)]

https://github.com/cobbler/cobbler/commit/b4ab497ae9f8a598a4165f6e953dc13179906e0b#diff-0ae5ba0dbee50061b41ad6084ad83836 This commit adds a feature for cobbler to specify auth_token_expiration in /etc/cobbler/settings

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60440
Committed: http://github.com/stackforge/fuel-astute/commit/3055cb18099db0037a37edaa45850cfdeecec03e
Submitter: Jenkins
Branch: master

commit 3055cb18099db0037a37edaa45850cfdeecec03e
Author: Vladimir <email address hidden>
Date: Fri Dec 6 12:23:40 2013 +0400

    Get new Cobbler token if old is unvalid

    Get token for every call to prevent token expiration

    Closes-Bug: #1256006

    Change-Id: Ibde35849837d8c724ee73379a6fd5f7084dde0bf