When bootstrapping on an Openstack network with Swift installed (plus S3 API emulation) and S3_URL set to the Swift service, the bootstrap node does not install correctly. "juju -v status" hangs on "Environment still initializing. Will wait.". The swift container (bucket) appears to be created correctly through the S3 emulation.
SSHing into the bootstrap node, /var/log/juju/machine-agent.log appears to error out with "juju.state.errors.MachineStateNotFound: Machine 0 was not found"
Note that this is limited to "openstack_s3" with Swift plus swift3. "openstack" with Swift works as expected (grabs Swift location from Keystone's object-store), as done "openstack_s3" pointed at a nova-objectstore.
Environment is Swift Folsom, along with the swift3 plugin from https://github.com/fujita/swift3 .
ryan@linda:~$ juju -v bootstrap
2012-10-08 11:44:30,357 DEBUG Initializing juju bootstrap runtime
2012-10-08 11:44:30,366 DEBUG openstack: using auth-mode 'userpass' with https://keystone.example.com:443/v2.0/
2012-10-08 11:44:31,199 DEBUG openstack: authenticated til u'2012-10-09T18:44:32Z'
2012-10-08 11:44:31,199 DEBUG openstack: GET 'https://nova-lcy02.example.com/v2/ACCOUNT/flavors'
2012-10-08 11:44:31,980 DEBUG openstack: 200 '{"flavors": [{"id": "1", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/1", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/1", "rel": "bookmark"}], "name": "m1.tiny"}, {"id": "2", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/2", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/2", "rel": "bookmark"}], "name": "m1.small"}, {"id": "3", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/3", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/3", "rel": "bookmark"}], "name": "m1.medium"}, {"id": "4", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/4", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/4", "rel": "bookmark"}], "name": "m1.large"}, {"id": "5", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/5", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/5", "rel": "bookmark"}], "name": "m1.xlarge"}]}'
2012-10-08 11:44:31,982 INFO Bootstrapping environment 'swifttest1' (origin: distro type: openstack_s3)...
2012-10-08 11:44:32,856 DEBUG Verifying writable storage
2012-10-08 11:44:33,732 DEBUG Launching juju bootstrap instance.
2012-10-08 11:44:33,766 DEBUG access compute @ https://nova-lcy02.example.com/v2/ACCOUNT/os-security-groups
2012-10-08 11:44:33,767 DEBUG openstack: GET 'https://nova-lcy02.example.com/v2/ACCOUNT/os-security-groups'
2012-10-08 11:44:34,553 DEBUG openstack: 200 '{"security_groups": [{"rules": [{"from_port": 22, "group": {}, "ip_protocol": "tcp", "to_port": 22, "parent_group_id": 843, "ip_range": {"cidr": "0.0.0.0/0"}, "id": 216}, {"from_port": -1, "group": {}, "ip_protocol": "icmp", "to_port": -1, "parent_group_id": 843, "ip_range": {"cidr": "0.0.0.0/0"}, "id": 217}], "tenant_id": "ACCOUNT", "id": 843, "name": "default", "description": "default"}]}'
2012-10-08 11:44:34,554 DEBUG Creating juju security group juju-swifttest1
2012-10-08 11:44:34,554 DEBUG openstack: POST 'https://nova-lcy02.example.com/v2/ACCOUNT/os-security-groups'
2012-10-08 11:44:35,362 DEBUG openstack: 200 '{"security_group": {"rules": [], "tenant_id": "ACCOUNT", "id": 1670, "name": "juju-swifttest1", "description": "juju group for swifttest1"}}'
2012-10-08 11:44:35,363 DEBUG openstack: POST 'https://nova-lcy02.example.com/v2/ACCOUNT/os-security-group-rules'
2012-10-08 11:44:36,246 DEBUG openstack: 200 '{"security_group_rule": {"from_port": 22, "group": {}, "ip_protocol": "tcp", "to_port": 22, "parent_group_id": 1670, "ip_range": {"cidr": "0.0.0.0/0"}, "id": 362}}'
2012-10-08 11:44:36,247 DEBUG openstack: POST 'https://nova-lcy02.example.com/v2/ACCOUNT/os-security-group-rules'
2012-10-08 11:44:37,163 DEBUG openstack: 200 '{"security_group_rule": {"from_port": 1, "group": {"tenant_id": "ACCOUNT", "name": "juju-swifttest1"}, "ip_protocol": "tcp", "to_port": 65535, "parent_group_id": 1670, "ip_range": {}, "id": 363}}'
2012-10-08 11:44:37,164 DEBUG openstack: POST 'https://nova-lcy02.example.com/v2/ACCOUNT/os-security-group-rules'
2012-10-08 11:44:38,053 DEBUG openstack: 200 '{"security_group_rule": {"from_port": 1, "group": {"tenant_id": "ACCOUNT", "name": "juju-swifttest1"}, "ip_protocol": "udp", "to_port": 65535, "parent_group_id": 1670, "ip_range": {}, "id": 364}}'
2012-10-08 11:44:38,054 DEBUG Creating machine security group juju-swifttest1-0
2012-10-08 11:44:38,054 DEBUG openstack: POST 'https://nova-lcy02.example.com/v2/ACCOUNT/os-security-groups'
2012-10-08 11:44:38,947 DEBUG openstack: 200 '{"security_group": {"rules": [], "tenant_id": "ACCOUNT", "id": 1671, "name": "juju-swifttest1-0", "description": "juju group for swifttest1 machine 0"}}'
2012-10-08 11:44:38,947 DEBUG access compute @ https://nova-lcy02.example.com/v2/ACCOUNT/flavors/detail
2012-10-08 11:44:38,947 DEBUG openstack: GET 'https://nova-lcy02.example.com/v2/ACCOUNT/flavors/detail'
2012-10-08 11:44:39,686 DEBUG openstack: 200 '{"flavors": [{"vcpus": 1, "disk": 0, "name": "m1.tiny", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/1", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/1", "rel": "bookmark"}], "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 0, "ram": 512, "id": "1", "swap": ""}, {"vcpus": 1, "disk": 10, "name": "m1.small", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/2", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/2", "rel": "bookmark"}], "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 20, "ram": 2048, "id": "2", "swap": ""}, {"vcpus": 2, "disk": 10, "name": "m1.medium", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/3", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/3", "rel": "bookmark"}], "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 40, "ram": 4096, "id": "3", "swap": ""}, {"vcpus": 4, "disk": 10, "name": "m1.large", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/4", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/4", "rel": "bookmark"}], "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 80, "ram": 8192, "id": "4", "swap": ""}, {"vcpus": 8, "disk": 10, "name": "m1.xlarge", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/5", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/5", "rel": "bookmark"}], "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 160, "ram": 16384, "id": "5", "swap": ""}]}'
2012-10-08 11:44:39,687 DEBUG access compute @ https://nova-lcy02.example.com/v2/ACCOUNT/servers
2012-10-08 11:44:39,687 DEBUG openstack: POST 'https://nova-lcy02.example.com/v2/ACCOUNT/servers'
2012-10-08 11:44:41,011 DEBUG openstack: 202 '{"server": {"OS-DCF:diskConfig": "MANUAL", "id": "94c92309-7860-44bd-b682-2da23034ffbc", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/servers/94c92309-7860-44bd-b682-2da23034ffbc", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/servers/94c92309-7860-44bd-b682-2da23034ffbc", "rel": "bookmark"}], "adminPass": "REMOVED"}}'
2012-10-08 11:44:42,787 INFO 'bootstrap' command finished successfully
ryan@linda:~$ juju -v status
2012-10-08 11:48:51,501 DEBUG Initializing juju status runtime
2012-10-08 11:48:51,510 DEBUG openstack: using auth-mode 'userpass' with https://keystone.example.com:443/v2.0/
2012-10-08 11:48:51,511 INFO Connecting to environment...
2012-10-08 11:48:53,204 DEBUG openstack: authenticated til u'2012-10-09T18:48:54Z'
2012-10-08 11:48:53,205 DEBUG access compute @ https://nova-lcy02.example.com/v2/ACCOUNT/servers/94c92309-7860-44bd-b682-2da23034ffbc
2012-10-08 11:48:53,206 DEBUG openstack: GET 'https://nova-lcy02.example.com/v2/ACCOUNT/servers/94c92309-7860-44bd-b682-2da23034ffbc'
2012-10-08 11:48:54,064 DEBUG openstack: 200 '{"server": {"OS-EXT-STS:task_state": null, "addresses": {"canonistack": [{"version": 4, "addr": "10.55.63.35"}]}, "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/servers/94c92309-7860-44bd-b682-2da23034ffbc", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/servers/94c92309-7860-44bd-b682-2da23034ffbc", "rel": "bookmark"}], "image": {"id": "4326c238-e834-450f-8b5d-29e045d1a4d2", "links": [{"href": "http://nova-lcy02.example.com/ACCOUNT/images/4326c238-e834-450f-8b5d-29e045d1a4d2", "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "flavor": {"id": "1", "links": [{"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/1", "rel": "bookmark"}]}, "id": "94c92309-7860-44bd-b682-2da23034ffbc", "user_id": "28b21464bfce4233b6f3cdf2d45b30e9", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "", "status": "ACTIVE", "updated": "2012-10-08T18:46:34Z", "hostId": "4cb114bd10f38d5a3c199af1a504d49d68c0acce6be97c743c1b90c5", "key_name": "", "name": "juju swifttest1 instance 0", "created": "2012-10-08T18:44:41Z", "tenant_id": "ACCOUNT", "metadata": {}}}'
2012-10-08 11:48:54,066 DEBUG Connecting to environment using 10.55.63.35...
2012-10-08 11:48:54,067 DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="10.55.63.35" remote_port="2181" local_port="39204".
2012-10-08 11:48:58,583:9323(0x7f16d2f87700):ZOO_INFO@log_env@658: Client environment:zookeeper.version=zookeeper C client 3.3.6
2012-10-08 11:48:58,583:9323(0x7f16d2f87700):ZOO_INFO@log_env@662: Client environment:host.name=linda
2012-10-08 11:48:58,583:9323(0x7f16d2f87700):ZOO_INFO@log_env@669: Client environment:os.name=Linux
2012-10-08 11:48:58,583:9323(0x7f16d2f87700):ZOO_INFO@log_env@670: Client environment:os.arch=3.5.0-17-generic
2012-10-08 11:48:58,583:9323(0x7f16d2f87700):ZOO_INFO@log_env@671: Client environment:os.version=#27-Ubuntu SMP Fri Oct 5 01:35:26 UTC 2012
2012-10-08 11:48:58,584:9323(0x7f16d2f87700):ZOO_INFO@log_env@679: Client environment:user.name=ryan
2012-10-08 11:48:58,584:9323(0x7f16d2f87700):ZOO_INFO@log_env@687: Client environment:user.home=/home/ryan
2012-10-08 11:48:58,584:9323(0x7f16d2f87700):ZOO_INFO@log_env@699: Client environment:user.dir=/home/ryan
2012-10-08 11:48:58,584:9323(0x7f16d2f87700):ZOO_INFO@zookeeper_init@727: Initiating client connection, host=localhost:39204 sessionTimeout=10000 watcher=0x7f16d2a70700 sessionId=0 sessionPasswd=<null> context=0x35ba180 flags=0
2012-10-08 11:48:58,585:9323(0x7f16ce548700):ZOO_INFO@check_events@1585: initiated connection to server [127.0.0.1:39204]
2012-10-08 11:48:59,889:9323(0x7f16ce548700):ZOO_INFO@check_events@1632: session establishment complete on server [127.0.0.1:39204], sessionId=0x13a41b4f0ca0003, negotiated timeout=10000
2012-10-08 11:49:00,079 DEBUG Environment still initializing. Will wait.
ubuntu@juju-swifttest1-instance-0:~$ cat /var/log/juju/machine-agent.log
2012-10-08 18:48:33,208: twisted@ERROR: Unhandled error in Deferred:
2012-10-08 18:48:33,237: twisted@ERROR: Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 391, in errback
self._startRunCallbacks(fail)
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 458, in _startRunCallbacks
self._runCallbacks()
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 545, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1095, in gotResult
_inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/python2.7/dist-packages/juju/agents/base.py", line 245, in startService
yield self.start()
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/python2.7/dist-packages/juju/agents/machine.py", line 63, in start
self.get_machine_id())
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1039, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/juju/state/machine.py", line 83, in get_machine_state
raise MachineStateNotFound(machine_id)
juju.state.errors.MachineStateNotFound: Machine 0 was not found
It appears the bootstrap node is trying to get $BUCKET/ juju_master_ id using an invalid query string authentication method.
JUJU_CLIENT_IP - - [09/Oct/ 2012:02: 40:33 +0000] "GET /juju-rfinnie- metadata- swift/provider- state HTTP/1.0" 404 4747 "-" "Twisted PageGetter" 2012:02: 40:34 +0000] "PUT /juju-rfinnie- metadata- swift/bootstrap -verify HTTP/1.0" 404 4747 "-" "Twisted PageGetter" 2012:02: 40:35 +0000] "PUT /juju-rfinnie- metadata- swift/ HTTP/1.0" 200 4678 "-" "Twisted PageGetter" 2012:02: 40:36 +0000] "PUT /juju-rfinnie- metadata- swift/bootstrap -verify HTTP/1.0" 200 4630 "-" "Twisted PageGetter" 2012:02: 40:42 +0000] "PUT /juju-rfinnie- metadata- swift/juju_ master_ id HTTP/1.0" 200 4630 "-" "Twisted PageGetter" 2012:02: 40:43 +0000] "PUT /juju-rfinnie- metadata- swift/provider- state HTTP/1.0" 200 4630 "-" "Twisted PageGetter" 2012:02: 44:38 +0000] "GET /juju-rfinnie- metadata- swift/juju_ master_ id?Signature= REMOVED HTTP/1.1" 401 4895 "-" "curl/7.22.0 (x86_64- pc-linux- gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"
JUJU_CLIENT_IP - - [09/Oct/
JUJU_CLIENT_IP - - [09/Oct/
JUJU_CLIENT_IP - - [09/Oct/
JUJU_CLIENT_IP - - [09/Oct/
JUJU_CLIENT_IP - - [09/Oct/
BOOTSTRAP_IP - - [09/Oct/
According to http:// s3.amazonaws. com/doc/ s3-developer- guide/RESTAuthe ntication. html, query string authentication requires AWSAccessKeyId, Expires and Signature, while the bootstrap node is only providing Signature. I'm guessing this is papered over in real S3 because Juju is setting the bucket to full global read (!!!), and swift3 does not support S3 ACLs, so the container/bucket is still restricted to only the owner.