DataSourceSmartOS needs locking
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
cloud-init |
Fix Released
|
Medium
|
Mike Gerdts | ||
cloud-init (Ubuntu) |
Fix Released
|
Undecided
|
Chad Smith | ||
Bug Description
I'm seeing the following while trying to read meta-data from SmartOS.
2018-01-31 21:36:03,554 - DataSourceSmart
" to metadata transport.
2018-01-31 21:36:03,995 - DataSourceSmart
2018-01-31 21:36:03,996 - handlers.py[DEBUG]: finish: init-local/
2018-01-31 21:36:03,996 - util.py[WARNING]: Getting data from <class 'cloudinit.
2018-01-31 21:36:03,996 - util.py[DEBUG]: Getting data from <class 'cloudinit.
Traceback (most recent call last):
File "/usr/lib/
if s.get_data():
File "/usr/lib/
return_value = self._get_data()
File "/usr/lib/
md[ci_noun] = self.md_
File "/usr/lib/
result = self.get(key, default=default)
File "/usr/lib/
val = self._get(key, strip=False, default=mdefault)
File "/usr/lib/
get(key, default=default, strip=strip))
File "/usr/lib/
result = self.request(
File "/usr/lib/
value = self._get_
File "/usr/lib/
frame_data = self.line_
AttributeError: 'NoneType' object has no attribute 'groupdict'
2018-01-31 21:36:04,004 - main.py[DEBUG]: No local datasource found
[root@7180e700-
sdc:routes
[root@7180e700-
[]
This seems to cause DataSourceSmartOS to fail completely, then it goes on to time out on EC2 and CloudStack.
This is using my own build with this changeset at HEAD.
$ git log -n 1
commit f7deaf15acf382d
Related branches
- Server Team CI bot: Approve (continuous-integration)
- cloud-init Commiters: Pending requested
-
Diff: 2991 lines (+1186/-256)75 files modified.pylintrc (+1/-1)
cloudinit/analyze/dump.py (+1/-1)
cloudinit/cmd/tests/test_main.py (+3/-3)
cloudinit/config/cc_apt_configure.py (+1/-1)
cloudinit/config/cc_bootcmd.py (+0/-1)
cloudinit/config/cc_disk_setup.py (+4/-8)
cloudinit/config/cc_emit_upstart.py (+1/-1)
cloudinit/config/cc_resizefs.py (+3/-5)
cloudinit/config/cc_rh_subscription.py (+8/-10)
cloudinit/config/cc_runcmd.py (+0/-1)
cloudinit/config/cc_set_passwords.py (+45/-60)
cloudinit/config/cc_snap.py (+2/-3)
cloudinit/config/cc_snappy.py (+2/-2)
cloudinit/config/cc_ubuntu_advantage.py (+2/-3)
cloudinit/config/schema.py (+2/-2)
cloudinit/config/tests/test_set_passwords.py (+71/-0)
cloudinit/config/tests/test_snap.py (+27/-2)
cloudinit/config/tests/test_ubuntu_advantage.py (+28/-2)
cloudinit/distros/freebsd.py (+1/-1)
cloudinit/distros/ubuntu.py (+1/-1)
cloudinit/net/__init__.py (+27/-1)
cloudinit/net/cmdline.py (+1/-1)
cloudinit/net/dhcp.py (+1/-1)
cloudinit/net/sysconfig.py (+1/-2)
cloudinit/net/tests/test_init.py (+1/-0)
cloudinit/reporting/events.py (+1/-1)
cloudinit/sources/DataSourceAliYun.py (+1/-1)
cloudinit/sources/DataSourceAltCloud.py (+1/-4)
cloudinit/sources/DataSourceAzure.py (+13/-20)
cloudinit/sources/DataSourceIBMCloud.py (+92/-14)
cloudinit/sources/DataSourceMAAS.py (+1/-1)
cloudinit/sources/DataSourceOVF.py (+1/-1)
cloudinit/sources/DataSourceOpenStack.py (+2/-2)
cloudinit/sources/DataSourceSmartOS.py (+13/-5)
cloudinit/sources/helpers/digitalocean.py (+3/-4)
cloudinit/sources/helpers/openstack.py (+1/-1)
cloudinit/sources/helpers/vmware/imc/config_nic.py (+1/-1)
cloudinit/sources/helpers/vmware/imc/config_passwd.py (+2/-2)
cloudinit/sources/helpers/vmware/imc/guestcust_util.py (+2/-2)
cloudinit/sources/tests/test_init.py (+1/-1)
cloudinit/ssh_util.py (+63/-7)
cloudinit/templater.py (+1/-1)
cloudinit/tests/helpers.py (+32/-2)
cloudinit/tests/test_util.py (+50/-1)
cloudinit/url_helper.py (+1/-1)
cloudinit/util.py (+16/-1)
debian/changelog (+25/-0)
doc/rtd/topics/datasources.rst (+1/-0)
doc/rtd/topics/datasources/aliyun.rst (+74/-0)
packages/debian/control.in (+1/-0)
tests/cloud_tests/bddeb.py (+1/-1)
tests/cloud_tests/collect.py (+2/-1)
tests/cloud_tests/platforms/instances.py (+1/-1)
tests/cloud_tests/platforms/lxd/instance.py (+4/-6)
tests/cloud_tests/setup_image.py (+5/-6)
tests/cloud_tests/testcases/base.py (+1/-1)
tests/cloud_tests/testcases/examples/including_user_groups.py (+1/-1)
tests/cloud_tests/testcases/modules/user_groups.py (+1/-1)
tests/cloud_tests/util.py (+1/-1)
tests/unittests/test__init__.py (+1/-1)
tests/unittests/test_datasource/test_azure.py (+2/-2)
tests/unittests/test_datasource/test_ibmcloud.py (+50/-0)
tests/unittests/test_datasource/test_maas.py (+2/-2)
tests/unittests/test_datasource/test_nocloud.py (+0/-3)
tests/unittests/test_datasource/test_smartos.py (+117/-4)
tests/unittests/test_ds_identify.py (+64/-8)
tests/unittests/test_handler/test_handler_apt_source_v3.py (+1/-1)
tests/unittests/test_handler/test_handler_bootcmd.py (+26/-8)
tests/unittests/test_handler/test_handler_ntp.py (+1/-1)
tests/unittests/test_handler/test_handler_runcmd.py (+26/-7)
tests/unittests/test_net.py (+127/-5)
tests/unittests/test_sshutil.py (+94/-3)
tests/unittests/test_templating.py (+2/-2)
tests/unittests/test_util.py (+3/-3)
tools/ds-identify (+20/-1)
- Douglas Jordan: Pending requested
- Chad Smith: Pending requested
-
Diff: 170 lines (+86/-8)2 files modifiedcloudinit/sources/DataSourceAzure.py (+14/-6)
tests/unittests/test_datasource/test_azure.py (+72/-2)
- Scott Moser: Approve
- Server Team CI bot: Approve (continuous-integration)
-
Diff: 118 lines (+67/-2)2 files modifiedcloudinit/sources/DataSourceSmartOS.py (+2/-0)
tests/unittests/test_datasource/test_smartos.py (+65/-2)
summary: |
- stack trace when sdc:* not defined + DataSourceSmartOS needs locking and retries |
Changed in cloud-init: | |
importance: | Undecided → Medium |
status: | New → Confirmed |
In a subsequent boot, I noticed a different stack:
2018-01-31 21:56:17,517 - DataSourceSmart OS.py[DEBUG] : Writing "V2 29 ae9f5912 d2a2a4ca GET c2RjOnJvdXRlcw== search- SmartOS: FAIL: no local data found from DataSourceSmartOS sources. DataSourceSmart OS.DataSourceSm artOS'> failed sources. DataSourceSmart OS.DataSourceSm artOS'> failed python2. 7/site- packages/ cloudinit/ sources/ __init_ _.py", line 434, in find_source python2. 7/site- packages/ cloudinit/ sources/ __init_ _.py", line 121, in get_data python2. 7/site- packages/ cloudinit/ sources/ DataSourceSmart OS.py", line 237, in _get_data client. get_json( smartos_ noun) python2. 7/site- packages/ cloudinit/ sources/ DataSourceSmart OS.py", line 406, in get_json python2. 7/site- packages/ cloudinit/ sources/ DataSourceSmart OS.py", line 552, in get python2. 7/site- packages/ cloudinit/ sources/ DataSourceSmart OS.py", line 538, in _get python2. 7/site- packages/ cloudinit/ sources/ DataSourceSmart OS.py", line 398, in get rtype=' GET', param=key) python2. 7/site- packages/ cloudinit/ sources/ DataSourceSmart OS.py", line 383, in request extend( self.fp. read(1) ) python2. 7/site- packages/ serial/ serialposix. py", line 453, in read
" to metadata transport.
2018-01-31 21:56:17,825 - handlers.py[DEBUG]: finish: init-local/
2018-01-31 21:56:17,825 - util.py[WARNING]: Getting data from <class 'cloudinit.
2018-01-31 21:56:17,826 - util.py[DEBUG]: Getting data from <class 'cloudinit.
Traceback (most recent call last):
File "/usr/lib/
if s.get_data():
File "/usr/lib/
return_value = self._get_data()
File "/usr/lib/
md[ci_noun] = self.md_
File "/usr/lib/
result = self.get(key, default=default)
File "/usr/lib/
val = self._get(key, strip=False, default=mdefault)
File "/usr/lib/
get(key, default=default, strip=strip))
File "/usr/lib/
result = self.request(
File "/usr/lib/
response.
File "/usr/lib/
buf = os.read(self.fd, size-len(read))
OSError: [Errno 11] Resource temporarily unavailable
If I understand it correctly, this time around there was a timeout while reading from the serial port. The timeout is 60 seconds and the machine was very lightly loaded. This is starting to feel more like there is something else going on. This is on bhyve, which is being ported to SmartOS. As such, there is a distinct possibility I am hitting a host or hypervisor issue.