persisting OpenStack metadata fails
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
cloud-init |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
Persistsing OpenStack metadata may fail.
OpenStack has an entry named "random_seed" in the metadata [1]. This entry is treaded specially in the openstack.py helper when the metadat is read [2]. The attempt to decode the read data my result in a string that is not utf-8 compliant which eventually results in an error when we attempt to persist the metadata as follows:
2018-11-02 12:14:13,755 - __init_
[1] https:/
[2] https:/
Related branches
- Robert Schweikert (community): Approve
- Server Team CI bot: Approve (continuous-integration)
- cloud-init Commiters: Pending requested
-
Diff: 108 lines (+50/-9)3 files modifiedcloudinit/sources/tests/test_init.py (+5/-7)
cloudinit/tests/test_util.py (+20/-0)
cloudinit/util.py (+25/-2)
- Server Team CI bot: Needs Fixing (continuous-integration)
- Ryan Harper: Approve
-
Diff: 2904 lines (+1300/-449)34 files modifiedcloudinit/cmd/main.py (+5/-4)
cloudinit/config/cc_ubuntu_advantage.py (+116/-109)
cloudinit/config/cc_ubuntu_drivers.py (+112/-0)
cloudinit/config/tests/test_ubuntu_advantage.py (+191/-156)
cloudinit/config/tests/test_ubuntu_drivers.py (+174/-0)
cloudinit/net/eni.py (+11/-5)
cloudinit/net/network_state.py (+33/-8)
cloudinit/net/sysconfig.py (+25/-9)
cloudinit/sources/DataSourceAzure.py (+168/-89)
cloudinit/sources/DataSourceEc2.py (+6/-2)
cloudinit/sources/DataSourceNoCloud.py (+3/-1)
cloudinit/sources/DataSourceScaleway.py (+2/-1)
cloudinit/sources/__init__.py (+3/-3)
cloudinit/sources/helpers/azure.py (+31/-0)
cloudinit/sources/tests/test_init.py (+15/-0)
cloudinit/util.py (+15/-0)
config/cloud.cfg.tmpl (+3/-0)
debian/changelog (+29/-0)
doc/rtd/topics/datasources/azure.rst (+35/-22)
doc/rtd/topics/datasources/nocloud.rst (+1/-1)
doc/rtd/topics/modules.rst (+1/-0)
tests/cloud_tests/testcases/modules/apt_pipelining_disable.yaml (+1/-2)
tests/cloud_tests/testcases/modules/apt_pipelining_os.py (+3/-3)
tests/cloud_tests/testcases/modules/apt_pipelining_os.yaml (+4/-5)
tests/data/azure/non_unicode_random_string (+1/-0)
tests/unittests/test_datasource/test_azure.py (+22/-2)
tests/unittests/test_datasource/test_nocloud.py (+42/-0)
tests/unittests/test_datasource/test_scaleway.py (+7/-0)
tests/unittests/test_distros/test_netconfig.py (+2/-0)
tests/unittests/test_ds_identify.py (+17/-0)
tests/unittests/test_handler/test_schema.py (+1/-0)
tests/unittests/test_net.py (+209/-15)
tools/ds-identify (+4/-3)
tox.ini (+8/-9)
summary: |
- persisting OpenStack metadat fails + persisting OpenStack metadata fails |
Example data for the case that triggered this issue:
From debug output inserted into util.load_json, which is called from read_v2() in openstack.py
"random_seed": "Cr7LysUKbg/ wjTuPKS2qq7rZ9B CWJQAnckkvxCv7f mfOWmHARAWj8mPE fIT+tl8Q1v9Kz3Q MKBC4SfgfjLF+ doX84pjcDw87lL4 UvDzzQfANmvUPeQ Sc8wpbMe9l74hDE vd/U7Rf1/ ag7E/tlQZRwlhQF cHrMI1TUSLKe70v SDspK0NNB1p1Gnw KB1Mp683iBtWcSW 9zURrdXmYGvl+ CvW/EJcesbC59XP X4AezVqejKwI4Tt FLDQZ+A9HNLJOXi 4E8/gFQO2hEhD56 vDzqh1VZNGj2aUd J7gBoBBE3qtEwda AkwWS2hS6BV6e1k Lgio2M6r/ Qz44uf8im9JJdyF lDsVycr6+ ZOdCIQTHdEhVQnj wvkxaZFFxp4CaVh zb75FgESkUoNzr/ Hr7GILAThJvPLB+ a94Q8uSjElSP9r5 +0LbB6CGNvEAXDE 0IlupzltNS1K9gH Vm7mDziA7wU6jtN rMb22gXgVs4pi3Q UJlBZYwR9dIPZIR wBCCrzWtyuW+ 0LMZmoADSosJ1ix FfX34thN/ 6CmVSjqOZNErBEc SnJ6apSQ8dpTgdn +yORZDs7O1cGGOz kMf4zS9t4TdLpmP TUyfFBi8ykjPLb0 Ec8eAAAK/ FwUN30LOqJ3GPRI 9SN6UBu+ 6feg96NAu9Surwp kucdg3L94Z0hNi2 1E7G9ET19Gw= "
After processing this turned into:
u'random_seed': '\xb8!\ ')\xe0\ x0e%<\xac) E\xf6\xe0. kt>V0\x88\ xdf\xb0\ x02:\xbc\ x192F+) P5\x9eL\ xab\xbd\ xf5\xa2g7\ x02J\xfb\ xd6\x14> ^\xe9\x87\ xf1\xa1LJ\ xd1\xbc\ xe3y\x0b\ xd73\x93\ xc3\xb8% 3\xac_\ x1cN\x02F\ xf3\xec\ xdaA.^\ x7f\x1e\ xf2\x958\ x7f\xc8_ f\x0c\xf3\ x9e\xce\ xb3\xe7> \xfa\x9a\ xe6\xbbX/ \x01>\r\ xc8\xa1\ xb1\x852\ x05@:\xbb\ xa3\x98\ xa1\xf7\ x11\x82\ xbeCN\xe9. \xd7\xa6\ x85\xf9\ x0b\t\xa7\ xcd\xc3U\ '\xac\xaa\ xc2\xea\ r\x11\xe1\ xdc\'\x0c\ x9a\x7f\ x85\xcc\ xe3\xac] 4S\xad\ x08V\xcc\ xba\xf6\ xf2\x89\ xc40\xad\ xfc%\xab? \tP\xc9\ x82\x87} z\x91\xd6\ x94\x17\ x8e\x83\ xa3\x15\ xc5\xfc\ x01\x81E\ xc0w|\x98Q4= \x9f\x07\ xf1\xa3ot\ xc6\x87X\ xd0\xe1\ x93\xae\ xbe\x0f\ \\xd3\x08VU\ x1d\xf8H\ xbb\xa4/ \xc1\x96\ x83\xb7\ xed\x89\ xb7\xa2\ x10O\x1c* \xc4\xbc\ xf6*2\x9cq\ x9c\x10\ xd8q\x93\ xec\xd2\ x0e\xd3\ xb3\xed\ xa2\xf8\ x8b\xcb1VT\ x9f 4\xdeg; \xf5\xe3\ xdb\xd9\ xb7\xe7\ x90\xf3@ \xf4qn\ xcf\xf0x\ xed*X`\ x13YS\x12\ xa6&\xfe\ xb5)/\xf5| \x8bp\t\ xc6\xfb3\ x03/\xaf\ xefa\x1b+ \xa2F7\ x10\x18\ x91?&\xe8\ x88\x1bY~ %|\xb1\ xe8\x1c\ x96\xdf\ xa1\xabH\ xf7\xab\ xb8\xc7\ xdfgT#\ xf2\xbd\ x198j%\ xba\x1f\ xea\x96\ x1e\r_\ xaeRe)\ xb5"-\xbf\ xe7\xb9\ x8c\xdb$ \xf2{\x98\ x04o\xd7D\ x86\xce\ x86\xf7F\ xa4\x8e: OZv@K0\ x0c\x0e\ xf4\xf6\ xce\x9c\ xc8\x8ai\ xa1\xf1Ec\ x95U\x88. \xc1\xce\ xb5\x92\ x98\xb5\ xe7\xf3\ xc4\xd5\ xc2\x1fR\ xe5`-\xbd\ xeaBC\xc0\ xad\xdda\ xb0oX\xf2b\ xf88a\x12\ x94R^\x0812\ x7f\xa2v\ x06Q\xc8\ x13X\x8e\ x8d-\x82\ x1b\x82\ x17\x18\ x1c\xdd> \x8c\x13\ x190\\" z\x8f\xa6\ x18\xf3\ xbf\x9e\ x95\xa7\ x8d\x89\ x91\x83\ x1a\xe5\ xbe\xcfK\ x08\xff[ !\xe5\xba\ x9f\xfah~ \x85\x16\ x18\xd9GX\ x9f\x0c\ xa4\xa9\ xb7\xdcn\ xd2\xaf5\ xc6\xbc' , u'uuid': u'4f9baeab- 1f8c-48ca- 8766-37c4c59927 cf'
per debug output inserted into _crawl_metadata(). With this data in the metadata dictionary persisting the data via persist_ instance_ data() fails with the message shown in the description.
The question is whether "random_seed" should be persisted in the first place. The value for "random_seed" is different every time the metadata server is accessed. Thus the value is not useful for comparison. However I do not know for what other reasons the persisted data would be used.