cloud-init-single-module-fails-to-merge

[Marco Morais]

I have a yaml file in a local directory (cloud-init.yaml) which contains 2 modules (write_files, runcmd). I am exercising the write_files module using the following command:

$ sudo cloud-init --debug --file cloud-init.yaml single --name cc_write_files

From the debug output I can see that the local file (/home/mmorais/cloud-init.yaml) is not getting merged into the master config file (/var/lib/cloud/instance/cloud-config.txt).

2014-05-05 21:25:45,367 - __init__.py[DEBUG]: Merging using located merger 'DictMerger: (method=no_replace,recurse_str=False,recurse_dict=True,recurse_array=False,allow_delete=False)' since it had method '_on_dict'
2014-05-05 21:25:45,368 - util.py[DEBUG]: Reading from /home/mmorais/cloud-init.yaml (quiet=False)
2014-05-05 21:25:45,368 - util.py[DEBUG]: Read 407 bytes from /home/mmorais/cloud-init.yaml
2014-05-05 21:25:45,368 - util.py[DEBUG]: < #cloud-config
2014-05-05 21:25:45,368 - util.py[DEBUG]: < write_files:
2014-05-05 21:25:45,368 - util.py[DEBUG]: < - content: |
2014-05-05 21:25:45,368 - util.py[DEBUG]: < #!/bin/sh
2014-05-05 21:25:45,368 - util.py[DEBUG]: < date +"%s" > /home/y/tmp/now.txt
2014-05-05 21:25:45,368 - util.py[DEBUG]: < path: /home/y/tmp/upload-boot-time.sh
2014-05-05 21:25:45,368 - util.py[DEBUG]: < permissions: '0755'
2014-05-05 21:25:45,369 - util.py[DEBUG]: < runcmd:
2014-05-05 21:25:45,369 - util.py[DEBUG]: < - ['/home/y/tmp/upload-boot-time.sh']
2014-05-05 21:25:45,369 - util.py[DEBUG]: Attempting to load yaml from string of length 407 with allowed root types (<type 'dict'>,)
2014-05-05 21:25:45,371 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/cloud-config.txt (quiet=False)
2014-05-05 21:25:45,371 - util.py[DEBUG]: Read 0 bytes from /var/lib/cloud/instance/cloud-config.txt
2014-05-05 21:25:45,371 - util.py[DEBUG]: Attempting to load yaml from string of length 0 with allowed root types (<type 'dict'>,)
2014-05-05 21:25:45,371 - util.py[DEBUG]: load_yaml given empty string, returning default

As a workaround I am copying /home/mmorais/cloud-init.yaml to /var/lib/cloud/instance/cloud-config.txt which defeats the purpose of being able to run cloud-init on arbitrary files for testing.

Also note that the command above does not work at all unless I specify *cc_write_files* instead of *write_files*.

I am running cloud-init (0.7.4).

[Scott Moser (smoser)]

$ cat my.cfg
write_files:
 - content: |
     hi mom
   owner: root:root
   path: /tmp/greeting.txt
   permissions: '0644'

$ sudo cloud-init --debug --file my.cfg single --name write_files --frequency=always ; echo $?
...
0

$ cat /tmp/greeting.txt
hi mom

So, that generally is working as designed.
I think that 2 things were tripping you up.
A.) you need to say --frequency=always or the write_files will run only with its default frequency (once per instance).
B.) merging in cloud-init with arrays is kind of annoying.

The default merge routine replaces arrays, where as I think you would rather have it append.
In recent versions of cloud-init, you *can* have it append, you just need to change to use cloud-config-jsonp format.

I tried quickly to ge tan example of doing that.. and my attempt looked like this:
$ cat my-patch.cfg
[
  {"op": "add", "path": "/write-files/-",
   "value": {"content": "hi mom", "path": "/tmp/new-greeting.txt"}}
]

using that fails though.

[Scott Moser (smoser)]

try again, same fail, but
$ cat my-patch.cfg
#cloud-config-jsonp
[
  {"op": "add", "path": "/write-files/-",
   "value": {"content": "hi mom", "path": "/tmp/new-greeting.txt"}}
]

[Scott Moser (smoser)]

relavent log from the attempt looks like:
$ sudo cloud-init --debug --file my-patch.cfg single --name write_files --frequency=always

2014-08-21 21:03:28,925 - util.py[DEBUG]: Reading from /home/ubuntu/my-patch.cfg (quiet=
False)
2014-08-21 21:03:28,925 - util.py[DEBUG]: Read 134 bytes from /home/ubuntu/my-patch.cfg
2014-08-21 21:03:28,925 - util.py[DEBUG]: Attempting to load yaml from string of length 134 with allowed root types (<type 'dict'>,)
2014-08-21 21:03:28,927 - util.py[WARNING]: Failed loading yaml blob
2014-08-21 21:03:28,927 - util.py[DEBUG]: Failed loading yaml blob
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 719, in load_yaml
    (allowed, type_utils.obj_name(converted)))
TypeError: Yaml load allows (<type 'dict'>,) root types, but got list instead
2014-08-21 21:03:28,927 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/cloud-config.txt (quiet=False)

[Joshua Harlow (harlowja)]

So try the following, this should work:

[harlowja@droppingstopping ~]$ cat /var/lib/cloud/instance/cloud-config.txt
#cloud-config
merge_how: "dict(allow_delete,recurse_array)+list(recurse_array,append)"
write_files:
 - content: "#!/bin/sh"
   path: /home/y/tmp/upload-boot-time.sh
   permissions: '0755'

[harlowja@droppingstopping ~]$ cat my.cfg
#cloud-config
write_files:
 - content: |
     hi mom
   owner: root:root
   path: /tmp/greeting.txt
   permissions: '0644'

[Scott Moser (smoser)]

and above, harlowja's 'merge-how' could also occur inside the 'my.cfg' (i expect, not tested). but thats the goal.

[Scott Moser (smoser)]

As this is right now, i'm going to make treat this bug as:
  "cloud-init --config=foo.cfg single" does not allow 'cloud-config-jsonp' in foo.cfg.

[Scott Moser (smoser)]

unless marco raises otherwise.