walinuxagent not downloading ssh certificates

[Scott Moser]

I launched an instance with:
  azure vm create --vm-size=extrasmall --vm-name=sm-testme0 "--location=East US" \
   --<email address hidden> --ssh=22 \
  --custom-data=/tmp/my.ud sm-testme0 \
  b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu_DAILY_BUILD-trusty-14_04-LTS-amd64-server-20140108-en-us-30GB \
  smoser PASS%word%123

Unless my custom-data provides some way to get in (ie, cloud-config 'ssh_import_id: smoser') then I'm not able to get to the instance. Looking at cloud-init.log, I see:

2014-01-09 17:56:59,819 - util.py[DEBUG]: Running command ['service', 'walinuxagent', 'start'] with allowed return codes [0] (shell=False, capture=True)
2014-01-09 17:58:00,588 - util.py[DEBUG]: waiting for files took 60.558 seconds
2014-01-09 17:58:00,589 - DataSourceAzure.py[WARNING]: Did not find files, but going on: set([u'/var/lib/waagent/6BE7A7C3C8A8F4B123CCA5D0C2F1BE4CA7B63ED7.crt'])
2014-01-09 17:58:00,597 - util.py[DEBUG]: Running command ['sh', '-c', 'openssl x509 -noout -pubkey < "$0" |ssh-keygen -i -m PKCS8 -f /dev/stdin', u'/var/lib/waagent/6BE7A7C3C8A8F4B123CCA5D0C2F1BE4CA7B63ED7.crt'] with allowed return codes [0] (shell=False, capture=True)
2014-01-09 17:58:00,697 - DataSourceAzure.py[WARNING]: failed to convert the crt files to pubkey: [<trimed>]
2014-01-09 17:58:00,716 - stages.py[DEBUG]: Loaded datasource DataSourceAzureNet - DataSourceAzureNet [seed=/dev/sr0]

The gist is that cloud-init ran walinuxagent, and expected it to produce /var/lib/waagent/BE7A7C3C8A8F4B123CCA5D0C2F1BE4CA7B63ED7.crt as that was mentioned in the ovf-env.xml. However, walinuxagent did not do that.

/var/log/waagent.log would normally say something like:
2014/01/09 18:10:27 Public cert with thumbprint: D3BCD6F2904D5E4B5E8155ED1E0A698C7B14F007 was retrieved.

but there isn't such a message in mine.

When I compare this to a system where it *did* have such a message, the HostingEnvironmentConfig.xml files differ.
The broken one is missing a section like:
  <StoredCertificates>
    <StoredCertificate name="Cert0My" certificateId="sha1:D3BCD6F2904D5E4B5E8155ED1E0A698C7B14F007" storeName="My" configurationLevel="System" />
  </StoredCertificates>

HostingEnvironmentConfig.xml is obtained by contacting the metadata service. Its possible the server side has changed its response, but its also possible that I had never previously tested providing both a password and a ssh key.

Possibly relevant information:
  * the '--custom-data' comes from patches at https://gist.github.com/smoser/5806147 .
  * Recently, it seems that in order to launch an instance with custom-data, server side validation is forcing you to also supply a password . That is just mentioned here as a reason for providing both password and ssh keys, which may be relevant.
  * I've had to change the azure/lib/services/management/servicemanagementservice.js to report itself as 2013-10-01 rather than 2013-06-01 in order to have custom-data allowed.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: walinuxagent 1.3.2-0ubuntu5 [modified: usr/sbin/waagent]
ProcVersionSignature: User Name 3.12.0-7.15-generic 3.12.4
Uname: Linux 3.12.0-7-generic x86_64
ApportVersion: 2.12.7-0ubuntu6
Architecture: amd64
Date: Thu Jan 9 18:38:22 2014
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: walinuxagent
UpgradeStatus: No upgrade log present (probably fresh install)

[Stephen A. Zarkos (stevez)]

Hi Scott,

I'm not able to repro this using the current dev branch of the CLI tools and the latest CustomData patches. Can you take a look at the latest CustomData patches at the pull requests below. They are mostly similar to yours and should port easily into the version of the xplat tools and SDK you are using:

https://github.com/WindowsAzure/azure-sdk-tools-xplat/pull/1048
https://github.com/WindowsAzure/azure-sdk-for-node/pull/1054

One notable change is that we re-arranged roleschema.json a bit and moved the CustomData section for both the Windows and Linux provisioning configuration (to make it consistent for both platforms). I'm not sure yet if the issue you are seeing is with the wire server or the agent, possibly the json is confusing the API and so we end up not getting the certificates we need.

The --no-ssh-password problem may be related to this issue:
https://github.com/WindowsAzure/azure-sdk-tools-xplat/issues/1003
https://github.com/WindowsAzure/azure-sdk-tools-xplat/pull/1004

That fix may actually be in the most recent release of the CLI tools, it does not repro in my installation. What version are you working with now?

We should see the feature branch happening soon-ish that will include the CustomData patches. I can let you know when that happens if you want to test it.

I hope this helps.

Steve

[Scott Moser (smoser)]

Invalid. i can't reproduce this.