Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] __init__.py[DEBUG]: Seeing if we can get any data from <class 'cloudinit.sources.DataSourceGCE.DataSourceGCE'>
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Checking DNS redirection...
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Trying does-not-exist.example.com....
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Not redirected
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Trying example.invalid....
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Not redirected
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Trying KgWv29ZuRsiJoFAGfGdwRj6To1L4b44q...
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Not redirected
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: DNS redirection checking complete.
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Doing actual check...
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Actual check successful.
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] url_helper.py[DEBUG]: [0/1] open 'http://metadata.google.internal/computeMetadata/v1/instance/id' with {'method': 'GET', 'url': 'http://metadata.google.internal
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] url_helper.py[DEBUG]: Read from http://metadata.google.internal/computeMetadata/v1/instance/id (200, 19b) after 1 attempts
And an instrumented yakkety gives:
Oct 03 09:41:19 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] __init__.py[DEBUG]: Seeing if we can get any data from <class 'cloudinit.sources.DataSourceGCE.DataSourceGCE'>
Oct 03 09:41:19 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Checking DNS redirection...
Oct 03 09:41:19 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Trying does-not-exist.example.com....
Oct 03 09:41:44 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Not redirected
Oct 03 09:41:44 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Trying example.invalid....
Oct 03 09:42:10 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Not redirected
Oct 03 09:42:10 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Trying DLMcOj0Uix9neTVLk3ksle1KNUDrSz5p...
Oct 03 09:42:35 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Not redirected
Oct 03 09:42:35 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: DNS redirection checking complete.
Oct 03 09:42:35 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Doing actual check...
Oct 03 09:42:35 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Actual check successful.
Oct 03 09:42:35 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] url_helper.py[DEBUG]: [0/1] open 'http://metadata.google.internal/computeMetadata/v1/instance/id' with {'url': 'http://metadata.google.internal/computeMetadata
Oct 03 09:42:35 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] url_helper.py[DEBUG]: Read from http://metadata.google.internal/computeMetadata/v1/instance/id (200, 19b) after 1 attempts
So it definitely looks like DNS resolution failure taking 25 seconds is A Thing (TM). It's also taking 25 seconds on both EC2 and GCE, which suggests the issue is somewhere in Ubuntu. It also is extremely unlikely to be that networking isn't up at all, because the successful check takes no time at all across several runs.
The above log and tests were on EC2.
On GCE, an instrumented xenial gives:
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] __init__.py[DEBUG]: Seeing if we can get any data from <class 'cloudinit. sources. DataSourceGCE. DataSourceGCE' > exist.example. com.... GfGdwRj6To1L4b4 4q... py[DEBUG] : [0/1] open 'http:// metadata. google. internal/ computeMetadata /v1/instance/ id' with {'method': 'GET', 'url': 'http:// metadata. google. internal py[DEBUG] : Read from http:// metadata. google. internal/ computeMetadata /v1/instance/ id (200, 19b) after 1 attempts
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Checking DNS redirection...
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Trying does-not-
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Not redirected
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Trying example.invalid....
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Not redirected
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Trying KgWv29ZuRsiJoFA
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Not redirected
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: DNS redirection checking complete.
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Doing actual check...
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] util.py[DEBUG]: Actual check successful.
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] url_helper.
Oct 03 09:40:43 xenial-161003-0956 cloud-init[1318]: [CLOUDINIT] url_helper.
And an instrumented yakkety gives:
Oct 03 09:41:19 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] __init__.py[DEBUG]: Seeing if we can get any data from <class 'cloudinit. sources. DataSourceGCE. DataSourceGCE' > exist.example. com.... Lk3ksle1KNUDrSz 5p... py[DEBUG] : [0/1] open 'http:// metadata. google. internal/ computeMetadata /v1/instance/ id' with {'url': 'http:// metadata. google. internal/ computeMetadata py[DEBUG] : Read from http:// metadata. google. internal/ computeMetadata /v1/instance/ id (200, 19b) after 1 attempts
Oct 03 09:41:19 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Checking DNS redirection...
Oct 03 09:41:19 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Trying does-not-
Oct 03 09:41:44 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Not redirected
Oct 03 09:41:44 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Trying example.invalid....
Oct 03 09:42:10 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Not redirected
Oct 03 09:42:10 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Trying DLMcOj0Uix9neTV
Oct 03 09:42:35 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Not redirected
Oct 03 09:42:35 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: DNS redirection checking complete.
Oct 03 09:42:35 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Doing actual check...
Oct 03 09:42:35 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] util.py[DEBUG]: Actual check successful.
Oct 03 09:42:35 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] url_helper.
Oct 03 09:42:35 yakkety-161003-0945 cloud-init[1274]: [CLOUDINIT] url_helper.
So it definitely looks like DNS resolution failure taking 25 seconds is A Thing (TM). It's also taking 25 seconds on both EC2 and GCE, which suggests the issue is somewhere in Ubuntu. It also is extremely unlikely to be that networking isn't up at all, because the successful check takes no time at all across several runs.