| 2017-11-30 04:27:44 |
Chad Smith |
bug |
|
|
added bug |
| 2017-11-30 04:29:10 |
Chad Smith |
attachment added |
|
cloud-init.log https://bugs.launchpad.net/cloud-init/+bug/1735331/+attachment/5016454/+files/cloud-init.log |
|
| 2017-11-30 04:46:27 |
Chad Smith |
description |
Saw an issue once on EC2 zesty image with 17.1.41 during SRU testing.
A traceback when trying to read the temporary pid file that was created by our dhclient run during Ec2Local setup. Maybe we exited out of the dhcp run before we could read the pid file?
...
2017-11-30 04:20:32,738 - util.py[DEBUG]: Running command ['ip', 'link', 'set', 'dev', 'eth0', 'up'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,744 - util.py[DEBUG]: Running command ['/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient', '-1', '-v', '-lf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhcp.leases', '-pf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid', 'eth0', '-sf', '/bin/true'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,768 - util.py[DEBUG]: Reading from /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid (quiet=False)
2017-11-30 04:20:32,768 - handlers.py[DEBUG]: finish: init-local/search-Ec2Local: FAIL: no local data found from DataSourceEc2Local
2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
2017-11-30 04:20:32,768 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 332, in find_source
if s.get_data():
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 378, in get_data
return super(DataSourceEc2Local, self).get_data()
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 100, in get_data
self.fallback_interface)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in maybe_perform_dhcp_discovery
return dhcp_discovery(dhclient_path, nic, tdir)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 124, in dhcp_discovery
pid = int(util.load_file(pid_file).strip())
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1257, in load_file
with open(fname, 'rb') as ifh:
FileNotFoundError: [Errno 2] No such file or directory: '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid' |
Saw an issue once on EC2 zesty image with 17.1.41 during SRU testing.
Looks like we hit an inability to create the pid file (from syslog)
#### syslog
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: Cloud-init v. 17.1 running 'init-local' at Thu, 30 Nov 2017 04:20:32 +0000. Up 7.16 seconds.
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: 2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Nov 30 04:20:35 ip-10-0-20-176 dhclient[669]: Can't create /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid: No such file or directory
#### end syslog
A traceback when trying to read the temporary pid file that was created by our dhclient run during Ec2Local setup. Maybe we exited out of the dhcp run before we could read the pid file?
...
2017-11-30 04:20:32,738 - util.py[DEBUG]: Running command ['ip', 'link', 'set', 'dev', 'eth0', 'up'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,744 - util.py[DEBUG]: Running command ['/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient', '-1', '-v', '-lf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhcp.leases', '-pf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid', 'eth0', '-sf', '/bin/true'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,768 - util.py[DEBUG]: Reading from /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid (quiet=False)
2017-11-30 04:20:32,768 - handlers.py[DEBUG]: finish: init-local/search-Ec2Local: FAIL: no local data found from DataSourceEc2Local
2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
2017-11-30 04:20:32,768 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 332, in find_source
if s.get_data():
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 378, in get_data
return super(DataSourceEc2Local, self).get_data()
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 100, in get_data
self.fallback_interface)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in maybe_perform_dhcp_discovery
return dhcp_discovery(dhclient_path, nic, tdir)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 124, in dhcp_discovery
pid = int(util.load_file(pid_file).strip())
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1257, in load_file
with open(fname, 'rb') as ifh:
FileNotFoundError: [Errno 2] No such file or directory: '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid' |
|
| 2017-11-30 05:12:59 |
Chad Smith |
summary |
ec2: temp dhclient.pid file issues |
ec2: zesty tempfile sandbox dhclient.pid file can't be created |
|
| 2017-11-30 06:20:22 |
Chad Smith |
cloud-init: status |
New |
Incomplete |
|
| 2017-11-30 06:20:24 |
Chad Smith |
cloud-init: status |
Incomplete |
Triaged |
|
| 2017-11-30 06:21:16 |
Chad Smith |
cloud-init: importance |
Undecided |
High |
|
| 2017-11-30 20:58:49 |
Chad Smith |
cloud-init: assignee |
|
Chad Smith (chad.smith) |
|
| 2017-11-30 20:58:51 |
Chad Smith |
cloud-init: status |
Triaged |
In Progress |
|
| 2017-11-30 21:06:14 |
Chad Smith |
merge proposal linked |
|
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/334543 |
|
| 2017-12-01 03:30:36 |
Scott Moser |
cloud-init: status |
In Progress |
Fix Committed |
|
| 2017-12-01 03:30:44 |
Scott Moser |
bug task added |
|
cloud-init (Ubuntu) |
|
| 2017-12-01 03:31:07 |
Scott Moser |
nominated for series |
|
Ubuntu Bionic |
|
| 2017-12-01 03:31:07 |
Scott Moser |
bug task added |
|
cloud-init (Ubuntu Bionic) |
|
| 2017-12-01 03:31:07 |
Scott Moser |
nominated for series |
|
Ubuntu Xenial |
|
| 2017-12-01 03:31:07 |
Scott Moser |
bug task added |
|
cloud-init (Ubuntu Xenial) |
|
| 2017-12-01 03:31:07 |
Scott Moser |
nominated for series |
|
Ubuntu Artful |
|
| 2017-12-01 03:31:07 |
Scott Moser |
bug task added |
|
cloud-init (Ubuntu Artful) |
|
| 2017-12-01 03:31:07 |
Scott Moser |
nominated for series |
|
Ubuntu Zesty |
|
| 2017-12-01 03:31:07 |
Scott Moser |
bug task added |
|
cloud-init (Ubuntu Zesty) |
|
| 2017-12-01 06:02:32 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/334565 |
|
| 2017-12-01 06:03:11 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/334566 |
|
| 2017-12-01 06:03:52 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/334567 |
|
| 2017-12-01 07:10:27 |
Launchpad Janitor |
cloud-init (Ubuntu Bionic): status |
New |
Fix Released |
|
| 2017-12-01 16:11:07 |
Scott Moser |
cloud-init (Ubuntu Xenial): status |
New |
Confirmed |
|
| 2017-12-01 16:11:10 |
Scott Moser |
cloud-init (Ubuntu Zesty): status |
New |
Confirmed |
|
| 2017-12-01 16:11:13 |
Scott Moser |
cloud-init (Ubuntu Artful): status |
New |
Confirmed |
|
| 2017-12-01 16:11:17 |
Scott Moser |
cloud-init (Ubuntu Xenial): importance |
Undecided |
Medium |
|
| 2017-12-01 16:11:19 |
Scott Moser |
cloud-init (Ubuntu Zesty): importance |
Undecided |
Medium |
|
| 2017-12-01 16:11:22 |
Scott Moser |
cloud-init (Ubuntu Artful): importance |
Undecided |
Medium |
|
| 2017-12-01 16:11:24 |
Scott Moser |
cloud-init (Ubuntu Bionic): importance |
Undecided |
Medium |
|
| 2017-12-01 16:55:20 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/334593 |
|
| 2017-12-01 17:01:04 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/334594 |
|
| 2017-12-01 17:03:56 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/334595 |
|
| 2017-12-01 17:05:50 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/334597 |
|
| 2017-12-01 22:30:20 |
Steve Langasek |
cloud-init (Ubuntu Artful): status |
Confirmed |
Fix Committed |
|
| 2017-12-01 22:30:24 |
Steve Langasek |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2017-12-01 22:30:28 |
Steve Langasek |
bug |
|
|
added subscriber SRU Verification |
| 2017-12-01 22:30:31 |
Steve Langasek |
tags |
|
verification-needed verification-needed-artful |
|
| 2017-12-01 22:32:16 |
Steve Langasek |
cloud-init (Ubuntu Zesty): status |
Confirmed |
Fix Committed |
|
| 2017-12-01 22:32:23 |
Steve Langasek |
tags |
verification-needed verification-needed-artful |
verification-needed verification-needed-artful verification-needed-zesty |
|
| 2017-12-04 22:53:27 |
Chad Smith |
attachment added |
|
ec2-sru-validate-17.1.46.log https://bugs.launchpad.net/cloud-init/+bug/1735331/+attachment/5018509/+files/ec2-sru-validate-17.1.46.log |
|
| 2017-12-04 22:55:27 |
Chad Smith |
description |
Saw an issue once on EC2 zesty image with 17.1.41 during SRU testing.
Looks like we hit an inability to create the pid file (from syslog)
#### syslog
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: Cloud-init v. 17.1 running 'init-local' at Thu, 30 Nov 2017 04:20:32 +0000. Up 7.16 seconds.
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: 2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Nov 30 04:20:35 ip-10-0-20-176 dhclient[669]: Can't create /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid: No such file or directory
#### end syslog
A traceback when trying to read the temporary pid file that was created by our dhclient run during Ec2Local setup. Maybe we exited out of the dhcp run before we could read the pid file?
...
2017-11-30 04:20:32,738 - util.py[DEBUG]: Running command ['ip', 'link', 'set', 'dev', 'eth0', 'up'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,744 - util.py[DEBUG]: Running command ['/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient', '-1', '-v', '-lf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhcp.leases', '-pf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid', 'eth0', '-sf', '/bin/true'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,768 - util.py[DEBUG]: Reading from /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid (quiet=False)
2017-11-30 04:20:32,768 - handlers.py[DEBUG]: finish: init-local/search-Ec2Local: FAIL: no local data found from DataSourceEc2Local
2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
2017-11-30 04:20:32,768 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 332, in find_source
if s.get_data():
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 378, in get_data
return super(DataSourceEc2Local, self).get_data()
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 100, in get_data
self.fallback_interface)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in maybe_perform_dhcp_discovery
return dhcp_discovery(dhclient_path, nic, tdir)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 124, in dhcp_discovery
pid = int(util.load_file(pid_file).strip())
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1257, in load_file
with open(fname, 'rb') as ifh:
FileNotFoundError: [Errno 2] No such file or directory: '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid' |
=== Begin SRU Template ===
[Impact]
Ec2 instances could hit race condition with tempdir removal where dhclient doesn't write a pidfile and DataSourceEc2Local hits a traceback trying to read that non-existent pidfile. This traceback causes the instance to fallback and get discovered in init-network stage as DataSourceEc2. The thrashing costs instances a couple extra seconds to boot while re-discovering in a different stage.
[Test Case]
# Launch instance under test
$ for release in xenial zesty artful; do
echo "Handling $release";
launch-ec2 --series $release;
ssh ubuntu@<ec2-address> cat /run/cloud-init/result.json;
ssh ubuntu@<ec2-address> grep Trace /var/log/cloud-init.log;
ssh ubuntu@<ec2-address> sudo sed 's/ $release / $release-proposed /' /etc/apt/sources.list;
ssh ubuntu@<ec2-address> sudo apt update;
ssh ubuntu@<ec2-address> sudo apt install cloud-init;
# Show upgrade without restart doesn't break
ssh ubuntu@<ec2-address> sudo cloud-init init;
# Show clean install doesn't break
ssh ubuntu@<ec2-address> 'sudo rm -rf /var/log/cloud-init* /var/lib/cloud; sudo reboot'
ssh ubuntu@<ec2-address> 'sudo cat /run/cloud-init/result.json
ssh ubuntu@<ec2-address> 'sudo grep Trace /var/log/cloud-init*';
# Asssert no intermittent tracebacks from dhcp_discovery and no leaked dhcpclients;
ssh ubuntu@<ec2-address> "sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()";
sudo ps -afe |grep dhclient;
done
[Regression Potential]
Regression would still result in Tracebacks in DataSourceEc2Local which would cause cloud-init to fallback to DataSourceEc2 in init-network stage.
[Other Info]
Upstream commit at
https://git.launchpad.net/cloud-init/commit/?id=7acc9e68
=== End SRU Template ===
=== Original Description ===
Saw an issue once on EC2 zesty image with 17.1.41 during SRU testing.
Looks like we hit an inability to create the pid file (from syslog)
#### syslog
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: Cloud-init v. 17.1 running 'init-local' at Thu, 30 Nov 2017 04:20:32 +0000. Up 7.16 seconds.
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: 2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Nov 30 04:20:35 ip-10-0-20-176 dhclient[669]: Can't create /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid: No such file or directory
#### end syslog
A traceback when trying to read the temporary pid file that was created by our dhclient run during Ec2Local setup. Maybe we exited out of the dhcp run before we could read the pid file?
...
2017-11-30 04:20:32,738 - util.py[DEBUG]: Running command ['ip', 'link', 'set', 'dev', 'eth0', 'up'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,744 - util.py[DEBUG]: Running command ['/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient', '-1', '-v', '-lf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhcp.leases', '-pf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid', 'eth0', '-sf', '/bin/true'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,768 - util.py[DEBUG]: Reading from /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid (quiet=False)
2017-11-30 04:20:32,768 - handlers.py[DEBUG]: finish: init-local/search-Ec2Local: FAIL: no local data found from DataSourceEc2Local
2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
2017-11-30 04:20:32,768 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 332, in find_source
if s.get_data():
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 378, in get_data
return super(DataSourceEc2Local, self).get_data()
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 100, in get_data
self.fallback_interface)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in maybe_perform_dhcp_discovery
return dhcp_discovery(dhclient_path, nic, tdir)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 124, in dhcp_discovery
pid = int(util.load_file(pid_file).strip())
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1257, in load_file
with open(fname, 'rb') as ifh:
FileNotFoundError: [Errno 2] No such file or directory: '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid' |
|
| 2017-12-04 22:56:28 |
Chad Smith |
description |
=== Begin SRU Template ===
[Impact]
Ec2 instances could hit race condition with tempdir removal where dhclient doesn't write a pidfile and DataSourceEc2Local hits a traceback trying to read that non-existent pidfile. This traceback causes the instance to fallback and get discovered in init-network stage as DataSourceEc2. The thrashing costs instances a couple extra seconds to boot while re-discovering in a different stage.
[Test Case]
# Launch instance under test
$ for release in xenial zesty artful; do
echo "Handling $release";
launch-ec2 --series $release;
ssh ubuntu@<ec2-address> cat /run/cloud-init/result.json;
ssh ubuntu@<ec2-address> grep Trace /var/log/cloud-init.log;
ssh ubuntu@<ec2-address> sudo sed 's/ $release / $release-proposed /' /etc/apt/sources.list;
ssh ubuntu@<ec2-address> sudo apt update;
ssh ubuntu@<ec2-address> sudo apt install cloud-init;
# Show upgrade without restart doesn't break
ssh ubuntu@<ec2-address> sudo cloud-init init;
# Show clean install doesn't break
ssh ubuntu@<ec2-address> 'sudo rm -rf /var/log/cloud-init* /var/lib/cloud; sudo reboot'
ssh ubuntu@<ec2-address> 'sudo cat /run/cloud-init/result.json
ssh ubuntu@<ec2-address> 'sudo grep Trace /var/log/cloud-init*';
# Asssert no intermittent tracebacks from dhcp_discovery and no leaked dhcpclients;
ssh ubuntu@<ec2-address> "sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()";
sudo ps -afe |grep dhclient;
done
[Regression Potential]
Regression would still result in Tracebacks in DataSourceEc2Local which would cause cloud-init to fallback to DataSourceEc2 in init-network stage.
[Other Info]
Upstream commit at
https://git.launchpad.net/cloud-init/commit/?id=7acc9e68
=== End SRU Template ===
=== Original Description ===
Saw an issue once on EC2 zesty image with 17.1.41 during SRU testing.
Looks like we hit an inability to create the pid file (from syslog)
#### syslog
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: Cloud-init v. 17.1 running 'init-local' at Thu, 30 Nov 2017 04:20:32 +0000. Up 7.16 seconds.
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: 2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Nov 30 04:20:35 ip-10-0-20-176 dhclient[669]: Can't create /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid: No such file or directory
#### end syslog
A traceback when trying to read the temporary pid file that was created by our dhclient run during Ec2Local setup. Maybe we exited out of the dhcp run before we could read the pid file?
...
2017-11-30 04:20:32,738 - util.py[DEBUG]: Running command ['ip', 'link', 'set', 'dev', 'eth0', 'up'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,744 - util.py[DEBUG]: Running command ['/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient', '-1', '-v', '-lf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhcp.leases', '-pf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid', 'eth0', '-sf', '/bin/true'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,768 - util.py[DEBUG]: Reading from /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid (quiet=False)
2017-11-30 04:20:32,768 - handlers.py[DEBUG]: finish: init-local/search-Ec2Local: FAIL: no local data found from DataSourceEc2Local
2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
2017-11-30 04:20:32,768 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 332, in find_source
if s.get_data():
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 378, in get_data
return super(DataSourceEc2Local, self).get_data()
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 100, in get_data
self.fallback_interface)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in maybe_perform_dhcp_discovery
return dhcp_discovery(dhclient_path, nic, tdir)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 124, in dhcp_discovery
pid = int(util.load_file(pid_file).strip())
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1257, in load_file
with open(fname, 'rb') as ifh:
FileNotFoundError: [Errno 2] No such file or directory: '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid' |
=== Begin SRU Template ===
[Impact]
Ec2 instances could hit race condition with tempdir removal where dhclient doesn't write a pidfile and DataSourceEc2Local hits a traceback trying to read that non-existent pidfile. This traceback causes the instance to fallback and get discovered in init-network stage as DataSourceEc2. The thrashing costs instances a couple extra seconds to boot while re-discovering in a different stage.
[Test Case]
# Launch instance under test
$ for release in xenial zesty artful; do
echo "Handling $release";
launch-ec2 --series $release;
ssh ubuntu@<ec2-address> cat /run/cloud-init/result.json;
ssh ubuntu@<ec2-address> grep Trace /var/log/cloud-init.log;
ssh ubuntu@<ec2-address> sudo sed 's/ $release / $release-proposed /' /etc/apt/sources.list;
ssh ubuntu@<ec2-address> sudo apt update;
ssh ubuntu@<ec2-address> sudo apt install cloud-init;
# Show upgrade without restart doesn't break
ssh ubuntu@<ec2-address> sudo cloud-init init;
# Show clean install doesn't break
ssh ubuntu@<ec2-address> 'sudo rm -rf /var/log/cloud-init* /var/lib/cloud; sudo reboot'
ssh ubuntu@<ec2-address> 'sudo cat /run/cloud-init/result.json
ssh ubuntu@<ec2-address> 'sudo grep Trace /var/log/cloud-init*';
# Asssert no intermittent tracebacks from dhcp_discovery and no leaked dhcpclients;
ssh ubuntu@<ec2-address> "sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()";
sudo ps -afe |grep dhclient;
done
[Regression Potential]
Regression would still result in Tracebacks in DataSourceEc2Local which would cause cloud-init to fallback to DataSourceEc2 in init-network stage.
[Other Info]
Upstream commit at
https://git.launchpad.net/cloud-init/commit/?id=16ad7eac
=== End SRU Template ===
=== Original Description ===
Saw an issue once on EC2 zesty image with 17.1.41 during SRU testing.
Looks like we hit an inability to create the pid file (from syslog)
#### syslog
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: Cloud-init v. 17.1 running 'init-local' at Thu, 30 Nov 2017 04:20:32 +0000. Up 7.16 seconds.
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: 2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Nov 30 04:20:35 ip-10-0-20-176 dhclient[669]: Can't create /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid: No such file or directory
#### end syslog
A traceback when trying to read the temporary pid file that was created by our dhclient run during Ec2Local setup. Maybe we exited out of the dhcp run before we could read the pid file?
...
2017-11-30 04:20:32,738 - util.py[DEBUG]: Running command ['ip', 'link', 'set', 'dev', 'eth0', 'up'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,744 - util.py[DEBUG]: Running command ['/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient', '-1', '-v', '-lf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhcp.leases', '-pf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid', 'eth0', '-sf', '/bin/true'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,768 - util.py[DEBUG]: Reading from /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid (quiet=False)
2017-11-30 04:20:32,768 - handlers.py[DEBUG]: finish: init-local/search-Ec2Local: FAIL: no local data found from DataSourceEc2Local
2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
2017-11-30 04:20:32,768 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 332, in find_source
if s.get_data():
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 378, in get_data
return super(DataSourceEc2Local, self).get_data()
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 100, in get_data
self.fallback_interface)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in maybe_perform_dhcp_discovery
return dhcp_discovery(dhclient_path, nic, tdir)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 124, in dhcp_discovery
pid = int(util.load_file(pid_file).strip())
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1257, in load_file
with open(fname, 'rb') as ifh:
FileNotFoundError: [Errno 2] No such file or directory: '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid' |
|
| 2017-12-04 22:57:41 |
Chad Smith |
description |
=== Begin SRU Template ===
[Impact]
Ec2 instances could hit race condition with tempdir removal where dhclient doesn't write a pidfile and DataSourceEc2Local hits a traceback trying to read that non-existent pidfile. This traceback causes the instance to fallback and get discovered in init-network stage as DataSourceEc2. The thrashing costs instances a couple extra seconds to boot while re-discovering in a different stage.
[Test Case]
# Launch instance under test
$ for release in xenial zesty artful; do
echo "Handling $release";
launch-ec2 --series $release;
ssh ubuntu@<ec2-address> cat /run/cloud-init/result.json;
ssh ubuntu@<ec2-address> grep Trace /var/log/cloud-init.log;
ssh ubuntu@<ec2-address> sudo sed 's/ $release / $release-proposed /' /etc/apt/sources.list;
ssh ubuntu@<ec2-address> sudo apt update;
ssh ubuntu@<ec2-address> sudo apt install cloud-init;
# Show upgrade without restart doesn't break
ssh ubuntu@<ec2-address> sudo cloud-init init;
# Show clean install doesn't break
ssh ubuntu@<ec2-address> 'sudo rm -rf /var/log/cloud-init* /var/lib/cloud; sudo reboot'
ssh ubuntu@<ec2-address> 'sudo cat /run/cloud-init/result.json
ssh ubuntu@<ec2-address> 'sudo grep Trace /var/log/cloud-init*';
# Asssert no intermittent tracebacks from dhcp_discovery and no leaked dhcpclients;
ssh ubuntu@<ec2-address> "sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()";
sudo ps -afe |grep dhclient;
done
[Regression Potential]
Regression would still result in Tracebacks in DataSourceEc2Local which would cause cloud-init to fallback to DataSourceEc2 in init-network stage.
[Other Info]
Upstream commit at
https://git.launchpad.net/cloud-init/commit/?id=16ad7eac
=== End SRU Template ===
=== Original Description ===
Saw an issue once on EC2 zesty image with 17.1.41 during SRU testing.
Looks like we hit an inability to create the pid file (from syslog)
#### syslog
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: Cloud-init v. 17.1 running 'init-local' at Thu, 30 Nov 2017 04:20:32 +0000. Up 7.16 seconds.
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: 2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Nov 30 04:20:35 ip-10-0-20-176 dhclient[669]: Can't create /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid: No such file or directory
#### end syslog
A traceback when trying to read the temporary pid file that was created by our dhclient run during Ec2Local setup. Maybe we exited out of the dhcp run before we could read the pid file?
...
2017-11-30 04:20:32,738 - util.py[DEBUG]: Running command ['ip', 'link', 'set', 'dev', 'eth0', 'up'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,744 - util.py[DEBUG]: Running command ['/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient', '-1', '-v', '-lf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhcp.leases', '-pf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid', 'eth0', '-sf', '/bin/true'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,768 - util.py[DEBUG]: Reading from /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid (quiet=False)
2017-11-30 04:20:32,768 - handlers.py[DEBUG]: finish: init-local/search-Ec2Local: FAIL: no local data found from DataSourceEc2Local
2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
2017-11-30 04:20:32,768 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 332, in find_source
if s.get_data():
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 378, in get_data
return super(DataSourceEc2Local, self).get_data()
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 100, in get_data
self.fallback_interface)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in maybe_perform_dhcp_discovery
return dhcp_discovery(dhclient_path, nic, tdir)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 124, in dhcp_discovery
pid = int(util.load_file(pid_file).strip())
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1257, in load_file
with open(fname, 'rb') as ifh:
FileNotFoundError: [Errno 2] No such file or directory: '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid' |
=== Begin SRU Template ===
[Impact]
Ec2 instances could hit race condition with tempdir removal where dhclient doesn't write a pidfile and DataSourceEc2Local hits a traceback trying to read that non-existent pidfile. This traceback causes the instance to fallback and get discovered in init-network stage as DataSourceEc2. The thrashing costs instances a couple extra seconds to boot while re-discovering in a different stage.
[Test Case]
# Launch instance under test
$ for release in xenial zesty artful; do
echo "Handling $release";
launch-ec2 --series $release;
ssh ubuntu@<ec2-address> cat /run/cloud-init/result.json;
ssh ubuntu@<ec2-address> grep Trace /var/log/cloud-init.log;
ssh ubuntu@<ec2-address> sudo sed 's/ $release / $release-proposed /' /etc/apt/sources.list;
ssh ubuntu@<ec2-address> sudo apt update;
ssh ubuntu@<ec2-address> sudo apt install cloud-init;
# Show upgrade without restart doesn't break
ssh ubuntu@<ec2-address> sudo cloud-init init;
# Show clean install doesn't break
ssh ubuntu@<ec2-address> 'sudo rm -rf /var/log/cloud-init* /var/lib/cloud; sudo reboot'
ssh ubuntu@<ec2-address> 'sudo cat /run/cloud-init/result.json
ssh ubuntu@<ec2-address> 'sudo grep Trace /var/log/cloud-init*';
# Asssert no intermittent tracebacks from dhcp_discovery and no leaked dhcpclients;
ssh ubuntu@<ec2-address> "sudo python3 -c 'from cloudinit.net.dhcp import maybe_perform_dhcp_discovery; maybe_perform_dhcp_discovery()";
sudo ps -afe |grep dhclient;
done
[Regression Potential]
Regression would still result in Tracebacks in DataSourceEc2Local which would cause cloud-init to fallback to DataSourceEc2 in init-network stage.
[Other Info]
Upstream commit at
https://git.launchpad.net/cloud-init/commit/?id=7acc9e68f
=== End SRU Template ===
=== Original Description ===
Saw an issue once on EC2 zesty image with 17.1.41 during SRU testing.
Looks like we hit an inability to create the pid file (from syslog)
#### syslog
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: Cloud-init v. 17.1 running 'init-local' at Thu, 30 Nov 2017 04:20:32 +0000. Up 7.16 seconds.
Nov 30 04:20:35 ip-10-0-20-176 cloud-init[440]: 2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Nov 30 04:20:35 ip-10-0-20-176 dhclient[669]: Can't create /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid: No such file or directory
#### end syslog
A traceback when trying to read the temporary pid file that was created by our dhclient run during Ec2Local setup. Maybe we exited out of the dhcp run before we could read the pid file?
...
2017-11-30 04:20:32,738 - util.py[DEBUG]: Running command ['ip', 'link', 'set', 'dev', 'eth0', 'up'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,744 - util.py[DEBUG]: Running command ['/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient', '-1', '-v', '-lf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhcp.leases', '-pf', '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid', 'eth0', '-sf', '/bin/true'] with allowed return codes [0] (shell=False, capture=True)
2017-11-30 04:20:32,768 - util.py[DEBUG]: Reading from /var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid (quiet=False)
2017-11-30 04:20:32,768 - handlers.py[DEBUG]: finish: init-local/search-Ec2Local: FAIL: no local data found from DataSourceEc2Local
2017-11-30 04:20:32,768 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
2017-11-30 04:20:32,768 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 332, in find_source
if s.get_data():
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 378, in get_data
return super(DataSourceEc2Local, self).get_data()
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py", line 100, in get_data
self.fallback_interface)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in maybe_perform_dhcp_discovery
return dhcp_discovery(dhclient_path, nic, tdir)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 124, in dhcp_discovery
pid = int(util.load_file(pid_file).strip())
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1257, in load_file
with open(fname, 'rb') as ifh:
FileNotFoundError: [Errno 2] No such file or directory: '/var/tmp/cloud-init/cloud-init-dhcp-hnatdvwi/dhclient.pid' |
|
| 2017-12-04 22:58:27 |
Chad Smith |
tags |
verification-needed verification-needed-artful verification-needed-zesty |
verification-done verification-done-artful verification-done-xenial verification-done-zesty |
|
| 2017-12-05 03:23:10 |
Chad Smith |
cloud-init (Ubuntu Xenial): status |
Confirmed |
Fix Committed |
|
| 2017-12-13 01:23:40 |
Launchpad Janitor |
cloud-init (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2017-12-13 01:24:00 |
Chris Halse Rogers |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2017-12-13 01:24:17 |
Launchpad Janitor |
cloud-init (Ubuntu Zesty): status |
Fix Committed |
Fix Released |
|
| 2017-12-13 01:24:45 |
Launchpad Janitor |
cloud-init (Ubuntu Artful): status |
Fix Committed |
Fix Released |
|
| 2017-12-14 21:03:11 |
Scott Moser |
cloud-init: status |
Fix Committed |
Fix Released |
|
| 2018-02-08 17:01:33 |
Stretch |
bug |
|
|
added subscriber Stretch |
| 2023-05-11 08:56:49 |
James Falcon |
bug watch added |
|
https://github.com/canonical/cloud-init/issues/3073 |
|
