No network in AWS (EC-Classic) after stopping and starting instance
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
cloud-init (Ubuntu) |
Fix Released
|
High
|
Guilherme G. Piccoli | ||
Xenial |
Fix Released
|
Low
|
Unassigned | ||
Bionic |
Fix Released
|
High
|
Unassigned | ||
Cosmic |
Fix Released
|
High
|
Unassigned |
Bug Description
I don't know is this cloud-init or netplan or what, but this is not good.
Background:
# lsb_release -rd
Description: Ubuntu 18.04.1 LTS
Release: 18.04
# apt-cache policy cloud-init
cloud-init:
Installed: 18.4-0ubuntu1~
Candidate: 18.4-0ubuntu1~
Version table:
*** 18.4-0ubuntu1~
500 http://
100 /var/lib/
18.
500 http://
1. Get newest image to use
$ aws --region eu-west-1 ec2 describe-images --owners 099720109477 --filters Name=root-
"ami-08596fdd2d
2. Start instance to EC2-Classic with that image.
3. Try to SSH. Everything is ok.
# cat /var/log/
Cloud-init v. 18.4-0ubuntu1~
Cloud-init v. 18.4-0ubuntu1~
ci-info: +++++++
ci-info: +------
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +------
ci-info: | eth0 | True | 10.74.200.25 | 255.255.255.192 | global | 22:00:0a:4a:c8:19 |
ci-info: | eth0 | True | fe80::2000:
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +------
...
Cloud-init v. 18.4-0ubuntu1~
Cloud-init v. 18.4-0ubuntu1~
Cloud-init v. 18.4-0ubuntu1~
4. Stop the instance.
5. Start the instance.
6. Try to SSH.
Expected to happen: Instance has network and is working.
What happens: Instance has no working network
Getting instance log we can see:
[ 11.342357] cloud-init[412]: Cloud-init v. 18.4-0ubuntu1~
[ OK ] Started Initial cloud-init job (pre-networking).
[ OK ] Reached target Network (Pre).
Starting Network Service...
[ OK ] Started Network Service.
Starting Network Name Resolution...
Starting Wait for Network to be Configured...
[ OK ] Started Wait for Network to be Configured.
Starting Initial cloud-init job (metadata service crawler)...
[ OK ] Started Network Name Resolution.
[ OK ] Reached target Host and Network Name Lookups.
[ OK ] Reached target Network.
[ 13.036207] cloud-init[637]: Cloud-init v. 18.4-0ubuntu1~
[ 13.052849] cloud-init[637]: ci-info: +++++++
[ 13.100325] cloud-init[637]: ci-info: +------
[ 13.121790] cloud-init[637]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 13.129189] cloud-init[637]: ci-info: +------
[ 13.144839] cloud-init[637]: ci-info: | eth0 | False | . | . | . | 22:00:0b:0a:cb:2d |[ OK ] Started Initial cloud-init job (metadata service crawler).
[ 13.158694] cloud-init
[ OK ] Reached target System Initialization.
[ OK ] Started Daily apt download activities.
[ 13.179053] ] Started Message of the Day.
cloud-init[637]: ci-info: | lo | True | ::1/128 | . | host | . |[ OK ] Started ACPI Events Check.
[ OK ] Reached target Paths.
[ 13.201012] cloud-init[637]: ci-info: +------
[ OK ] Listening on Open-iSCSI iscsid Socket.
[ 13.213993] cloud-init[ OK ] Listening on D-Bus System Message Bus Socket.[637]: ci-info: +++++++
Starting Socket activation for snappy daemon.
[ 13.229707] cloud-init[637]: ci-info: +------
[ OK ] Started Daily apt upgrade and clean activities.
[ 13.244949] cloud-init[637]: ci-info: | Route | Destination | Gateway | Interface | Flags |] Listening on UUID daemon activation socket.
Starting LXD - unix socket.
[ OK ] Started Daily Cleanup of Temporary Directories.[ 13.256281] cloud-init[637]: ci-info: +------
[ OK ] Started Discard unused blocks once a week.
[ OK ] Reached target Timers.
[ OK ] Reached target Cloud-config availability.
[ 13.286424] cloud-init[637]:
[ OK ] Reached target Network is Online.ci-info: +------
It would be nice that the instances would work also after stop&start as they used to.
My speculation for the problem is that /etc/netplan/
match:
Which changes in the stop&start and it is handled in wrong order. File is not generated before trying to get the network up and there is no device for that macaddress. But without console access and internal knowledge how this netplan/
pinpoint the problematic thing.
But I did a small test. Edited /usr/lib/
if if_type == 'physical':
# required_keys = ['name', 'mac_address']
eth = {
}
if eth['match'] is None:
+ del eth['match']
+ del eth['set-name']
And then run:
cloud-init clean
cloud-init init
# cat /etc/netplan/
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/
# network: {config: disabled}
network:
version: 2
ethernets:
eth0:
dhcp4: true
And then stopped the instance and started it.. It gets the network and works. And stopped again just to be sure it wasn't one time magic. Started and it works.
So the problem really seems that the match/macaddress but how one should properly fix that, I'll leave for people who have made it misbehave like this.
But I think there might be some pretty scared and annoyed people after stopping the instance and starting it, the instance is unreachable. Also depending on their skills to troubleshoot the problem and mount the volume to another instance and fix it (if it's ebs backed, if not, sorry, make a new instance).
Related branches
- Server Team CI bot: Approve (continuous-integration)
- Chad Smith: Approve
- Ryan Harper: Pending requested
-
Diff: 101 lines (+56/-0)3 files modifiedcloudinit/sources/DataSourceEc2.py (+21/-0)
doc/rtd/topics/datasources/ec2.rst (+11/-0)
tests/unittests/test_datasource/test_ec2.py (+24/-0)
- Chad Smith: Needs Fixing
- Ryan Harper: Needs Fixing
-
Diff: 101 lines (+56/-0)3 files modifiedcloudinit/sources/DataSourceEc2.py (+21/-0)
doc/rtd/topics/datasources/ec2.rst (+11/-0)
tests/unittests/test_datasource/test_ec2.py (+24/-0)
summary: |
- No network in AWS (EC-Classic) after stopping and starging instance + No network in AWS (EC-Classic) after stopping and starting instance |
Changed in cloud-init (Ubuntu): | |
status: | Confirmed → Fix Committed |
And little bit more testing. It really seems to be that cloud-init does not generate the file again if not run clout-init clean before. So it doesn't generate the file again after stop&start and macaddress differs and no network, and no network.
Let's test it by editing the file by hand:
$ cat /etc/netplan/ 50-cloud- init.yaml cloud.cfg. d/99-disable- network- config. cfg with the following:
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/
# network: {config: disabled}
# EDITED
network:
version: 2
ethernets:
eth0:
dhcp4: true
$ sudo cloud-init init
$ cat /etc/netplan/ 50-cloud- init.yaml cloud.cfg. d/99-disable- network- config. cfg with the following:
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/
# network: {config: disabled}
# EDITED
network:
version: 2
ethernets:
eth0:
dhcp4: true
$ sudo cloud-init clean
$ cat /etc/netplan/ 50-cloud- init.yaml cloud.cfg. d/99-disable- network- config. cfg with the following:
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/
# network: {config: disabled}
# EDITED
network:
version: 2
ethernets:
eth0:
dhcp4: true
$ sudo cloud-init init
$ cat /etc/netplan/ 50-cloud- init.yaml cloud.cfg. d/99-disable- network- config. cfg with the following:
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/
# network: {config: disabled}
network:
version: 2
ethernets:
eth0:
dhcp4: true
So clean and init is needed for the file to be generated again, and probably normal boot does just the init and yes, there is wrong macaddress, so no network for you.