[2.0] TFTP issues when rackd — Missing connections to 1 region controller(s).
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Fix Released
|
Critical
|
Blake Rouse |
Bug Description
We started seeing our EFI systems failing to deploy. It looks to be an issue with the TFTP exchange with the maas server. the errors started happening around 1800 UTC on 9/21 and appears to impact EFI systems only.
Per juju status, this was the error:
machines:
"0":
juju-status:
current: pending
since: 22 Sep 2016 13:03:55Z
dns-name: 10.245.0.153
instance-id: 4y3xat
machine-status:
message: 'Failed deployment: bootx64.efi'
since: 22 Sep 2016 13:45:27Z
series: trusty
containers:
0/lxd/0:
current: pending
since: 22 Sep 2016 13:04:11Z
current: pending
since: 22 Sep 2016 13:04:11Z
series: trusty
0/lxd/1:
current: pending
since: 22 Sep 2016 13:04:13Z
current: pending
since: 22 Sep 2016 13:04:13Z
series: trusty
0/lxd/2:
current: pending
since: 22 Sep 2016 13:04:15Z
current: pending
since: 22 Sep 2016 13:04:15Z
series: trusty
In the rackd.log, only seeing:
2016-09-22 16:36:44 [TFTP (UDP)] Datagram received from ('10.245.0.226', 1705): <RRQDatagram(
and monitoring the console of the booting server:
..PXE-E23: Client received TFTP error from server.
On the rackd page, it shows:
Region Controller
regiond
bind9
proxy
Rack Controller
rackd — Missing connections to 1 region controller(s).
http
tftp
dhcpd
dhcpd6
tgt
ubuntu@
Desired=
| Status=
|/ Err?=(none)
||/ Name Version Architecture Description
+++-===
ii maas 2.0.0+bzr5189-
ii maas-cli 2.0.0+bzr5189-
un maas-cluster-
ii maas-common 2.0.0+bzr5189-
ii maas-dhcp 2.0.0+bzr5189-
ii maas-dns 2.0.0+bzr5189-
ii maas-proxy 2.0.0+bzr5189-
ii maas-rack-
ii maas-region-api 2.0.0+bzr5189-
ii maas-region-
un maas-region-
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-
ii python3-django-maas 2.0.0+bzr5189-
ii python3-maas-client 2.0.0+bzr5189-
ii python3-
Related branches
- Mike Pontillo (community): Approve
-
Diff: 441 lines (+134/-46)15 files modifiedsrc/provisioningserver/rackdservices/dhcp_probe_service.py (+1/-1)
src/provisioningserver/rackdservices/image_download_service.py (+1/-1)
src/provisioningserver/rackdservices/lease_socket_service.py (+1/-1)
src/provisioningserver/rackdservices/networks_monitoring_service.py (+28/-18)
src/provisioningserver/rackdservices/ntp.py (+1/-1)
src/provisioningserver/rackdservices/service_monitor_service.py (+1/-1)
src/provisioningserver/rackdservices/tests/test_dhcp_probe_service.py (+8/-7)
src/provisioningserver/rackdservices/tests/test_image_download_service.py (+2/-2)
src/provisioningserver/rackdservices/tests/test_lease_socket_service.py (+1/-1)
src/provisioningserver/rackdservices/tests/test_ntp.py (+1/-1)
src/provisioningserver/rackdservices/tests/test_service_monitor_service.py (+1/-1)
src/provisioningserver/rackdservices/tests/test_tftp.py (+8/-6)
src/provisioningserver/rackdservices/tftp.py (+10/-5)
src/provisioningserver/rpc/clusterservice.py (+21/-0)
src/provisioningserver/rpc/tests/test_clusterservice.py (+49/-0)
- Blake Rouse (community): Approve
-
Diff: 441 lines (+134/-46)15 files modifiedsrc/provisioningserver/rackdservices/dhcp_probe_service.py (+1/-1)
src/provisioningserver/rackdservices/image_download_service.py (+1/-1)
src/provisioningserver/rackdservices/lease_socket_service.py (+1/-1)
src/provisioningserver/rackdservices/networks_monitoring_service.py (+28/-18)
src/provisioningserver/rackdservices/ntp.py (+1/-1)
src/provisioningserver/rackdservices/service_monitor_service.py (+1/-1)
src/provisioningserver/rackdservices/tests/test_dhcp_probe_service.py (+8/-7)
src/provisioningserver/rackdservices/tests/test_image_download_service.py (+2/-2)
src/provisioningserver/rackdservices/tests/test_lease_socket_service.py (+1/-1)
src/provisioningserver/rackdservices/tests/test_ntp.py (+1/-1)
src/provisioningserver/rackdservices/tests/test_service_monitor_service.py (+1/-1)
src/provisioningserver/rackdservices/tests/test_tftp.py (+8/-6)
src/provisioningserver/rackdservices/tftp.py (+10/-5)
src/provisioningserver/rpc/clusterservice.py (+21/-0)
src/provisioningserver/rpc/tests/test_clusterservice.py (+49/-0)
- MAAS Maintainers: Pending requested
-
Diff: 3861 lines (+2983/-3) (has conflicts)80 files modifieddocs/changelog.rst (+148/-0)
src/maasserver/api/spaces.py (+7/-0)
src/maasserver/api/subnets.py (+49/-0)
src/maasserver/forms/tests/test_interface_link.py (+6/-1)
src/maasserver/models/staticipaddress.py (+160/-0)
src/maasserver/models/tests/test_staticipaddress.py (+4/-0)
src/maasserver/static/assets/images/icons/account.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/add-logical-volume.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/add-partition.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/add.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/cross-orange.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/cross.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/cross_orange.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/debug.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/delete.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/edit.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/green-tick.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/help.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/info-mono.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/info.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/logical-volume.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/magnifying_glass.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/mount.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/partition.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/power-error.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/power-off.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/power-on.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/remove.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/settings.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/success-mono.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/success.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/success_colour_black.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/success_colour_white.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/success_mono.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/sync.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/system-shutdown.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/tags.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/tick.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/tooltip.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/unmount.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/warning-mono.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/warning.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/warning_colour_black.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/warning_colour_white.svg.OTHER (+1/-0)
src/maasserver/static/assets/images/icons/warning_mono.svg.OTHER (+1/-0)
src/maasserver/static/css/build.css (+1/-0)
src/maasserver/static/js/angular/maas.js (+5/-0)
src/maasserver/static/partials/dashboard.html (+207/-2)
src/maasserver/static/partials/domain-details.html (+4/-0)
src/maasserver/static/partials/domains-list.html (+4/-0)
src/maasserver/static/partials/error.html (+4/-0)
src/maasserver/static/partials/fabric-details.html (+8/-0)
src/maasserver/static/partials/intro-user.html (+33/-0)
src/maasserver/static/partials/intro.html (+71/-0)
src/maasserver/static/partials/networks-list.html (+4/-0)
src/maasserver/static/partials/node-details.html (+8/-0)
src/maasserver/static/partials/node-events.html (+4/-0)
src/maasserver/static/partials/node-result.html (+8/-0)
src/maasserver/static/partials/nodes-list.html (+4/-0)
src/maasserver/static/partials/settings.html (+4/-0)
src/maasserver/static/partials/space-details.html (+8/-0)
src/maasserver/static/partials/ssh-keys.html (+11/-0)
src/maasserver/static/partials/subnet-details.html (+12/-0)
src/maasserver/static/partials/vlan-details.html (+10/-0)
src/maasserver/static/scss/_base.forms.scss.OTHER (+377/-0)
src/maasserver/static/scss/_components.buttons.scss.OTHER (+149/-0)
src/maasserver/static/scss/_components.flashmessages.scss.OTHER (+59/-0)
src/maasserver/static/scss/_components.forms.scss.OTHER (+201/-0)
src/maasserver/static/scss/_components.icons.scss.OTHER (+274/-0)
src/maasserver/static/scss/_components.tables.scss.OTHER (+668/-0)
src/maasserver/static/scss/_helpers.display.scss.OTHER (+101/-0)
src/maasserver/static/scss/_helpers.spacing.scss.OTHER (+48/-0)
src/maasserver/static/scss/_patterns.page-header.scss.OTHER (+177/-0)
src/maasserver/templates/maasserver/base.html (+4/-0)
src/maasserver/tests/test_preseed.py (+17/-0)
src/metadataserver/api.py (+30/-0)
src/provisioningserver/boot/uefi_amd64.py (+4/-0)
src/provisioningserver/import_images/boot_resources.py (+15/-0)
src/provisioningserver/import_images/tests/test_boot_resources.py (+28/-0)
versions.cfg (+8/-0)
- Andres Rodriguez (community): Approve
- Lee Trager (community): Approve
-
Diff: 184 lines (+82/-10)6 files modifiedsrc/maasserver/models/node.py (+5/-7)
src/maasserver/models/tests/test_node.py (+5/-3)
src/maasserver/rpc/regionservice.py (+13/-0)
src/maasserver/rpc/tests/test_regionservice.py (+29/-0)
src/provisioningserver/utils/ps.py (+10/-0)
src/provisioningserver/utils/tests/test_ps.py (+20/-0)
- Blake Rouse (community): Approve
-
Diff: 196 lines (+94/-10)6 files modifiedsrc/maasserver/models/node.py (+5/-7)
src/maasserver/models/tests/test_node.py (+5/-3)
src/maasserver/rpc/regionservice.py (+13/-0)
src/maasserver/rpc/tests/test_regionservice.py (+29/-0)
src/provisioningserver/utils/ps.py (+14/-0)
src/provisioningserver/utils/tests/test_ps.py (+28/-0)
summary: |
- [2.0] PXE EFI boot failures: message: 'Failed deployment: bootx64.efi' - - PXE-E23 Client received TFTP error from server + [2.0] 'Failed deployment: bootx64.efi' - rackd — Missing connections to + 1 region controller(s). |
summary: |
- [2.0] 'Failed deployment: bootx64.efi' - rackd — Missing connections to - 1 region controller(s). + [2.0] TFTP issues when rackd — Missing connections to 1 region + controller(s). |
Changed in maas: | |
importance: | Undecided → Critical |
status: | New → Confirmed |
Changed in maas: | |
status: | Confirmed → Triaged |
Changed in maas: | |
milestone: | 2.1.0 → 2.1.1 |
Changed in maas: | |
milestone: | 2.1.1 → 2.1.2 |
Changed in maas: | |
milestone: | 2.1.2 → 2.1.3 |
tags: | added: uosci |
no longer affects: | maas/2.0 |
no longer affects: | maas/2.1 |
This actually turns out to be re-creatable. It happens when doing an image upload.
I recreated it with:
maas root boot-resources create name=windows/ win2012hvr2 title="Windows \"Hyper-V 2012 R2\"" architecture= amd64/generic filetype=ddtgz content@ =/home/ ubuntu/ images/ win2012hvr2- dd-img- root-dd --debug
In retrospect, original issue happened around the time that I was loading the previous 1.6Gig image. This image is larger 2.9Gig and I saw unresponsiveness immediately after starting the upload and seeing the multiple disconnections and attempts to reconnect. Attaching the latest logs..
For 2.1, I uploaded the same issue and did not observe this.