AttributeError: 'Port' object has no attribute 'socket'

[Andres Rodriguez]

Seems some machines didn't start/finish commissioning like the other ones as I saw this issue:

-02-23 12:10:28+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49153): <RRQDatagram(filename=b'/boot/isolinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:10:40+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49160): <RRQDatagram(filename=b'pxelinux.cfg/01-74-d4-35-89-bc-23', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:10:40+0800 [HTTPPageGetter,client] RemoteOriginReadSession starting on 53521
2016-02-23 12:10:40+0800 [HTTPPageGetter,client] Starting protocol <tftp.bootstrap.RemoteOriginReadSession object at 0x7f4311638320>
2016-02-23 12:10:40+0800 [RemoteOriginReadSession (UDP)] (UDP Port 53521 Closed)
2016-02-23 12:10:40+0800 [RemoteOriginReadSession (UDP)] Stopping protocol <tftp.bootstrap.RemoteOriginReadSession object at 0x7f4311638320>
2016-02-23 12:10:41+0800 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 364, in startReactor
            self.config, oldstdout, oldstderr, self.profiler, reactor)
          File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 285, in runReactorWithLogging
            reactor.run()
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1194, in run
            self.mainLoop()
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1203, in mainLoop
            self.runUntilCurrent()
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 825, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/usr/lib/python3/dist-packages/tftp/util.py", line 81, in _call_and_schedule
            self.callable(*self.callable_args, **self.callable_kwargs)
          File "/usr/lib/python3/dist-packages/twisted/internet/udp.py", line 268, in write
            return self.socket.send(datagram)
        builtins.AttributeError: 'Port' object has no attribute 'socket'

[...]

2016-02-23 12:13:33+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:13:37+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:13:42+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:13:48+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:13:56+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:14:05+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:14:15+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>

=====

TFTP Request - pxelinux.cfg/01-74-d4-35-89-bc-3b Tue, 23 Feb. 2016 11:58:22
TFTP Request - pxelinux.cfg/3bbc8935-d474-8001-e311-1ea200eefb38 Tue, 23 Feb. 2016 11:58:22
TFTP Request - ubuntu/amd64/generic/xenial/daily/boot-kernel Tue, 23 Feb. 2016 11:58:22
TFTP Request - pxelinux.cfg/01-74-d4-35-89-bc-3b Tue, 23 Feb. 2016 11:58:22
TFTP Request - /syslinuxldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - /boot/isolinux/ldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - /syslinux/ldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - /boot/syslinux/ldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - /syslinux/ldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - ldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - pxelinux.0 Tue, 23 Feb. 2016 11:58:21
TFTP Request - /isolinux/ldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - pxelinux.0 Tue, 23 Feb. 2016 11:58:21
PXE Request - commissioning Tue, 23 Feb. 2016 11:58:22
PXE Request - commissioning Tue, 23 Feb. 2016 11:58:22
Node powered on Tue, 23 Feb. 2016 11:56:43

[Newell Jensen (newell-jensen)]

This error shows up for me at the very end of deployment. The node still reaches the "Deployed" state, but this exception is being raised.

[Jason Hobbs (jason-hobbs)]

I usually see this when I have issues with NAT interfering with TFTP. Is that a possibility in this case?

[Raphaël Badin (rvb)]

We're still seeing this. It's a crash so marking this 'Critical'.

[Andres Rodriguez (andreserl)]

Marking incompelte as I believe this has already been fixed, but maybe another bug was attached to it.

[Andres Rodriguez (andreserl)]

In fact, marking this invalid.

[Raghuram Kota (rkota)]

Andres can you please let us know which version of MaaS is this fixed in ? we are still seeing this 1.8

[dann frazier (dannf)]

fyi, we're using 1.8.0+bzr4001-0ubuntu2

[Larry Michel (lmic)]

I am hitting this issue with1.9.1+bzr4543-0ubuntu1~trusty. I was going to open a bug, but I am seeing that one is already opened.

[Larry Michel (lmic)]

I am also hitting it with 1.8.3+bzr4053-0ubuntu1~trusty1 for ppc64el system in powerNV mode. When I hit it, the wrong ephemeral image got loaded (amd64 instead of ppc64el).

[Ryan Beisner (1chb1n)]

In trying to enlist and commission a Tuleta Power8 machine with MAAS 1.9.4, I get the following and cannot seem to find a way to get the machine to enlist.

# package version info
ubuntu@lescina:/var/log/maas$ apt-cache policy maas
maas:
  Installed: 1.9.4+bzr4592-0ubuntu1~trusty1
  Candidate: 1.9.4+bzr4592-0ubuntu1~trusty1
  Version table:
 *** 1.9.4+bzr4592-0ubuntu1~trusty1 0
        500 http://ppa.launchpad.net/maas/proposed/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     1.9.4+bzr4592-0ubuntu1~14.04.1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
     1.5.4+bzr2294-0ubuntu1.2 0
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     1.5+bzr2252-0ubuntu1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

# clusterd.log
2016-10-02 00:34:06+0000 [RemoteOriginReadSession (UDP)] (UDP Port 45741 Closed)
2016-10-02 00:34:06+0000 [RemoteOriginReadSession (UDP)] Stopping protocol <tftp.bootstrap.RemoteOriginReadSession instance at 0x7ffaa1206b48>
2016-10-02 00:34:06+0000 [RemoteOriginReadSession (UDP)] Final ACK received, transfer successful
2016-10-02 00:34:06+0000 [-] (UDP Port 47215 Closed)
2016-10-02 00:34:06+0000 [-] Stopping protocol <tftp.bootstrap.RemoteOriginReadSession instance at 0x7ffaa120ccb0>
2016-10-02 00:34:07+0000 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 392, in startReactor
            self.config, oldstdout, oldstderr, self.profiler, reactor)
          File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 313, in runReactorWithLogging
            reactor.run()
          File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
            self.mainLoop()
          File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
            self.runUntilCurrent()
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/usr/lib/python2.7/dist-packages/tftp/util.py", line 80, in _call_and_schedule
            self.callable(*self.callable_args, **self.callable_kwargs)
          File "/usr/lib/python2.7/dist-packages/twisted/internet/udp.py", line 254, in write
            return self.socket.send(datagram)
        exceptions.AttributeError: 'Port' object has no attribute 'socket'

[Ryan Beisner (1chb1n)]

FYI, we require MAAS 1.9.x in order to validate with Juju 1.x.

[Junien F (axino)]

I hit this bug with MAAS 2.1, the server fails commissioning.

rackd.log :

2016-11-03 16:36:05 [TFTP (UDP)] Datagram received from ('10.189.142.2', 1332): <RRQDatagram(filename=b'bootx64.efi', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1468')]))>
2016-11-03 16:36:05 [-] RemoteOriginReadSession starting on 37207
2016-11-03 16:36:05 [-] Starting protocol <tftp.bootstrap.RemoteOriginReadSession object at 0x7fa1226dcda0>
2016-11-03 16:36:05 [RemoteOriginReadSession (UDP)] (UDP Port 37207 Closed)
2016-11-03 16:36:05 [RemoteOriginReadSession (UDP)] Stopping protocol <tftp.bootstrap.RemoteOriginReadSession object at 0x7fa1226dcda0>
2016-11-03 16:36:06 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 364, in startReactor
            self.config, oldstdout, oldstderr, self.profiler, reactor)
          File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 285, in runReactorWithLogging
            reactor.run()
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1194, in run
            self.mainLoop()
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1203, in mainLoop
            self.runUntilCurrent()
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 825, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/usr/lib/python3/dist-packages/tftp/util.py", line 81, in _call_and_schedule
            self.callable(*self.callable_args, **self.callable_kwargs)
          File "/usr/lib/python3/dist-packages/twisted/internet/udp.py", line 268, in write
            return self.socket.send(datagram)
        builtins.AttributeError: 'Port' object has no attribute 'socket'

tcpdump attached.

[Junien F (axino)]

I forgot to add : the machine above is an HP DL360 Gen9

I successfully enlisted it in MAAS 2.1 last week. Since then, I upgraded the firmware of the iLO to 2.50 (not sure what it was before), which could be the cause of this new behaviour ?

[Lee Trager (ltrager)]

This may be an issue with the external library MAAS uses to handle TFTP. I found an upstream bug with a fix that looks similar to this.

https://github.com/shylent/python-tx-tftp/issues/15

[Junien F (axino)]

Hi,

This is different, my backtrace doesn't involve session.py

I tried the proposed fix, and I have the same failures.

[Junien F (axino)]

I found out the issue : improper IP configuration.

My MAAS server is on IP_A, but on the same interface there is another IP, IP_B. I had tweaked up things and IP_B was the primary IP of the interface, and apparently python-txtftp doesn't like that.

[Narinder Gupta (narindergupta)]

At HPE lab i am hitting the same issue and this happens at any time and started seeing now with commisoning process. MAAS version is 2.1

maas logs
http://paste.ubuntu.com/24330558/
http://paste.ubuntu.com/24330584/
http://paste.ubuntu.com/24330585/

[Andy Whitcroft (apw)]

Hello Andres, or anyone else affected,

Accepted python-tx-tftp into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-tx-tftp/0.1~bzr47-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-tx-tftp - 0.1~bzr47-0ubuntu1

---------------
python-tx-tftp (0.1~bzr47-0ubuntu1) zesty; urgency=medium

  * New upstream revision that only includes bug fixes and changes made
    in previous Ubuntu packaging (lp:python-tfx-tftp) (LP: #1682187).
    - Include fixes to setup.py and tox (bzr43).
    - Include port to python3 (bzr44).
    - Mention object transfer in README (bzr45)
    - Fix handle non-RRQ/WRQ datagrams on default port (bzr46) (LP: #1682139)
    - Fix tracebacks that cause tftp to temporarily stop with
      missing 'stopListening', 'socket' (LP: #1376483, LP: #1381129)
  * debian/patches:
    - 04-setup-py-and-tox.patch: Drop, merged upstream (bzr43)
    - 05-python3-support.patch: Drop, merged upstream (bzr44)
    - 99-drop-trial.patch: Drop setuptools_trial, provided that it is not
      in the Ubuntu archive.

 -- Andres Rodriguez <email address hidden> Tue, 11 Apr 2017 11:03:56 -0400

[Łukasz Zemczak (sil2100)]

As part of a recent change in the Stable Release Update verification policy we would like to inform that for a bug to be considered verified for a given release a verification-done-$RELEASE tag needs to be added to the bug where $RELEASE is the name of the series the package that was tested (e.g. verification-done-xenial). Please note that the global 'verification-done' tag can no longer be used for this purpose.

Thank you!

[Ubuntu Foundations Team Bug Bot (crichton)]

The fix for this bug has been awaiting testing feedback in the -proposed repository for zesty for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

[Sanji Vinsmoke (masterofcloud1)]

Having the same issue, but on VM'S.
I have MaaS installed on VM, and I'm trying to pxe boot other vms with it.
But it is not working.
On rackd.log :
2018-06-14 08:16:31 -: [critical] Unhandled Error
 Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 364, in startReactor
     self.config, oldstdout, oldstderr, self.profiler, reactor)
   File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 285, in runReactorWithLogging
     reactor.run()
   File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1194, in run
     self.mainLoop()
   File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1203, in mainLoop
     self.runUntilCurrent()
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 825, in runUntilCurrent
     call.func(*call.args, **call.kw)
   File "/usr/lib/python3/dist-packages/tftp/util.py", line 81, in _call_and_schedule
     self.callable(*self.callable_args, **self.callable_kwargs)
   File "/usr/lib/python3/dist-packages/twisted/internet/udp.py", line 268, in write
     return self.socket.send(datagram)
 builtins.AttributeError: 'Port' object has no attribute 'socket'

Both MaaS and python-twisted is up to date.

[Mike Pontillo (mpontillo)]

Can those of you who are still encountering this issue try the latest version of python3-txtftp? It seems that it may still need to be backported to Xenial.

It can be downloaded directly from the archive at:

http://archive.ubuntu.com/ubuntu/pool/main/p/python-tx-tftp/python3-txtftp_0.1~bzr47-0ubuntu2_all.deb