AttributeError: 'Port' object has no attribute 'socket'

Bug #1376483 reported by Andres Rodriguez on 2014-10-01
64
This bug affects 10 people
Affects Status Importance Assigned to Milestone
MAAS
Critical
Andres Rodriguez
python-tx-tftp (Ubuntu)
Critical
Andres Rodriguez
Zesty
Undecided
Unassigned

Bug Description

Seems some machines didn't start/finish commissioning like the other ones as I saw this issue:

-02-23 12:10:28+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49153): <RRQDatagram(filename=b'/boot/isolinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:10:40+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49160): <RRQDatagram(filename=b'pxelinux.cfg/01-74-d4-35-89-bc-23', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:10:40+0800 [HTTPPageGetter,client] RemoteOriginReadSession starting on 53521
2016-02-23 12:10:40+0800 [HTTPPageGetter,client] Starting protocol <tftp.bootstrap.RemoteOriginReadSession object at 0x7f4311638320>
2016-02-23 12:10:40+0800 [RemoteOriginReadSession (UDP)] (UDP Port 53521 Closed)
2016-02-23 12:10:40+0800 [RemoteOriginReadSession (UDP)] Stopping protocol <tftp.bootstrap.RemoteOriginReadSession object at 0x7f4311638320>
2016-02-23 12:10:41+0800 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 364, in startReactor
            self.config, oldstdout, oldstderr, self.profiler, reactor)
          File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 285, in runReactorWithLogging
            reactor.run()
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1194, in run
            self.mainLoop()
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1203, in mainLoop
            self.runUntilCurrent()
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 825, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/usr/lib/python3/dist-packages/tftp/util.py", line 81, in _call_and_schedule
            self.callable(*self.callable_args, **self.callable_kwargs)
          File "/usr/lib/python3/dist-packages/twisted/internet/udp.py", line 268, in write
            return self.socket.send(datagram)
        builtins.AttributeError: 'Port' object has no attribute 'socket'

[...]

2016-02-23 12:13:33+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:13:37+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:13:42+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:13:48+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:13:56+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:14:05+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-02-23 12:14:15+0800 [TFTP (UDP)] Datagram received from ('192.168.10.254', 49155): <RRQDatagram(filename=b'/boot/syslinux/ldlinux.c32', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>

=====

TFTP Request - pxelinux.cfg/01-74-d4-35-89-bc-3b Tue, 23 Feb. 2016 11:58:22
TFTP Request - pxelinux.cfg/3bbc8935-d474-8001-e311-1ea200eefb38 Tue, 23 Feb. 2016 11:58:22
TFTP Request - ubuntu/amd64/generic/xenial/daily/boot-kernel Tue, 23 Feb. 2016 11:58:22
TFTP Request - pxelinux.cfg/01-74-d4-35-89-bc-3b Tue, 23 Feb. 2016 11:58:22
TFTP Request - /syslinuxldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - /boot/isolinux/ldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - /syslinux/ldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - /boot/syslinux/ldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - /syslinux/ldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - ldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - pxelinux.0 Tue, 23 Feb. 2016 11:58:21
TFTP Request - /isolinux/ldlinux.c32 Tue, 23 Feb. 2016 11:58:22
TFTP Request - pxelinux.0 Tue, 23 Feb. 2016 11:58:21
PXE Request - commissioning Tue, 23 Feb. 2016 11:58:22
PXE Request - commissioning Tue, 23 Feb. 2016 11:58:22
Node powered on Tue, 23 Feb. 2016 11:56:43

Newell Jensen (newell-jensen) wrote :

This error shows up for me at the very end of deployment. The node still reaches the "Deployed" state, but this exception is being raised.

Jason Hobbs (jason-hobbs) wrote :

I usually see this when I have issues with NAT interfering with TFTP. Is that a possibility in this case?

Changed in maas:
status: New → Triaged
importance: Undecided → Low
Raphaël Badin (rvb) wrote :

We're still seeing this. It's a crash so marking this 'Critical'.

Changed in maas:
importance: Low → Critical
Andres Rodriguez (andreserl) wrote :

Marking incompelte as I believe this has already been fixed, but maybe another bug was attached to it.

Changed in maas:
status: Triaged → Incomplete
Andres Rodriguez (andreserl) wrote :

In fact, marking this invalid.

Raghuram Kota (rkota) wrote :

Andres can you please let us know which version of MaaS is this fixed in ? we are still seeing this 1.8

dann frazier (dannf) wrote :

fyi, we're using 1.8.0+bzr4001-0ubuntu2

Gavin Panella (allenap) on 2015-09-03
Changed in maas:
status: Incomplete → Invalid
tags: added: tech-debt
no longer affects: maas/1.8
Changed in maas:
status: Invalid → New
milestone: none → 1.10.0
milestone: 1.10.0 → 2.0.0
description: updated
description: updated
description: updated
description: updated
Gavin Panella (allenap) on 2016-02-24
Changed in maas:
status: New → Triaged
Larry Michel (lmic) wrote :

I am hitting this issue with1.9.1+bzr4543-0ubuntu1~trusty. I was going to open a bug, but I am seeing that one is already opened.

tags: added: oil
Larry Michel (lmic) wrote :

I am also hitting it with 1.8.3+bzr4053-0ubuntu1~trusty1 for ppc64el system in powerNV mode. When I hit it, the wrong ephemeral image got loaded (amd64 instead of ppc64el).

no longer affects: maas/1.10
Ryan Beisner (1chb1n) wrote :

In trying to enlist and commission a Tuleta Power8 machine with MAAS 1.9.4, I get the following and cannot seem to find a way to get the machine to enlist.

# package version info
ubuntu@lescina:/var/log/maas$ apt-cache policy maas
maas:
  Installed: 1.9.4+bzr4592-0ubuntu1~trusty1
  Candidate: 1.9.4+bzr4592-0ubuntu1~trusty1
  Version table:
 *** 1.9.4+bzr4592-0ubuntu1~trusty1 0
        500 http://ppa.launchpad.net/maas/proposed/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     1.9.4+bzr4592-0ubuntu1~14.04.1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
     1.5.4+bzr2294-0ubuntu1.2 0
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     1.5+bzr2252-0ubuntu1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

# clusterd.log
2016-10-02 00:34:06+0000 [RemoteOriginReadSession (UDP)] (UDP Port 45741 Closed)
2016-10-02 00:34:06+0000 [RemoteOriginReadSession (UDP)] Stopping protocol <tftp.bootstrap.RemoteOriginReadSession instance at 0x7ffaa1206b48>
2016-10-02 00:34:06+0000 [RemoteOriginReadSession (UDP)] Final ACK received, transfer successful
2016-10-02 00:34:06+0000 [-] (UDP Port 47215 Closed)
2016-10-02 00:34:06+0000 [-] Stopping protocol <tftp.bootstrap.RemoteOriginReadSession instance at 0x7ffaa120ccb0>
2016-10-02 00:34:07+0000 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 392, in startReactor
            self.config, oldstdout, oldstderr, self.profiler, reactor)
          File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 313, in runReactorWithLogging
            reactor.run()
          File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
            self.mainLoop()
          File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
            self.runUntilCurrent()
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/usr/lib/python2.7/dist-packages/tftp/util.py", line 80, in _call_and_schedule
            self.callable(*self.callable_args, **self.callable_kwargs)
          File "/usr/lib/python2.7/dist-packages/twisted/internet/udp.py", line 254, in write
            return self.socket.send(datagram)
        exceptions.AttributeError: 'Port' object has no attribute 'socket'

Ryan Beisner (1chb1n) wrote :

FYI, we require MAAS 1.9.x in order to validate with Juju 1.x.

Junien Fridrick (axino) wrote :

I hit this bug with MAAS 2.1, the server fails commissioning.

rackd.log :

2016-11-03 16:36:05 [TFTP (UDP)] Datagram received from ('10.189.142.2', 1332): <RRQDatagram(filename=b'bootx64.efi', mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1468')]))>
2016-11-03 16:36:05 [-] RemoteOriginReadSession starting on 37207
2016-11-03 16:36:05 [-] Starting protocol <tftp.bootstrap.RemoteOriginReadSession object at 0x7fa1226dcda0>
2016-11-03 16:36:05 [RemoteOriginReadSession (UDP)] (UDP Port 37207 Closed)
2016-11-03 16:36:05 [RemoteOriginReadSession (UDP)] Stopping protocol <tftp.bootstrap.RemoteOriginReadSession object at 0x7fa1226dcda0>
2016-11-03 16:36:06 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 364, in startReactor
            self.config, oldstdout, oldstderr, self.profiler, reactor)
          File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 285, in runReactorWithLogging
            reactor.run()
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1194, in run
            self.mainLoop()
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1203, in mainLoop
            self.runUntilCurrent()
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 825, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/usr/lib/python3/dist-packages/tftp/util.py", line 81, in _call_and_schedule
            self.callable(*self.callable_args, **self.callable_kwargs)
          File "/usr/lib/python3/dist-packages/twisted/internet/udp.py", line 268, in write
            return self.socket.send(datagram)
        builtins.AttributeError: 'Port' object has no attribute 'socket'

tcpdump attached.

Junien Fridrick (axino) wrote :

I forgot to add : the machine above is an HP DL360 Gen9

I successfully enlisted it in MAAS 2.1 last week. Since then, I upgraded the firmware of the iLO to 2.50 (not sure what it was before), which could be the cause of this new behaviour ?

Lee Trager (ltrager) wrote :

This may be an issue with the external library MAAS uses to handle TFTP. I found an upstream bug with a fix that looks similar to this.

https://github.com/shylent/python-tx-tftp/issues/15

Junien Fridrick (axino) wrote :

Hi,

This is different, my backtrace doesn't involve session.py

I tried the proposed fix, and I have the same failures.

Junien Fridrick (axino) wrote :

I found out the issue : improper IP configuration.

My MAAS server is on IP_A, but on the same interface there is another IP, IP_B. I had tweaked up things and IP_B was the primary IP of the interface, and apparently python-txtftp doesn't like that.

Changed in maas:
milestone: 2.0.0 → 2.2.0rc2
Changed in maas:
assignee: nobody → Gavin Panella (allenap)
Gavin Panella (allenap) on 2017-04-10
Changed in maas:
status: Triaged → In Progress
Gavin Panella (allenap) on 2017-04-11
Changed in maas:
assignee: Gavin Panella (allenap) → nobody
status: In Progress → Triaged
Changed in maas:
assignee: nobody → Blake Rouse (blake-rouse)
Changed in maas:
assignee: Blake Rouse (blake-rouse) → Andres Rodriguez (andreserl)
Changed in maas:
status: Triaged → In Progress
Changed in python-tx-tftp (Ubuntu):
importance: Undecided → Critical
status: New → Triaged
assignee: nobody → Andres Rodriguez (andreserl)
status: Triaged → In Progress
Narinder Gupta (narindergupta) wrote :

At HPE lab i am hitting the same issue and this happens at any time and started seeing now with commisoning process. MAAS version is 2.1

maas logs
http://paste.ubuntu.com/24330558/
http://paste.ubuntu.com/24330584/
http://paste.ubuntu.com/24330585/

Changed in maas:
status: In Progress → Fix Committed

Hello Andres, or anyone else affected,

Accepted python-tx-tftp into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-tx-tftp/0.1~bzr47-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-tx-tftp (Ubuntu Zesty):
status: New → Fix Committed
tags: added: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-tx-tftp - 0.1~bzr47-0ubuntu1

---------------
python-tx-tftp (0.1~bzr47-0ubuntu1) zesty; urgency=medium

  * New upstream revision that only includes bug fixes and changes made
    in previous Ubuntu packaging (lp:python-tfx-tftp) (LP: #1682187).
    - Include fixes to setup.py and tox (bzr43).
    - Include port to python3 (bzr44).
    - Mention object transfer in README (bzr45)
    - Fix handle non-RRQ/WRQ datagrams on default port (bzr46) (LP: #1682139)
    - Fix tracebacks that cause tftp to temporarily stop with
      missing 'stopListening', 'socket' (LP: #1376483, LP: #1381129)
  * debian/patches:
    - 04-setup-py-and-tox.patch: Drop, merged upstream (bzr43)
    - 05-python3-support.patch: Drop, merged upstream (bzr44)
    - 99-drop-trial.patch: Drop setuptools_trial, provided that it is not
      in the Ubuntu archive.

 -- Andres Rodriguez <email address hidden> Tue, 11 Apr 2017 11:03:56 -0400

Changed in python-tx-tftp (Ubuntu):
status: In Progress → Fix Released
Changed in maas:
status: Fix Committed → Fix Released

As part of a recent change in the Stable Release Update verification policy we would like to inform that for a bug to be considered verified for a given release a verification-done-$RELEASE tag needs to be added to the bug where $RELEASE is the name of the series the package that was tested (e.g. verification-done-xenial). Please note that the global 'verification-done' tag can no longer be used for this purpose.

Thank you!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.