Ubuntu 9.04 Ekiga 3.2.0 If PDU is way larger than standard MTU (1500), even if local MTU is large enough (7000), connection timeout.

[BrainwreckedTech]

== THINGS NEEDED TO REPRODUCE ==
Ubuntu 8.10 and Ubuntu 9.04 and their respective versions of Ekiga
Ekiga.net account
Diamondcard.us account
Two consumer-grade routers (my case, a Buffalo AirStation WHR-G45S and Asus WL-520G, both running DD-WRT v24-sp1 (07/27/08) std)
Set one router to connect to the Cable/DSL modem
Set the second router to treat the the first router as a WAN
Forwarding of ports 5000-5100 UDP so that the second router can receive SIP traffic and forward it to a particular internal IP address.

== WORKING SOLUTION ==
Ubuntu 8.10 with Ekiga 2.0 (default version that ships with Ubuntu 8.10)
Set up Ekiga with an Ekiga.net account
Ekiga.net login works as expected
Ekiga.net echo test works as exepected
Setup Ekiga with a Diamondcard.us account
Diamondcard.us login works as expected
Diamondcard.us echo test works as expected

== WORKING SOLUTION (UNCONFIRMED) ==

[deleted July 10 2009 -- not needed]

== UNTESTED SOLUTION ==

[deleted July 10 2009 -- not needed]

== NON-WORKING SOLUTION ==
Ubuntu 9.04 and Ekiga 3.2 (default version that ships with Ubuntu 9.04)
Set up Ekiga with an Ekiga.net account
Ekiga.net login works as expected
Ekiga.net echo test fails (User not available)
Set up Diamondcard.us account
Diamondcard.us login works as expected
Ekiga receives call through the Diamondcard.us account as expected
During received calls, sound only comes in. The person calling in cannot hear the recipient behind Ekiga.

========== UPDATE JULY 10 2009 ==========
After some discussion, the problem I was having was not with a double-NAT set up but rather an MTU setting. If you have optimized your MTU packet size for a local gigabit LAN, chances are it is higher than the Internet standard of 1500. Currently, Ekiga 3.2.0 only supports communications using UDP packets, and they cannot be fragmented into smaller packets. TCP packets can be fragmented, but TCP is currently not implemented (though high on the TODO). If you issue the command ifconfig [interface] mtu 1500 (where [interface] is something along the lines of eth0) and everything starts working, then this was your problem. If you're still having problems, try the Ekiga Echo Test. If the Ekiga Echo Test works and the other SIP server/peer does not, chances are that you and this other SIP server/peer you are connecting to cannot negotiate a codec that will fit inside this limit.
=========================================

[BrainwreckedTech (paulhinchberger3+ubuntulaunchpad)]

Found this PPA for Ekiga 3.0 SVN and Ubuntu 8.10: https://launchpad.net/~sevmek/+archive/ppa
Will try this when I can. Hopefully a working, easily-replicated scenario helps in narrowing things down a bit.

[BrainwreckedTech (paulhinchberger3+ubuntulaunchpad)]

Scratch that. The PPA's version of Ekiga is 2.9.90. This version connects correctly, but incoming sound cannot be heard, much like non-working Ubuntu 9.04/Ekiga 3.2 setup with Diamondcard. Test cases are piling up, and as I used to have Ekiga 3.0SVN working on Ubuntu 8.10, I don't think this test case should be considered. Correct me if anyone thinks otherwise.

[Alvin (purplealvin-deactivatedaccount)]

The following maybe the cause??:

Running Ekiga 3.2 (on jaunty) using ekiga -d 5 > out
I see the following line:
"2009/07/05 23:07:53.315 0:51.397 Housekeeper:0xb5555b90 SIP PDU is likely too large (1550 bytes) for UDP datagram."

which is likely to be this known issue:
http://wiki.ekiga.org/index.php/Troubleshooting#I_hear_the_ring.2C_but_the_communication_is_not_established

and upstream bug:
http://bugzilla.gnome.org/show_bug.cgi?id=341518

and for the record - test call to echo (<email address hidden>) fails with "user is not available" as reported above.

[Yannick Defais (sevmek)]

"2009/07/05 23:07:53.315 0:51.397 Housekeeper:0xb5555b90 SIP PDU is likely too large (1550 bytes) for UDP datagram." is indeed an issue here.

Solution: disable some codecs in preferences until the PDU is lower than 1500.

[BrainwreckedTech (paulhinchberger3+ubuntulaunchpad)]

Alvin:

Running ekiga -d 5 > out, the output still goes to the console. If I remember anything correctly about C programming, ekiga is directly dumping output to the console instead of using stdout. You have to use ekiga -d5 2> out.

Alvin & Yannick Defais:

The PDU 1500/1550 thing makes a heckuva lot of sense. The computer running Ubuntu 8.10/Ekiga 2 did not have it's MTU touched, so it's at the default 1500. The computer running Ubuntu 9.04/Ekiga 3 has its MTU set at 7500 because its /home is on an NFS share over gigabit networking. If I'm recalling this correctly (again), UDP packets don't support fragmentation while TCP packets do.

After issuing ifconfig eth0 mtu 1500, Ubuntu 9.04/Ekiga 3 works as expected. But should users be expected to fiddle with MTU settings just for one program? Shouldn't Ekiga be able to pick up on the error and say, "Oh, the packets are too big? Let's try smaller, standard ones." Or is the packet size completely out of the hands of the application?

As it stands now, my options aren't nice:

* take a 50%-66% hit in local gigabit network performance to run Ekiga all the time
* run Ekiga with a script that'll change the MTU as needed (only dragging performance while Ekiga is running)
* set up another computer just to run Ekiga all the time (which is a bit of a waste)
* I don't have dual NICs, so setting one up for the local network and one for the Internet goes right out the window

[Yannick Defais (sevmek)]

BrainwreckedTech,

Ekiga cannot try to reduce the packet size because of the SIP protocol: it has to deal with a codec negociation, and it respect user choice to present all codecs available (and there is many codecs available in Ekiga) and selected by the user. Reducing the packet will mean dropping some codecs.

As you explained UDP packets cannot be fragmented, while TPC packets can. The SIP standard thus ask the application to switch to TCP if the packet size do not fit in UDP. The issue here is Ekiga do not support this feature yet, which lead to this bad situation. It was not an issue until Ekiga supported so many codecs as it is able to do nowadays. Supporting TCP as well as UDP is a main priority for the project (in fact for the OPAL lib ekiga uses).

Until this bug is fixed, the best solution is to deselect some codecs in Ekiga until it fits the UDP packet size.

Best regards,
Yannick

[Alvin (purplealvin-deactivatedaccount)]

Thanks Yannick.
I understand the issue a lot better now.

I wonder if there is a more immediate issue here with regard to Ubuntu packaging. On Jaunty (netbook remix) Ekiga comes 'out-of-the-box' with all codecs pre-selected which is not a working configuration.

This may also be a nice config mod to be nominated for Karmic Koala.

[Yannick Defais (sevmek)]

Alvin,

AFAIK, Ubuntu ships only the free codecs and if you select them all it fits in UDP, the issue occurs when you add the non-free codecs.

Do you use the package from Ubuntu? If so, does the PDU exceed 1500?

Best regards,
Yannick

[Alvin (purplealvin-deactivatedaccount)]

Yannick,

No I hadn't installed any non-free codecs (not knowingly anyway).

I wanted to confirm this so I booted from clean live USB memory stick image (Jaunty).
Selected ekiga for installation.
Synaptic informs that the following are required: libgsm1, libopal3.6.1, libpt2.6.1, libpt2.6.1-plugins-alsa, libpt2.6.1-plugins-v4l2. (All of these come from the Main repository).

Once above are installed: Ekiga has the same problem as described above (PDU exceed 1500) and same solution as you describe above also works.

Out-of-the-box Audio codecs: G722, Speex(16kHz), PCMA, PCMU, G726-16, G726-24, G726-32, G726-40, gsm, ms-gsm, Speex(8kHz).

Out-of-the-box Video codecs: h261, theora.

So, in answer to your question, only using default packages from Ubuntu is sufficient for PDU to exceed 1500.

[BrainwreckedTech (paulhinchberger3+ubuntulaunchpad)]

I can confirm with Alvin, insofar as I did not install any additional codecs (knowingly). Granted I haven't gone so far as to test a fresh install, so there exists a possibility that additional codecs were available to me as a side-effect of installing other packages on top of the default ubuntu install. Here's what I got (all enabled):

* Speex 8/16 kHz, PCMA, PCMU, G726-16/24/32/40, GSM, MS-GSM, G722

Ekiga can connect to the Echo Test and work as expected when the local machine's MTU is set to 1500. Funny things happen when the the local machine's MTU is set too high (common on machines that are on a local gigabit network but also connect to the Internet.)

I take exception with the view that Ekiga must honor the user's wish to use whatever codecs the user specifies. The view by itself is justified and agreeable, but ignores the ultimate wish of every user of every program ever in existence -- the user wants the program to actually work. If those two ideals happen to collide, the latter should take precedence over the former. Having said that, I don't think the following would be a problem:

* Offer a checkbox that will offer to limit UDP packet sizes to the standard size found on the Internet in the same place where users can enable/disable codecs.
* Warn users that this may disable the use of some codecs the user has chosen to use which might result in not being able to connect to some SIP servers/peers.
* When the user makes a call and is unable to negotiate the use of a codec with the SIP server/peer, tell them so. Don't tell them "user not available." That's how this thread started in the first place. Tell them with as much specificity as you can -- "unable to negotiate codec with server" or something similar.

While Ekiga gets TCP support, there very well could be some servers that are UDP-only. This is the only reason I'm going into the diatribe. I've seen too many bug reports and release notes for SIP-related software/firmware. For varying reasons, shortcuts get taken. To the user, if one implementation works and another does not, the non-working implementation looks broken even when the non-working implementation is not at fault.

And last but not least, I just want to make sure that it's understood that I'm not holding anyone personally responsible for my gripes with Ekiga -- Yannick or otherwise. After taking a few read-overs on this post, I realized what I wrote could be taken that way.

[Alvin (purplealvin-deactivatedaccount)]

Confirming & modified title to reflect issue as it is now known.
Please feel free to revert/modify if you prefer.

[BrainwreckedTech (paulhinchberger3+ubuntulaunchpad)]

Alvin

We seem to be having two different problems.

Mine was due to the local machine's MTU setting and was fixed when the MTU was brought down to 1500. My specific issue here was that Ekiga could very well work with the local machine's MTU set to 7000 as it's called *Maximum* Transmission Unit not *Only* Transmission Unit. As evidenced in my test cases, a computer can receive packets smaller than its MTU setting with no problem. That's why conversation went one way in one of my test cases - Ekiga was receiving 1500-byte (or smaller) packets but sending out 7000-byte packets and the packets where being dropped (too large) once they hit the public Internet.

However, you seem to be saying that no matter what you do, you can't get the Ekiga Echo Test to work. Please reply and verify:

* that your MTU setting is 1500 in the output that ifconfig gives you. Look for the MTU in the 4th line.

  Example:
     eth0 Link encap:Ethernet HWaddr 00:17:31:3b:d0:ee
               inet addr:192.168.1.101 Bcast:192.168.1.255 Mask:255.255.255.0
               inet6 addr: fe80::217:31ff:fe3b:d0ee/64 Scope:Link
               UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

* that you still can't connection when the only codec you tell Ekiga to use is GSM

When I got this working, I swear this was the protocol Ekiga said it was using. I have to go by memory on this one, because it seems that the Ekiga registration server is down at the moment. (Both my desktop and netbook fail Ekiga registration with "MonSock UDP Port on remote not ready." in the debug output.)

* try an MTU of 1400.

While 1500 is the standard, some hops on the net are less. If your ISP is one of those hops, you'll never get 1500-byte UDP packets out to the Internet.

From http://kb.netgear.com/app/answers/detail/a_id/1153 :

# 1500. The largest Ethernet packet size; it is also the default value. This is the typical setting for non-PPPoE, non-VPN connections.
# 1492. The size PPPoE prefers.
# 1472. Maximum size to use for pinging. (Bigger packets are fragmented.)
# 1468. The size DHCP prefers.
# 1460. Usable by AOL if you don't have large email attachments, etc.
# 1430. The size VPN and PPTP prefer.
# 1400. Maximum size for AOL DSL.

If you still can't connect, I think this might be another bug and it'd be best to file another one for it.

[Alvin (purplealvin-deactivatedaccount)]

Ah. I see. Thanks for correcting.
Given, the detail you've provided I now wonder whether the upstream bug I refered to earlier fully encompasses your issue. You've given some good detail & ideas here which perhaps should be seen upstream. Just a suggestion.

[Yannick Defais (sevmek)]

Hi,

I'm part of upstream, we are discussing this issue in the dev-mailing list.

Best regards,
Yannick

[Yannick Defais (sevmek)]

Hi,

A synthesis of our discussion upstream:

TCP is mandatory and it is a main focus of the project. The switch from UDP to TCP is also part of the SIP standard. Reason: in some case (too many codecs enabled in Ekiga's preferences) Ekiga needs a bigger datagrams than the one usually provided by the MTU (usually 1500).

UDP is datagram based ("Packets are sent individually and are checked for integrity only if they arrive." cf. http://en.wikipedia.org/wiki/User_Datagram_Protocol#Comparison_of_UDP_and_TCP ), thus meaningful related informations must stick together in the same "datagram", which size is at maximum the size provided by MTU.

While TCP is streaming ("TCP uses a sequence number to identify each byte of data. The sequence number identifies the order of the bytes sent from each computer so that the data can be reconstructed in order, regardless of any fragmentation, disordering, or packet loss that may occur during transmission." cf. http://en.wikipedia.org/wiki/Transmission_Control_Protocol#Reliable_transmission )

TCP and UDP are at the same layer level, i.e. the "Transport Layer". The underlying level is the "Internet Layer" (see http://en.wikipedia.org/wiki/Internet_Protocol_Suite ). Thus UDP and TCP are encapsulated in the IP layer, which provide "IP fragmentation" *for both*. see: http://en.wikipedia.org/wiki/IP_fragmentation

Relevant part for our issue is:
"In a case where a router receives a protocol data unit (PDU) larger than the next hop's MTU, it has two options if the transport is IPv4. Drop the PDU and send an Internet Control Message Protocol (ICMP) message which indicates "Packet too Big", or to fragment the IP packet and send over the link with a smaller MTU. IPv6 hosts are supposed to determine the optimal Path MTU before sending packets."

Thus even UDP packets will be fragmented if the next hop's MTU is smaller. There is techniques for determining the smallest MTU along the path (cf. http://tools.ietf.org/html/rfc4821 ), but it is only optimization and even if Ekiga use the MTU from its localhost, it should not prevent the UDP packet to reach the destination, no matter what MTUs are in the path or destination.

This is the theory.

I still do not understand why a UDP packets from a large MTU like 7500 do not reach a smaller MTU like 1500. The IP layer should take care of that automatically.

Best regards,
Yannick

[Yannick Defais (sevmek)]

BrainwreckedTech,

Could you attach a full -d 5 output in the case of failing MTU 7500 to MTU 1500? I'm still investigating some corner cases...

Best regards,
Yannick

[BrainwreckedTech (paulhinchberger3+ubuntulaunchpad)]

Of note here: I was wrong about GSM. In the working scenario, Ekiga choses PCMA for the Ekiga Echo Test.

[BrainwreckedTech (paulhinchberger3+ubuntulaunchpad)]

I wish I could go back and add more descriptive descriptions

First log: All codecs, MTU 7000. Does not work - User not available.
Second log: GSM only, MTU 7000. Does not work - Call connects but no sound and no codec selected.
Third log: All codecs, MTU 1500. Works as expected.
Fourth log: PCMA only, MTU 7000. Works as expected, but two noticeable drop-outs in the incoming sound.

Given that last result, the idea I'm getting here is that Ekiga is not properly negotiating codecs. PCMA is working not because Ekiga is doing anything different, but because the 8kHz sample rate is resulting in 99% of the packets being under 1500 bytes anyway. However, as noted with GSM, not all 8kHz codecs will work.

I also noted in the logs "PDU is too large (1615 bytes) trying compact form." Does this mean Ekiga does in fact try something smaller, but is failing?

[Yannick Defais (sevmek)]

BrainwreckedTech ,

Sorry for the long delay.

I cannot really explain why the first log (MTU 7000) is a failure.
The PDU is bigger than 1500, Ekiga is sending the packet. But the connection timeout. Something in path is eating the packet, most probably because of MTU reason.

Indeed, in the second log, with a PDU smaller than 1500, Ekiga connect to the echo test. Thus if the size of the packets is smaller than 1500 (approximately, we will discuss this later), the connection works.
The issue with no sound is not related to the MTU issue; the echo test simply do not support the GSM codec and SIP is designed as a best effort: thus as Ekiga and the echo test can agree on video, you only get video. This is something you can see on the panel call, below the video are the codecs in use; GSM was probably not written there.

The interesting point here is even if the MTU is large enough for the PDU Ekiga use (PDU ~1600 and MTU 7000) the connection failed because of a timeout. Something in the path did nor react well to the PDU size, i do not know what it is (might even be our server...). This need further investigation.But I've no clue how to do this. I'll ask our dev, but he is on holidays now and it is somewhat tricky...

The third log is also interesting; it prove if the PDU is a little bit bigger than 1500, the connection can work. I observe the same using my own setup. 1500 is not a hard threshold. I've no clue why.

The 4th log works as the PDU is smaller than 1500 and the echo test accept the audio codec proposed. The noticeable drop out are explained as some kind of network congestion. Ekiga is designed to not allow a delay bigger than 500milliseconds (above that is not good for audio communication; you will experience a long delay before the other side answer you), if the delay is bigger it drops packets. The PCMA codec is the simplest codecs, it has no mechanism to correct the perception if packets are dropped, thus the noticeable drop out. Other codecs more sophisticated have mechanism to support packets drop (at some extend), e.g. speex.

This is expressed in the log:
2009/07/10 18:28:36.451 0:25.460 Media Patch:0xef4af950 RTP Jitter buffer length exceeded
2009/07/10 18:28:36.451 0:25.460 Media Patch:0xef4af950 RTP Jitter buffer oldest packet (78720 < 79040) too late, throwing away
2009/07/10 18:28:36.451 0:25.460 Media Patch:0xef4af950 RTP Jitter buffer oldest packet (78880 < 79040) too late, throwing away
2009/07/10 18:28:36.451 0:25.460 Media Patch:0xef4af950 RTP Jitter buffer size increased to 4000 (500ms)

2009/07/10 18:28:45.810 0:34.819 Media Patch:0xef4af950 RTP Jitter buffer length exceeded
2009/07/10 18:28:45.811 0:34.820 Media Patch:0xef4af950 RTP Jitter buffer oldest packet (133120 < 153440) too late, throwing away
2009/07/10 18:28:45.811 0:34.820 Media Patch:0xef4af950 RTP Jitter buffer oldest packet (133280 < 153440) too late, throwing away
2009/07/10 18:28:45.811 0:34.820 Media Patch:0xef4af950 RTP Jitter buffer size increased to 4000 (500ms)

2009/07/10 18:28:46.051 0:35.060 Media Patch:0xef4af950 RTP Jitter buffer length exceeded
2009/07/10 18:28:46.051 0:35.060 Media Patch:0xef4af950 RTP Jit...

[Yannick Defais (sevmek)]

Hi,

Even if it is not directly related to the issue in the first log, TCP support in Ekiga will probably solve Alvin issue as well as the issue in the first log.

Thus I link the upstream bug related to TCP support here.

Best regards,
Yannick

[BrainwreckedTech (paulhinchberger3+ubuntulaunchpad)]

Thanks for looking into this at all. I wasn't expecting this to be easy. If it was something that could be solved with a few hours of pounding one's head into solid objects, I would have done that already and put the solution up on my blog, along with a thank you to the responsible party for making me a little less sane. If an Ekiga developer can't explain log #1, that means I haven't lost all sanity just yet. :)

So I guess the solution for now is to experiment with the codecs. Have the user enable codecs one-by-one on a one-and-only-one basis, keep notes, and enable only those that will work. Not the best way to do things, but a work-around is a work-around.

What I suspect is happening now:

Ekiga established UDP connection
Ekiga negotiates codecs with SIP server
Ekiga uses first mutually-agreeable codec without checking if resulting UDP packet sizes might result in a drop
Packets get dropped
Ekiga says, "Hey, I tried!"
User goes "WTF?"
Devs grumble

What I suspect will happen with TCP support:

Ekiga negotiates TCP or UDP with SIP server
If both are supported, Ekiga uses TCP because of the problems with UDP.
Ekiga negotiates codecs with SIP server
Ekiga uses first mutually-agreeable codec
Packets get fragmented instead of dropped
Ekiga works.
Both users and devs happy happy

Scenario I'm afraid of with TCP support:

Ekiga negotiates UDP or TCP with SIP server
Ekiga finds a UDP-only SIP server
Ekiga negotiates codecs with SIP server
Ekiga uses first mutually-agreeable codec without checking if resulting UDP packet sizes might result in a drop
Packets get dropped
Ekiga says, "Hey, I tried!"
Users and devs both go, "WTF?"

The addition of TCP support would definitely minimize the impact of UDP packet sizes, and admittedly knock the priority of the bug way down on the list due to that minimized impact. But someone, somewhere, is going to run into that last scenario. I'll bet an entire case of Monster (or your favorite) energy drinks on it.

[Yannick Defais (sevmek)]

Hi,

You do not need to enable just 1 codec, SIP is a negotiation mechanism; it will find an agreement if you enable several codecs. There is also priority order: in Ekiga preferences the more you push a codec on top of the list, the more it get higher priority.

The issue here is a threshold issue: the standard MTU is set to 1500 (with some exceptions). If there is too much codecs the packet size will be higher than that threshold and the packet will not reach the destination, thus preventing the negotiation leading to a failure.

Adding TCP support will remove this threshold, thus you'll be able to select all the codecs. The workaround for now is not to enable just 1 codec, the workaround is to disable a few codecs (I'm not sure about the exact number, maybe 2 or 3 will be enough).

It seems you are confusing the SIP negociation mechanism, which is clear test based and goal is to control/negotiate and the communication mechanism (the audio and video streams during a call) which use a completely different mechanism based on the RTP protocol http://en.wikipedia.org/wiki/Real-time_Transport_Protocol

AFAIK there is no issue with RTP related to MTU size. The sample rate of a codec (8 kHz, 16 kHz, etc.) is not related with your issue.

The standard define strategies for dealing with the MTU:
http://tools.ietf.org/html/rfc3261#section-18.1.1
"If a request is within 200 bytes of the path MTU, or if it is larger
   than 1300 bytes and the path MTU is unknown, the request MUST be sent
   using an RFC 2914 [43] congestion controlled transport protocol, such
   as TCP. If this causes a change in the transport protocol from the
   one indicated in the top Via, the value in the top Via MUST be
   changed. This prevents fragmentation of messages over UDP and
   provides congestion control for larger messages. However,
   implementations MUST be able to handle messages up to the maximum
   datagram packet size. For UDP, this size is 65,535 bytes, including
   IP and UDP headers."
(I will add this to the upstream bug)

Thus, as of now, Ekiga is not standard compliant in this matter. It is a hard question to have Ekiga dealing gracefully in case of a connection to a non-standard compliant client. The standard itself, which is already very complexe (SIP is way much more than a single RFC, there is many texts (i.e. RFCs) defining the standard), was written to deal with capabilities negotiation. Adding more cases in situation of non-standard compliant clients (i.e. obscure behaviour) is really hard.

As we have limited resources (really), our efforts are mainly to stick to the standard and try our best to be compliant. Make no mistake, I'm not refusing any such tricks. We have done some changes like that already in the past (i.e. changing Ekiga behaviour to deal with not fully compliant clients, Ekiga remaining compliant). My point is: this a much more complex than just let's downgrade gracefully. Change has to remain standard compliant or it will lead to a situation where it will improve the situation for some people and will be a regression for other. In practice, it is often a best effort to narrow the issue window until there is no more issue.

Best reg...