SCP over IPv6 address is very Slow. Takes Hours

[Prasad Tadi]

Binary package hint: openssh-server

Hi,

I am using Ubutu Hardy release 8.04 and openssh 4.7P1

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 8.04.1
Release: 8.04
Codename: hardy
root@htaf1:~#
OpenSSH_4.7p1 Debian-8ubuntu1.2, OpenSSL 0.9.8g 19 Oct 2007

When I copy a 50+ MB file from one system to another Ubuntu

Using IPv4 address it took just less than 3 sec.
Same thing over IPv6 took more than an hour ( 1 Hr and 20 Min)

I copied the file using local IPv6 address. It ran fine. May be it is using loop back, instead of real IPv6 address

When I initiate the same from a free BSD box to BSD It took less than 3 sec on both Ipv4 and IPv6.
I can also copy from Ubuntu to a FreeBSD box faster both Ipv4 and IPv6 addreses.

I just can't copy the same from a Ubutu to Ubutu or other machines to Ubuntu over IPv6 address.

It is very critical for our product :-(

I even tried getting the latest OpenSSH package from openssh.org and compiled that module and ran that sshd.
Still no use. Same old behaviour.

It starts negotiating at 1.5 Mb/ sec and slowly drops to few KB / sec and stalls the copy process during the time.

Any updates or known issues.

thanks in advance for your information on this issue.

[Bryan McLellan (btm)]

Using only linklocal addresses I've sustained near 40MB/s copying a 10GB file in the following patterns:

Ubuntu 8.10 -> Ubuntu 9.04
Debian 4.0 -> Ubuntu 9.04
Ubuntu 9.04 -> Ubuntu 8.10

It seems more likely that your problem would be a hardware issue.

Have you tried connecting the two Ubuntu machines directly together to rule out any network equipment between the two?

Have you tried to copy from your FreeBSD machine to more than one Ubuntu server to rule out a problem with that installation or hardware?

Any other testing that could possibly narrow it down? Thanks!

[Prasad Tadi (ptadi)]

Bryan,

My hardware is little more complicated, I will explain that below...
But wants to make sure that I understood "Local Link".

Hope you are not referring to IPv4 address.
scp over IPv4 is fine. No issues at all. Only when I use the IPv6, it
chokes.

scp /tmp/x.iso 172.16.109.50:/tmp completes in less than 3 sec.
So almost the same speed that you are referring to.

When I use the same two systems and use the second network card
(IPv6), then it chokes.

ex:
 scp /tmp/x.iso [a:b:c:d::e]:/tmp/ FAILS or completes in 1+ Hr :-(

So The issue I am referring is all about the IPv6 network addresses.

Coming to my hardware....

It is all on a ESX server. All these VMs ( freebsd, Ubuntu Linux
boxes are on a single ESX server with VMware VM's.)
Logically speaking they are all on a single Hardware.

I tried between Multiple VMs freeBSD to different Ubuntu Servers
( within the same ESX and also VM resides in different ESX. No issues.

When I copy from one Ubuntu to other Ubuntu server using IPv6, then
only the issues shows up.

I will attache/share some more facts / data tomorrow.

HTH
Prasad

On Apr 2, 2009, at 8:45 PM, Bryan McLellan wrote:

> Using only linklocal addresses I've sustained near 40MB/s copying a
> 10GB
> file in the following patterns:
>
> Ubuntu 8.10 -> Ubuntu 9.04
> Debian 4.0 -> Ubuntu 9.04
> Ubuntu 9.04 -> Ubuntu 8.10
>
> It seems more likely that your problem would be a hardware issue.
>
> Have you tried connecting the two Ubuntu machines directly together to
> rule out any network equipment between the two?
>
> Have you tried to copy from your FreeBSD machine to more than one
> Ubuntu
> server to rule out a problem with that installation or hardware?
>
> Any other testing that could possibly narrow it down? Thanks!
>
> --
> SCP over IPv6 address is very Slow. Takes Hours
> https://bugs.launchpad.net/bugs/352841
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in “openssh” source package in Ubuntu: New
>
> Bug description:
> Binary package hint: openssh-server
>
> Hi,
>
> I am using Ubutu Hardy release 8.04 and openssh 4.7P1
>
>
> lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description: Ubuntu 8.04.1
> Release: 8.04
> Codename: hardy
> root@htaf1:~#
> OpenSSH_4.7p1 Debian-8ubuntu1.2, OpenSSL 0.9.8g 19 Oct 2007
>
>
> When I copy a 50+ MB file from one system to another Ubuntu
>
> Using IPv4 address it took just less than 3 sec.
> Same thing over IPv6 took more than an hour ( 1 Hr and 20 Min)
>
> I copied the file using local IPv6 address. It ran fine. May be it
> is using loop back, instead of real IPv6 address
>
>
> When I initiate the same from a free BSD box to BSD It took less
> than 3 sec on both Ipv4 and IPv6.
> I can also copy from Ubuntu to a FreeBSD box faster both Ipv4 and
> IPv6 addreses.
>
> I just can't copy the same from a Ubutu to Ubutu or other machines
> to Ubuntu over IPv6 address.
>
> It is very critical for our product :-(
>
> I even tried getting the latest OpenSSH package from openssh.org
> and compiled that module and ran that sshd.
> Still no use. Same old behaviour.
>
...

[Bryan McLellan (btm)]

On Thu, Apr 2, 2009 at 7:33 PM, Prasad Tadi <email address hidden> wrote:
> My hardware is little more complicated, I will explain that below...
> But wants to make sure that I understood "Local Link".

"Link Local" is the ipv6 address assigned to the interface that starts
with fe80:: [1] You don't have to configure these addresses, and hosts
on the same physical network should be able to communicate using them.

scp -6 disk0.qcow2 \[fe80::215:c5ff:fee3:f9cd%br0.4\]:/tmp

> When I copy from one Ubuntu to other Ubuntu server using IPv6, then
> only the issues shows up.

You only see the performance problem when going from Ubuntu 8.04 to
8.04? Have you tried any combinations with other versions of Ubuntu?

[1] http://en.wikipedia.org/wiki/IPv6#Link-local_addresses_and_zone_indices

[Prasad Tadi (ptadi)]

Hi,

I am not really an expert, but I think the "Local link" may use Loop back, instead of actually testing the Network / IPv6 stack.
I have requested my network guru (james) to look at it. We are going to try with new OS also and will update. Here is some info for clarity...

James says...
Link-local is the fe80 addresses that refer to devices on a particular data link, such as a single Ethernet network, or a point-to-point serial connection. If you're using your own link-local address, then yes, that traffic will - I believe - traverse the loopback interface.
I've downloaded the beta version of Ubuntu 9 but haven't yet installed it. I've also got the most recent kernel source but keep running out of space trying to get it built and installed.
=======

Yes, I have tried earlier with locallink. It works fine.

here is my network info.
root@cspf1:~# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:50:56:b5:24:d9
          inet addr:172.16.109.90 Bcast:172.16.0.0 Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:feb5:24d9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3640571 errors:0 dropped:0 overruns:0 frame:0
          TX packets:639330 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1420069146 (1.3 GB) TX bytes:194595857 (185.5 MB)
          Base address:0x1070 Memory:f4820000-f4840000

eth1 Link encap:Ethernet HWaddr 00:50:56:b5:52:c1
          inet6 addr: fdf8:c879:493e:1:250:56ff:feb5:52c1/64 Scope:Global
          inet6 addr: fe80::250:56ff:feb5:52c1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3755999 errors:0 dropped:0 overruns:0 frame:0
          TX packets:146318 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:588116930 (560.8 MB) TX bytes:37499108 (35.7 MB)
          Base address:0x1078 Memory:f4840000-f4860000

root@cspf1:~# scp /ifs/qa/Package/cc/Hunter/branch/build-6634/desktone.tar 172.16.109.90:/tmp
The authenticity of host '172.16.109.90 (172.16.109.90)' can't be established.
RSA key fingerprint is 39:5c:3d:5d:e4:08:16:d1:0e:6e:36:f1:c4:bd:f8:b5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.109.90' (RSA) to the list of known hosts.
root@172.16.109.90's password:
desktone.tar 100% 71MB 35.3MB/s 00:02

NOW with IPv6 of eth1 network card....

scp -6 /ifs/qa/Package/cc/Hunter/branch/build-6634/desktone.tar [fdf8:c879:493e:1:250:56ff:feb5:52c1]:/tmp
root@fdf8:c879:493e:1:250:56ff:feb5:52c1's password:
desktone.tar 100% 71MB 35.3MB/s 00:02
root@cspf1:~#

Following is the IPv6 address of the next node.

I have taken different snapshots for the IPv6 failure to get some good idea.
scp -6 /ifs/qa/Package/cc/Hunter/branch/build-6634/desktone.tar [fdf8:c879:493e:1:250:56ff:feb5:68c5]:/tmp
root@fdf8:c879:493e:1:250:56ff:feb5:68c5's password:
desktone.tar ...

[Bryan McLellan (btm)]

On Fri, Apr 3, 2009 at 12:23 PM, Prasad Tadi <email address hidden> wrote:
> I am not really an expert, but I think the "Local link" may use Loop back
>
> Link-local is the fe80 addresses that refer to devices on a particular data link, such as a single Ethernet network, or a point-to-point serial connection.  If you're using your own link-local address, then yes, that traffic will - I believe - traverse the loopback interface.

This is not correct. If you read the link I provided earlier it starts
with "All interfaces have an associated link-local address". Loopback
is it's own interface (lo) and is independent of your ethN interfaces.
The loopback interface also has it's own inet6 address (ifconfig lo0 |
grep inet6) in the specified ipv6 loopback address range.

> eth1      Link encap:Ethernet  HWaddr 00:50:56:b5:52:c1
>          inet6 addr: fdf8:c879:493e:1:250:56ff:feb5:52c1/64 Scope:Global
>          inet6 addr: fe80::250:56ff:feb5:52c1/64 Scope:Link

The second address, that starts with fe80::, and additionally says
"Scope: Link" is the "Link Local" address. This is because this
address is for the local link only.

Try copying between two hosts using these address with Scope:Link to
rule out any addressing problems. Because these are local addresses,
you may need to specify the interface you want scp to use by adding
'%interface' to the end of the address, such as:

scp -6 desktone.tar [fdf8:c879:493e:1:250:56ff:feb5:52c1%eth1]:/tmp

It may be helpful to run 'sudo tshark ip6' on the originating host
while copying the file to look for any network errors.

[Prasad Tadi (ptadi)]

Bryan,

Thanks for the details....

Here is some more data following your mail.
The problem still persists...

However, when I run the command on local link (local machine's) , It
completes and not even a single trace on tshark ip6 -i eth1

But when I traced on " lo", it did show the data.

cruise@CC-branch:~/newcc/projects/branch-build/dt-RelEng$ scp
desktone.tar [fe80::250:56ff:fe8d:1ddc%eth1]:/tmp/x.tar
cruise@fe80::250:56ff:fe8d:1ddc%eth1's password:
desktone.tar
                                  100% 71MB 35.3MB/s 00:02
cruise@CC-branch:~/newcc/projects/branch-build/dt-RelEng$

# tshark ip6 -i lo
<SNIP>

  8.381329 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc SSHv2
Encrypted response packet len=48
   8.381401 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc
SSHv2 Encrypted request packet len=32
   8.382301 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc
SSHv2 Encrypted response packet len=128
   8.383901 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc
SSHv2 Encrypted request packet len=32
   8.383932 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc TCP
42214 > ssh [FIN, ACK] Seq=74203897 Ack=30225 Win=39040 Len=0
TSV=7656054 TSER=7656054
   8.419376 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc TCP
ssh > 42214 [ACK] Seq=30225 Ack=74203898 Win=852224 Len=0 TSV=7656058
TSER=7656054
   8.469483 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc TCP
ssh > 42214 [FIN, ACK] Seq=30225 Ack=74203898 Win=852224 Len=0
TSV=7656063 TSER=7656054
   8.469504 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc TCP
42214 > ssh [ACK] Seq=74203898 Ack=30226 Win=39040 Len=0 TSV=7656063
TSER=7656063

So I think though the Document suggests that it should use its own,
the OS /APP is intelligent to route the packets through LOOP BACK (lo).

Now the data while transferring to next node.....

I tried both eth1 and eth0.

First the SCP command
ruise@CC-branch:~/newcc/projects/branch-build/dt-RelEng$ scp
desktone.tar [fe80::250:56ff:fe8d:403c%eth0]:/tmp/x.tar
cruise@fe80::250:56ff:fe8d:403c%eth0's password:
desktone.tar
                                    2% 2112KB 2.1MB/s 00:33 ETA
desktone.tar
                                    2% 2112KB 1.7MB/s 00:41 ETA
desktone.tar
                                    2% 2112KB 1.5MB/s 00:45 ETA
desktone.tar
                                    2% 2112KB 1.2MB/s - stalled -
desktone.tar
                                    2% 2112KB 1.1MB/s - stalled -
desktone.tar
                                    3% 2208KB 918.8KB/s 01:16 ETA
desktone.tar
                                    3% 2208KB 744.2KB/s 01:34
ETAcruise@CC-branch:~/newcc/projects/branch-buil...

[Prasad Tadi (ptadi)]

Bryan,

Have a good news. I installed Jaunt (9.04) on a new VM and I could
push the Large size files without any issues over IPv6.

So it looks like OS/ Kernl level issue.

I can use the new machine (Jaunt) and can push to both Hardy and
Jaunt machines.

At least it solved one issue, but we may need to re-test our product
with new OS.

Thanks
Prasad

On Apr 3, 2009, at 8:18 PM, Prasad Tadi wrote:

>
> Bryan,
>
> Thanks for the details....
>
> Here is some more data following your mail.
> The problem still persists...
>
>
> However, when I run the command on local link (local machine's) , It
> completes and not even a single trace on tshark ip6 -i eth1
>
> But when I traced on " lo", it did show the data.
>
> cruise@CC-branch:~/newcc/projects/branch-build/dt-RelEng$ scp
> desktone.tar [fe80::250:56ff:fe8d:1ddc%eth1]:/tmp/x.tar
> cruise@fe80::250:56ff:fe8d:1ddc%eth1's password:
> desktone.tar
> 100% 71MB 35.3MB/s 00:02
> cruise@CC-branch:~/newcc/projects/branch-build/dt-RelEng$
>
> # tshark ip6 -i lo
> <SNIP>
>
> 8.381329 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc SSHv2
> Encrypted response packet len=48
> 8.381401 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc
> SSHv2 Encrypted request packet len=32
> 8.382301 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc
> SSHv2 Encrypted response packet len=128
> 8.383901 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc
> SSHv2 Encrypted request packet len=32
> 8.383932 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc TCP
> 42214 > ssh [FIN, ACK] Seq=74203897 Ack=30225 Win=39040 Len=0
> TSV=7656054 TSER=7656054
> 8.419376 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc TCP
> ssh > 42214 [ACK] Seq=30225 Ack=74203898 Win=852224 Len=0 TSV=7656058
> TSER=7656054
> 8.469483 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc TCP
> ssh > 42214 [FIN, ACK] Seq=30225 Ack=74203898 Win=852224 Len=0
> TSV=7656063 TSER=7656054
> 8.469504 fe80::250:56ff:fe8d:1ddc -> fe80::250:56ff:fe8d:1ddc TCP
> 42214 > ssh [ACK] Seq=74203898 Ack=30226 Win=39040 Len=0 TSV=7656063
> TSER=7656063
>
> So I think though the Document suggests that it should use its own,
> the OS /APP is intelligent to route the packets through LOOP BACK
> (lo).
>
>
> Now the data while transferring to next node.....
>
> I tried both eth1 and eth0.
>
> First the SCP command
> ruise@CC-branch:~/newcc/projects/branch-build/dt-RelEng$ scp
> desktone.tar [fe80::250:56ff:fe8d:403c%eth0]:/tmp/x.tar
> cruise@fe80::250:56ff:fe8d:403c%eth0's password:
> desktone.tar
> 2% 2112KB 2.1MB/s 00:33 ETA
> desktone.tar
> 2% 2112KB 1.7MB/s 00:41 ETA
> desktone.tar
> 2% 2112KB 1.5MB/s 00:45 ETA
> desktone.tar
> 2% 2112KB 1.2MB/s - stalled -
> desktone.tar
> 2% 2112KB 1.1MB/s - stalled -
> desktone.tar
> 3% 2208KB 918.8KB/s 01:16 ETA
> desktone.tar
> 3% 2208KB 744.2KB/s 01:34
> ETAcruise@CC-branch:~/newcc/...

[Bryan McLellan (btm)]

On Fri, Apr 3, 2009 at 5:18 PM, Prasad Tadi <email address hidden> wrote:
> However, when I run the command on local link (local machine's) , It
> completes and not even a single trace on tshark ip6 -i eth1
>
> But when I traced on  " lo",  it did show the data.

Right, because you're using the local machines link local address. You
need to use the local link address of the remote machine you want to
copy the file to.

I just built two hardy guests using python-vm-builder and ran them
under KVM using bridging. I had no problems with sustained traffic
ipv6 traffic between the two over scp using the link local addresses.

> tshark ip6 output.... on the originator.... No Errors
>
>  16.867469 fe80::250:56ff:fe8d:5c00 -> fe80::250:56ff:fe8d:403c
> SSHv2 [TCP Retransmission] Encrypted request packet len=1428

Actually, that is indicative of an error. A retransmission occurs when
the host sending the data doesn't get an acknowledgement that the
remote host received the data, so it sends it again. Either the
sending host isn't seeing them, or the receiving host isn't sending
them. This can be caused by a network or hardware problem on either
host or the equipment (or virtualized equipment) in between.

On Mon, Apr 6, 2009 at 9:35 AM, Prasad Tadi <email address hidden> wrote:
> Have a good news. I installed Jaunt (9.04) on a new VM and I could
> push the Large size files without any issues over IPv6.
>
> So it looks like OS/ Kernl level issue.

Perhaps, glad it's working for you now though.

[Chuck Short (zulcss)]

Fixed according to the user.

Regards
chuck