ip crashes after a few times adding and removing network namespaces

Bug #1615209 reported by KJH
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
iproute2 (Ubuntu)
Expired
High
Unassigned

Bug Description

# which ip
/sbin/ip
# valgrind ip netns add black2
==22804== Memcheck, a memory error detector
==22804== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==22804== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==22804== Command: ip netns add black2
==22804==
==22804== Invalid write of size 1
==22804== at 0x4031F43: memcpy (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==22804== by 0x8059C83: ??? (in /bin/ip)
==22804== by 0x805AF9A: netns_map_init (in /bin/ip)
==22804== by 0x805B01F: do_netns (in /bin/ip)
==22804== by 0x804DF67: ??? (in /bin/ip)
==22804== by 0x804DA11: main (in /bin/ip)
==22804== Address 0x4227094 is 0 bytes after a block of size 28 alloc'd
==22804== at 0x402D17C: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==22804== by 0x8059C5E: ??? (in /bin/ip)
==22804== by 0x805AF9A: netns_map_init (in /bin/ip)
==22804== by 0x805B01F: do_netns (in /bin/ip)
==22804== by 0x804DF67: ??? (in /bin/ip)
==22804== by 0x804DA11: main (in /bin/ip)
==22804==
Cannot create namespace file "/var/run/netns/black2": File exists
==22804==
==22804== HEAP SUMMARY:
==22804== in use at exit: 28 bytes in 1 blocks
==22804== total heap usage: 2 allocs, 1 frees, 32,824 bytes allocated
==22804==
==22804== LEAK SUMMARY:
==22804== definitely lost: 0 bytes in 0 blocks
==22804== indirectly lost: 0 bytes in 0 blocks
==22804== possibly lost: 0 bytes in 0 blocks
==22804== still reachable: 28 bytes in 1 blocks
==22804== suppressed: 0 bytes in 0 blocks
==22804== Rerun with --leak-check=full to see details of leaked memory
==22804==
==22804== For counts of detected and suppressed errors, rerun with: -v
==22804== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Revision history for this message
KJH (foxkjhermans) wrote :

# uname -a
Linux charcoal 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:28 UTC 2016 i686 i686 i686 GNU/Linux

Revision history for this message
KJH (foxkjhermans) wrote :

# cat /etc/issue
Ubuntu 16.04.1 LTS \n \l

Revision history for this message
KJH (foxkjhermans) wrote :

# ip -V
ip utility, iproute2-ss151103

Revision history for this message
Nish Aravamudan (nacc) wrote :

Hello,

Thank you for filing this bug report! I am unable to reproduce this in 16.10, which has the same version of iproute2 currently (4.3.0-1ubuntu3). Do you have any local configuration that might be relevant?

Thanks,
Nish

Changed in iproute2 (Ubuntu):
status: New → Incomplete
Revision history for this message
KJH (foxkjhermans) wrote :

I have no idea if or whether the 16.10 and 16.04 versions are somehow the same. I'll see if I can write a script to reproduce the bug (which I forgot to post: this causes a segfault).

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for iproute2 (Ubuntu) because there has been no activity for 60 days.]

Changed in iproute2 (Ubuntu):
status: Incomplete → Expired
Revision history for this message
KJH (foxkjhermans) wrote :

I found the bug: in the package iproute2 version 2-4.3, in the file ip/ipnetns.c, line 175 must read:

c = malloc(sizeof(*c) + strlen(name) + 1);

instead of:

c = malloc(sizeof(*c) + strlen(name));

Changed in iproute2 (Ubuntu):
status: Expired → In Progress
Revision history for this message
KJH (foxkjhermans) wrote : Re: [Bug 1615209] Re: ip crashes after a few times adding and removing network namespaces

Please reopen as I've found the cause of the bug and posted it to the forum.

2016-10-26 6:17 GMT+02:00 Launchpad Bug Tracker <email address hidden>
:

> [Expired for iproute2 (Ubuntu) because there has been no activity for 60
> days.]
>
> ** Changed in: iproute2 (Ubuntu)
> Status: Incomplete => Expired
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1615209
>
> Title:
> ip crashes after a few times adding and removing network namespaces
>
> Status in iproute2 package in Ubuntu:
> Expired
>
> Bug description:
> # which ip
> /sbin/ip
> # valgrind ip netns add black2
> ==22804== Memcheck, a memory error detector
> ==22804== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
> ==22804== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright
> info
> ==22804== Command: ip netns add black2
> ==22804==
> ==22804== Invalid write of size 1
> ==22804== at 0x4031F43: memcpy (in /usr/lib/valgrind/vgpreload_
> memcheck-x86-linux.so)
> ==22804== by 0x8059C83: ??? (in /bin/ip)
> ==22804== by 0x805AF9A: netns_map_init (in /bin/ip)
> ==22804== by 0x805B01F: do_netns (in /bin/ip)
> ==22804== by 0x804DF67: ??? (in /bin/ip)
> ==22804== by 0x804DA11: main (in /bin/ip)
> ==22804== Address 0x4227094 is 0 bytes after a block of size 28 alloc'd
> ==22804== at 0x402D17C: malloc (in /usr/lib/valgrind/vgpreload_
> memcheck-x86-linux.so)
> ==22804== by 0x8059C5E: ??? (in /bin/ip)
> ==22804== by 0x805AF9A: netns_map_init (in /bin/ip)
> ==22804== by 0x805B01F: do_netns (in /bin/ip)
> ==22804== by 0x804DF67: ??? (in /bin/ip)
> ==22804== by 0x804DA11: main (in /bin/ip)
> ==22804==
> Cannot create namespace file "/var/run/netns/black2": File exists
> ==22804==
> ==22804== HEAP SUMMARY:
> ==22804== in use at exit: 28 bytes in 1 blocks
> ==22804== total heap usage: 2 allocs, 1 frees, 32,824 bytes allocated
> ==22804==
> ==22804== LEAK SUMMARY:
> ==22804== definitely lost: 0 bytes in 0 blocks
> ==22804== indirectly lost: 0 bytes in 0 blocks
> ==22804== possibly lost: 0 bytes in 0 blocks
> ==22804== still reachable: 28 bytes in 1 blocks
> ==22804== suppressed: 0 bytes in 0 blocks
> ==22804== Rerun with --leak-check=full to see details of leaked memory
> ==22804==
> ==22804== For counts of detected and suppressed errors, rerun with: -v
> ==22804== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/
> 1615209/+subscriptions
>

Revision history for this message
KJH (foxkjhermans) wrote :

Well, either that or you change struct nsid_cache (line 138) to

struct nsid_cache {
        struct hlist_node nsid_hash;
        struct hlist_node name_hash;
        int nsid;
        char name[1];
};

I think that that would do the same.

Revision history for this message
KJH (foxkjhermans) wrote :

By the way, the bug was fixed already:

http://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/commit/?id=a1b4a274d4917be06b3060d3c8e95aba5c92a9a9

So by upgrading to ip version 4.5 this should be fixed.

Robie Basak (racb)
Changed in iproute2 (Ubuntu):
status: In Progress → Triaged
importance: Undecided → High
tags: added: server-next
Revision history for this message
Jon Grimm (jgrimm) wrote :

I'll take a look at this one.

Revision history for this message
Jon Grimm (jgrimm) wrote :

While I can visually confirm that the fix in comment #10 is now in the zesty source code (and the fix is not in the xenial/yakkety source code), I'm unable to actually recreate the experience of a valgrind failure on xenial (I've attempted with both i386 and amd64 xenial containers). I additionally rebuilt the amd64 source with -O0 -g but that did not affect the valgrind run as well.

Reporter, can you trigger this bug on the Ubuntu Xenial (16.04)? To get this accepted through the SRU approval process we should really have some testcase that actually triggers the issue.

root@x2:~# ip -V
ip utility, iproute2-ss151103
root@x2:~# valgrind ip netns add black2
==916== Memcheck, a memory error detector
==916== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==916== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==916== Command: ip netns add black2
==916==
==916==
==916== HEAP SUMMARY:
==916== in use at exit: 0 bytes in 0 blocks
==916== total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==916==
==916== All heap blocks were freed -- no leaks are possible
==916==
==916== For counts of detected and suppressed errors, rerun with: -v
==916== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
root@x2:~#

So where this sits:
1) Appears fixed in zesty
2) Needs a testcase/recreate scenario if desired fixed in Yakkety/Xenial

Changed in iproute2 (Ubuntu):
status: Triaged → Incomplete
Revision history for this message
KJH (foxkjhermans) wrote :

The thing is, I rely on 'ip' a lot. So, not wanting to have to rely on code I knew was buggy, and how it was buggy, I apt-get sourced it, fixed it, and installed it on all my machines. I have now no machines with that particular bug left. My hunch is that when using network namespaces with large(r) names, adding and removing them a few times, should trigger the bug.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi KJH,
I appreciate all your help and I really think Jon is really trying to help.

And also from the source I agree.
But the rules dictate that one is supposed to verify the issue and the fix of it.

So a question might be "how often is often" and "how long is long".
Because for me as well e.g. the following just runs fine.
$ for i in $(seq 1 10); do echo $i; name=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1); echo "${i}:${name}"; for j in $(seq 1 100); do printf "%s " $j; sudo ip netns add ${name}; sudo ip netns del ${name}; done; printf "\n"; done;

Which is 10x100 add/removes of rather long names.

I'd have expected a segfault there, and running valgrind against them does not change a bit.

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for iproute2 (Ubuntu) because there has been no activity for 60 days.]

Changed in iproute2 (Ubuntu):
status: Incomplete → Expired
Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

I also tried to reproduce the mentioned failure in a clean Xenial container with no success. Since this is an old bug and no one replied to it for years I am removing the server-next tag.

tags: removed: server-next
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.