ip crashes after a few times adding and removing network namespaces

[KJH]

# which ip
/sbin/ip
# valgrind ip netns add black2
==22804== Memcheck, a memory error detector
==22804== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==22804== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==22804== Command: ip netns add black2
==22804==
==22804== Invalid write of size 1
==22804== at 0x4031F43: memcpy (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==22804== by 0x8059C83: ??? (in /bin/ip)
==22804== by 0x805AF9A: netns_map_init (in /bin/ip)
==22804== by 0x805B01F: do_netns (in /bin/ip)
==22804== by 0x804DF67: ??? (in /bin/ip)
==22804== by 0x804DA11: main (in /bin/ip)
==22804== Address 0x4227094 is 0 bytes after a block of size 28 alloc'd
==22804== at 0x402D17C: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==22804== by 0x8059C5E: ??? (in /bin/ip)
==22804== by 0x805AF9A: netns_map_init (in /bin/ip)
==22804== by 0x805B01F: do_netns (in /bin/ip)
==22804== by 0x804DF67: ??? (in /bin/ip)
==22804== by 0x804DA11: main (in /bin/ip)
==22804==
Cannot create namespace file "/var/run/netns/black2": File exists
==22804==
==22804== HEAP SUMMARY:
==22804== in use at exit: 28 bytes in 1 blocks
==22804== total heap usage: 2 allocs, 1 frees, 32,824 bytes allocated
==22804==
==22804== LEAK SUMMARY:
==22804== definitely lost: 0 bytes in 0 blocks
==22804== indirectly lost: 0 bytes in 0 blocks
==22804== possibly lost: 0 bytes in 0 blocks
==22804== still reachable: 28 bytes in 1 blocks
==22804== suppressed: 0 bytes in 0 blocks
==22804== Rerun with --leak-check=full to see details of leaked memory
==22804==
==22804== For counts of detected and suppressed errors, rerun with: -v
==22804== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

[KJH (foxkjhermans)]

# uname -a
Linux charcoal 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:28 UTC 2016 i686 i686 i686 GNU/Linux

[KJH (foxkjhermans)]

# cat /etc/issue
Ubuntu 16.04.1 LTS \n \l

[KJH (foxkjhermans)]

# ip -V
ip utility, iproute2-ss151103

[Nish Aravamudan (nacc)]

Hello,

Thank you for filing this bug report! I am unable to reproduce this in 16.10, which has the same version of iproute2 currently (4.3.0-1ubuntu3). Do you have any local configuration that might be relevant?

Thanks,
Nish

[KJH (foxkjhermans)]

I have no idea if or whether the 16.10 and 16.04 versions are somehow the same. I'll see if I can write a script to reproduce the bug (which I forgot to post: this causes a segfault).

[Launchpad Janitor (janitor)]

[Expired for iproute2 (Ubuntu) because there has been no activity for 60 days.]

[KJH (foxkjhermans)]

I found the bug: in the package iproute2 version 2-4.3, in the file ip/ipnetns.c, line 175 must read:

c = malloc(sizeof(*c) + strlen(name) + 1);

instead of:

c = malloc(sizeof(*c) + strlen(name));

[KJH (foxkjhermans)]

Please reopen as I've found the cause of the bug and posted it to the forum.

2016-10-26 6:17 GMT+02:00 Launchpad Bug Tracker <email address hidden>
:

> [Expired for iproute2 (Ubuntu) because there has been no activity for 60
> days.]
>
> ** Changed in: iproute2 (Ubuntu)
> Status: Incomplete => Expired
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1615209
>
> Title:
> ip crashes after a few times adding and removing network namespaces
>
> Status in iproute2 package in Ubuntu:
> Expired
>
> Bug description:
> # which ip
> /sbin/ip
> # valgrind ip netns add black2
> ==22804== Memcheck, a memory error detector
> ==22804== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
> ==22804== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright
> info
> ==22804== Command: ip netns add black2
> ==22804==
> ==22804== Invalid write of size 1
> ==22804== at 0x4031F43: memcpy (in /usr/lib/valgrind/vgpreload_
> memcheck-x86-linux.so)
> ==22804== by 0x8059C83: ??? (in /bin/ip)
> ==22804== by 0x805AF9A: netns_map_init (in /bin/ip)
> ==22804== by 0x805B01F: do_netns (in /bin/ip)
> ==22804== by 0x804DF67: ??? (in /bin/ip)
> ==22804== by 0x804DA11: main (in /bin/ip)
> ==22804== Address 0x4227094 is 0 bytes after a block of size 28 alloc'd
> ==22804== at 0x402D17C: malloc (in /usr/lib/valgrind/vgpreload_
> memcheck-x86-linux.so)
> ==22804== by 0x8059C5E: ??? (in /bin/ip)
> ==22804== by 0x805AF9A: netns_map_init (in /bin/ip)
> ==22804== by 0x805B01F: do_netns (in /bin/ip)
> ==22804== by 0x804DF67: ??? (in /bin/ip)
> ==22804== by 0x804DA11: main (in /bin/ip)
> ==22804==
> Cannot create namespace file "/var/run/netns/black2": File exists
> ==22804==
> ==22804== HEAP SUMMARY:
> ==22804== in use at exit: 28 bytes in 1 blocks
> ==22804== total heap usage: 2 allocs, 1 frees, 32,824 bytes allocated
> ==22804==
> ==22804== LEAK SUMMARY:
> ==22804== definitely lost: 0 bytes in 0 blocks
> ==22804== indirectly lost: 0 bytes in 0 blocks
> ==22804== possibly lost: 0 bytes in 0 blocks
> ==22804== still reachable: 28 bytes in 1 blocks
> ==22804== suppressed: 0 bytes in 0 blocks
> ==22804== Rerun with --leak-check=full to see details of leaked memory
> ==22804==
> ==22804== For counts of detected and suppressed errors, rerun with: -v
> ==22804== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/
> 1615209/+subscriptions
>

[KJH (foxkjhermans)]

Well, either that or you change struct nsid_cache (line 138) to

struct nsid_cache {
        struct hlist_node nsid_hash;
        struct hlist_node name_hash;
        int nsid;
        char name[1];
};

I think that that would do the same.

[KJH (foxkjhermans)]

By the way, the bug was fixed already:

http://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/commit/?id=a1b4a274d4917be06b3060d3c8e95aba5c92a9a9

So by upgrading to ip version 4.5 this should be fixed.

[Jon Grimm (jgrimm)]

I'll take a look at this one.

[Jon Grimm (jgrimm)]

While I can visually confirm that the fix in comment #10 is now in the zesty source code (and the fix is not in the xenial/yakkety source code), I'm unable to actually recreate the experience of a valgrind failure on xenial (I've attempted with both i386 and amd64 xenial containers). I additionally rebuilt the amd64 source with -O0 -g but that did not affect the valgrind run as well.

Reporter, can you trigger this bug on the Ubuntu Xenial (16.04)? To get this accepted through the SRU approval process we should really have some testcase that actually triggers the issue.

root@x2:~# ip -V
ip utility, iproute2-ss151103
root@x2:~# valgrind ip netns add black2
==916== Memcheck, a memory error detector
==916== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==916== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==916== Command: ip netns add black2
==916==
==916==
==916== HEAP SUMMARY:
==916== in use at exit: 0 bytes in 0 blocks
==916== total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==916==
==916== All heap blocks were freed -- no leaks are possible
==916==
==916== For counts of detected and suppressed errors, rerun with: -v
==916== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
root@x2:~#

So where this sits:
1) Appears fixed in zesty
2) Needs a testcase/recreate scenario if desired fixed in Yakkety/Xenial

[KJH (foxkjhermans)]

The thing is, I rely on 'ip' a lot. So, not wanting to have to rely on code I knew was buggy, and how it was buggy, I apt-get sourced it, fixed it, and installed it on all my machines. I have now no machines with that particular bug left. My hunch is that when using network namespaces with large(r) names, adding and removing them a few times, should trigger the bug.

[Christian Ehrhardt  (paelzer)]

Hi KJH,
I appreciate all your help and I really think Jon is really trying to help.

And also from the source I agree.
But the rules dictate that one is supposed to verify the issue and the fix of it.

So a question might be "how often is often" and "how long is long".
Because for me as well e.g. the following just runs fine.
$ for i in $(seq 1 10); do echo $i; name=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1); echo "${i}:${name}"; for j in $(seq 1 100); do printf "%s " $j; sudo ip netns add ${name}; sudo ip netns del ${name}; done; printf "\n"; done;

Which is 10x100 add/removes of rather long names.

I'd have expected a segfault there, and running valgrind against them does not change a bit.

[Launchpad Janitor (janitor)]

[Expired for iproute2 (Ubuntu) because there has been no activity for 60 days.]

[Lucas Kanashiro (lucaskanashiro)]

I also tried to reproduce the mentioned failure in a clean Xenial container with no success. Since this is an old bug and no one replied to it for years I am removing the server-next tag.