race condition using groupadd causes /etc/group to be overwritten

[Javier Sánchez]

Description: Ubuntu 12.04.2 LTS
Release: 12.04
adduser:
  Installed: 3.113ubuntu2
  Candidate: 3.113ubuntu2
  Version table:
 *** 3.113ubuntu2 0
        500 http://es.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status

The server is running under a VMWare ESXi 5.1 VM version 9. This might be relevant.

Environment
We are using ISPConfig to run a small ISP. Our current configuration uses a main or central server and several "slave" servers. Users configure their features through the administration web site and then ISPConfig distributes the changes to the apropriate servers. ISPConfig has a tool for administrators which allows them to do full synchronization of some of the features. One of these features is related to websites synchronization.

During this administrative operation, ISPConfig tries to create a user for the website and a group for the client. This is an extract of the auth.log during this operation.
Apr 16 16:58:18 fasnia groupadd[10103]: group added to /etc/group: name=client323, GID=1000
Apr 16 16:58:18 fasnia groupadd[10103]: group added to /etc/gshadow: name=client323
Apr 16 16:58:18 fasnia groupadd[10103]: new group: name=client323, GID=1000
Apr 16 16:58:20 fasnia groupadd[10170]: group added to /etc/group: name=client432, GID=1001
Apr 16 16:58:20 fasnia groupadd[10170]: group added to /etc/gshadow: name=client432
Apr 16 16:58:20 fasnia groupadd[10170]: new group: name=client432, GID=1001
Apr 16 16:58:20 fasnia groupadd[10172]: group added to /etc/group: name=client535, GID=1002
Apr 16 16:58:20 fasnia groupadd[10172]: group added to /etc/gshadow: name=client535
Apr 16 16:58:20 fasnia groupadd[10172]: new group: name=client535, GID=1002

The problem
We have detected that the /etc/group file can be destroyed during this operation. We suspect this is the result of a race condition which might happen (we weren't able to confirm this 100% but we are reasonably sure as to file a bug) when two different administrators request a synchronization.

While a single synchronization tries to issue the groupadd commands sequentially (see the auth.log extract), the second synchronization might produce that two groupadd commands might be run at the same time. The result is that the original /etc/group disappears losing all the standard groups (root, daemon, bin, sys, adm, tty...) and most of the groups added by the groupadd commands. The new /etc/group contains just some of the groups created by the groupadd commands (client323, client432, etc...).

You can imagine what kind of disaster we handled the first time. Not even with a single user boot we were able to restore the file. Using VMWare we managed to attach the disk to another machine and then copy the basic part of the /etc/group. Now we keep a copy of the /etc/group and this allows us to restore it very quickly.

And yes, it happens from time to time. We have accounted five events in the last three months.

Other remarks
There is no other write access to the /etc/group file.
Log files don't show nothing abnormal, just a set of successful groupadd commands, as reported.
There are equivalent useradd commands, but they had never resulted in a broken /etc/passwd file.

What we expect:
Independently of the number and timing of the groupadd commands, the file locking mechanism should protect /etc/group from being overwritten or destroyed as described.

We can provide any additional information upon request and, of course, testing at your will with the standard safeguards and delays of a production system.

Thanks