I've been able to finish the analysis of the bug, this is the summary:
- nmap includes an old version of lpeg (0.12 ~Trusty/oldoldstable) in all releases (all files merged in lpeg.c)
- Debian introduced a patch that links nmap's build against an external lua-lpeg lib because lpeg is properly packaged (a hygiene measure according to Debian's maintainer)
- Upstream introduced a patch, available in B+, that fixed a FTBFS regarding lpeg (undefined reference for luaopen_lpeg())
- The version of lua-lpeg in X/B/E has a recursion error
- When both the upstream commit and the external linking patch are available, local lpeg is used
This results in:
- X fails because it uses lua-lpeg (no upstream commit in the build)
- B works because it uses local lpeg (upstream commit available)
- E is a special case in my reproducer: the debian patch removes #include "lpeg.c" so it uses the external lua-lpeg, but works because the scanned service has a fingerprint and avoids the crash. Removing the fingerprint from /usr/share/nmap/nmap-service-probes makes it crash as expected
The best way to fix this bug will be to fix the recursion error in lua-lpeg so nmap would work regardless of the version of lua-lpeg it uses.
I've been able to finish the analysis of the bug, this is the summary:
- nmap includes an old version of lpeg (0.12 ~Trusty/ oldoldstable) in all releases (all files merged in lpeg.c)
- Debian introduced a patch that links nmap's build against an external lua-lpeg lib because lpeg is properly packaged (a hygiene measure according to Debian's maintainer)
- Upstream introduced a patch, available in B+, that fixed a FTBFS regarding lpeg (undefined reference for luaopen_lpeg())
- The version of lua-lpeg in X/B/E has a recursion error
- When both the upstream commit and the external linking patch are available, local lpeg is used
This results in:
- X fails because it uses lua-lpeg (no upstream commit in the build) nmap/nmap- service- probes makes it crash as expected
- B works because it uses local lpeg (upstream commit available)
- E is a special case in my reproducer: the debian patch removes #include "lpeg.c" so it uses the external lua-lpeg, but works because the scanned service has a fingerprint and avoids the crash. Removing the fingerprint from /usr/share/
The best way to fix this bug will be to fix the recursion error in lua-lpeg so nmap would work regardless of the version of lua-lpeg it uses.