There are two schools of thought here. Some people prefer linking to the libraries directly (the saner approach for a Linux distribution) and some people prefer to load depending libraries with LoadFile (to make it easier to avoid loading to different versions of the same library in the same process). Mod_security used to take the second approach. If that has changed, or the packaging modifies it to link to libxml, then simply removing the LoadFile line is the logical thing to do.
There are two schools of thought here. Some people prefer linking to the libraries directly (the saner approach for a Linux distribution) and some people prefer to load depending libraries with LoadFile (to make it easier to avoid loading to different versions of the same library in the same process). Mod_security used to take the second approach. If that has changed, or the packaging modifies it to link to libxml, then simply removing the LoadFile line is the logical thing to do.