Recompiling with --disable-libudev gets rid of the 100% CPU problem and I can still use my smartcard.
Recompiling with --disable-libudev gets rid of the 100% CPU problem and I can still use my smartcard.