Sorry, did not notice you did have installed ath_pci. But can you compare ath5k vs. ath_pci ping performance. When I get toping google.com
PING google.com (74.125.45.100) 56(84) bytes of data.
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=1 ttl=240 time=40.7 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=2 ttl=240 time=40.4 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=3 ttl=240 time=40.2 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=4 ttl=240 time=40.1 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=5 ttl=240 time=40.2 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=6 ttl=240 time=39.9 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=7 ttl=240 time=40.3 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=8 ttl=240 time=40.1 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=9 ttl=240 time=40.3 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=10 ttl=240 time=39.9 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=11 ttl=240 time=39.7 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=12 ttl=240 time=39.8 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=13 ttl=240 time=39.7 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=14 ttl=240 time=39.8 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=15 ttl=240 time=39.8 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=16 ttl=240 time=39.7 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=17 ttl=240 time=39.9 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=18 ttl=240 time=40.2 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=19 ttl=240 time=40.8 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=20 ttl=240 time=40.2 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=21 ttl=240 time=39.7 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=22 ttl=240 time=40.0 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=23 ttl=240 time=39.7 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=24 ttl=240 time=39.7 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=25 ttl=240 time=40.0 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=26 ttl=240 time=40.1 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=27 ttl=240 time=39.7 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=28 ttl=240 time=40.1 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=29 ttl=240 time=40.2 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=30 ttl=240 time=40.2 ms
64 bytes from yx-in-f100.google.com (74.125.45.100): icmp_seq=31 ttl=240 time=40.0 ms
^C
--- google.com ping statistics ---
31 packets transmitted, 31 received, 0% packet loss, time 30044ms
rtt min/avg/max/mdev = 39.709/40.080/40.829/0.357 ms
, I most of the time see 0% loss. 

Sometimes though, I get up to 5%. I do not see much difference between them now. I had, when noticing almost 0 upload rate on ath5k. This might have been fixed by now....