In an attempt to figure out the issue I decided to change the IO scheduler thinking it might help considering the contents of the trace. I set one group of nodes to noop and another to deadline. I have seen panics on both groups of machines since doing so. From the (partial) traces I've gotten from those machines the noop trace looks quite a bit different while the deadline trace looks pretty similar with lots of bits regarding xfs.
In an attempt to figure out the issue I decided to change the IO scheduler thinking it might help considering the contents of the trace. I set one group of nodes to noop and another to deadline. I have seen panics on both groups of machines since doing so. From the (partial) traces I've gotten from those machines the noop trace looks quite a bit different while the deadline trace looks pretty similar with lots of bits regarding xfs.
Screenshots: img.skitch. com/20100816- buaaqf6ggdfp6m8 y41x4wfyhfy. jpg img.skitch. com/20100816- dnwh8sijt8jnck5 k18ewrdcdeu. jpg
http://
http://
Unfortunately the console in the remote management card cuts the top of the trace off.