Here are 'perf top -p `pgrep qemu` -a` output; I met tcmalloc problem on the osd host and fix it with a larger TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES. The perf top of tcmalloc problem is a little bit different of my problem.
------------------------------------------------------------------
Samples: 15M of event 'cycles', Event count (approx.): 463535858808
87.68% [kernel] [k] _raw_spin_lock
1.02% [kernel] [k] futex_wait_setup
0.66% [kernel] [k] futex_wake
0.47% libc-2.19.so [.] malloc
0.42% [kernel] [k] try_to_wake_up
0.22% [kernel] [k] __unqueue_futex
0.22% [kernel] [k] _raw_spin_lock_irq
0.18% libpthread-2.19.so [.] pthread_mutex_lock
0.18% libc-2.19.so [.] 0x00000000000f4dff
0.17% libc-2.19.so [.] 0x00000000000ea96f
0.15% [kernel] [k] idle_cpu
0.15% libpthread-2.19.so [.] __pthread_mutex_unlock_usercnt
0.14% [kernel] [k] get_futex_value_locked
0.13% [kernel] [k] get_futex_key_refs.isra.13
0.13% libc-2.19.so [.] free
0.12% [kernel] [k] _raw_spin_lock_irqsave
0.11% [kernel] [k] update_curr
0.11% [kernel] [k] select_task_rq_fair
0.10% [kernel] [k] native_write_msr_safe
0.10% [kernel] [k] futex_wait_queue_me
0.10% [kernel] [k] __schedule
0.09% [kernel] [k] wake_futex
Press '?' for help on key bindings
----------------------------------------------------------------
Tcmalloc problem perf top (log from ceph bug tracer email, same with tcmalloc problem i met)
-------------------------------------------------------------
34.37% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::FetchFromSpans
18.06% libtcmalloc.so.4.1.2 [.] tcmalloc::ThreadCache::ReleaseToCentralCache
13.76% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseToSpans
1.45% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::RemoveRange
-------------------------------------------------------------
Here are 'perf top -p `pgrep qemu` -a` output; I met tcmalloc problem on the osd host and fix it with a larger TCMALLOC_ MAX_TOTAL_ THREAD_ CACHE_BYTES. The perf top of tcmalloc problem is a little bit different of my problem. ------- ------- ------- ------- ------- ------- ------- ------- --- mutex_unlock_ usercnt value_locked key_refs. isra.13 lock_irqsave write_msr_ safe ------- ------- ------- ------- ------- ------- ------- ------- -
-------
Samples: 15M of event 'cycles', Event count (approx.): 463535858808
87.68% [kernel] [k] _raw_spin_lock
1.02% [kernel] [k] futex_wait_setup
0.66% [kernel] [k] futex_wake
0.47% libc-2.19.so [.] malloc
0.42% [kernel] [k] try_to_wake_up
0.22% [kernel] [k] __unqueue_futex
0.22% [kernel] [k] _raw_spin_lock_irq
0.18% libpthread-2.19.so [.] pthread_mutex_lock
0.18% libc-2.19.so [.] 0x00000000000f4dff
0.17% libc-2.19.so [.] 0x00000000000ea96f
0.15% [kernel] [k] idle_cpu
0.15% libpthread-2.19.so [.] __pthread_
0.14% [kernel] [k] get_futex_
0.13% [kernel] [k] get_futex_
0.13% libc-2.19.so [.] free
0.12% [kernel] [k] _raw_spin_
0.11% [kernel] [k] update_curr
0.11% [kernel] [k] select_task_rq_fair
0.10% [kernel] [k] native_
0.10% [kernel] [k] futex_wait_queue_me
0.10% [kernel] [k] __schedule
0.09% [kernel] [k] wake_futex
Press '?' for help on key bindings
-------
Tcmalloc problem perf top (log from ceph bug tracer email, same with tcmalloc problem i met) ------- ------- ------- ------- ------- ------- ------- ----- so.4.1. 2 [.] tcmalloc: :CentralFreeLis t::FetchFromSpa ns so.4.1. 2 [.] tcmalloc: :ThreadCache: :ReleaseToCentr alCache so.4.1. 2 [.] tcmalloc: :CentralFreeLis t::ReleaseToSpa ns so.4.1. 2 [.] tcmalloc: :CentralFreeLis t::RemoveRange ------- ------- ------- ------- ------- ------- ------- -----
-------
34.37% libtcmalloc.
18.06% libtcmalloc.
13.76% libtcmalloc.
1.45% libtcmalloc.
-------