I get this every couple of days on our CI - from processes which access the GPU. nvidia-driver 396.24.02. Here are some dmesg warning from various failures

[Mon Nov  5 18:43:29 2018] INFO: task kworker/4:2:25281 blocked for more than 120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic #153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Nov  5 18:43:29 2018] kworker/4:2     D ffff88026a5cbb68     0 25281      2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff88026a5cbb68 0000000000000036 ffff880827175a00 ffff88026a5cc000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 ffff880827175a00
[Mon Nov  5 18:43:29 2018]  ffff88026a5cbb80 ffffffff81818105 7fffffffffffffff ffff88026a5cbc28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b73bf>] ? enqueue_entity+0x3af/0xbe0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b33978>] ? _nv007828rm+0x38/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3701>] ? kmem_cache_alloc+0x191/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task kworker/4:1:3562 blocked for more than 120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic #153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Nov  5 18:43:29 2018] kworker/4:1     D ffff880104db3b68     0  3562      2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff880104db3b68 ffffffff81817b46 ffff88044e1f8f00 ffff880104db4000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 ffff88044e1f8f00
[Mon Nov  5 18:43:29 2018]  ffff880104db3b80 ffffffff81818105 7fffffffffffffff ffff880104db3c28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b33978>] ? _nv007828rm+0x38/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3700>] ? kmem_cache_alloc+0x190/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task kworker/0:3:16616 blocked for more than 120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic #153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Nov  5 18:43:29 2018] kworker/0:3     D ffff8805f97a3b68     0 16616      2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff8805f97a3b68 ffffffff81817b46 ffff8805f5952d00 ffff8805f97a4000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 ffff8805f5952d00
[Mon Nov  5 18:43:29 2018]  ffff8805f97a3b80 ffffffff81818105 7fffffffffffffff ffff8805f97a3c28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b33978>] ? _nv007828rm+0x38/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3701>] ? kmem_cache_alloc+0x191/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task kworker/0:4:16617 blocked for more than 120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic #153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Nov  5 18:43:29 2018] kworker/0:4     D ffff8805f978bb68     0 16617      2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff8805f978bb68 ffffffff810b73bf ffff8805f5951e00 ffff8805f978c000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 ffff8805f5951e00
[Mon Nov  5 18:43:29 2018]  ffff8805f978bb80 ffffffff81818105 7fffffffffffffff ffff8805f978bc28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b73bf>] ? enqueue_entity+0x3af/0xbe0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b8c6d>] ? check_preempt_wakeup+0x12d/0x210
[Mon Nov  5 18:43:29 2018]  [<ffffffff810aa577>] ? check_preempt_curr+0x87/0x90
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b33978>] ? _nv007828rm+0x38/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3700>] ? kmem_cache_alloc+0x190/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task py.test:8373 blocked for more than 120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic #153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Nov  5 18:43:29 2018] py.test         D ffff880377c1bb68     0  8373   8372 0x00000000
[Mon Nov  5 18:43:29 2018]  ffff880377c1bb68 0000000000000000 ffff88071774cb00 ffff880377c1c000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 ffff88071774cb00
[Mon Nov  5 18:43:29 2018]  ffff880377c1bb80 ffffffff81818105 7fffffffffffffff ffff880377c1bc28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725100>] ? os_free_semaphore+0x10/0x10 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cc6008>] ? _nv007547rm+0xa8/0x150 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d5a7c0>] ? _nv001114rm+0x4c0/0x850 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d656f3>] ? rm_ioctl+0x73/0x100 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc071d601>] ? nvidia_ioctl+0x91/0x6c0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc071d601>] ? nvidia_ioctl+0x91/0x6c0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc071db11>] ? nvidia_ioctl+0x5a1/0x6c0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0719083>] ? nvidia_frontend_unlocked_ioctl+0x43/0x50 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81219157>] ? do_vfs_ioctl+0x2e7/0x4d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c8ca7>] ? percpu_up_read+0x17/0x40
[Mon Nov  5 18:43:29 2018]  [<ffffffff81208131>] ? __sb_end_write+0x21/0x30
[Mon Nov  5 18:43:29 2018]  [<ffffffff81206195>] ? vfs_write+0x155/0x1a0
[Mon Nov  5 18:43:29 2018]  [<ffffffff812193b9>] ? SyS_ioctl+0x79/0x90
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181ba08>] ? entry_SYSCALL_64_fastpath+0x1c/0xbb
[Mon Nov  5 18:43:29 2018] INFO: task py.test:8603 blocked for more than 120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic #153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Nov  5 18:43:29 2018] py.test         D ffff88011416bcc8     0  8603   8372 0x00000000
[Mon Nov  5 18:43:29 2018]  ffff88011416bcc8 ffff88011416bdb0 ffff88008f51da00 ffff88011416c000
[Mon Nov  5 18:43:29 2018]  ffff880825b56638 0000000000000002 0000000000000000 ffff88008f51da00
[Mon Nov  5 18:43:29 2018]  ffff88011416bce0 ffffffff81818105 7fffffffffffffff ffff88011416bd88
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810fe25d>] ? futex_wait+0x1ad/0x270
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc071d5d0>] nvidia_ioctl+0x60/0x6c0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b46>] ? __schedule+0x296/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0719083>] nvidia_frontend_unlocked_ioctl+0x43/0x50 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81219157>] do_vfs_ioctl+0x2e7/0x4d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b86>] ? __schedule+0x2d6/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff811009b1>] ? SyS_futex+0x71/0x150
[Mon Nov  5 18:43:29 2018]  [<ffffffff812193b9>] SyS_ioctl+0x79/0x90
[Mon Nov  5 18:43:29 2018]  [<ffffffff810eeb21>] ? posix_get_monotonic_raw+0x11/0x20
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181ba08>] entry_SYSCALL_64_fastpath+0x1c/0xbb
[Mon Nov  5 18:43:29 2018] INFO: task kworker/0:0:8594 blocked for more than 120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic #153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Nov  5 18:43:29 2018] kworker/0:0     D ffff8804c115fb68     0  8594      2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff8804c115fb68 ffff8804c115fb70 ffff880823618000 ffff8804c1160000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 ffff880823618000
[Mon Nov  5 18:43:29 2018]  ffff8804c115fb80 ffffffff81818105 7fffffffffffffff ffff8804c115fc28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b73bf>] ? enqueue_entity+0x3af/0xbe0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b475fc>] ? _nv007760rm+0x3c/0x180 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3701>] ? kmem_cache_alloc+0x191/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task kworker/0:2:14982 blocked for more than 120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic #153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Nov  5 18:43:29 2018] kworker/0:2     D ffff880101fb7b68     0 14982      2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff880101fb7b68 ffff880101fb7b70 ffff8807025c6900 ffff880101fb8000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 ffff8807025c6900
[Mon Nov  5 18:43:29 2018]  ffff880101fb7b80 ffffffff81818105 7fffffffffffffff ffff880101fb7c28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b5b40>] ? update_curr+0x80/0x170
[Mon Nov  5 18:43:29 2018]  [<ffffffff8104f870>] ? native_smp_send_reschedule+0x50/0x70
[Mon Nov  5 18:43:29 2018]  [<ffffffff810a9b06>] ? resched_curr+0xb6/0xd0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b33978>] ? _nv007828rm+0x38/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3700>] ? kmem_cache_alloc+0x190/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task kworker/4:0:15088 blocked for more than 120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic #153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Nov  5 18:43:29 2018] kworker/4:0     D ffff88010d7afb68     0 15088      2 0x00000000
[Mon Nov  5 18:43:29 2018] Workqueue: events os_execute_work_item [nvidia]
[Mon Nov  5 18:43:29 2018]  ffff88010d7afb68 ffff88010d7afb70 ffff880716fc0000 ffff88010d7b0000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 ffff880716fc0000
[Mon Nov  5 18:43:29 2018]  ffff88010d7afb80 ffffffff81818105 7fffffffffffffff ffff88010d7afc28
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b5b40>] ? update_curr+0x80/0x170
[Mon Nov  5 18:43:29 2018]  [<ffffffff8104f870>] ? native_smp_send_reschedule+0x50/0x70
[Mon Nov  5 18:43:29 2018]  [<ffffffff810a9b06>] ? resched_curr+0xb6/0xd0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0b33978>] ? _nv007828rm+0x38/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d62ad4>] ? _nv001065rm+0x84/0xe0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d663f9>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff811e3700>] ? kmem_cache_alloc+0x190/0x200
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725101>] ? os_execute_work_item+0x1/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725146>] ? os_execute_work_item+0x46/0x70 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099716>] ? process_one_work+0x156/0x400
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109a0fa>] ? worker_thread+0x11a/0x480
[Mon Nov  5 18:43:29 2018]  [<ffffffff81099fe0>] ? rescuer_thread+0x310/0x310
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f5d8>] ? kthread+0xd8/0xf0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81817b52>] ? __schedule+0x2a2/0x820
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181be75>] ? ret_from_fork+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8109f500>] ? kthread_park+0x60/0x60
[Mon Nov  5 18:43:29 2018] INFO: task ffmpeg:15818 blocked for more than 120 seconds.
[Mon Nov  5 18:43:29 2018]       Tainted: P           OE   4.4.0-127-generic #153~14.04.1-Ubuntu
[Mon Nov  5 18:43:29 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Nov  5 18:43:29 2018] ffmpeg          D ffff88012f2cf958     0 15818  15825 0x00000000
[Mon Nov  5 18:43:29 2018]  ffff88012f2cf958 ffffffff810b73bf ffff880114a2da00 ffff88012f2d0000
[Mon Nov  5 18:43:29 2018]  ffff88082370a768 0000000000000002 0000000000000000 ffff880114a2da00
[Mon Nov  5 18:43:29 2018]  ffff88012f2cf970 ffffffff81818105 7fffffffffffffff ffff88012f2cfa18
[Mon Nov  5 18:43:29 2018] Call Trace:
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b73bf>] ? enqueue_entity+0x3af/0xbe0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81818105>] schedule+0x35/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181aafb>] schedule_timeout+0x23b/0x2d0
[Mon Nov  5 18:43:29 2018]  [<ffffffff810b8c89>] ? check_preempt_wakeup+0x149/0x210
[Mon Nov  5 18:43:29 2018]  [<ffffffff810aa577>] ? check_preempt_curr+0x87/0x90
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819d85>] __down_common+0xa6/0xf9
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0725100>] ? os_free_semaphore+0x10/0x10 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff81819df5>] __down+0x1d/0x1f
[Mon Nov  5 18:43:29 2018]  [<ffffffff810c88e1>] down+0x41/0x50
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0724d97>] os_acquire_mutex+0x37/0x40 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0cdb9fc>] _nv031564rm+0x5c/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0d68960>] ? rm_get_gpu_uuid_raw+0x70/0x120 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff810ab555>] ? wake_up_process+0x15/0x20
[Mon Nov  5 18:43:29 2018]  [<ffffffffc071aae2>] ? nv_open_device+0x4c2/0x700 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc071b118>] ? nvidia_open+0x148/0x2f0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffffc0719388>] ? nvidia_frontend_open+0x58/0xa0 [nvidia]
[Mon Nov  5 18:43:29 2018]  [<ffffffff8120a884>] ? chrdev_open+0xb4/0x190
[Mon Nov  5 18:43:29 2018]  [<ffffffff81203d2a>] ? do_dentry_open+0x22a/0x320
[Mon Nov  5 18:43:29 2018]  [<ffffffff8120a7d0>] ? cdev_put+0x30/0x30
[Mon Nov  5 18:43:29 2018]  [<ffffffff81204df5>] ? vfs_open+0x55/0x80
[Mon Nov  5 18:43:29 2018]  [<ffffffff81210d97>] ? may_open+0x57/0xe0
[Mon Nov  5 18:43:29 2018]  [<ffffffff8121328d>] ? path_openat+0x1ad/0x12e0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81221d69>] ? atime_needs_update+0x69/0xd0
[Mon Nov  5 18:43:29 2018]  [<ffffffff8121602e>] ? do_filp_open+0x7e/0xd0
[Mon Nov  5 18:43:29 2018]  [<ffffffff81223486>] ? __alloc_fd+0x46/0x180
[Mon Nov  5 18:43:29 2018]  [<ffffffff81205179>] ? do_sys_open+0x129/0x270
[Mon Nov  5 18:43:29 2018]  [<ffffffff812052de>] ? SyS_open+0x1e/0x20
[Mon Nov  5 18:43:29 2018]  [<ffffffff8181ba08>] ? entry_SYSCALL_64_fastpath+0x1c/0xbb