kernel error running mosquitto_pub for extended period
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Raspbian |
Expired
|
Undecided
|
Unassigned | ||
mosquitto |
Invalid
|
Undecided
|
Unassigned |
Bug Description
I am running mosquitto 0.15 on my RaspberryPi running Raspbian. I am constantly using MQTT approx 20 pubs per minute.
After a period of time - usually 3/4/5 days the OS throws the following errors around mosquitto
Aug 5 08:16:17 raspberrypi kernel: [ 3050.821012] Modules linked in: snd_bcm2835 snd_pcm snd_page_alloc snd_seq snd_seq_device snd_timer snd pl2303 evdev usbserial
Aug 5 08:16:17 raspberrypi kernel: [ 3050.830793] CPU: 0 Not tainted (3.1.9+ #171)
Aug 5 08:16:17 raspberrypi kernel: [ 3050.840548] PC is at inet_dump_
Aug 5 08:16:17 raspberrypi kernel: [ 3050.849992] LR is at add_preempt_
Aug 5 08:16:18 raspberrypi kernel: [ 3050.859253] pc : [<c03a2a78>] lr : [<c03fe868>] psr: 60000013
Aug 5 08:16:18 raspberrypi kernel: [ 3050.859266] sp : c3fe7c58 ip : c3fe7c48 fp : c3fe7ca4
Aug 5 08:16:18 raspberrypi kernel: [ 3050.877527] r10: cdaf3480 r9 : cdaf3480 r8 : 00000000
Aug 5 08:16:18 raspberrypi kernel: [ 3050.886282] r7 : 00000001 r6 : cc57ec60 r5 : 00000001 r4 : cd8b0c00
Aug 5 08:16:18 raspberrypi kernel: [ 3050.894922] r3 : c05cae94 r2 : 00000000 r1 : 00000000 r0 : 311a4880
Aug 5 08:16:18 raspberrypi kernel: [ 3050.903484] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
Aug 5 08:16:18 raspberrypi kernel: [ 3050.911843] Control: 00c5387d Table: 0c60c008 DAC: 00000015
Aug 5 08:16:18 raspberrypi kernel: [ 3051.176035] [<c03a2a78>] (inet_dump_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.183757] [<c035a4f0>] (rtnl_dump_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.191333] [<c036893c>] (netlink_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.199048] [<c0368cdc>] (netlink_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.206680] [<c0336ba4>] (sock_recvmsg+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.214175] [<c0336870>] (__sys_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.221646] [<c0338dec>] (sys_recvmsg+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.253777] ---[ end trace ec7eb1d2e181cb2b ]---
Aug 5 08:16:18 raspberrypi kernel: [ 3051.261429] note: mosquitto_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.276733] Modules linked in: snd_bcm2835 snd_pcm snd_page_alloc snd_seq snd_seq_device snd_timer snd pl2303 evdev usbserial
Aug 5 08:16:18 raspberrypi kernel: [ 3051.285423] [<c00153d4>] (unwind_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.293245] [<c03f95c0>] (dump_stack+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.301188] [<c00216d0>] (__schedule_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.309078] [<c03f9e28>] (__schedule+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.317061] [<c0021fb0>] (__cond_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.324946] [<c03f9fe4>] (_cond_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.332695] [<c00ded60>] (unmap_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.340556] [<c00e4c24>] (exit_mmap+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.348391] [<c002b078>] (mmput+0x44/0x118) from [<c002fb94>] (exit_mm+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.356182] [<c002fb94>] (exit_mm+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.363831] [<c0031a40>] (do_exit+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.371623] [<c001277c>] (die+0x2c4/0x388) from [<c0019054>] (__do_kernel_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.379390] [<c0019054>] (__do_kernel_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.387080] [<c00190c0>] (do_bad_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.394771] [<c03fe788>] (do_translation
Aug 5 08:16:18 raspberrypi kernel: [ 3051.402415] [<c000836c>] (do_DataAbort+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.410109] Exception stack(0xc3fe7c10 to 0xc3fe7c58)
Aug 5 08:16:18 raspberrypi kernel: [ 3051.417709] 7c00: 311a4880 00000000 00000000 c05cae94
Aug 5 08:16:18 raspberrypi kernel: [ 3051.425383] 7c20: cd8b0c00 00000001 cc57ec60 00000001 00000000 cdaf3480 cdaf3480 c3fe7ca4
Aug 5 08:16:18 raspberrypi kernel: [ 3051.432950] 7c40: c3fe7c48 c3fe7c58 c03fe868 c03a2a78 60000013 ffffffff
Aug 5 08:16:18 raspberrypi kernel: [ 3051.440677] [<c03fc85c>] (__dabt_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.448562] [<c03a2a78>] (inet_dump_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.456467] [<c035a4f0>] (rtnl_dump_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.464216] [<c036893c>] (netlink_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.472088] [<c0368cdc>] (netlink_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.480008] [<c0336ba4>] (sock_recvmsg+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.487945] [<c0336870>] (__sys_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.495950] [<c0338dec>] (sys_recvmsg+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.548242] ------------[ cut here ]------------
Aug 5 08:16:18 raspberrypi kernel: [ 3051.556342] WARNING: at net/netlink/
Aug 5 08:16:18 raspberrypi kernel: [ 3051.564090] Modules linked in: snd_bcm2835 snd_pcm snd_page_alloc snd_seq snd_seq_device snd_timer snd pl2303 evdev usbserial
Aug 5 08:16:18 raspberrypi kernel: [ 3051.572847] [<c00153d4>] (unwind_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.581089] [<c03f95c0>] (dump_stack+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.589403] [<c002d828>] (warn_slowpath_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.597765] [<c002d86c>] (warn_slowpath_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.606175] [<c0368fac>] (netlink_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.614392] [<c033b154>] (__sk_free+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.622713] [<c033b368>] (sk_free+0x38/0x3c) from [<c036a600>] (netlink_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.631071] [<c036a600>] (netlink_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.639533] [<c03377e8>] (sock_release+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.648018] [<c03378a0>] (sock_close+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.656336] [<c00f82f0>] (fput+0xc0/0x218) from [<c00f4730>] (filp_close+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.664659] [<c00f4730>] (filp_close+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.672876] [<c003139c>] (put_files_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.681296] [<c0031484>] (exit_files+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.689716] [<c00315e4>] (do_exit+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.698146] [<c001277c>] (die+0x2c4/0x388) from [<c0019054>] (__do_kernel_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.706512] [<c0019054>] (__do_kernel_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.714810] [<c00190c0>] (do_bad_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.723001] [<c03fe788>] (do_translation
Aug 5 08:16:18 raspberrypi kernel: [ 3051.731318] [<c000836c>] (do_DataAbort+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.739613] Exception stack(0xc3fe7c10 to 0xc3fe7c58)
Aug 5 08:16:18 raspberrypi kernel: [ 3051.747850] 7c00: 311a4880 00000000 00000000 c05cae94
Aug 5 08:16:18 raspberrypi kernel: [ 3051.756155] 7c20: cd8b0c00 00000001 cc57ec60 00000001 00000000 cdaf3480 cdaf3480 c3fe7ca4
Aug 5 08:16:18 raspberrypi kernel: [ 3051.764285] 7c40: c3fe7c48 c3fe7c58 c03fe868 c03a2a78 60000013 ffffffff
Aug 5 08:16:18 raspberrypi kernel: [ 3051.772573] [<c03fc85c>] (__dabt_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.780946] [<c03a2a78>] (inet_dump_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.789338] [<c035a4f0>] (rtnl_dump_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.797826] [<c036893c>] (netlink_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.806298] [<c0368cdc>] (netlink_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.814677] [<c0336ba4>] (sock_recvmsg+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.822900] [<c0336870>] (__sys_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.831268] [<c0338dec>] (sys_recvmsg+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.839540] ---[ end trace ec7eb1d2e181cb2c ]---
Aug 5 08:16:18 raspberrypi kernel: [ 3051.856101] Modules linked in: snd_bcm2835 snd_pcm snd_page_alloc snd_seq snd_seq_device snd_timer snd pl2303 evdev usbserial
Aug 5 08:16:18 raspberrypi kernel: [ 3051.865179] [<c00153d4>] (unwind_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.873619] [<c03f95c0>] (dump_stack+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.882102] [<c00216d0>] (__schedule_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.890577] [<c03f9e28>] (__schedule+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.899043] [<c0021fb0>] (__cond_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.907563] [<c03f9fe4>] (_cond_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.916082] [<c00313a0>] (put_files_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.924417] [<c0031484>] (exit_files+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.932960] [<c00315e4>] (do_exit+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.941463] [<c001277c>] (die+0x2c4/0x388) from [<c0019054>] (__do_kernel_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.949952] [<c0019054>] (__do_kernel_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.958446] [<c00190c0>] (do_bad_
Aug 5 08:16:18 raspberrypi kernel: [ 3051.966868] [<c03fe788>] (do_translation
Aug 5 08:16:18 raspberrypi kernel: [ 3051.975203] [<c000836c>] (do_DataAbort+
Aug 5 08:16:18 raspberrypi kernel: [ 3051.983383] Exception stack(0xc3fe7c10 to 0xc3fe7c58)
Aug 5 08:16:18 raspberrypi kernel: [ 3051.991557] 7c00: 311a4880 00000000 00000000 c05cae94
Aug 5 08:16:18 raspberrypi kernel: [ 3051.999887] 7c20: cd8b0c00 00000001 cc57ec60 00000001 00000000 cdaf3480 cdaf3480 c3fe7ca4
Aug 5 08:16:18 raspberrypi kernel: [ 3052.008159] 7c40: c3fe7c48 c3fe7c58 c03fe868 c03a2a78 60000013 ffffffff
Aug 5 08:16:18 raspberrypi kernel: [ 3052.016451] [<c03fc85c>] (__dabt_
Aug 5 08:16:18 raspberrypi kernel: [ 3052.024768] [<c03a2a78>] (inet_dump_
Aug 5 08:16:18 raspberrypi kernel: [ 3052.033023] [<c035a4f0>] (rtnl_dump_
Aug 5 08:16:18 raspberrypi kernel: [ 3052.041342] [<c036893c>] (netlink_
Aug 5 08:16:18 raspberrypi kernel: [ 3052.049708] [<c0368cdc>] (netlink_
Aug 5 08:16:18 raspberrypi kernel: [ 3052.058003] [<c0336ba4>] (sock_recvmsg+
Aug 5 08:16:18 raspberrypi kernel: [ 3052.066335] [<c0336870>] (__sys_
Aug 5 08:16:18 raspberrypi kernel: [ 3052.074447] [<c0338dec>] (sys_recvmsg+
A reboot is required to get the OS fully working again.
summary: |
- mosquitto_pub[22867] exited with preempt_count 1 + kernel error running mosquitto_pub for extended period |
That sounds quite serious. I have to be honest and say that I don't see there is any way that anything mosquitto related could be affecting the kernel like this. What is more likely is that what you're doing with mosquitto is exposing a bug in the system or a memory problem perhaps.
I know some people with RPis though, so I might be able to persuade them to help reproduce the problem. Could you give a bit more detail of the kind of thing you're doing? I notice that the program listed as crashing above is mosquitto_pub - are you sending each of your 20 messages/s with a separate mosquitto_pub, or is it bridged, that sort of thing. Any detail you can provide would be helpful.
On the subject of mosquitto_pub - you're probably better off building a client that can stay connected to the broker and send messages where possible. mosquitto_pub generally (unless you use the stdin option) only sends a single message per connection which is quite inefficient.