Unable to start elasticsearch after upgrade to kernal 4.13.0-31

[Rob Nichols]

OS version:
Linux Mint 18.3 Sylvia
Kernal 4.13.0-31-generic

Elasticsearch version:
Version: 2.4.1, Build: c67dc32/2016-09-27T18:57:55Z, JVM: 1.8.0_151

JVM version (java -version):

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

Description of the problem:
I start elasticsearch with sudo systemctl restart elasticsearch.service and this seems to behave as normal. However, when I go to http://localhost:9200 I get connection refused errors.

I think the error has been triggered by an upgrade to kernal 4.13.0-31-generic. I have reported the error in elasticsearch issues and they have diagnosed a kernal fault:

https://github.com/elastic/elasticsearch/issues/28354

I cannot see any errors in the elasticsearch logs, but get the following in the system log

System logs:

Jan 24 09:54:49 Q957 systemd[1]: Starting Elasticsearch...
Jan 24 09:54:49 Q957 systemd[1]: Started Elasticsearch.
Jan 24 09:54:49 Q957 kernel: [ 831.017466] kernel tried to execute NX-protected page - exploit attempt? (uid: 124)
Jan 24 09:54:49 Q957 kernel: [ 831.018446] BUG: unable to handle kernel paging request at 00007f3a99f4f7a0
Jan 24 09:54:49 Q957 kernel: [ 831.019422] IP: 0x7f3a99f4f7a0
Jan 24 09:54:49 Q957 kernel: [ 831.020399] PGD 8000000172776067
Jan 24 09:54:49 Q957 kernel: [ 831.020399] P4D 8000000172776067
Jan 24 09:54:49 Q957 kernel: [ 831.021375] PUD 154036067
Jan 24 09:54:49 Q957 kernel: [ 831.022318] PMD 154037067
Jan 24 09:54:49 Q957 kernel: [ 831.023228] PTE 8000000155b48867
Jan 24 09:54:49 Q957 kernel: [ 831.024102]
Jan 24 09:54:49 Q957 kernel: [ 831.025764] Oops: 0011 [#5] SMP PTI
Jan 24 09:54:49 Q957 kernel: [ 831.026546] Modules linked in: ccm rfcomm arc4 bnep binfmt_misc nls_iso8859_1 snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic iwlmvm intel_rapl mac80211 snd_hda_intel snd_hda_codec snd_hda_core x86_pkg_temp_thermal snd_hwdep intel_powerclamp coretemp snd_pcm snd_seq_midi kvm_intel snd_seq_midi_event snd_rawmidi iwlwifi kvm snd_seq snd_seq_device irqbypass snd_timer input_leds intel_cstate cfg80211 intel_rapl_perf snd btusb hci_uart btrtl serdev ucsi_acpi btbcm mei_me typec_ucsi btqca soundcore shpchp mei typec btintel bluetooth fujitsu_laptop ecdh_generic sparse_keymap intel_lpss_acpi intel_lpss mac_hid acpi_als tpm_crb kfifo_buf industrialio acpi_pad parport_pc ppdev lp parport autofs4 btrfs xor raid6_pq algif_skcipher af_alg dm_crypt dm_mirror dm_region_hash dm_log i915 crct10dif_pclmul
Jan 24 09:54:49 Q957 kernel: [ 831.030005] crc32_pclmul ghash_clmulni_intel pcbc i2c_algo_bit hid_generic drm_kms_helper aesni_intel syscopyarea aes_x86_64 e1000e crypto_simd sysfillrect glue_helper sysimgblt cryptd fb_sys_fops ptp nvme drm ahci pps_core nvme_core usbhid libahci wmi i2c_hid hid video
Jan 24 09:54:49 Q957 kernel: [ 831.031867] CPU: 0 PID: 5399 Comm: java Tainted: G D 4.13.0-31-generic #34~16.04.1-Ubuntu
Jan 24 09:54:49 Q957 kernel: [ 831.032813] Hardware name: FUJITSU ESPRIMO Q957/D3513-A1, BIOS V5.0.0.12 R1.9.0 for D3513-A1x 06/12/2017
Jan 24 09:54:49 Q957 kernel: [ 831.033761] task: ffff9185924f2f80 task.stack: ffffa78383758000
Jan 24 09:54:49 Q957 kernel: [ 831.034716] RIP: 0010:0x7f3a99f4f7a0
Jan 24 09:54:49 Q957 kernel: [ 831.035642] RSP: 0018:ffffa7838375bf50 EFLAGS: 00010202
Jan 24 09:54:49 Q957 kernel: [ 831.036566] RAX: 00000000000003e7 RBX: 0000000000000000 RCX: 00007f3a9987c4d9
Jan 24 09:54:49 Q957 kernel: [ 831.037494] RDX: 00007f3a99f4ff50 RSI: 00007f3a99f50030 RDI: 0000000000000000
Jan 24 09:54:49 Q957 kernel: [ 831.038445] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000000c
Jan 24 09:54:49 Q957 kernel: [ 831.039368] R10: 00007f3a99f4f7a0 R11: ffff9185924f2f80 R12: 0000000000000000
Jan 24 09:54:49 Q957 kernel: [ 831.040293] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Jan 24 09:54:49 Q957 kernel: [ 831.041224] FS: 00007f3a99f51700(0000) GS:ffff91866e400000(0000) knlGS:0000000000000000
Jan 24 09:54:49 Q957 kernel: [ 831.042207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 24 09:54:49 Q957 kernel: [ 831.043135] CR2: 00007f3a99f4f7a0 CR3: 0000000136b7c002 CR4: 00000000003606f0
Jan 24 09:54:49 Q957 kernel: [ 831.044068] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 09:54:49 Q957 kernel: [ 831.045001] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jan 24 09:54:49 Q957 kernel: [ 831.045924] Call Trace:
Jan 24 09:54:49 Q957 kernel: [ 831.046888] ? entry_SYSCALL_64_fastpath+0x33/0xa3
Jan 24 09:54:49 Q957 kernel: [ 831.047806] Code: Bad RIP value.
Jan 24 09:54:49 Q957 kernel: [ 831.048723] RIP: 0x7f3a99f4f7a0 RSP: ffffa7838375bf50
Jan 24 09:54:49 Q957 kernel: [ 831.049647] CR2: 00007f3a99f4f7a0
Jan 24 09:54:49 Q957 kernel: [ 831.050619] ---[ end trace f522e6a598391b45 ]---

[ergo (ergo14)]

Can confirm I'm also affected by this bug:

Jan 24 17:04:23 ergo-desktop kernel: [ 9640.453538] kernel tried to execute NX-protected page - exploit attempt? (uid: 1001)

[Jeffrey Bouter (jbouter)]

Same here.

-- Unit elasticsearch.service has begun starting up.
Jan 24 18:34:26 logness systemd[1]: Started Elasticsearch.
-- Subject: Unit elasticsearch.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has finished starting up.
--
-- The start-up result is done.
Jan 24 18:34:27 logness kernel: kernel tried to execute NX-protected page - exploit attempt? (uid: 111)
Jan 24 18:34:27 logness kernel: BUG: unable to handle kernel paging request at 00007f8429f767a0
Jan 24 18:34:27 logness kernel: IP: 0x7f8429f767a0
Jan 24 18:34:27 logness kernel: PGD 8000000136028067
Jan 24 18:34:27 logness kernel: P4D 8000000136028067
Jan 24 18:34:27 logness kernel: PUD 139ea5067
Jan 24 18:34:27 logness kernel: PMD 139f07067
Jan 24 18:34:27 logness kernel: PTE 8000000109a6f867
Jan 24 18:34:27 logness kernel:
Jan 24 18:34:27 logness kernel: Oops: 0011 [#9] SMP PTI
Jan 24 18:34:27 logness kernel: Modules linked in: ppdev snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep input_leds snd_pcm joydev serio_raw snd_timer snd i2c_piix4 soundcore
Jan 24 18:34:27 logness kernel: linear hid_generic usbhid hid qxl ttm drm_kms_helper syscopyarea sysfillrect psmouse virtio_blk virtio_net sysimgblt fb_sys_fops drm pata_acpi floppy
Jan 24 18:34:27 logness kernel: CPU: 0 PID: 2305 Comm: java Tainted: G D 4.13.0-31-generic #34~16.04.1-Ubuntu
Jan 24 18:34:27 logness kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
Jan 24 18:34:27 logness kernel: task: ffff984ff604d800 task.stack: ffffac7b41dfc000
Jan 24 18:34:27 logness kernel: RIP: 0010:0x7f8429f767a0
Jan 24 18:34:27 logness kernel: RSP: 0018:ffffac7b41dfff50 EFLAGS: 00010202
Jan 24 18:34:27 logness kernel: RAX: 00000000000003e7 RBX: 0000000000000000 RCX: 00007f84298884d9
Jan 24 18:34:27 logness kernel: RDX: 00007f8429f76f50 RSI: 00007f8429f77030 RDI: 0000000000000000
Jan 24 18:34:27 logness kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 000000000000000c
Jan 24 18:34:27 logness kernel: R10: 00007f8429f767a0 R11: ffff984ff604d800 R12: 0000000000000000
Jan 24 18:34:27 logness kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Jan 24 18:34:27 logness kernel: FS: 00007f8429f78700(0000) GS:ffff984fffc00000(0000) knlGS:0000000000000000
Jan 24 18:34:27 logness kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 24 18:34:27 logness kernel: CR2: 00007f8429f767a0 CR3: 0000000136022000 CR4: 00000000000006f0
Jan 24 18:34:27 logness kernel: Call Trace:
Jan 24 18:34:27 logness kernel: ? entry_SYSCALL_64_fastpath+0x33/0xa3
Jan 24 18:34:27 logness kernel: Code: Bad RIP value.
Jan 24 18:34:27 logness kernel: RIP: 0x7f8429f767a0 RSP: ffffac7b41dfff50
Jan 24 18:34:27 logness kernel: CR2: 00007f8429f767a0
Jan 24 18:34:27 logness kernel: ---[ end trace 2b5dfc9417bb8d5f ]---

[GJ (testguest)]

Me too.

And I fixed it by rolling back the kernel. I verified this by using the GRUB menu to boot into a previous kernel (26-generic) and ES started again. I then purged the tricky kernel. Here's how:

sudo apt-get purge linux-image-4.13.0-31-generic
sudo apt-get purge linux-headers-4.13.0-31-generic

and then reboot

because we haven’t purged the linux-generic package, Ubuntu will still try to get a new kernel when one is available, skipping the purged one.

[Rob Nichols (s-rob-t)]

I confirm that rolling back the kernal fixes the problem.