Comment 5 for bug 1863086

Yes, you are absolutely correct. I can reproduce this now, when kernel.core_pattern is set to "|".

I can also confirm that the first kernel that this is broken in is 5.3, as it works fine in 5.2 and below.

I will look into this and hopefully get this fixed for you. Thanks for reporting!

Engineering Notes:

RIP: 0010:do_coredump+0x536/0xb30

$ eu-addr2line do_coredump+0x536 -e vmlinux-5.3.0-29-generic
/build/linux-OZAq_R/linux-5.3.0/include/linux/fs.h:2841:7

This is the function file_start_write():

2839 static inline void file_start_write(struct file *file)
2840 {
2841 if (!S_ISREG(file_inode(file)->i_mode))
2842 return;
2843 __sb_start_write(file_inode(file)->i_sb, SB_FREEZE_WRITE, true);
2844 }

This is called from do_coredump():

565 void do_coredump(const kernel_siginfo_t *siginfo)
566 {
...
788 if (!dump_interrupted()) {
789 file_start_write(cprm.file);
790 core_dumped = binfmt->core_dump(&cprm);
791 file_end_write(cprm.file);
792 }
...
810 }

On kernels 5.2 and before, kernel.core_pattern=| normally outputs:

[ 39.328638] Core dump to | pipe failed

This is output is from the pipe section of do_coredump():

565 void do_coredump(const kernel_siginfo_t *siginfo)
566 {
...
623 ispipe = format_corename(&cn, &cprm, &argv, &argc);
624
625 if (ispipe) {
...
681 sub_info = call_usermodehelper_setup(helper_argv[0],
682 helper_argv, NULL, GFP_KERNEL,
683 umh_pipe_setup, NULL, &cprm);
684 if (sub_info)
685 retval = call_usermodehelper_exec(sub_info,
686 UMH_WAIT_EXEC);
687
688 kfree(helper_argv);
689 if (retval) {
690 printk(KERN_INFO "Core dump to |%s pipe failed\n",
691 cn.corename);
692 goto close_fail;
693 }
...
810 }

With kernel.core_pattern=|, format_corename() still interprets this as the pipe path with:

191 static int format_corename(struct core_name *cn, struct coredump_params *cprm,
192 size_t **argv, int *argc)
193 {
...
196 int ispipe = (*pat_ptr == '|');
...
335 return ispipe;
336 }

Which checks the first byte of the core_pattern string to see if it is a "|".

This means we really want the call to call_usermodehelper_exec() in do_coredump() to fail, so we take the if(retval) step and output "Core dump to |%s pipe failed\n".

Note that cn.corename is NULL when kernel.core_pattern=|.

For some reason with 5.3 and later, this call to call_usermodehelper_exec() is not failing, and execution continues on and breaks at file_start_write(cprm.file); at the end of do_coredump().

call_usermodehelper_exec() calls umh_pipe_setup() and creates the pipe used. Something has likely changed in the pipe subsystem.

I will update when I figure out what.