Broken pipe due to grep -q

[Chris White]

I kept getting a writing to broken pipe error from dpkg-query.

Eventually tracked this down to grep quitting on first match when the -q option was used.

Fixed it by removing the -q and instead redirecting any grep output to null, i.e. "&> /dev/null", wherever grep is used.

[Jarno Suni (jarnos)]

Oh, I can not reproduce the issue. I guess it is due to race condition. Does the error make script somehow fail or is it just an annoying error message?

[Jarno Suni (jarnos)]

I think it would be better just to redirect stderr of dpkg-query like this: "2>/dev/null"

[Jarno Suni (jarnos)]

Which version of dpkg are you using?

[Jarno Suni (jarnos)]

Broken pipe error is not dangerous in this case, because no more information is needed from dpkg-query and the error does not affect to the exit code of pipeline (because pipefail option is not set in shell). So I suppress the error messages by redirecting in the following commit. The side effect is that no other possible error messages of dpkg-query are shown either, but such error is very rare in this case, I think.

Here is the commit
https://git.launchpad.net/linux-purge/commit/?id=c757fc1568d1a395628f843c62caa77f550aab1a

I still wonder why you get the error message and I do not. Maybe I still change the fix later...

[Jarno Suni (jarnos)]

Now broken pipe error message may also happen, if you press q quickly after launching
linux-purge --info
(in a terminal window that does have few rows) but it is harmless.

[Jarno Suni (jarnos)]

I reverted the commit mentioned in #4 because it will suppress possible other error messages as well.

I dealt with the broken pipe errors referred in #5 in another new commit.

I am still waiting Chris White to respond.

[Jarno Suni (jarnos)]

Which Ubuntu release you used when this happened?

[Jarno Suni (jarnos)]

I have made some changes. Does this bug still occur to you? I know that your fix works, but it does some extra computing, and I have still not been able to reproduce the issue.

What does 'trap -p PIPE' output in the Bash shell you call linux-purge from? Or do you have some other environment?

[Launchpad Janitor (janitor)]

[Expired for linux-purge because there has been no activity for 60 days.]

[Chris White (whitecf)]

Apologies as I missed any notice for the fix having been made. Just updated my installed version,

> linux-purge --version
linux-purge (Linux Purge for Ubuntu) 1.0.110-d5962da
Copyright (C) 2017-2020 Jarno Ilari Suni
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Jarno Ilari Suni.

Still seem to be getting the error but also have some more stuff now,

> linux-purge --auto-only --keep=2 --yes
dpkg-query: error: error writing to '<standard output>': Broken pipe
[linux-purge] ERROR: There are no meta-kernels so can not decide which kernels
to keep.

[Chris White (whitecf)]

FYI,

> bash -x linux-purge --auto-only --keep=2 --yes
+ set -u -e
+ export LC_ALL=C
+ LC_ALL=C
+ export TMPDIR=/dev/shm
+ TMPDIR=/dev/shm
+ readonly program_name=linux-purge 'program_full_name=Linux Purge for Ubuntu' program_version=1.0.110-d5962da copyright_years=2017-2020 MB=1000000 width=72
+ program_name=linux-purge
+ program_full_name='Linux Purge for Ubuntu'
+ program_version=1.0.110-d5962da
+ copyright_years=2017-2020
+ MB=1000000
+ width=72
++ uname -r
++ stat --format=%m /boot
+ readonly 'w_error=Terminal screen not wide enough. Minimum is 72 columns.' 'h_error=Not enough lines in terminal screen.' lockfile=/var/lib/dpkg/lock current=4.15.0-115-generic boot=/boot
+ w_error='Terminal screen not wide enough. Minimum is 72 columns.'
+ h_error='Not enough lines in terminal screen.'
+ lockfile=/var/lib/dpkg/lock
+ current=4.15.0-115-generic
+ boot=/boot
+ readonly 'cs=;'
+ cs=';'
++ mktemp /dev/shm/linux-purge.error_XXX
+ readonly error_file=/dev/shm/linux-purge.error_ZxP
+ error_file=/dev/shm/linux-purge.error_ZxP
+ keep=
+ auto_only=
+ clear_boot=
+ choose=
+ debug=
+ debug_file=
+ show_debug_filename=
+ fix=
+ info=
+ simulate=
+ yes=
+ manual=
+ legend=1
+ list=
+ optimize=
+ interface=d
+ operation=purge
+ update_grub_disabled=
+ update_grub=
+ unset -v LatestReleases ManualReleases HoldReleases
+ trap finish EXIT
+ trap 'finish INT; trap - INT; kill -s INT -- $$' INT
+ env -u GETOPT_COMPATIBLE getopt --test
+ [[ 4 -eq 4 ]]
++ env -u GETOPT_COMPATIBLE getopt -o abcd::fhi:k:lmnosrvy -l auto-only,clear-boot,choose,debug::,fix,help,info,interface:,keep:,list,manual,no-legend,optimize,remove,simulate,version,yes --name '[linux-purge] ERROR' -- --auto-only --keep=2 --yes
+ params=' --auto-only --keep '\''2'\'' --yes --'
+ eval set -- ' --auto-only --keep '\''2'\'' --yes --'
++ set -- --auto-only --keep 2 --yes --
+ :
+ case ${1-} in
+ auto_only=1
+ shift
+ :
+ case ${1-} in
+ keep=1
+ n=2
+ shift
+ shift
+ :
+ case ${1-} in
+ yes=-y
+ shift
+ :
+ case ${1-} in
+ shift
+ break
+ [[ 0 -eq 0 ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -n 1 ]]
+ [[ 2 =~ ^[0-9]+$ ]]
+ [[ -n 1 ]]
+ [[ ! -n 1 ]]
+ [[ -n 1 ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -z '' ]]
+ [[ 0 -ne 0 ]]
+ [[ -n '' ]]
++ command -v update-grub
+ update_grub=/usr/sbin/update-grub
+ [[ -n /usr/sbin/update-grub ]]
+ chmod +x /usr/sbin/update-grub
+ [[ d = d ]]
+ hash dialog
+ [[ -n '' ]]
+ operate
+ [[ -n '' ]]
+ purge_normally
++ dpkg-query -W '-f=${Package} ${db:Status-Abbrev}\n'
+++ literalize 4.15.0-115-generic
+++ printf %s 4.15.0-115-generic
+++ sed -e 's/[^^]/[&]/g' -e 's/\^/\\^/g'
++ grep -qE '^linux-image-(unsigned-)?[4][.][1][5][.][0][-][1][1][5][-][g][e][n][e][r][i][c] [ih]i $'
dpkg-query: error: error writing to '<standard output>': Broken pipe
++ dpkg-query -W '-f=${Package} ${Status}\n' 'linux-*'
++ awk '$3!="ok" || ($4!="installed" && $4!="not-installed" && $4!="config-files")
  '
+ Pkgs=
+ [[ -n '' ]]
++ dpkg-query -W '-f=${db:Status-Abbrev} ${Package}\n'
++ sed -nr 's/^[ih]i linux-image-(unsigned-)?([0-9]+\..+)/\2/p'
++ linux-version sort
+ Releases='4.15.0-55-generic
4.15.0-106-generic
4.15.0-107-generic
4.15.0-112-generic
4.15.0-115-generic'
+ [[ -n '' ]]
+ [[ ...

[Chris White (whitecf)]

From the online man page for grep,

"... Portable shell scripts should avoid both -q and -s and should redirect standard and error output to /dev/null instead. ..."

[Jarno Suni (jarnos)]

As for portability, linux-purge is meant for Ubuntu, so I think I can assume you are using GNU grep. (You can run "grep --version" to verify that.)

Please update linux-purge and run

linux-purge --auto-only --keep=2 --simulate --debug

Can you confirm that the broken pipe error is gone?

Please attach the file whose path is show finally.

[Chris White (whitecf)]

I've made two attempts to post a comment with the file attached but as best I can tell this just isn't working so I'll paste the text into this comment,

> linux-purge --auto-only --keep=2 --simulate --debug
dpkg-query: error: error writing to '<standard output>': Broken pipe
[linux-purge] ERROR: There are no meta-kernels so can not decide which kernels
to keep.
[linux-purge] NOTE: The debug information is saved in
/dev/shm/linux-purge.debug_B83

> cat /dev/shm/linux-purge.debug_B83
## start time: Mon Sep 7 11:33:33 BST 2020
## linux-purge version: 1.0.110-d5962da
## options: --auto-only --keep '2' --simulate --debug '' --
## OS: Linux Mint 18.3 Sylvia
## awk executable: /usr/bin/gawk
## awk version:
GNU Awk 4.1.3, API: 1.1 (GNU MPFR 3.1.4, GNU MP 6.1.0)
Copyright (C) 1989, 1991-2015 Free Software Foundation.

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/.
## linux packages:
ii linux-base
ii linux-firmware
un linux-firmware-snapdragon
un linux-headers
un linux-headers-2.6-686
un linux-headers-2.6-amd64
un linux-headers-3.0
ii linux-headers-4.15.0-112
ii linux-headers-4.15.0-112-generic
ii linux-headers-4.15.0-115
ii linux-headers-4.15.0-115-generic
ii linux-headers-4.15.0-55
ii linux-headers-4.15.0-55-generic
un linux-headers-686-pae
un linux-headers-amd64
un linux-headers-generic
un linux-headers-generic-pae
un linux-hwe-tools
un linux-image
ii linux-image-4.15.0-112-generic
ii linux-image-4.15.0-115-generic
un linux-image-4.15.0-55-generic
un linux-image-unsigned-4.15.0-112-generic
un linux-image-unsigned-4.15.0-115-generic
ii linux-image-unsigned-4.15.0-55-generic
un linux-initramfs-tool
un linux-kernel-headers
un linux-kernel-log-daemon
ii linux-libc-dev
ii linux-modules-4.15.0-112-generic
ii linux-modules-4.15.0-115-generic
ii linux-modules-4.15.0-55-generic
ii linux-modules-extra-4.15.0-112-generic
ii linux-modules-extra-4.15.0-115-generic
un linux-restricted-common
ii linux-sound-base

## trace and output:
+ debug() 147: operate
++ debug() 147: tee -ia -- /dev/shm/linux-purge.debug_B83
+ operate() 909: [[ -n '' ]]
+ operate() 912: purge_normally
++ debug() 147: tee -ia -- /dev/shm/linux-purge.debug_B83
++ purge_normally() 748: dpkg-query -W '-f=${Package} ${db:Status-Abbrev}\n'
+++ purge_normally() 748: literalize 4.15.0-115-generic
+++ literalize() 247: printf %s 4.15.0-115-generic
+++ literalize() 247: sed -e 's/[^^]/[&]/g' -e 's/\^/\\^/g'
++ purge_normally() 748: grep -qE '^linux-image-(unsigned-)?[4][.][1][5][.][0][-][1][1][5][-][g][e][n][e][r][i][c] [ih]i $'
dpkg-query: error: error writing to '<standard output>': Broken pipe
++ purge_normal...

[Jarno Suni (jarnos)]

You used the old version for the debug information. Please update by reinstalling using the instructions in https://launchpad.net/linux-purge/+announcement/15313

I do not understand how you get kernel updates as there seems not to be any depending meta kernel installed. Please also show the output of:

apt-cache rdepends --important linux-image-4.15.0-115-generic

(or whichever is the latest kernel)
You seem to have some unsigned kernel installed, too, for some reason. Have you used some other tool to remove kernels besides linux-purge?

[Chris White (whitecf)]

> sh -c 'cd /tmp && wget -N https://git.launchpad.net/linux-purge/plain/install-linux-purge.sh && chmod +x ./install-linux-purge.sh && sudo ./install-linux-purge.sh && rm ./install-linux-purge.sh' && [ "$BASH" ] && echo Replaced current shell in order to make the Bash completion work. && exec bash
--2020-09-07 12:46:38-- https://git.launchpad.net/linux-purge/plain/install-linux-purge.sh
Resolving git.launchpad.net (git.launchpad.net)... 91.189.94.79, 91.189.94.77
Connecting to git.launchpad.net (git.launchpad.net)|91.189.94.79|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 959 [text/plain]
Saving to: ‘install-linux-purge.sh’

install-linux-purge.sh 100%[=================================================>] 959 --.-KB/s in 0s

2020-09-07 12:46:39 (155 MB/s) - ‘install-linux-purge.sh’ saved [959/959]

--2020-09-07 12:46:39-- https://git.launchpad.net/linux-purge/plain/linux-purge
Resolving git.launchpad.net (git.launchpad.net)... 91.189.94.79, 91.189.94.77
Connecting to git.launchpad.net (git.launchpad.net)|91.189.94.79|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 35346 (35K) [text/plain]
Saving to: ‘linux-purge’

linux-purge 100%[=================================================>] 34.52K --.-KB/s in 0.009s

2020-09-07 12:46:39 (3.71 MB/s) - ‘linux-purge’ saved [35346/35346]

--2020-09-07 12:46:39-- https://git.launchpad.net/linux-purge/plain/debian/linux-purge.8
Resolving git.launchpad.net (git.launchpad.net)... 91.189.94.77, 91.189.94.79
Connecting to git.launchpad.net (git.launchpad.net)|91.189.94.77|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4493 (4.4K) [text/plain]
Saving to: ‘linux-purge.8’

linux-purge.8 100%[=================================================>] 4.39K --.-KB/s in 0s

2020-09-07 12:46:39 (1.00 GB/s) - ‘linux-purge.8’ saved [4493/4493]

--2020-09-07 12:46:43-- https://git.launchpad.net/linux-purge/plain/debian/linux-purge.bash-completion
Resolving git.launchpad.net (git.launchpad.net)... 91.189.94.79, 91.189.94.77
Connecting to git.launchpad.net (git.launchpad.net)|91.189.94.79|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1517 (1.5K) [text/plain]
Saving to: ‘linux-purge.bash-completion’

linux-purge.bash-completion 100%[=================================================>] 1.48K --.-KB/s in 0s

2020-09-07 12:46:43 (445 MB/s) - ‘linux-purge.bash-completion’ saved [1517/1517]

Replaced current shell in order to make the Bash completion work.

[Chris White (whitecf)]

> linux-purge --version
linux-purge (Linux Purge for Ubuntu) 1.0.113-c15c34a
Copyright (C) 2017-2020 Jarno Ilari Suni
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Jarno Ilari Suni.

[Chris White (whitecf)]

> linux-purge --auto-only --keep=2 --simulate --debug[linux-purge] ERROR: There are no meta-kernels so can not decide which kernels
to keep.
[linux-purge] NOTE: The debug information is saved in
/dev/shm/linux-purge.debug_b3n

> cat /dev/shm/linux-purge.debug_b3n
## start time: Mon Sep 7 12:47:59 BST 2020
## linux-purge version: 1.0.113-c15c34a
## options: --auto-only --keep '2' --simulate --debug '' --
## OS: Linux Mint 18.3 Sylvia
## awk executable: /usr/bin/gawk
## awk version:
GNU Awk 4.1.3, API: 1.1 (GNU MPFR 3.1.4, GNU MP 6.1.0)
Copyright (C) 1989, 1991-2015 Free Software Foundation.

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/.
## linux packages:
ii linux-base
ii linux-firmware
un linux-firmware-snapdragon
un linux-headers
un linux-headers-2.6-686
un linux-headers-2.6-amd64
un linux-headers-3.0
ii linux-headers-4.15.0-112
ii linux-headers-4.15.0-112-generic
ii linux-headers-4.15.0-115
ii linux-headers-4.15.0-115-generic
ii linux-headers-4.15.0-55
ii linux-headers-4.15.0-55-generic
un linux-headers-686-pae
un linux-headers-amd64
un linux-headers-generic
un linux-headers-generic-pae
un linux-hwe-tools
un linux-image
ii linux-image-4.15.0-112-generic
ii linux-image-4.15.0-115-generic
un linux-image-4.15.0-55-generic
un linux-image-unsigned-4.15.0-112-generic
un linux-image-unsigned-4.15.0-115-generic
ii linux-image-unsigned-4.15.0-55-generic
un linux-initramfs-tool
un linux-kernel-headers
un linux-kernel-log-daemon
ii linux-libc-dev
ii linux-modules-4.15.0-112-generic
ii linux-modules-4.15.0-115-generic
ii linux-modules-4.15.0-55-generic
ii linux-modules-extra-4.15.0-112-generic
ii linux-modules-extra-4.15.0-115-generic
un linux-restricted-common
ii linux-sound-base

## trace and output:
+ debug() 147: operate
++ debug() 147: tee -ia -- /dev/shm/linux-purge.debug_b3n
+ operate() 909: [[ -n '' ]]
+ operate() 912: purge_normally
++ debug() 147: tee -ia -- /dev/shm/linux-purge.debug_b3n
++ purge_normally() 748: dpkg-query -W '-f=${Package} ${db:Status-Abbrev}\n'
+++ purge_normally() 748: literalize 4.15.0-115-generic
+++ literalize() 247: printf %s 4.15.0-115-generic
+++ literalize() 247: sed -e 's/[^^]/[&]/g' -e 's/\^/\\^/g'
++ purge_normally() 748: grep -qE '^linux-image-(unsigned-)?[4][.][1][5][.][0][-][1][1][5][-][g][e][n][e][r][i][c] [ih]i $'
++ purge_normally() 758: dpkg-query -W '-f=${Package} ${Status}\n' 'linux-*'
++ purge_normally() 760: awk '$3!="ok" || ($4!="installed" && $4!="not-installed" && $4!="config-files")
  '
+ purge_normally() 758: Pkgs=
+ purge_normally() 759: [[ -n '' ]]
++ purge_normally() 773: dpkg-query -W '-f=${db:Status-A...

[Chris White (whitecf)]

> apt-cache rdepends --important linux-image-4.15.0-115-generic
linux-image-4.15.0-115-generic
Reverse Depends:
  linux-image-generic-hwe-16.04
 |linux-modules-extra-4.15.0-115-generic
  linux-image-virtual-hwe-16.04-edge
  linux-image-virtual-hwe-16.04
  linux-image-generic-hwe-16.04-edge

[Chris White (whitecf)]

> hostnamectl
   Static hostname: Blodwyn
         Icon name: computer-desktop
           Chassis: desktop
        Machine ID: 6a4e92f40a6e4d7e8a650c6c44d79ab2
           Boot ID: 926b1423adde41c6b5d98960051de513
  Operating System: Linux Mint 18.3
            Kernel: Linux 4.15.0-115-generic
      Architecture: x86-64

[Chris White (whitecf)]

"... Have you used some other tool to remove kernels besides linux-purge?"

Just the Linux Mint "Update Manager", mintUpdate 5.3.2.5, as linux-purge wasn't removing anything.

[Jarno Suni (jarnos)]

I think you should install package linux-generic-hwe-16.04 to have further kernel updates if you upgrade system by e.g. sudo apt-get update; sudo apt-get dist-upgrade, but I am not familiar with mintUpdate. Anyway, installing it helps linux-purge to work with --keep.

The broken pipe issue seems to have disappeared.

[Jarno Suni (jarnos)]

You should now update linux-purge. See: https://git.launchpad.net/linux-purge/commit/?id=a520d707ac59a2e041f87237a6644ca93844a3cf
In your case you want to have linux-generic-hwe-16.04 installed, not linux-generic.

[Jarno Suni (jarnos)]

Is it working now?

[Chris White (whitecf)]

I'm still looking into if it it's ok for me to install linux-generic-hwe-16.04 rather than just going with what the Mint update manager does. I assume I won't be able to verify any fixes you've applied until I resolve that.

[Jarno Suni (jarnos)]

You could run the following scripts in shell to find out, if/when linux-generic-hwe-16.04 was purged:

grep -FH 'purge linux-generic-hwe-16.04' /var/log/dpkg.log /var/log/dpkg.log.?

for i in /var/log/dpkg.log.*.gz; do gzip -cd $i | grep --label=$i -FH 'purge linux-generic-hwe-16.04'; done

If it prints some match from time when you were using version 1.0.110-d5962da or 1.0.113-c15c34a of linux-purge, the package was probably purged by linux-purge due to a bug that is fixed in later version, see https://launchpad.net/linux-purge/+announcement/27825
Can you show the output?

P.S. As for easy updating linux-purge, see https://launchpad.net/linux-purge/+announcement/27826

[Chris White (whitecf)]

> grep -FH 'purge linux-generic-hwe-16.04' /var/log/dpkg.log /var/log/dpkg.log.*

> for i in /var/log/dpkg.log.*.gz; do gzip -cd $i | grep --label=$i -FH 'purge linux-generic-hwe-16.04'; done

No matches on either search I'm afraid.

[Jarno Suni (jarnos)]

That is odd. It seems like it was never installed then.

The --keep functionality of linux-purge relies on meta kernels, but you should be able to use linux-purge without --keep even if you do not have a meta kernel installed.

[Jarno Suni (jarnos)]

Did you find out, if you should install (or have installed) linux-generic-hwe-16.04 or some other meta kernel in Linux Mint?

Do you still have an issue with 'grep -q' with the latest version of linux-purge? If there is, please tell what grep --version reports.

[Chris White (whitecf)]

I've recently upgraded to Mint 20.1 and don't know what changes this might have made but linux-purge seems to be working ok now.

System: Kernel: 5.4.0-67-generic x86_64 bits: 64 compiler: gcc v: 9.3.0 Desktop: Cinnamon 4.8.6
           wm: muffin dm: LightDM Distro: Linux Mint 20.1 Ulyssa base: Ubuntu 20.04 focal

> linux-purge --version
linux-purge (Linux Purge for Ubuntu) 1.0.119-c64222c
Copyright (C) 2017-2020 Jarno Ilari Suni
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Jarno Ilari Suni.
> grep -FH 'purge linux-generic-hwe' /var/log/dpkg.log /var/log/dpkg.log.*
> for i in /var/log/dpkg.log.*.gz; do gzip -cd $i | grep --label=$i -FH 'linux-generic-hwe'; done
> grep --version
grep (GNU grep) 3.4
Copyright © 2020 Free Software Foundation, Inc.
Licence GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others; see
<https://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.
>

[Jarno Suni (jarnos)]

I suppose you have linux-generic installed.
I recommend that you upgrade to the latest version of linux-purge.

[Jarno Suni (jarnos)]

I am marking this as fixed. Please create another bug report if you still can reproduce the bug in some system with the latest version of linux-purge.