ls --color=always outputs possibly-extraneous ESC [ m on separate line, confusing less -FXR

Bug #494663 reported by jimav
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
coreutils
Fix Released
Undecided
Unassigned
coreutils (Ubuntu)
Fix Released
Low
Unassigned

Bug Description

Binary package hint: coreutils

ls --color=always outputs a (seemingly) extraneous ESC [ m after everything else, which makes less -RX show an extra blank line.

Note that ESC [ m is the ANSI escape sequence ESC [ <parameters> m with empty <parameters>.

Does a "null mode set" do anything? I can't find any documentation saying that it's other han a no-op,
but maybe some terminal somewhere requires (required?) this "no-op" to behave correctly (?)

In any case, the extra "no-op" escape sequence appears after the final newline, and therefore
constitutes an extra unterminated line. In an interactive shell this is invisible, but if the output it piped
through a pager such as less -RX then the extra line can cause an extraneous blank line to appear.

SUGGESTED SOLUTIONS:
If the "no op" escape sequence is really a no-op, then don't output it;
If it is necessary, then output it before the newline of the final item, so it will not constitute an unterminated extra line.

I'll attach a tar archive of the test directory used in the following examples:

bash>env - /bin/ls -l --color=always | less -FXR
total 120
-rw-r--r-- 1 jima jima 0 Dec 9 09:57 afile
-rwxr-xr-x 1 jima jima 114032 Oct 6 04:06 bexecutable
-rw-r--r-- 1 jima jima 0 Dec 9 09:58 bfile
-rw-r--r-- 1 jima jima 0 Dec 9 09:58 cfile

bash> #note EXTRA BLANK LINE above

bash>(env - /bin/ls -l --color=always; echo ==AFTER==) | cat -teuv
total 120$
-rw-r--r-- 1 jima jima 0 Dec 9 09:57 afile$
-rwxr-xr-x 1 jima jima 114032 Oct 6 04:06 ^[[0m^[[01;32mbexecutable^[[0m$
-rw-r--r-- 1 jima jima 0 Dec 9 09:58 bfile$
-rw-r--r-- 1 jima jima 0 Dec 9 09:58 cfile$
^[[m==AFTER==$
bash> # Note the colorization of the executable, which is correct, and the final ESC [ m which causes trouble.

ProblemType: Bug
Architecture: amd64
Date: Wed Dec 9 10:21:20 2009
DistroRelease: Ubuntu 9.10
NonfreeKernelModules: fglrx
Package: coreutils 7.4-2ubuntu1
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-16.52-generic
SourcePackage: coreutils
Uname: Linux 2.6.31-16-generic x86_64
XsessionErrors:
 (gnome-settings-daemon:2128): GLib-CRITICAL **: g_propagate_error: assertion `src != NULL' failed
 (gnome-settings-daemon:2128): GLib-CRITICAL **: g_propagate_error: assertion `src != NULL' failed
 (nautilus:2197): Eel-CRITICAL **: eel_preferences_get_boolean: assertion `preferences_is_initialized ()' failed
 (polkit-gnome-authentication-agent-1:2229): GLib-CRITICAL **: g_once_init_leave: assertion `initialization_value != 0' failed
 (rhythmbox:2224): Rhythmbox-WARNING **: Could not open device /dev/radio0

Revision history for this message
jimav (james-avera) wrote :
Revision history for this message
jimav (james-avera) wrote :
Revision history for this message
C de-Avillez (hggdh2) wrote :

Thank you for opening this bug and helping make Ubuntu better. The "ESC[m" sequence at the end of a 'ls --color' (or --color=auto, or --color=always) is added to reset the colouring to the default values.

I am not sure, however, on how necessary this is -- a _few_ tests I performed, on a hacked version of 'ls' that does *not* add in the reset sequence showed no problems -- but, again, (1) few tests; (2) rather incomplete; (3) and I am quite sure this is there because some terminal needs it...

Nevertheless, I cannot repeat your problem -- if I pipe 'ls' output to another programme, I see *no* colouring codes being output. As such, I wonder if you are forcing '--color=always' to the pipe. This would, most probably, be a bad move.

Changed in coreutils (Ubuntu):
importance: Undecided → Low
status: New → Incomplete
Revision history for this message
jimav (james-avera) wrote : Re: [Bug 494663] Re: ls --color=always outputs possibly-extraneous ESC [ m on separate line, confusing less -FXR
Download full text (4.8 KiB)

Hi C-de-Avillez,

It sounds like you didn't actually try the exact command shown in the bug report. Please try it,
and post whether you can reproduce the problem.

Note that --color=always *is* necessary when piping the output through a pager such as 'less -FXR'.
The point is that the pager is confused by the final escape sequence appearing on an extra "line" (which has no terminating newline).

If the final escape were sent before the final newline then the problem should go away.
You could simulate that change like this:
   ls -l --color=always >file
   vim -b file, and edit to move the final escape sequence to be before the final newline
   cat file | less -FXR

________________________________
From: C de-Avillez <email address hidden>
To: <email address hidden>
Sent: Wed, December 30, 2009 4:41:02 PM
Subject: [Bug 494663] Re: ls --color=always outputs possibly-extraneous ESC [ m on separate line, confusing less -FXR

Thank you for opening this bug and helping make Ubuntu better. The
"ESC[m" sequence at the end of a 'ls --color' (or --color=auto, or
--color=always) is added to reset the colouring to the default values.

I am not sure, however, on how necessary this is -- a _few_ tests I
performed, on a hacked version of 'ls' that does *not* add in the reset
sequence showed no problems -- but, again, (1) few tests; (2) rather
incomplete; (3) and I am quite sure this is there because some terminal
needs it...

Nevertheless, I cannot repeat your problem -- if I pipe 'ls' output to
another programme, I see *no* colouring codes being output. As such, I
wonder if you are forcing '--color=always' to the pipe. This would, most
probably, be a bad move.

** Changed in: coreutils (Ubuntu)
   Importance: Undecided => Low

** Changed in: coreutils (Ubuntu)
       Status: New => Incomplete

--
ls --color=always outputs possibly-extraneous ESC [ m on separate line, confusing less -FXR
https://bugs.launchpad.net/bugs/494663
You received this bug notification because you are a direct subscriber
of the bug.

Status in “coreutils” package in Ubuntu: Incomplete

Bug description:
Binary package hint: coreutils

ls --color=always outputs a (seemingly) extraneous ESC [ m after everything else, which makes less -RX show an extra blank line.

Note that ESC [ m is the ANSI escape sequence ESC [ <parameters> m with empty <parameters>.

Does a "null mode set" do anything? I can't find any documentation saying that it's other han a no-op,
but maybe some terminal somewhere requires (required?) this "no-op" to behave correctly (?)

In any case, the extra "no-op" escape sequence appears after the final newline, and therefore
constitutes an extra unterminated line. In an interactive shell this is invisible, but if the output it piped
through a pager such as less -RX then the extra line can cause an extraneous blank line to appear.

SUGGESTED SOLUTIONS:
If the "no op" escape sequence is really a no-op, then don't output it;
If it is necessary, then output it before the newline of the final item, so it will not constitute an unterminated extra line.

I'll attach a tar archive of the test directory used in the following examples:

bash>env - /bin/ls -l --color=always ...

Read more...

Revision history for this message
C de-Avillez (hggdh2) wrote :

Yes, I tried it. For the record, this is what is produced for the different runs:

cerdea@xango:/usr/src/buildd/coreutils/src$ ls --color=auto|tail -1 | od -Xc
0000000 2e736579 00000a6f
          y e s . o \n
0000006
cerdea@xango:/usr/src/buildd/coreutils/src$ ls --color=always|tail -2 | od -Xc
0000000 2e736579 5b1b0a6f 0000006d
          y e s . o \n 033 [ m
0000011
cerdea@xango:/usr/src/buildd/coreutils/src$ ls --color|tail -2 | od -Xc
0000000 2e736579 5b1b0a6f 0000006d
          y e s . o \n 033 [ m
0000011
cerdea@xango:/usr/src/buildd/coreutils/src$ ls --color=auto|tail -2 | od -Xc
0000000 2e736579 65790a63 0a6f2e73
          y e s . c \n y e s . o \n
0000014
cerdea@xango:/usr/src/buildd/coreutils/src$

Also, when I run 'ls --color|less -FXR' all I see is an additional (looking empty) last line. By the way, 'ls --color' is identical with 'ls --color=always'. The --color parameter defaults to 'always'.

So, what do you mean as "confusing less -FXR"?

Revision history for this message
C de-Avillez (hggdh2) wrote :

Just in case, shot a question over to the Coreutils bug list, waiting for a reply.

Changed in coreutils:
status: New → Incomplete
Revision history for this message
C de-Avillez (hggdh2) wrote :

Marking Triaged/Low, per feedback from Jim (see the thread at http://lists.gnu.org/archive/html/bug-coreutils/2009-12/msg00322.html).

Changed in coreutils (Ubuntu):
status: Incomplete → Triaged
Revision history for this message
C de-Avillez (hggdh2) wrote :

I have tried upstream's patch, and it seems to work OK. Upstream is considering it for inclusion on next maintenance release (8.3). If it makes it in 8.3, there is a good chance we will have it on Lucid.

@jimav: there you go. If you want to check the patch, please grab it from the email thread I reference above.

Changed in coreutils:
status: Incomplete → In Progress
Revision history for this message
C de-Avillez (hggdh2) wrote :

For the record, my PPA has a GIT coreutils package for Lucid with this patch in (https://edge.launchpad.net/~hggdh2/+archive/ppa).

Revision history for this message
C de-Avillez (hggdh2) wrote :

This bug has been fixed on coreutils 8.4.x (development). It should land on Ubuntu when we upgrade to 8.5 or later, probably Lucid+1.

Changed in coreutils:
status: In Progress → Fix Released
Revision history for this message
C de-Avillez (hggdh2) wrote :

This has been fixed on coreutils 8.5, available on Maverick 10.10.

Changed in coreutils (Ubuntu):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.