ksh2020 has massive memory issues

[Chris Bertin]

I have a complex ksh environment that I have built over 40 years and it's worked fine (with occasional tweaks) on most systems (SunOS, Solaris, BSD, HP-UX, AIX, NCR, many Linux versions). I just upgraded to kubuntu 20.04 and my 'xterm -ls' windows failed to open because the shell was crashing. I reverted to ksh93 and it worked again. Errors with ksh2020 were false reports of mismatched single or double quotes, among others. I stripped down my environment and managed to use 2020. But when I run some functions, it still blows up. I ran it with 'valgrind' (something that every single app should be run with before any release, especially a shell!) and it came up with hundreds of errors. Ksh93 only has a handful. Valgrind output is attached. My environment is probably too big to share with you but fixing all the glaring issues exposed by valgrind may resolve my problems. For now, I am back on ksh93.

[Anuradha Weeraman (aweeraman)]

Chris, thanks for the report and Valgrind observations.

We have reverted ksh2020 back to ksh93u+, in Debian and Ubuntu with the latest version 2020.0.0+really93u+20120801-8 as of this writing available for Groovy Gorilla when you install the "ksh" package.

Could you pick up this version and let us know if you continue to run into problems? Happy to help resolve and/or bring them to the attention of folks who may be able to help you move forward.

[Chris Bertin (chris-bertin)]

Anarudha, I am on Focal Fossa and the ksh93 version is:

$ ksh93 --version
  version sh (AT&T Research) 93u+ 2012-08-01

Ksh93 is pretty clean and it has only one major bug that I never filed a
ticket for: the handling 'command | while read ...'. A simple example is:

 $ ls | while read f; do [ -f "$f" ] || continue; cp $f /tmp/test/.; done
cp: cp: cannot execute [Exec format error]
cp: cp: cannot execute [Exec format error]
cp: cp: cannot execute [Exec format error]
cp: cp: cannot execute [Exec format error]
cp: cp: cannot execute [Exec format error]
cp: cp: cannot execute [Exec format error]
cp: cp: cannot execute [Exec format error]
cp: cp: cannot execute [Exec format error]
cp: cp: cannot execute [Exec format error]
cp: cp: cannot execute [Exec format error]
cp: cp: cannot execute [Exec format error]
cp: cp: cannot execute [Exec format error]
cp: cp: cannot execute [Exec format error]

Whenever I am hit with this issue, I just run the command in a sub-shell:

$ ksh -c 'ls | while read f; do [ -f "$f" ] || continue; cp $f /tmp/test/.;
done'

The workaround is simple enough that I never bothered to file a ticket.
But, just trying it again, this bug has gotten worse with the FF version!:
this kind of loop now hangs after it's completed, whether or not it is run
in a sub-shell!

The valgrind output of 'ksh93 -l' is attached.

If you send me a copy of the Groovy Gorilla ksh93, I can give it a spin
and, if it's not fixed, I can file a ticket. Or I can file a ticket for the
FF version right now.

Thanks,

Chris (<email address hidden> or <email address hidden>)

On Fri, Jul 17, 2020 at 7:50 AM Anuradha Weeraman <
<email address hidden>> wrote:

> Chris, thanks for the report and Valgrind observations.
>
> We have reverted ksh2020 back to ksh93u+, in Debian and Ubuntu with the
> latest version 2020.0.0+really93u+20120801-8 as of this writing
> available for Groovy Gorilla when you install the "ksh" package.
>
> Could you pick up this version and let us know if you continue to run
> into problems? Happy to help resolve and/or bring them to the attention
> of folks who may be able to help you move forward.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1887863
>
> Title:
> ksh2020 has massive memory issues
>
> Status in ksh package in Ubuntu:
> New
>
> Bug description:
> I have a complex ksh environment that I have built over 40 years and
> it's worked fine (with occasional tweaks) on most systems (SunOS,
> Solaris, BSD, HP-UX, AIX, NCR, many Linux versions). I just upgraded
> to kubuntu 20.04 and my 'xterm -ls' windows failed to open because the
> shell was crashing. I reverted to ksh93 and it worked again. Errors
> with ksh2020 were false reports of mismatched single or double quotes,
> among others. I stripped down my environment and managed to use 2020.
> But when I run some functions, it still blows up. I ran it with
> 'valgrind' (something that every single app should be run with before
> any release, especially a shell!) and it came up with hundreds of
> errors. Ksh93 only has a handful. Valgrind output is attached. My...

[Anuradha Weeraman (aweeraman)]

I believe the issue you're referring to is similar to this issue that was filed recently regarding the "Exec format error":

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965072

Will bring it up with the folks working on ksh93 to see if there's a resolution that can be found. Let me know if there's a github id that I can tag you on for any updates related to it.

Since you're on focal, you can use the 'ksh93' package, which is basically ksh93u+ that was spun off as a result of the transition to ksh2020 on the 'ksh' package. Recently, we have reverted ksh2020 back to ksh93 and removed the "ksh93" package in Debian. So please track the following package in Ubuntu for updates as this is back to ksh93 and should address your original issue on this bug report.

https://launchpad.net/ubuntu/+source/ksh

[Chris Bertin (chris-bertin)]

OK, Anuradha. Yes, 965072 looks like my issue. Regarding the hang, I had a
typo in the command. That should have generated a syntax error but,
instead, it hung. No big deal, though, and not worth filing a defect,
something I try to avoid unless it's a major issue.

I did 'apt install' the ksh93 package, a better solution than just copying
the shell and updating /etc/alternatives.

I created a github account. My ID is 'chrisbertin'.

Thanks!

Chris (<email address hidden> or <email address hidden>)

On Fri, Jul 17, 2020 at 12:40 PM Anuradha Weeraman <
<email address hidden>> wrote:

> I believe the issue you're referring to is similar to this issue that
> was filed recently regarding the "Exec format error":
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965072
>
> Will bring it up with the folks working on ksh93 to see if there's a
> resolution that can be found. Let me know if there's a github id that I
> can tag you on for any updates related to it.
>
> Since you're on focal, you can use the 'ksh93' package, which is
> basically ksh93u+ that was spun off as a result of the transition to
> ksh2020 on the 'ksh' package. Recently, we have reverted ksh2020 back to
> ksh93 and removed the "ksh93" package in Debian. So please track the
> following package in Ubuntu for updates as this is back to ksh93 and
> should address your original issue on this bug report.
>
> https://launchpad.net/ubuntu/+source/ksh
>
> ** Bug watch added: Debian Bug tracker #965072
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965072
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1887863
>
> Title:
> ksh2020 has massive memory issues
>
> Status in ksh package in Ubuntu:
> New
>
> Bug description:
> I have a complex ksh environment that I have built over 40 years and
> it's worked fine (with occasional tweaks) on most systems (SunOS,
> Solaris, BSD, HP-UX, AIX, NCR, many Linux versions). I just upgraded
> to kubuntu 20.04 and my 'xterm -ls' windows failed to open because the
> shell was crashing. I reverted to ksh93 and it worked again. Errors
> with ksh2020 were false reports of mismatched single or double quotes,
> among others. I stripped down my environment and managed to use 2020.
> But when I run some functions, it still blows up. I ran it with
> 'valgrind' (something that every single app should be run with before
> any release, especially a shell!) and it came up with hundreds of
> errors. Ksh93 only has a handful. Valgrind output is attached. My
> environment is probably too big to share with you but fixing all the
> glaring issues exposed by valgrind may resolve my problems. For now, I
> am back on ksh93.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/ksh/+bug/1887863/+subscriptions
>

[Anuradha Weeraman (aweeraman)]

Great, I will tag you on future communication on the exec format issue. Hopefully we'll get to a resolution. Thanks for your time on reporting the issue.

[Chris Bertin (chris-bertin)]

Sounds good.

Chris (<email address hidden> or <email address hidden>)

On Fri, Jul 17, 2020 at 1:50 PM Anuradha Weeraman <
<email address hidden>> wrote:

> Great, I will tag you on future communication on the exec format issue.
> Hopefully we'll get to a resolution. Thanks for your time on reporting
> the issue.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1887863
>
> Title:
> ksh2020 has massive memory issues
>
> Status in ksh package in Ubuntu:
> New
>
> Bug description:
> I have a complex ksh environment that I have built over 40 years and
> it's worked fine (with occasional tweaks) on most systems (SunOS,
> Solaris, BSD, HP-UX, AIX, NCR, many Linux versions). I just upgraded
> to kubuntu 20.04 and my 'xterm -ls' windows failed to open because the
> shell was crashing. I reverted to ksh93 and it worked again. Errors
> with ksh2020 were false reports of mismatched single or double quotes,
> among others. I stripped down my environment and managed to use 2020.
> But when I run some functions, it still blows up. I ran it with
> 'valgrind' (something that every single app should be run with before
> any release, especially a shell!) and it came up with hundreds of
> errors. Ksh93 only has a handful. Valgrind output is attached. My
> environment is probably too big to share with you but fixing all the
> glaring issues exposed by valgrind may resolve my problems. For now, I
> am back on ksh93.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/ksh/+bug/1887863/+subscriptions
>

[Anuradha Weeraman (aweeraman)]

Could you please try upgrading to the latest ksh93u+m (https://launchpad.net/ubuntu/+source/ksh93u+m), and report if you're continuing to see any issues?

[Chris Bertin (chris-bertin)]

Hi Anuradha,

I tested your new shell and it seems to be working fine.

Thanks,

Chris (<email address hidden> or chris.bertin@proton
<email address hidden>.me)

On Sun, Aug 7, 2022 at 7:20 AM Anuradha Weeraman <email address hidden>
wrote:

> Could you please try upgrading to the latest ksh93u+m
> (https://launchpad.net/ubuntu/+source/ksh93u+m), and report if you're
> continuing to see any issues?
>
> ** Changed in: ksh (Ubuntu)
> Status: New => Fix Released
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1887863
>
> Title:
> ksh2020 has massive memory issues
>
> Status in ksh package in Ubuntu:
> Fix Released
>
> Bug description:
> I have a complex ksh environment that I have built over 40 years and
> it's worked fine (with occasional tweaks) on most systems (SunOS,
> Solaris, BSD, HP-UX, AIX, NCR, many Linux versions). I just upgraded
> to kubuntu 20.04 and my 'xterm -ls' windows failed to open because the
> shell was crashing. I reverted to ksh93 and it worked again. Errors
> with ksh2020 were false reports of mismatched single or double quotes,
> among others. I stripped down my environment and managed to use 2020.
> But when I run some functions, it still blows up. I ran it with
> 'valgrind' (something that every single app should be run with before
> any release, especially a shell!) and it came up with hundreds of
> errors. Ksh93 only has a handful. Valgrind output is attached. My
> environment is probably too big to share with you but fixing all the
> glaring issues exposed by valgrind may resolve my problems. For now, I
> am back on ksh93.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/ksh/+bug/1887863/+subscriptions
>
>