Error is sort function in buildscripts

[Debian Bug Importer]

Automatically imported from Debian bug report #282582 http://bugs.debian.org/282582

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Sun, 21 Nov 2004 19:40:59 +0100
From: Frans Pop <email address hidden>
To: <email address hidden>
Cc: Christian Perrier <email address hidden>
Subject: Testresults for console-data_2002.12.04dbs-46.2

Package: console-data
Version: 2002.12.04dbs-46.2
Severity: grave

On Tuesday 16 November 2004 06:35, Christian Perrier wrote:
> Folks, If I'm correct, the sid_d-i builds of 20041116 will include the
> new console-keymap udebs.
>
> Please test the keyboard selection as much as possible with these
> builds, especially the new features bringed by this new console-data:

I've done some tests, and here are the results:
- Start d-i in medium priority and select Dutch/Netherlands; choose
  PC-stijl-toetsenbord: no keymap list is shown, but 'us' keymap is
  selected automatically.
  The keymap list _should_ be shown here.
  If I select USB/Mac, the keymap list _is_ shown (with correct sorting).
  If I select English/US, the list is also shown.
- Start d-i in default priority and select Dutch/Netherlands; keymap
  selection appears to be skipped altogether (consistent with item above).
- Start d-i in medium priority and select Dutch/Netherlands; choose
  USB/Mac: Portuguees Macintosh is selected by default instead of US.
  This means #280069 is not fixed correctly. It looks like this is maybe
  an issue related to sorting as the entry in console-keymaps-usb looks
  correct.
- There are several countries that have two apparently identical entries
  in the keymap lists (in English):
  - AT: Croatian
  - USB/Mac: German
  Descriptions should be changed to indicate the differences.
- USB/Mac Portuguese keymap still segfaults, see #275086
  (please leave this bug open for the segfault issue)

(Setting priority grave to make sure package will not propagate to Sarge
 in this condition.)

--
Cheers,
FJP

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Mon, 22 Nov 2004 07:39:40 +0100
From: Christian Perrier <email address hidden>
To: Frans Pop <email address hidden>, <email address hidden>
Cc: <email address hidden>
Subject: Re: Bug#282379: Testresults for console-data_2002.12.04dbs-46.2

tags 282379 sid
thanks

Quoting Frans Pop (<email address hidden>):
> Package: console-data
> Version: 2002.12.04dbs-46.2
> Severity: grave

Of course, this bug only applies to the version of console-data in
sid.

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Mon, 22 Nov 2004 18:52:51 +0100
From: Christian Perrier <email address hidden>
To: Frans Pop <email address hidden>, <email address hidden>
Cc: <email address hidden>
Subject: Re: Bug#282379: Testresults for console-data_2002.12.04dbs-46.2

clone 282379 -1 -2
retitle 282379 Keymap list not shown for AT-style keyboard in Dutch and Norwegian
retitle -1 Croatian keymap listed twice in AT-style keymap list
severity -1 minor
retitle -2 Identical English translations for two different German layouts in USB keyboards
severity -2 normal
thanks

> - Start d-i in medium priority and select Dutch/Netherlands; choose
> USB/Mac: Portuguees Macintosh is selected by default instead of US.
> This means #280069 is not fixed correctly. It looks like this is maybe
> an issue related to sorting as the entry in console-keymaps-usb looks
> correct.

I can't reproduce that one. If I choose Dutch, then Netherlands, then
USB keyboard, I get American English kbd chosen.

Or do you mention this happens indeed on a PPC machine when a Mac USB
kbd is detected�?

> - There are several countries that have two apparently identical entries
> in the keymap lists (in English):
> - AT: Croatian

This is because, the original templates indeed have the same choice
twice....No-one ever noticed because before the sorting patch these
choices were never close together..:-)

> - USB/Mac: German
> Descriptions should be changed to indicate the differences.

There are two German layouts for USB keyboard, so I guess this is not
a problem. The problem is that English "translations" both mention "German"

Both these problems are indeed here for a long time..:-)

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Mon, 22 Nov 2004 18:34:22 +0100
From: Christian Perrier <email address hidden>
To: Frans Pop <email address hidden>, <email address hidden>
Cc: =?iso-8859-15?Q?Bj=F8rn?= Steensrud <email address hidden>,
 <email address hidden>
Subject: Re: Bug#282379: Testresults for console-data_2002.12.04dbs-46.2

Quoting Frans Pop (<email address hidden>):
> Package: console-data
> Version: 2002.12.04dbs-46.2
> Severity: grave
>
> On Tuesday 16 November 2004 06:35, Christian Perrier wrote:
> > Folks, If I'm correct, the sid_d-i builds of 20041116 will include the
> > new console-keymap udebs.
> >
> > Please test the keyboard selection as much as possible with these
> > builds, especially the new features bringed by this new console-data:
>
> I've done some tests, and here are the results:
> - Start d-i in medium priority and select Dutch/Netherlands; choose
> PC-stijl-toetsenbord: no keymap list is shown, but 'us' keymap is
> selected automatically.
> The keymap list _should_ be shown here.
> If I select USB/Mac, the keymap list _is_ shown (with correct sorting).
> If I select English/US, the list is also shown.
> - Start d-i in default priority and select Dutch/Netherlands; keymap
> selection appears to be skipped altogether (consistent with item above).

All these seem to be the same problem.

I've narrowed it and possibly found the source of the problem.

It happens only for Dutch and Norwegian as far as I can see. These two
languages are the ones and only the ones which have used some commas
in keyboard layout names translations.

The sorting script does escape commas and, indeed, in the templates
file, these commas are really escaped in Choices-nl, Choices-nb (and
Choices-no) fields.

However, for whatever reasons, cdebconf breaks on these templates. The
consequence is that the template is not shown...but no error is
reported�: a "safe" default is used.

A simple short term solution is probably by removing the commas in
these translations.

I suggest replacing them by dashes:

Braziliaans, ABNT2-indeling (br-abnt2)

--> Braziliaans - ABNT2-indeling (br-abnt2)

and so on.

If the Dutch team and Bj�o not object, I can take care of it in
both translations.

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 23 Nov 2004 07:06:34 +0100
From: Christian Perrier <email address hidden>
To: <email address hidden>
Cc: Frans Pop <email address hidden>, <email address hidden>
Subject: Re: Bug#282379: Testresults for console-data_2002.12.04dbs-46.2

clone 282379 -1
reassign -1 cdebconf
severity -1 normal
retitle -1 Some functions in cdebconf improperly handle escaped commas
thanks

Frans Pop suggested in #debian-boot

[20:30:49] <fjp> bubulle: I think the program that causes the problems
with comma's is cdebconf's "strutl.c". The functions strgetargc and
strchoicesplit (and maybe str(un)escape) don't know about escaped commas.

So, I duplicate this bug and assign it to cdebconf. Severity normal as
the consequences may vary from program to program and there's indeed
no real urgency for fixing it as workarounds exist (just don't put
commas in translated strings for __Choices).

282379 will be handled by this workaround in console-data (remove
commas).

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 23 Nov 2004 21:43:55 +0100
From: <email address hidden> (Denis Barbier)
To: Christian Perrier <email address hidden>, <email address hidden>
Cc: Frans Pop <email address hidden>
Subject: Re: Bug#282582: Some functions in cdebconf improperly handle escaped commas

On Tue, Nov 23, 2004 at 07:06:34AM +0100, Christian Perrier wrote:
> Frans Pop suggested in #debian-boot
>
> [20:30:49] <fjp> bubulle: I think the program that causes the problems
> with comma's is cdebconf's "strutl.c". The functions strgetargc and
> strchoicesplit (and maybe str(un)escape) don't know about escaped commas.

It may be buggy, but there is definitely code to deal with escaped
commas in these functions.

Denis

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Tue, 23 Nov 2004 23:46:31 +0100
From: Frans Pop <email address hidden>
To: <email address hidden>,
 <email address hidden>
Cc: Christian Perrier <email address hidden>,
 <email address hidden> (Denis Barbier)
Subject: Re: Bug#282582: Some functions in cdebconf improperly handle escaped commas

reassign 282582 console-data
retitle 282582 Error is sort function in buildscripts
thanks

On Tuesday 23 November 2004 21:43, Denis Barbier wrote:
> On Tue, Nov 23, 2004 at 07:06:34AM +0100, Christian Perrier wrote:
> > Frans Pop suggested in #debian-boot
> >
> > [20:30:49] <fjp> bubulle: I think the program that causes the problems
> > with comma's is cdebconf's "strutl.c". The functions strgetargc and
> > strchoicesplit (and maybe str(un)escape) don't know about escaped commas.
>
> It may be buggy, but there is definitely code to deal with escaped
> commas in these functions.

Ah, yes. (/me learns to read C a little better again)
Sorry bout that. Analysis was a little to quick...

Looking at this again I got a little brainstorm and I think _this_ should be
the real cause of the problem :-)

Looking at the template, I noticed there are only 43 Choices-nl, but the
Indices-nl runs from 1 to 46!
Which probably means there is an easy to fix little error in the sorting script
that also adds an entry in Indices-nl for escaped comma's.

Which also means the translations for nl and no probably don't need to be
changed at all...

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Wed, 24 Nov 2004 06:54:04 +0100
From: Christian Perrier <email address hidden>
To: Frans Pop <email address hidden>, <email address hidden>
Cc: Christian Perrier <email address hidden>,
        Denis Barbier <email address hidden>
Subject: Re: Bug#282582: Some functions in cdebconf improperly handle escaped commas

> Looking at the template, I noticed there are only 43 Choices-nl, but the
> Indices-nl runs from 1 to 46!
> Which probably means there is an easy to fix little error in the sorting script
> that also adds an entry in Indices-nl for escaped comma's.

Dammit.....I indeed *counted* myself and found no error. Looks like I
was not awaken enough. I had the very same problem with choose-mirror
and know that having different number of choices in Indices and
Choices-xx fields is Bad.

I'll look again at the sort-keymap script. Though it is a copy of the
sort-countries script in choose-mirror (which properly deals with
commas), I may have done something wrong here.

In the future, something should be done for cdebconf handling this
situation better, imho. Probably make it fail rather than silently
exit.

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Wed, 24 Nov 2004 20:39:15 +0100
From: <email address hidden> (Denis Barbier)
To: Christian Perrier <email address hidden>
Cc: Frans Pop <email address hidden>, <email address hidden>
Subject: Re: Bug#282582: Some functions in cdebconf improperly handle escaped commas

On Wed, Nov 24, 2004 at 06:54:04AM +0100, Christian Perrier wrote:
[...]
> In the future, something should be done for cdebconf handling this
> situation better, imho. Probably make it fail rather than silently
> exit.

No, otherwise we would have to add lots of run-time checks. A better
solution is to check that generated templates files are right, e.g.
with help from linda/lintian.

Denis

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Wed, 24 Nov 2004 23:45:03 +0100
From: Christian Perrier <email address hidden>
To: <email address hidden>, <email address hidden>
Cc: <email address hidden>
Subject: Patch for fixing the sorting bug in console-data

--L6iaP+gRLNZHKoI4
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Attached is a proposed patch to the sort-keymaps script so that
escaped commas are temporarily replaced while the sorting algorithm
operates, and put back after it.

Frans, I used your suggestion of "#@#" as a replacement string.

I have checked it works by building a d-i mini.iso for i386 one can
download at http://www.perrier.eu.org/~bubulle/mini.iso.

Boot with "linux DEBCONF_PRIORITY=medium", then Dutch, Nederlands, PC
kbd and the list now appears...

Norwegian has been checked as well.

--

--L6iaP+gRLNZHKoI4
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="patch.282582"

--- sort-keymaps.old 2004-11-24 23:39:43.000000000 +0100
+++ sort-keymaps 2004-11-24 23:33:11.000000000 +0100
@@ -9,6 +9,7 @@
     mkdir $pkgname || true
     cd $pkgname
     sed -n -e "/^Template: $pkgname\\/keymap/,/^Description:/p" ../../$pkgname/DEBIAN/templates |
+ sed -e "s/\\\\,/#@#/g" |
     sed -e "s/\.UTF-8:/\.UTF-8,/g" |
     perl -p -e '
     chomp;
@@ -39,7 +40,7 @@
     # files have to be added to the $pkgname/keymap template.
     sed -e "/$pkgname.*keymap/{n;q;}" ../../$pkgname/DEBIAN/templates >templates.tmp
     cat sorted* >>templates.tmp
- sed -e "/$pkgname.*keymap/!d;/$pkgname.*keymap/{:end;n;b end}" ../../$pkgname/DEBIAN/templates | sed '1,2d'>>templates.tmp
+ sed -e "/$pkgname.*keymap/!d;/$pkgname.*keymap/{:end;n;b end}" ../../$pkgname/DEBIAN/templates | sed '1,2d' | sed 's/#@#/\\\\,/g' >>templates.tmp
     mv templates.tmp ../../$pkgname/DEBIAN/templates
     cd ..
 done

--L6iaP+gRLNZHKoI4--

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Thu, 25 Nov 2004 08:06:24 +0100
From: Christian Perrier <email address hidden>
To: <email address hidden>
Subject: tagging 282582

# Automatically generated email from bts, devscripts version 2.8.5
tags 282582 patch

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Thu, 25 Nov 2004 08:07:21 +0100
From: Christian Perrier <email address hidden>
To: <email address hidden>
Subject: severity of 282582 is grave

# Automatically generated email from bts, devscripts version 2.8.5
severity 282582 grave

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Thu, 25 Nov 2004 08:07:43 +0100
From: Christian Perrier <email address hidden>
To: <email address hidden>
Subject: merging 282379 282582

# Automatically generated email from bts, devscripts version 2.8.5
merge 282379 282582

[Debian Bug Importer (debzilla)]

Marking as duplicate based on debbugs merge (282379,282582)

This bug has been marked as a duplicate of bug 10544.

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Thu, 25 Nov 2004 18:27:43 +0100
From: Christian Perrier <email address hidden>
To: <email address hidden>
Subject: tagging 282379

# Automatically generated email from bts, devscripts version 2.8.5
tags 282379 pending

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Sat, 27 Nov 2004 07:02:21 -0500
From: Christian Perrier <email address hidden>
To: <email address hidden>
Cc: Christian Perrier <email address hidden>, Alastair McKinstry <email address hidden>
Subject: Fixed in NMU of console-data 2002.12.04dbs-46.3

tag 275033 + fixed
tag 282379 + fixed
tag 282514 + fixed
tag 282515 + fixed
tag 282884 + fixed

quit

This message was generated automatically in response to a
non-maintainer upload. The .changes file follows.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.7
Date: Thu, 25 Nov 2004 20:04:18 +0100
Source: console-data
Binary: console-keymaps-dec console-keymaps-mac console-data console-keymaps-usb console-keymaps-at console-keymaps-atari console-keymaps-sun console-keymaps-acorn console-keymaps-amiga
Architecture: source all
Version: 2002.12.04dbs-46.3
Distribution: unstable
Urgency: low
Maintainer: Alastair McKinstry <email address hidden>
Changed-By: Christian Perrier <email address hidden>
Description:
 console-data - Keymaps, fonts, charset maps, fallback tables for console-tools
 console-keymaps-acorn - Keymaps for Acorn RISC-PC keyboards. (udeb)
 console-keymaps-amiga - Keymaps for Amiga keyboards (udeb)
 console-keymaps-at - Keymaps for PC-style (PS/2 and AT) keyboards (udeb)
 console-keymaps-atari - Keymaps for Atari keyboards. (udeb)
 console-keymaps-dec - Keymaps for Dec keyboards (udeb)
 console-keymaps-mac - Keymaps for Apple keyboards (udeb)
 console-keymaps-sun - Keymaps for Sun keyboards (udeb)
 console-keymaps-usb - Keymaps for USB keyboards (udeb)
Closes: 275033 282379 282514 282515 282884
Changes:
 console-data (2002.12.04dbs-46.3) unstable; urgency=low
 .
   * Non-maintainer upload targeted at Debian Installer
   * Christian Perrier
     - Correct Portuguese USB keymap file so that it does not hang
       anymore
       Closes: #275033
     - Hack sort-keymaps for properly handling escaped commas
       Closes: #282379
     - Give safe defaults to all d-i supported languages in an attempt
       to avoid breaking fully automated installs
       Closes: #282884
     - Remove duplicate "croat" entries
       Closes: #282514
     - Correct English "translation" of some keyboard names for avoiding
       duplicate entries in some lists
       Closes: #282515
     - Correct Dutch translation from comments sent by Frans Pop
       and the Dutch team
Files:
 7a31455d260c1a15c80e647a0976b1f1 812 utils important console-data_2002.12.04dbs-46.3.dsc
 0919cf8e43da6891f2ab57b0346f82e6 1273442 utils important console-data_2002.12.04dbs-46.3.tar.gz
 908ed60912f93d9202ccf328e4bf6311 1142472 utils important console-data_2002.12.04dbs-46.3_all.deb
 25f151705024694548cb925e62e44d09 95782 debian-installer optional console-keymaps-at_2002.12.04dbs-46.3_all.udeb
 55cc96153bf58be524684c3577d32a9e 14914 debian-installer extra console-keymaps-mac_2002.12.04dbs-46.3_all.udeb
 5f692fca0fb8964d07f39d3898d2f924 29776 debian-installer extra console-keymaps-sun_2002.12.04dbs-46.3_all.udeb
 d3758616f5dd938448e1d50ab027fce9 89160 debian-installer extra console-keymaps-acorn_2002.12.04dbs-46.3_all.udeb
 794f88364a9762ec4488fe15dbd701e3 13742 debi...

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Sun, 28 Nov 2004 08:53:56 +0100
From: <email address hidden> (Denis Barbier)
To: Christian Perrier <email address hidden>
Cc: Frans Pop <email address hidden>, <email address hidden>
Subject: Re: Bug#282582: Some functions in cdebconf improperly handle escaped commas

--mP3DRpeJDSE+ciuQ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Wed, Nov 24, 2004 at 08:39:15PM +0100, Denis Barbier wrote:
> On Wed, Nov 24, 2004 at 06:54:04AM +0100, Christian Perrier wrote:
> [...]
> > In the future, something should be done for cdebconf handling this
> > situation better, imho. Probably make it fail rather than silently
> > exit.
>
> No, otherwise we would have to add lots of run-time checks. A better
> solution is to check that generated templates files are right, e.g.
> with help from linda/lintian.

Never mind, I was confused by mixing several bugreports.
Here is a tested patch to ignore Indices fields when they contain
invalid values.
When a (multi)select field has N items, only the first N items of
Default and Indices fields are considered, others are silently ignored.
Changing this behaviour might break other parts, so this is a feature
for now.

Denis

--mP3DRpeJDSE+ciuQ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="indices.patch"

Index: packages/cdebconf/src/strutl.c
===================================================================
--- packages/cdebconf/src/strutl.c (revision 24026)
+++ packages/cdebconf/src/strutl.c (working copy)
@@ -249,6 +249,14 @@
     return argc;
 }

+static int reset_index(int *oindex, size_t maxnarg)
+{
+ int i;
+ for (i = 0; i < maxnarg; i++)
+ oindex[i] = i;
+ return maxnarg;
+}
+
 int strchoicesplitsort(const char *origbuf, const char *transbuf, const char *indices, char **oargv, char **targv, int *oindex, size_t maxnarg)
 {
     int i;
@@ -265,27 +273,32 @@
         return DC_NOTOK;
     if (strchoicesplit(transbuf, targv, maxnarg) != maxnarg)
         return DC_NOTOK;
- if (indices == NULL || *indices == '\0') {
- for (i = 0; i < maxnarg; i++)
- oindex[i] = i;
- } else {
- cindex = malloc(sizeof(char *) * maxnarg);
- if (strchoicesplit(indices, cindex, maxnarg) != maxnarg)
- return DC_NOTOK;
- sorted_targv = malloc(sizeof(char *) * maxnarg);
- for (i = 0; i < maxnarg; i++) {
- oindex[i] = strtol(cindex[i], NULL, 10) - 1;
- if (oindex[i] < 0 || oindex[i] >= maxnarg)
- return DC_NOTOK;
- sorted_targv[i] = STRDUP(targv[oindex[i]]);
- }
- for (i = 0; i < maxnarg; i++) {
- free(targv[i]);
- targv[i] = sorted_targv[i];
- }
- free(sorted_targv);
+ if (indices == NULL || *indices == '\0')
+ return reset_index(oindex, maxnarg);
+
+ cindex = malloc(sizeof(char *) * maxnarg);
+ if (strchoicesplit(indices, cindex, maxnarg) != maxnarg) {
         free(cindex);
+ INFO(INFO_WARN, "Wrong number of arguments in Indices field");
+ return reset_index(oindex, maxnarg);
     }
+ for (i = 0; i < maxnarg; i++) {
+ ...

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Sun, 06 Feb 2005 08:24:32 +0000
From: Alastair McKinstry <email address hidden>
To: <email address hidden>
Subject: closing; fixed in NMU upload

Fixed in NMU upload 2002.12.04dbs-46.1.

Regards
Alastair

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Sun, 06 Feb 2005 08:25:23 +0000
From: Alastair McKinstry <email address hidden>
To: <email address hidden>
Subject: closing; fixed in NMU upload

Fixed in NMU upload 2002.12.04dbs-46.2

Regards
Alastair