IPC-D-356 Fab Output not UTF-8 Compliant

[cflin]

When the labels of nets in the schematic include unicode characters, those names are properly displayed in pcb_new but show up as question marks in the IPC-D-356 fab output. This bug may be related to https://bugs.launchpad.net/kicad/+bug/1667686 .

Information about my KiCAD, running on Ubuntu 16.04.3LTS:

Application: kicad
Version: 4.0.7-e2-6376~58~ubuntu16.04.1 release build
wxWidgets: Version 3.0.2 (debug,wchar_t,compiler with C++ ABI 1009,GCC 5.4.0,wx containers,compatible with 2.8)
Platform: Linux 4.13.0-36-generic x86_64, 64 bit, Little endian, wxGTK
Boost version: 1.58.0
Curl version: libcurl/7.47.0 OpenSSL/1.0.2g zlib/1.2.8 libidn/1.32 librtmp/2.3
         USE_WX_GRAPHICS_CONTEXT=OFF
         USE_WX_OVERLAY=OFF
         KICAD_SCRIPTING=ON
         KICAD_SCRIPTING_MODULES=ON
         KICAD_SCRIPTING_WXPYTHON=ON
         USE_FP_LIB_TABLE=HARD_CODED_ON
         BUILD_GITHUB_PLUGIN=ON

[Seth Hillbrand (sethh)]

There have been a number of changes to the IPC-D-356 output. Can you test with a recent master? I am unable to reproduce with current master on Linux.

[cflin (cflin)]

I guess that when you say "current master," you mean "master" as in git? If so, I will attempt to try it. I am still using 4.0.7 because I literally make a living using KiCAD (and FreeCAD) and I am reluctant to switch to the development branch for fear of messing up my stable installation. But maybe this is enough impetus to take the plunge and try ver. 5.

I cannot do it right away, though, because I have to secure of my component libraries first. My intention was to file this bug against the stable version 4.0.7.

[Nick Østergaard (nickoe)]

That is understandable. If you just backup your old environment you should be safe. As we are nearing a new stable release, help verifying this bug is appreciated.

[Seth Hillbrand (sethh)]

You can also download a precompiled nightly build from the KiCad website to test this.

As Nick says, be sure to backup both your old config files and keep a copy of your project before opening in the nightly version. Files created or modified in nightly may not open in 4.0.7

[cflin (cflin)]

Before filing this bug, I opened a discussion thread about it here: https://forum.kicad.info/t/ipc-d-356-output-not-utf-8-compliant-in-4-0-7/9908 . Now, in that thread [davidsrsb] pointed out this code commit: https://github.com/KiCad/kicad-source-mirror/commit/243ce6979bf837ca860830f92514980e92e0ccc0 . If I read the C++ code correctly, this is actually the INTENDED behavior? To block unicode characters? So, how come [sethh] cannot reproduce this behavior?

[jean-pierre charras (jp-charras)]

This commit does not change the previous behavior. It fixes a potential incorrect behavior.

IPC-D-356 is a ASCII file, with a very poor syntax.
IPC-D-356A is not really better.

Documentation does not explain how to code non ASCII7 chars (unicode chars if you want) in ASCII files.
UTF8 is one way to code them.
Gerber files use an other way.
Unfortunately, UTF8 is not used in Windows, and gives not easily readable ASCII files, unless you have a text editor on Windows that can use UTF8 coding.

I am reluctant to use in a fabrication file a coding convention not usable on Windows, but only on Unix.

Net names in netlist are not free. They have serious constrains, depending on the software that use this netlist.

[cflin (cflin)]

It seems to me that the rest of KiCAD ***IS*** UTF-8 compliant on Linux, though. I think it should be left up to the users to decide whether they want to use UTF-8 or not; the program should not make that decision for them and could pop a warning that the non-ASCII characters may be problematic.

Fortunately, KiCAD is open source. I can get into the source code and remove that if block of C++ code. The down side is that I cannot just install KiCAD from the Ubuntu ppa's and will have to compile it from source.

[Jeff Young (jeyjey)]

KiCad is only UTF-8 compliant *where the extra-ASCII encoding is specified to be UTF-8*.

If the encoding is *not* specified, then KiCad doesn't know whether to write UTF-8, UTF-16, UCS-2, UCS-4, ISO-2022, or EBCDIC. There's no way to know *how* to be compliant if the extra-ASCII encoding isn't defined.

[Lorenzo Marcantonio (l-marcantonio)]

It is intended to block non-ASCII characters. IPC-356 is a punched-card style format (I'm not joking) and in fact it only handles upper case letters: I'm not sure but I think the 'safe' character set is 0-9A-Z plus _, + and -; MAYBE other characters are supported but I wouldn't count on it.

Since it's column based an encoding like UTF8 is not feasible, and, anyway, CAM350 chokes even with simple latin1 characters.

So, yes, I did it for a purpose! There is a 356B around but nobody uses it (even 356A is not widely accepted)