Grouped-by-value BOM generators don't sort references correctly within a group

[Ricardo Crudo]

The BOM files generated by the python scripts does not sort the references correctly.

For example, the following references:

R1, R2, R3, R4, R10, R11, R12, R20

will be sorted as:

R1, R10, R11, R12, R2, R20, R3, R4

[Wayne Stambaugh (stambaughw)]

Please either submit a patch or create a merge request on launchpad
against your branch so I can see the changes without having to merge to
my local repo and then figure out what you've changed. Thanks.

On 6/12/2015 9:22 PM, Launchpad Bug Tracker wrote:
> ** Branch linked: lp:~ricardocrudo/kicad/fix1464805
>

[Jeff Young (jeyjey)]

"bom_csv_sorted_by_ref" sorts correctly. Please re-open with more specifics if there's another one that does not.

[Aleksandr Sh (dsa-t)]

Here is what I found (I have custom field "Link" on some components):
bom_csv_sorted_by_ref.py sorts properly but does not output custom fields.

bom_csv_grouped_by_value.py sorts properly and outputs custom fields, but does not group.
Example output:
"Item","Qty","Reference(s)","Value","LibPart","Footprint","Datasheet","Link"
"","","C1","10V 100uF","Device:CP","Capacitor_THT:CP_Radial_D8.0mm_P3.50mm","",""
"","","C2","100nF","Device:C","Capacitor_SMD:C_0603_1608Metric","",""
"","","C3","100nF","Device:C","Capacitor_SMD:C_0603_1608Metric","",""
"","","C4","100nF","Device:C","Capacitor_SMD:C_0603_1608Metric","",""
"","","C5","100nF","Device:C","Capacitor_SMD:C_0603_1608Metric","",""
"","","C6","100nF","Device:C","Capacitor_SMD:C_0603_1608Metric","",""
"","","C7","100nF","Device:C","Capacitor_SMD:C_0603_1608Metric","",""
"","","C8","100nF","Device:C","Capacitor_SMD:C_0603_1608Metric","",""
"","","C9","15pF","Device:C","Capacitor_SMD:C_0603_1608Metric","",""
"","","C10","15pF","Device:C","Capacitor_SMD:C_0603_1608Metric","",""
"","","C11","10uF 25V","Device:C","Capacitor_SMD:C_0603_1608Metric","",""
"","","C12","10uF 25V","Device:C","Capacitor_SMD:C_0603_1608Metric","",""
"","","C13","10uF 25V","Device:C","Capacitor_SMD:C_0603_1608Metric","",""
"","","C14","10uF 25V","Device:C","Capacitor_SMD:C_0603_1608Metric","",""

bom_csv_grouped_by_value_with_fp.py groups but does not sort and does not output custom fields. (And creates unneccessary trailing ", ")
Example output:
"Ref","Qnty","Value","Cmp name","Footprint","Description","Vendor"
"C1, ","1","10V 100uF","CP","Capacitor_THT:CP_Radial_D8.0mm_P3.50mm","Polarised capacitor",""
"C3, C7, C8, C6, C5, C4, C2, ","7","100nF","C","Capacitor_SMD:C_0603_1608Metric","Unpolarized capacitor",""
"C10, C9, ","2","15pF","C","Capacitor_SMD:C_0603_1608Metric","Unpolarized capacitor",""
"C11, C12, C13, C14, ","4","10uF 25V","C","Capacitor_SMD:C_0603_1608Metric","Unpolarized capacitor",""
"C15, C24, ","2","100uF","CP","Capacitor_THT:CP_Radial_D8.0mm_P3.50mm","Polarised capacitor",""
"C19, C18, ","2","0.1uF","C","Capacitor_SMD:C_0603_1608Metric","Unpolarized capacitor",""
"C21, C20, ","2","10uF 10V","C","Capacitor_SMD:C_0603_1608Metric","Unpolarized capacitor",""
"C22, ","1","33pF","C","Capacitor_SMD:C_0603_1608Metric","Unpolarized capacitor",""
"C23, ","1","1.8nF","C","Capacitor_SMD:C_0603_1608Metric","Unpolarized capacitor",""
"C25, C26, C16, C17, ","4","22uF","C","Capacitor_SMD:C_1206_3216Metric","Unpolarized capacitor",""

bom_html_grouped_by_value.py groups and sorts properly but does not output custom fields.
bom_html_with_advanced_grouping.py groups and sorts properly but does not output custom fields.
bom_sorted_by_ref.py sorts properly but does not output custom fields.
bom_with_title_block_2_csv.xsl does not sort and does not output custom fields.
bom2csv.xsl does not sort, but outputs custom fields

bom2grouped_csv.xsl outputs custom fields but sorts like this:
D1 D12 D13 D2 D23 D24 D34 D35 D45 D46 D47 D48 D49 D50 D62 D63 D64 D65 D66 D67 D79 D80 D81 D86 D87 D88
And the header format is incorrect:
Reference, Quantity, Value, Footprint, Datasheet, Link
(The leading whitespaces are unneccessary)

[Jeff Young (jeyjey)]

@Alexsandr, thanks for the summary.

The exact format requirements (such as which custom fields are output) is going to vary by user. This is why we moved the BOM generators to Python and XSLT. The ones that ship with the product are examples more than anything else.

Things like whether or not custom fields are output (or /which/ custom fields are output) are going to vary by user (which is why the BOM generators are in Python and XSLT). Think of the ones that are shipped more as examples.

That being said, the two grouped_by_value scripts would certainly be better examples if they sorted the references within the group correctly.

[Nick Østergaard (nickoe)]

@Alexander, what version are you on?

[Aleksandr Sh (dsa-t)]

Application: kicad
Version: (5.0.0-rc2-dev-293-gfb882633b), release build
Libraries:
    wxWidgets 3.0.3
    libcurl/7.54.1 OpenSSL/1.0.2l zlib/1.2.11 libssh2/1.8.0 nghttp2/1.23.1 librtmp/2.3
Platform: Windows 8 (build 9200), 64-bit edition, 64 bit, Little endian, wxMSW
Build Info:
    wxWidgets: 3.0.3 (wchar_t,wx containers,compatible with 2.8)
    Boost: 1.60.0
    Curl: 7.54.1
    Compiler: GCC 7.1.0 with C++ ABI 1011

Build settings:
    USE_WX_GRAPHICS_CONTEXT=OFF
    USE_WX_OVERLAY=OFF
    KICAD_SCRIPTING=ON
    KICAD_SCRIPTING_MODULES=ON
    KICAD_SCRIPTING_WXPYTHON=ON
    KICAD_SCRIPTING_ACTION_MENU=ON
    BUILD_GITHUB_PLUGIN=ON
    KICAD_USE_OCE=ON
    KICAD_SPICE=ON

[Aleksandr Sh (dsa-t)]

Command line:
"C:\Program Files\Saxonica\SaxonHE9.5N\bin\Transform.exe" -t "%I" -xsl:"-path-" -o:"%O.csv"

[Aleksandr Sh (dsa-t)]

xsltproc -o "%O.csv" "-path-" "%I"
Gives the same results

[Martin Thomas (mtlaunchpad)]

I had written a patch to sort the references

[Martin Thomas (mtlaunchpad)]

I forgot to mention that we adapt the BOM creation with PCB new. It fits better to our manufacturer needs.
Maybe you consider it as handy and integrate it to the master branch. I created a git patch and also put the according files into the zip file.
The BOM gui form need an additional radio button or something else to chose between the both lists. Currently I coded it hard on list 1 (see line 135 in build_BOM_from_board.cpp).

So you can create a list as usal but sorted references or almost the same list but distinguished by the side, were the part is placed.

[Wayne Stambaugh (stambaughw)]

@Martin, the zip file you provided has a bunch of patches. We only need the one that fixes you problem. I cannot tell which patch is correct and I don't have time to sift through all of them. Please attach only the patches required to resolve the issue.

[Martin Thomas (mtlaunchpad)]

OK I changed the patch accordingly.

[Martin Thomas (mtlaunchpad)]

I filed a new bug as this patch does not fit here.
https://bugs.launchpad.net/kicad/+bug/1787329

[KiCad Janitor (kicad-janitor)]

Fixed in revision a96ebe8facb243f528ab47d1e9775497162d7150
https://git.launchpad.net/kicad/patch/?id=a96ebe8facb243f528ab47d1e9775497162d7150

[Piotr Gałka (piotrgalka)]

5.1.4 stable Win 7
*.py files dated 2019-08-10 17:13:08
I have noticed today that bom_csv_grouped_by_value_with_fp.py don't sorts correctly.
It is even not a alphabetic sort.
I get the following sequence:

C1, C2, C3, C4, C30, C31, C32, C23, C21, C22, C13, C8, C7, C33, C35

I absolutelly don't know Python.
During:
https://forum.kicad.info/t/to-exclude-virtual-footprints-from-bom/19410
I ended with replacing in my (little modified) copy of bom_csv_grouped_by_value_with_fp.py
gropued = net.groupComponents()
with
grouped = net.groupComponents(net.getInterestingComponents())
and noticed that except expected elimination from BOM elements marked with NU in field Installed I also get elements sorted correctly.
In getInterestedComponents() there is used ret.sort(...) and
in groupComponents() there is used g = sorted(g, ...) with the same function sortKey.
I don't know if that are two different sorting algorithms or two ways of calling the same.

So I ended with double sorting - first with ret.sort(...) and next with g = sorted(g,...) and getting the correctly sorted elements.

A moment ago I replaced in kicad_netlist_reader.py in function groupComponents the line:
g = sorted(g, key=lambda g: sortKey(g.getRef()))
with:
g.sort(key=lambda g: sortKey(g.getRef()))
and it looks that groupComponents alone (without getInterestedComponents) sorts references correctly.
I don't know if what I have done is save from Python point of view.

I think that it is good idea to eliminate uninstalled elements (like holes, but not only) from BOM so I think the functionality of getInterestedComponents() should be by default present in bom_csv_grouped_by_value_with_fp.py and others.
My first idea was to eliminate elements having footprint with Fabrication Attributes set to Virtual.
I'm not sure - may be it is simpler/better then adding extra field "Installed" and give it a value "NU".

Application: KiCad
Version: (5.1.4)-1, release build
Libraries:
    wxWidgets 3.0.4
    libcurl/7.61.1 OpenSSL/1.1.1 (WinSSL) zlib/1.2.11 brotli/1.0.6 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) nghttp2/1.34.0
Platform: Windows 7 (build 7601, Service Pack 1), 64-bit edition, 64 bit, Little endian, wxMSW
Build Info:
    wxWidgets: 3.0.4 (wchar_t,wx containers,compatible with 2.8)
    Boost: 1.68.0
    OpenCASCADE Community Edition: 6.9.1
    Curl: 7.61.1
    Compiler: GCC 8.2.0 with C++ ABI 1013

Build settings:
    USE_WX_GRAPHICS_CONTEXT=OFF
    USE_WX_OVERLAY=OFF
    KICAD_SCRIPTING=ON
    KICAD_SCRIPTING_MODULES=ON
    KICAD_SCRIPTING_PYTHON3=OFF
    KICAD_SCRIPTING_WXPYTHON=ON
    KICAD_SCRIPTING_WXPYTHON_PHOENIX=OFF
    KICAD_SCRIPTING_ACTION_MENU=ON
    BUILD_GITHUB_PLUGIN=ON
    KICAD_USE_OCE=ON
    KICAD_USE_OCC=OFF
    KICAD_SPICE=ON

[Ian McInerney (imcinerney)]

@piotr Please file that issue as a new bug report.

[Jeff Young (jeyjey)]

I'm pretty sure that one is already logged....