Kicad error/crash when browsing to change footprint

[Oivind Toien]

In normal mode in Pcbnew (BZR 5743, also some earlier versions tested), after opening "edit pars" on a footprint, then pressing change footprint button, and then the browse button in the Exchange Footprints dialog, the opening of the window that lists all footprints often fails and an error message appears: "Errors were encountered loading footprints - IOERROR: std::bad_alloc.." etc. I have attached a screenshot. After pressing the OK button Pcbnew will typically crash on next operation, for instance trying to switch canvas to OpenGL mode.

This is on a Win7 64 bit system, Kicad built with latest Winbuilder release. I have both local footprints and footprints referenced on github, thus there are a very large number of footprints listed if the listing is successful (ca 850 as I recall ). A thought is that it had been better if the access/selection of footprint was consequent: Instead of presenting the very long list of footprints that are directly searchable, I think it would be better if it used the selection method used when placing new footprint. Why should they be different? Thereby there is no need to initially search through the whole library for each footprint both locally and on github before making a selection. (It takes quite some time if successful.)

[Chris Pavlina (pavlina-chris)]

I'm thinking you might actually be running out of memory. How much free RAM do you have when this happens?

Also, I submitted a patch to make the change you suggest - I agree.

[Oivind Toien (otoien)]

The computer is a quad core Thinkpad T440p with 16GB of RAM. There is 10-12 GB spare RAM. The browsing operation takes 500-600MB. All 8 virtual processor cores are active during the caching operation at 100%. Could it be running out of handles?
 I am attaching a screenshot from an unsuccessful browse with the task manager resource screen. During another test when the browse was successful, when switching to OpenGL mode after this there was only a rudimentary display without footprints, and Kicad crashed when I tried to do something.

[Nick Østergaard (nickoe)]

Could you try the builds on https://www2.futureware.at/~nickoe/, and see if you can reproduce the problem?

[Oivind Toien (otoien)]

Just to clarify, are you looking for me to check that winbuilder does not cause build problems? Otherwise I can just do an update from winbuilder once I get to a connection with higher bandwidth. I assume in both cases we are talking about testing the very latest production release.

[Wayne Stambaugh (stambaughw)]

Can you open the footprint viewer and view any footprints? This looks
suspiciously like a github plugin error when attempting to load the
footprint information for all of the libraries in the footprint library
table. The other option might be to try disabling the multithreaded
footprint loading in common/footprint_info.cpp by setting #define
USE_WORKER_THREADS to 0 instead of 1 and rebuilding kicad to see if that
works.

On 6/14/2015 5:47 PM, Oivind Toien wrote:
> The computer is a quad core Thinkpad T440p with 16GB of RAM. There is 10-12 GB spare RAM. The browsing operation takes 500-600MB. All 8 virtual processor cores are active during the caching operation at 100%. Could it be running out of handles?
> I am attaching a screenshot from an unsuccessful browse with the task manager resource screen. During another test when the browse was successful, when switching to OpenGL mode after this there was only a rudimentary display without footprints, and Kicad crashed when I tried to do something.
>
> ** Attachment added: "Screenshot with Kicad and Task Manager"
> https://bugs.launchpad.net/kicad/+bug/1464821/+attachment/4414826/+files/Kicad-Change-footprint-browse-crash-004.jpg
>

[Oivind Toien (otoien)]

I can confirm that I have no problems viewing footprints on github through the footprint viewer. This includes displaying the github 3d models (where available), which I store locally as only one root location for 3d models can be specified.

But of course the viewer only access one footprint library (.pretty folder) at a time, not the whole list of everything. If Chris Pavlinas suggestion/patch get into the production branch it might not be a problem any more if it invokes the regular way of adding a new footprint instead of the current list with everything in it used by change footprint browsing.

[Wayne Stambaugh (stambaughw)]

Then it is definitely a problem with the multithreaded footprint info
code. Even if Chris's changes fix you immediate problem. This code is
used elsewhere in KiCad so it should be addressed. We may have to add
an option to set this at configure time for certain platforms and/or
hardware combinations. I don't have this issue on my system with a core
i7 and 16G of memory so it may be your winbuilder build. I'm using
msys2 to create both mingw32 and mingw64 builds so there may be a
difference there.

On 6/14/2015 7:18 PM, Oivind Toien wrote:
> I can confirm that I have no problems viewing footprints on github
> through the footprint viewer. This includes displaying the github 3d
> models (where available), which I store locally as only one root
> location for 3d models can be specified.
>
> But of course the viewer only access one footprint library (.pretty
> folder) at a time, not the whole list of everything. If Chris Pavlinas
> suggestion/patch get into the production branch it might not be a
> problem any more if it invokes the regular way of adding a new footprint
> instead of the current list with everything in it used by change
> footprint browsing.
>

[Oivind Toien (otoien)]

OK, I will try the pre-built version once I get to to a connection with higher bandwidth.

[jean-pierre charras (jp-charras)]

The footprint viewer access only one library, but Cvpcb loads all libraries, like the browse button in the Exchange Footprints dialog.

Have you this issue with Cvpcb, when launching it from Eeschema ?

[Oivind Toien (otoien)]

A quick test: The 8 core 100% CPU load occurred in the same way when caching footprints in Cvpcb launched from the schematic editor, loading of the list was successful, but when I opened pcbnew after that and tried switching to OpenGL mode, Pcbnew crashed. (I normally prefer to assign footprints directly in the schmatic edtor and not Cvpcb.)

[Wayne Stambaugh (stambaughw)]

I took a more careful look at your screen shot and this is not a crash.
 This is the footprint information code complaining that something has
gone wrong while attempting to load the footprint information for every
footprint in every footprint library defined in your footprint library
table. Are you on a slow or intermittent internet connection? If so,
this could be causing your problem. If you do not have a reliable
internet connection, consider using git to clone the KiCad footprint
libraries to your system so you can access them locally without the
internet.

On 6/14/2015 5:47 PM, Oivind Toien wrote:
> The computer is a quad core Thinkpad T440p with 16GB of RAM. There is 10-12 GB spare RAM. The browsing operation takes 500-600MB. All 8 virtual processor cores are active during the caching operation at 100%. Could it be running out of handles?
> I am attaching a screenshot from an unsuccessful browse with the task manager resource screen. During another test when the browse was successful, when switching to OpenGL mode after this there was only a rudimentary display without footprints, and Kicad crashed when I tried to do something.
>
> ** Attachment added: "Screenshot with Kicad and Task Manager"
> https://bugs.launchpad.net/kicad/+bug/1464821/+attachment/4414826/+files/Kicad-Change-footprint-browse-crash-004.jpg
>

[Nick Østergaard (nickoe)]

FWIW you can also download them all on http://ci.kicad-pcb.org/job/any-kicad-pretty-lib-nightlies/lastSuccessfulBuild/artifact/*zip*/archive.zip

[Oivind Toien (otoien)]

Thank for the link Nick. I think it is better to store the footprints locally as all the other ones I have. (I use Walter's excellent library modules as primary source together with my own store, and a few more. I also have the original Eagle libraries working with the plugin, just in case there is need for something special, although I hardly ever use them as source for my own).

Wayne, you are correct, the title and name of the image file is a bit misleading. It should rather read: Browse fails and Kicad crash on further operations. I am on a fast university connection now, during this try with BZR5763 built with Winbuilder, the caching was "successful" (at 100% cpu activity as before on all 8 cores), but as soon as I tried switching to OpenGL mode, Kicad crashed (attached image). I will do further test with the nightly builds. Can they be installed in parallel to the one with Winbuilder, or is it best to test it in a VM?

[Oivind Toien (otoien)]

I tried installing the daily build in a VM, however that build (BZR5762) does not seem to understand windows paths, it inserts a '/' in the path names (unix style) instead of a '\'. Thus it cannot find locations for the local libraries. I am attaching a file that demonstrates it. Access to the github libraries worked OK.

This problem is not present in the Winbuilder version. (I did however notice one peculiarity: I have specified my own folders for libraries in the RunKicad.bat file but in the Kicad preferences the paths remains at defaults. KiCad seems to ignore these, which is good in this case. Perhaps this is the way it is meant to work.)

I will reconfigure to have the github libraries locally in the winbuilder version and see how that works.

[Oivind Toien (otoien)]

I tried installing the daily build in a VM, however that build (BZR5762) does not seem to understand windows paths, it inserts a '/' in the path names (unix style) instead of a '\'. Thus it cannot find locations for the local libraries. I am attaching a file that demonstrates it. Access to the github libraries worked OK.

This problem is not present in the Winbuilder version. (I did however notice one peculiarity: I have specified my own folders for libraries in the RunKicad.bat file but in the Kicad preferences the paths remains at defaults. KiCad seems to ignore these, which is good in this case. Perhaps this is the way it is meant to work.)

I will reconfigure to have the github libraries locally in the winbuilder version and see how that works.

[Oivind Toien (otoien)]

I am getting the same symptoms with all the footprints including the github ones stored and references locally in the winbuilder BZR 5763 version. If caching is successful without error message, I will get an immediate crash if I after closing the dialogs try switching to OpenGL mode as described above.

[Wayne Stambaugh (stambaughw)]

I cannot reproduce this bug on either 32 or 64 bit mingw builds. Here
are the steps I used to attempt to reproduce the bug;

1) Open any board in Pcbnew.
2) Use the edit hot key (E) on any footprint to open the footprint
   edit dialog.
3) Click the "Change Footprint(s)" button in the "Footprint Properties"
   dialog.
4) Click the "Browse" button in the "Exchange Footprints" dialog.
5) Click the "Cancel" button when the "Footprints" dialog is shown.
6) Click the "Close" button in the exchange footprints dialog.
7) Select "Switch Canvas to OpenGL" from the "View" menu.

If you are doing something different, please indicate what that is.
Also, please paste a copy of the KiCad version build information in your
bug report.

On 6/17/2015 4:42 AM, Oivind Toien wrote:
> I am getting the same symptoms with all the footprints including the
> github ones stored and references locally in the winbuilder BZR 5763
> version. If caching is successful without error message, I will get an
> immediate crash if I after closing the dialogs try switching to OpenGL
> mode as described above.
>

[Oivind Toien (otoien)]

There is a possibility that the problem is the Eagle plugin. Did you have the original eagle libraries that require the plugin in your list? I edited them out of the fp-lib-table and this time I did not get up to 100% on 8 cores. The browse was sucessful, and no crash when switching to OpenGL mode. This could of course be because I am accessing fewer libraries too, but the more normal cpu load now (<50% average) could perhaps indicate that it is the eagle plugin and related code that should be looked at.

[Nick Østergaard (nickoe)]

You could try with only the eagle libs enabled.

[Oivind Toien (otoien)]

Thanks for the suggestion. Doing this confirmed that the 100% cpu load on 8 virtual cores (and long time for buffering) was related loading eagle footprints through the plugin. The crash on performing #7 above was less consequent, but occurred if I had the schematic editor and the pcbnew 3dview open at the same time. Reverting back to the non-eagle fplib table and having similar Kicad elements open before the test did not result in crash on #7.

However: The standard eagle library alone is 13096 footprints vs. 4016 footprints in the non-eagle libraries. (a lot of the eagle components have multiple footprints associated with each library part, so that might be why there are so many of them).

Both of these numbers were listed in Cvpcb invoked from the schematic editor. After this listing of the eagle library, invoking Pcbnew, and directly trying to swith to OpenGL mode caused an error message : "Error during data upload to the GPU memory" and pressing OK then results in a display with the gray air lines and a very few text items that belongs to the footprints. So after all this can be a memory allocation related problem, although it occurs long before available ram is used up.

This was with BZR5766/Win7 pro.
[Btw. very nice with the new color choices in 3Dview!]

[Maciej Suminski (orsonmmz)]

Hi Oivind,

Is there any chance we can get the files that cause the described behaviour? It is much easier to debug when we can recreate the problem on our machines.

Regards,
Orson

[Oivind Toien (otoien)]

I have temporarily uploaded the files to a dropbox folder:
https://www.dropbox.com/sh/4loiwzjud49hzad/AAAD_rlD-67SC0gGj8-mY4yha?dl=0
Please notify me if you cannot access it.
The eagle library is in a zip file (there are also some sub-folders, but I do not think those are listed in the fplib file).
There is the fplib-table file. (Note that I changed nicknames to place them below other libraries when these are combined with other fplib entries.).
The readme text file has the path where the eagle library resides. For the testing one should probably place the library in a location that has at least as long path as this.

[Wayne Stambaugh (stambaughw)]

Thank you for the files. I just finished testing them and they do not
cause my home computer (core i7, 16 memory) to crash but it takes almost
two minutes to parse every single eagle library. I'm guessing you (or
any one else for that matter) don't want to wait that long (or longer)
for the footprint search library to appear so Chris's footprint viewer
patch seems like a better option. The other option is to pair down your
global footprint library table to the minimum libraries you use on every
project and then add any additional library requirements to your project
specific footprint library table. This is typically how work which is
why I don't see this kind of lag. At most, I only have a dozen or so
libraries total for any given project.

On 6/18/2015 11:33 PM, Oivind Toien wrote:
> I have temporarily uploaded the files to a dropbox folder:
> https://www.dropbox.com/sh/4loiwzjud49hzad/AAAD_rlD-67SC0gGj8-mY4yha?dl=0
> Please notify me if you cannot access it.
> The eagle library is in a zip file (there are also some sub-folders, but I do not think those are listed in the fplib file).
> There is the fplib-table file. (Note that I changed nicknames to place them below other libraries when these are combined with other fplib entries.).
> The readme text file has the path where the eagle library resides. For the testing one should probably place the library in a location that has at least as long path as this.
>

[Oivind Toien (otoien)]

DId you make sure to have just as long path for the home location of the eagle footprints?

I do have an "ownmods" folder listed in the fp-lib table that contains the footprints I have checked/modified and use for projects (and also footprints I created myself). However there is need to be able to browse a wider selection of footprints to use as starting points for these in the beginning of a project. Having to make edits to the fplib-table file for every project isn't very practical and user-friendly. Thus I have just kept a rather wide selection available to use for browsing and copying.

That said, It would probably not be a big loss if I edited out 90% of the eagle footprints from the fplib table, and it is very rare that I use them as starting point. It was just convenient to copy in the whole "official" table without thinking about sorting things. The eagle footprint fplib table is distributed in the "share" folder of the Winbuilder project. Thus it is very natural to just copy the whole table to one's own fplib table.

Perhaps there should have been two levels of footprint access: One wider for browsing/copying only and a narrower for the more used parts that change footprints and Cvpcb lists.

But I understand Cvpcb is on the way out? (I personally do not use it and rather do direct assignments in the schematic editor). If so, and if the change footprint feature modified to use the same one as for add footprints then we will not see these problems at all.

[Wayne Stambaugh (stambaughw)]

On 6/19/2015 7:09 PM, Oivind Toien wrote:
> DId you make sure to have just as long path for the home location of the
> eagle footprints?

No. I only used the Eagle libs you copied to dropbox.

>
> I do have an "ownmods" folder listed in the fp-lib table that contains
> the footprints I have checked/modified and use for projects (and also
> footprints I created myself). However there is need to be able to browse
> a wider selection of footprints to use as starting points for these in
> the beginning of a project. Having to make edits to the fplib-table
> file for every project isn't very practical and user-friendly. Thus I
> have just kept a rather wide selection available to use for browsing and
> copying.

This is your choice to include all of your libraries but you will pay
the price for library load times. You do not have to edit your global
library table for each new project. Only the project library table
needs to be changed per project. They are completely separate tables
that get merged when the project is loaded. This is all documented in
both the CvPcb and Pcbnew reference manuals.

>
> That said, It would probably not be a big loss if I edited out 90% of
> the eagle footprints from the fplib table, and it is very rare that I
> use them as starting point. It was just convenient to copy in the whole
> "official" table without thinking about sorting things. The eagle
> footprint fplib table is distributed in the "share" folder of the
> Winbuilder project. Thus it is very natural to just copy the whole table
> to one's own fplib table.
>
> Perhaps there should have been two levels of footprint access: One wider
> for browsing/copying only and a narrower for the more used parts that
> change footprints and Cvpcb lists.
>
> But I understand Cvpcb is on the way out? (I personally do not use it
> and rather do direct assignments in the schematic editor). If so, and if
> the change footprint feature modified to use the same one as for add
> footprints then we will not see these problems at all.
>

[Oivind Toien (otoien)]

I cannot reproduce this in the latest Windows nightly (2017-12-27 revision fab6fc69e). This bug report should be closed as invalid or fixed (as I understand there has been later changes to handling of Eagle files at some point that can have affected this).

[Nick Østergaard (nickoe)]

It is unclear when or if this was fixed directly. Also the original post is quite old now, so I will go with invalid status.