Pcbnew error with fp-lib-table entry but no folder

[Evan Shultz]

As the footprint library has grown, there are more parts that need categorized. The librarians have also come up with better ways to classify footprints. Taken all together, this means we sometimes add new footprint libraries.

When this happens, the fp-lib-table file gets a new entry for the new .pretty lib. That's when we run into a problem.

Pcbnew (or the footprint editor) will show the new .pretty lib, but clicking on that library will cause the error in the screenshot I'll post in the reply. This is because fp-lib-table says that a .pretty lib is present, but the folder doesn't exist on the HDD. To be more salient, fp-lib-table shows Connectors_USB.pretty exists at a certain path on the HDD, but that folder isn't actually present.

A couple of things could fix this:
-KiCad could create the folder automatically. An empty .pretty folder is fine. However, there may be rights/access issues with creating a folder.
-KiCad could check to see if the folder is present and, if not, hide the library from Pcbnew. I would think in this case some message (maybe in the window's status bar?) should let the user know of the problem. At least this way there's no blocking dialog but the user isn't completely in the dark.
-I'm a lowly human so you probably can think of many other better ways to handle this.

At any rate, adding/removing/changing footprint libraries is inevitable and currently KiCad doesn't handle it very gracefully.

I've only tried this on Windows, so behavior and rights/access may be different in other OSes.

Application: kicad
Version: 4.0.6 release build
wxWidgets: Version 3.0.2 (debug,wchar_t,compiler with C++ ABI 1010,GCC 6.3.0,wx containers,compatible with 2.8)
Platform: Windows 7 (build 7601, Service Pack 1), 64-bit edition, 64 bit, Little endian, wxMSW
Boost version: 1.60.0
Curl version: libcurl/7.52.1 OpenSSL/1.0.2k zlib/1.2.11 libssh2/1.8.0 nghttp2/1.19.0 librtmp/2.3
         USE_WX_GRAPHICS_CONTEXT=OFF
         USE_WX_OVERLAY=OFF
         KICAD_SCRIPTING=ON
         KICAD_SCRIPTING_MODULES=ON
         KICAD_SCRIPTING_WXPYTHON=ON
         USE_FP_LIB_TABLE=HARD_CODED_ON
         BUILD_GITHUB_PLUGIN=ON

[Evan Shultz (evan-shultz)]

Here is the screenshot, as promised.

[Wayne Stambaugh (stambaughw)]

I'm not sure exactly how you expect kicad to know what the user intended in the case of bad footprint library table entries but I don't find either of your proposed solutions acceptable. We already have a footprint library wizard to download and maintain footprint libraries and a footprint library table editor to edit the footprint library table. This is one case where a dialog is justified because something is seriously broken in the footprint library configuration and needs to be fixed. If we put a warning in the message panel or status bar, the user may not see it before it gets overwritten by the next action that changes the text. If a user configures a footprint library table incorrectly or deletes a properly configured library accidentally, I don't see that as a problem for kicad to fix or work around. Unless someone comes up with a sane way to handle this issue, I'm tempted to mark this as invalid or wont fix.

[Oliver (schrodingersgat)]

Evan, I'm afraid I misunderstood your issue when we discussed this on the GitHub page. I thought you were saying that the error occurs when a .pretty dir exists but is empty.

If this is not the case, I agree with Wayne here that a missing .pretty dir is not something that should be handled differently than it already is.

Wayne,

I feel that this and similar issues would be better presented to the user with a saner error message:

Currently the format is:

TERSE_ERROR_MSG:
<Location in code where the error occurred>

This is useful for dev feedback but looks to the user to be jargon. Could we have the <error location> hidden in a drop-down box for these sorts of messages?

[Evan Shultz (evan-shultz)]

Thanks Wayne and Oliver for commenting.

fp-lib-table is updated by librarians when needed and it's not synchronized to the KiCad release schedule. For example, just this morning there was a load of conversation (not that it will happen) that would cause significant reorganization. See https://github.com/KiCad/Connectors_WAGO.pretty/issues/1#issuecomment-316956512.

This happens as the library grows and is improved, as mentioned above. It's out-of-band compared to the KiCad release schedule.

For example, recently some deprecated libraries were removed from KiCad (https://github.com/KiCad/kicad-library/commit/00fea12881013e615a6667fe552d574f407b6f82). When I use the Footprint Libraries Wizard in Pcbnew to update the libraries from GitHub, I still see the removed libraries in Pcbnew. I still have Air_Coils_SML_NEOSID. That tells me that the updater may touch the .pretty repos, but it doesn't update fp-lib-table. Maybe I'm wrong, but on my PC that's what I see when I tried just now.

So that means the librarians could add, change, or remove .pretty repos but that doesn't affect the fp-lib-table on the user's PC. There may be parts hidden from the user since they've moved around on GH but the fp-lib-table on the user's PC hasn't been updated.

I would assume it's best for fp-lib-table and the repos so stay in sync. **From what I see, this would mean KiCad needs a code change to pull down the fp-lib-table file when the libraries get updated from GH.** In that case, if fp-lib-table was updated by the user trigger in Pcbnew, we'd have the situation I described above where a folder doesn't exist but fp-lib-table is asking for it. The error message attached above is then displayed because no folder would exist by fp-lib-table has an entry for a repo that was added due to library changes.

To clarify, Oliver, there's no issue I can find with having empty directories on the user's PC. I do not know about empty GH repos.

I know this is a bit off the original post I made. I should have written this at a higher level the whole time so it was more inclusive of mismatches between fp-lib-table and the GH repos, with the desired outcome being that the users are always getting the latest and greatest from GH when they choose to use that feature to update their footprint libraries.

Any of the above might be wrong. Apologies if I've misunderstood.

[Jeff Young (jeyjey)]

@Wayne, is there an action for us here, or is the GitHub stuff in "limited maintenance" mode?

[Wayne Stambaugh (stambaughw)]

There is no action for us that I can think of or am willing to commit to. Footprint libraries will come and go. It is the responsibility of the user to maintain their footprint library table as libraries change over time.

[Evan Shultz (evan-shultz)]

I still have a different viewpoint, but perhaps it doesn't matter! :)

GH (or pulled from GH) is the current, freshest librarians. That's where the librarians are accepting contributions. And the sym-lib-table and fp-lib-table should always be updated to be in sync with the repo.

The issue is when the local libraries don't match GH, which seems like something KiCad could handle more gracefully. There could be an option in KiCad that would suck down the libraries from GH, overwriting/adding/deleting where necessary so that the local libs match GH. (It would also be reasonable to only touch libs that are in GH, or allow libs to be protected, so that a user's special corporate library won't be touched.) It wouldn't need to be the default, but giving the user the option to update their lib to the newest one in the KiCad UI seems like an obvious feature. Users that don't build their own library symbols/footprints, or do and want to see their contributions in the official lib as presented in KiCad, could click a button to sync their local libs.

Wayne, you mentioned at FOSDEM earlier this year that the GH libs won't be the default anymore. In that case, if the libs are shipped in a frozen state, how does the user get any lib updates? Will all users be stuck on the lib version that is at the HEAD of GH when the KiCad release shipped? The libs change much more quickly than KiCad releases (no offense) and requiring users to download the GH libs manually and overwrite their local lib seems convoluted. KiCad could make it easier for users that want to use the official libs in GH.

Certainly some users won't want to use the GH library and that's OK too. In that case, they just don't pull the libraries from GH.

This may seem like a deviation from the original topic, but the only thing I'm trying to get across is that if the user is using the GH libs, and they change, things can break. And when that happens KiCad isn't as helpful as it could be in keeping things working and updated. Consider the influx of EAGLE users.

[Wayne Stambaugh (stambaughw)]

On 3/15/2018 12:47 PM, Evan Shultz wrote:
> I still have a different viewpoint, but perhaps it doesn't matter! :)
>
> GH (or pulled from GH) is the current, freshest librarians. That's where
> the librarians are accepting contributions. And the sym-lib-table and
> fp-lib-table should always be updated to be in sync with the repo.

This would overwrite any custom library entries the user has added to
their global footprint library table. I don't think anyone would accept
this solution.

>
> The issue is when the local libraries don't match GH, which seems like
> something KiCad could handle more gracefully. There could be an option
> in KiCad that would suck down the libraries from GH,
> overwriting/adding/deleting where necessary so that the local libs match
> GH. (It would also be reasonable to only touch libs that are in GH, or
> allow libs to be protected, so that a user's special corporate library
> won't be touched.) It wouldn't need to be the default, but giving the
> user the option to update their lib to the newest one in the KiCad UI
> seems like an obvious feature. Users that don't build their own library
> symbols/footprints, or do and want to see their contributions in the
> official lib as presented in KiCad, could click a button to sync their
> local libs.

This has never been the goal of the KiCad project. There is a simple
reason for this. Corporate users (like me) cannot have there footprints
constantly changing while designing a board. Updating libraries has
been left up to the user's discretion. I'm not opposed to adding a
plugin to handle git repos directly but I would never allow any commit
that would automatically do this without the user expressly requesting it.

>
> Wayne, you mentioned at FOSDEM earlier this year that the GH libs won't
> be the default anymore. In that case, if the libs are shipped in a
> frozen state, how does the user get any lib updates? Will all users be
> stuck on the lib version that is at the HEAD of GH when the KiCad
> release shipped? The libs change much more quickly than KiCad releases
> (no offense) and requiring users to download the GH libs manually and
> overwrite their local lib seems convoluted. KiCad could make it easier
> for users that want to use the official libs in GH.

Existing users will have to manually update their footprint library
tables accordingly or use the footprint library wizard to walk them
through the process.

>
> Certainly some users won't want to use the GH library and that's OK too.
> In that case, they just don't pull the libraries from GH.
>
> This may seem like a deviation from the original topic, but the only
> thing I'm trying to get across is that if the user is using the GH libs,
> and they change, things can break. And when that happens KiCad isn't as
> helpful as it could be in keeping things working and updated. Consider
> the influx of EAGLE users.
>

This will only apply to existing users. New users will get the default
global footprint library table which includes all of the footprint
libraries distributed with the release of KiCad. We will also likely
provide periodic library updates for users who don't want to deal with
git to mainta...