Implement notification updates for footprint/symbol libraries

[Piotr Gałka]

I have moved (using file management programm) some footprints betwean libraries and now the footprint lists showed by Footprint Editor disagree with files list in directories.
I include screen shots of directory lists and Footprint editor window.

I just don't believe in what is going on. Noticing the problem I have switched the computer off and on to force KiCad to reread directories. I have done it twice but with no effect.
I just supposed that KiCad reads the footprint lists from directories each time I run Footprint editor.

But now during writeing here I should say something - at second computer I have also the same KiCad version installed but not with my libarary list, but with ortiginal lists. When I run Footprint editor at that computer it behaves diferently from day to day. Sometimes it looks that it hangs during importing footprints at the beginning, but after more than one minute it finishes and looks OK. Sometimes it takes about 30 seconds to run Footprint Editor and sometims it looks like it needs no time - just run, and in one or two seconds it is ready.

Now come back to my working computer (it is not connected to internet so Windows 7 is not updated since 2 years, but I would not suppose it would be a source of problem).

Today I decided to move all my PLxxyy.kicad_mod from my S library to my H library.
I started doing it with KiCad using 'Save as' and 'Delete' - this way I moved PL0512, PL0513, PL0613. Then decided to move rest faster using file management program (FreeCommander). I have done it (probably not closing Footprint Editor, but not sure). And now the situation is as you can see in attached screenshots.
When I try to enter (double click) the unexisting footprint - just nothing happenes.
I have reopened Footprint Editor, and whole KiCad several times, I have swithced off and on computer - no effect.
I tried to find "PL0815" text in my KiCad installation directory and in my AppData/Roaming/kicad directory to discover where from KiCad can get this old directory situation. I didn't found.
To be sure I am not cheat by FreeCommander I looked at directory contenets using Windows Explorer - all files are where I think they are.
The most I am surprised is - where from KiCad could get libabry lists it showes.
I just don't believe in what is going on.

I plan to move files back, and then move them using KiCad and hope problem will gone. But certainly something is going wrong.
I have copied kicad_mod files from KiCad libararies to my libraraies many times duirng last month with absolutelly no problem. After reopening Footprint Editor the footprint list was updated.

Application: kicad
Version: (5.1.2)-1, release build
Libraries:
    wxWidgets 3.0.4
    libcurl/7.61.1 OpenSSL/1.1.1 (WinSSL) zlib/1.2.11 brotli/1.0.6 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) nghttp2/1.34.0
Platform: Windows 7 (build 7601, Service Pack 1), 64-bit edition, 64 bit, Little endian, wxMSW
Build Info:
    wxWidgets: 3.0.4 (wchar_t,wx containers,compatible with 2.8)
    Boost: 1.68.0
    OpenCASCADE Community Edition: 6.9.1
    Curl: 7.61.1
    Compiler: GCC 8.2.0 with C++ ABI 1013

Build settings:
    USE_WX_GRAPHICS_CONTEXT=OFF
    USE_WX_OVERLAY=OFF
    KICAD_SCRIPTING=ON
    KICAD_SCRIPTING_MODULES=ON
    KICAD_SCRIPTING_PYTHON3=OFF
    KICAD_SCRIPTING_WXPYTHON=ON
    KICAD_SCRIPTING_WXPYTHON_PHOENIX=OFF
    KICAD_SCRIPTING_ACTION_MENU=ON
    BUILD_GITHUB_PLUGIN=ON
    KICAD_USE_OCE=ON
    KICAD_USE_OCC=OFF
    KICAD_SPICE=ON

[Piotr Gałka (piotrgalka)]

I moved all PLxxxx back to S library.
I opened Footprint Editor, double clicked one of them - I sow it - everything works.
I copied (not moved) all PLxxxx from S to H.
I opened Footprint Editor - OK they are in both directories.
I deleted them from S directory.
I opened Footprint Editor - OK they are only in H.

So I have everything correct.

I tried to repeat the bug.
I moved one of them from H to S (footprint editor was closed during it) and then opened footprint editor - it has the wrong lists.

So the summary of all that.
If (Footprint Editor closed) I move footprint from one library to another then footprint editor has some problem with it.
When I copy footprint to second libarary then open and close footprint editor and then delete footprint from first library then everything is correct.

[Seth Hillbrand (sethh)]

Please delete fp-info-cache from your project directory and restart KiCad. Let us know if that doesn't fix it.

[Piotr Gałka (piotrgalka)]

If I close Footprint Editor but left main Kicad window then deleting fp-info-cache does't help.
But if I close all KiCad and then delete fp-info-cache - it helps.

I didn't serched the reason in project as in my imagination managing my libraries has nothing to currently opened project (as KiCad don't allow to run library editors without project I have one opened).

I am at stage of preparing my libs before starting any serious PCB.
I don't understand why to manage my libraries I need to have opened a PCB project. I don't see logical arguments for that. If any information of my library managing actions is important for project it will be writen to only one. May be to others when they will be opened. So why not allow for library managing with no project opened.

After I will finish with libraries I would like to make some PCB files without schematics but with all information about PCB and case around it.
I don't think such PCBs need a project, but suppose they will - we will see.
Then when I have schematic ready and decide what case I will just copy case pcb file, rename and go on.
If such PCB wil need a project file then I will don't know if to the destination project I should take project generated with schematic or generated with PCB. But all that is in category "we will see".

I am writeing all these just to say that it looks for me (as a new in KiCad world) that there are too many dependencies betwean files I would not expect.

[Seth Hillbrand (sethh)]

I've modified the title to reflect the solution for this. The cache has been specifically created to avoid the very common slow-down on windows that comes from re-checking the potentially thousands of files.

Implementing an inotify-type system will allow us to update on-demand and will resolve the issue you encounter. This is a potentially invasive change and will have many system-dependent differences, so the task is not small. Until then, we'll update the documentation to address this particular use case.

[Piotr Gałka (piotrgalka)]

OK. But...

As dividing the move operation in two steps (copy, show to KiCad, delete) is OK then it is possible to notice that change and reread.
Isn't it possible to notice move?

[Piotr Gałka (piotrgalka)]

I think that cashed footprint lists should be logically connected with PcbNew and Footprint Editor but not with project. So it should be in AppData/Roaming/kicad directory.

As change made out of KiCad can be difficult to notice (but deleting and copying is noticed) why not opening everything using cached footprint directory then in background check if lists need not to be updated.

I suppose that opening Footprint editor takes sometimes so long because it loads all footprints. If it would read only directories it would not take so much time, I think).

In FreeCommander there is an option of showing the directory sizes. When I have this option on I sometimes see it needs some time to calculate sum of all files in directory and its subdirectories. But is is never more than few seconds for one directory even with so many files as KiCad install directory.

I suppose the time is lost on interpretting the footprint files. I think that makeing at opening the program only the library lists can be really fast without any new system (I'm not programist - I don't know what is inotify-type system).

[Jeff Young (jeyjey)]

The cache is dependant on the library tables, one of which is project-specific. Therefore the caches must be project-dependant.

We do have another feature request somewhere for loading them in a background thread.

[KiCad Janitor (kicad-janitor)]

KiCad bug tracker has moved to Gitlab. This report is now available here: https://gitlab.com/kicad/code/kicad/-/issues/2440