Schematic choose component slow for first time

[David Pearce]

Windows 64 Nightly (for several months)
When you open a schematic and "choose component", for the first time in a session, the dialog freezes for several minutes on a I5 desktop. Disk access is busy, but Windows Task Manager shows very low cpu use, so anti-virus is unlikely to be the cause.
I have 30 symbol libraries with 3982 components.
Afterwards selecting components is fast, even if I close KiCad and reopen it. Restart Windows and back to the slow state again.
I suspect that KiCad is opening the library files component by component, causing a very inefficient disk access.

Application: kicad
Version: (2017-07-12 revision 459fd9e58)-makepkg, release build
Libraries: wxWidgets 3.0.2
           libcurl/7.54.0 OpenSSL/1.0.2k zlib/1.2.11 libssh2/1.8.0 nghttp2/1.19.0 librtmp/2.3
Platform: Windows 7 (build 7601, Service Pack 1), 64-bit edition, 64 bit, Little endian, wxMSW
- Build Info -
wxWidgets: 3.0.2 (wchar_t,wx containers,compatible with 2.8)
Boost: 1.63.0
Curl: 7.54.0
KiCad - Compiler: GCC 6.3.0 with C++ ABI 1010
        Settings: USE_WX_GRAPHICS_CONTEXT=OFF
                  USE_WX_OVERLAY=OFF
                  KICAD_SCRIPTING=ON
                  KICAD_SCRIPTING_MODULES=ON
                  KICAD_SCRIPTING_WXPYTHON=ON
                  KICAD_SCRIPTING_ACTION_MENU=ON
                  BUILD_GITHUB_PLUGIN=ON
                  KICAD_USE_OCE=ON

[David Pearce (halzia)]

Process Monitor shows that reading the .libs takes much less than a second. The delay is due to reading every footprint, one file at a time

[Lorenzo Marcantonio (l-marcantonio)]

On Fri, Jul 14, 2017 at 07:03:57AM -0000, David Pearce wrote:
> Process Monitor shows that reading the .libs takes much less than a
> second. The delay is due to reading every footprint, one file at a time

Could be interesting to know if it's the parsing (which *can* be
optimized, IIRC it was already done in the past) or if Windows has an
horrible file opening overhead (wouldn't be surprised of that).

--
Lorenzo Marcantonio

[Nick Østergaard (nickoe)]

There is no such things as ".libs", what do you really mean? Is that your folders containing the .kicad_mod files?

[David Pearce (halzia)]

I meant the 30 libraries that I have active.
After the libraries have been read and a lot of Python activity, all inside a second or two, I then see minutes of .kicad_mod file opening.
What might help would be for a .pretty directory contents only to be read when a library that uses the footprint is expanded.

[Chris Pavlina (pavlina-chris)]

This is known. Reading the footprints all at once is unavoidable due to the new footprint selection features, which are still being worked on. I'll probably look into optimizing that bit a little more; unfortunately there are some parts that would require _serious_ refactoring of the parser to optimize properly, and I've already put a lot of work into it. We're going to have to settle for _some_ slowness on first load. Subsequent loads should be very fast.

I'm closing this as not a bug, I don't think this is a particularly big deal. It is on my radar though.

[jean-pierre charras (jp-charras)]

I don't think the parser creates issues, at least on Windows.
For me this is due to the disk access when a lot of files are loaded.

For instance, to load the same set of .pretty libs, the *same* project takes more than 30s if libs are stored on a usual hard disk, and 3s if libs are stored on a SSD.

[David Pearce (halzia)]

I am seeing minutes on a fast Windows PC with conventional drive. Maybe I have a fragmented disk structure and a slow antivirus

[jean-pierre charras (jp-charras)]

This is heavily depending on the number and size of .pretty libs to load
My test was made with loading 32 libraries (and no invasive antivirus), not the full available set.
But it shows the fact the parser is not the (main) culprit.