custom Environment variables are not used immediately

[Jan Krieger]

Hi!

I'm using 64-bit KiCAD Nightly 2017-12-21 (7586afd53) on win7-64bit. As a library maintainer I'm using customized environment variable to make KiCAD use libs that I checked out somewhere on my system.

I set them using the KiCAD menu entry "Preferences | Configure Paths". They point to checkouts of the KiCAD libs that I made below D:\KiCAD. I.e. ${KICAD_SYMBOL_DIR} = "D:\KiCAD\kicad-symbols", see attached screenshot

Now I'm doing the following:
1. Start KiCAD, with the last project I used (sometimes I'm using a local sym-lib-table sometimes not, the global sym-lib-table is emotied, The sym-lib-table always references using environment vars, e.g. "${KICAD_SYMBOL_DIR}/Amplifier_Buffer.lib")
2. Opening the library editor, I see a full list of all libs in
3. Now I see a list of all the libs that are currently referenced by the currently used sym-lib-table, but when I try to open the libs in the "Search Tree" the nodes are empty, i.e. the libs do not expand and show NO entries. This is true for all libs that not part of the default libs installed with the nightly (remember I'm a lib maintainer, so I'm working with cutting-edge versions and PRs that cannot yet be part of the nightly package). I can though expand libraries that are part of the standard installation and then see the contents in the installed libraries, NOT in the downloaded.
4. Now I go to the KiCAD main screen, click on "Preferences | Configure Paths", click OK and reopen the library editor. After it reloads the libs, I can now expand a lib and its contents from the checkout is shown.

To me this looks like KiCAD does not use the values for the environment variables set in its dialog "Preferences | Configure Paths", but only after the dialog has been opened. Until then it seems KiCAD uses the system environment variable, as it was generated during Install, which points to the installed libs.

This behaviour is a bit unexpected ... True, I can easily fix it by setting the system environment variables, but since I CAN set them in KiCAD, I would expect KiCAD to use these right away.

BTW: Would it be possible to set these environment variable on a per-project basis? That would be great for library review (I could have a project for libs in state 1 and another projet for libs in state 2 ... but I also see that that's maybe a special request that not many people will profit from ;-)

Would be great to see this fixed. Unfortunately I currently don't have enough time to look into the code myself and propose a patch :-(

PS: To all that do celebrate it: Merry christmas!

[Nick Østergaard (nickoe)]

Could you please try the latest nightly? The configure paths dialog was added to eeschema after your build. Maybe this fixes it. Keep in mind that no libs bare bundled with that install, but it seems you are managing the libs youself han have them elsewhere anyways. :)

[Jan Krieger (jkriege2)]

I downloaded, installed and tested the lates nightly "kicad-r9037.570866557-x86_64" and it shows the same behaviour. I also checked eeschema (in addition to the lib editor) and it shows the same behaviour.

I just overwrote the old install, meaning I have the libs inside the install dir (somewhere on C:) and the windows environment variable pointing there. In addition I have my checkouts on D:\KiCAD and overwrote the environment variables in KiCAD with the pathes on D:

The only difference in the new version is that the pathes now appear grayed out (see screenshot).

Is there a way to see which pathes are currently used by KiCAD (LOG-files or something)?

[Nick Østergaard (nickoe)]

They grey ones are those that are defined by the system. So for it to work in your case, you should unset the variables such that it will use the ones in the table.

[Jan Krieger (jkriege2)]

Yes, I know how to fix that, but why can I set them in KiCAD (and did), but this only takes effect AFTER I open and close the environment varoables dialog every time I start KiCAD. That's quite unexpected behaviour to me. I would expect this:
1. If they are set by the system and not set in KiCAD: USE system-values
2. If they are not set in system, but in KiCAD: use KiCAD-values
3. If they are set in system AND in KiCAD: use KiCAD-values (or warn user that there are inconsistent values, but since the dialog is there, I expect to be able to overwrite them)

[Nick Østergaard (nickoe)]

Ad point 3, IMHO systems variables should ALWAYS override.

From #3 I was not under the impression that you needed to open the env dialog. Is this also required in pcbnew?

[Jan Krieger (jkriege2)]

This issue is also there for pcbnew (just tested with recent nightly above). If I try to select a footprint, it accesses the path set by environment variables, which does not exist there! After opening and closing the env var dialog it accesses the correct location (set in the dialog).

> Ad point 3, IMHO systems variables should ALWAYS override.
You mean that they should always take precedence over what a user sets in KiCAD? I would disagree here. I would see it like this:

KiCAD should use system environment variables, when nothing is set inside KiCAD. If I (as a user) set something else in KiCAD, that should either overwrite the system variables (expected behaviour #1, IMHO), or store the new value in the KiCAD config-files and apply it to overwrite whatever is set in the system env. vars. (expected behaviour #2, this seems to be what KiCAD does at the moment, except the overwrite-part, which only works when opening and closing the env. var. dialog).

The current behaviour is somehow inconsistent, as it depends on whether I open a dialog (to view the variables) or not.

[Chris Pavlina (pavlina-chris)]

> Ad point 3, IMHO systems variables should ALWAYS override.

What? That's not how environment variables have ever worked. The parent process sets the environment first, then the child overrides it with any variables it explicitly sets. "System" variables come from the session manager and then KiCad overrides with any the user has set to non-default. This is how every program works by default on everything but Windows (which doesn't _have_ per-application env variables IIRC) and I see no reason for KiCad to do the opposite.

[Oivind Toien (otoien)]

I encountered this problem very long time ago, long before sym-lib-table, may be even in the stable version. Since then I have always made sure to untick "Environment variables" during install, and it has worked for me ever since. Why add them during install if you know you are not going to use the standard paths?

In a Win7 VM where that previously worked (and no previous system environment variables), I just did a test install of the last windows nightly (2017-12-24 revision 570866557) with the "Environment variables" ticked to on, and can confirm that this is what creates the behavior described above (I also have all the official libraries under in my home folder tree). There are however signs in the custom path setup dialog where the overridden paths become dark gray, and when I exit the dialog there is a warning message about the paths have been defined externally an may be temporarily overwritten. So KiCad is sort of honest about it.

I tried a reinstall (without uninstall) with this option ticked off again but the environment variables seem to still stick even after a reboot of the VM and copying back a backup of the ~\Appdata\Roaming\KiCad folder. Even a full uninstall and reinstall without the environment variables on did not fix it. I had to uninstall, go into the registry and search for "C:\Program Files\KiCad\share" and delete them under Session management - Environment, and then reboot before reinstall.

This is problematic, because users will have a hard time recovering from environment variables installed by accident.

Also since this is a recurring problem for users who use custom paths (second bug report mentioning it in a short time and me commenting on it), perhaps something should be done to it or to documentation during install. At least some instructions should be available on how to unintall the system environment variables (maybe it is?, I have not checked the wiki).

Perhaps it would be better if the system environment variables are not installed at all, but if the installer found that the standard custom paths were empty, it would fill in the standard paths with the default paths under C:\Program Files\KiCad\share (or whatever the location would be on the system).

[Wayne Stambaugh (stambaughw)]

This has been discussed before and I really don't want to rehash it. The current behavior is by design to allow developers to easily override the current settings when testing new symbol libraries. Honestly, I would rather not even expose users to environment variables. My original intent was to make this feature only available in debug builds for devs but we ran into an issue on windows where installs are relocatable so the internal default paths would be wrong.

[Oivind Toien (otoien)]

I see, I was going to suggest that the option to install environment variables should not be ticked on as default during install then, at least in the Windows version. If off users might then understand it is special option [only intended for developers].

I did a test install of the latest (2017-12-26 revision 90818afed) in a clean VM with only the footprint wizard option ticked on. However the base resulting KiCad share path (C:\msys64\mingw64\share) differ from that of the standard KiCad install (C:\Program Files\KiCad\share), so if this is going to work, one would if possible need to fix the default install paths so that those installing libraries with KiCad in the program folder have a correct paths predefined when system environment variables have not been installed.

Another idea is that if the library installs becomes a separate process, the library installer would define these paths for the kicad settings accordingly.

[Jeff Young (jeyjey)]

The logic is broken.

The first time you run KiCad, we apply the env-vars-override-kicad-settings rule.

However, if you ever open the Configure Paths dialog and click OK, we'll write ALL the path definitions into the settings file -- but you only get a warning if you modified one of the externally set ones.

After that we always overwrite the env-vars with the settings file, so there are no more externally-defined flags. And you therefore won't get a warning if you change one of them. (Nor will kicad pay any attention to changes made in the env vars.)

You might say: only write out the changed paths, and this is fine. Except that everyone out there already has them all written into their settings files, and we probably don't want to just blast those. Beside, the warning says changes only take effect until the next run -- which means we shouldn't be saving those at all -- even if changed.

Shall I reverse the logic so that we only pay attention to the settings file if the variable was NOT set externally?

[Wayne Stambaugh (stambaughw)]

If this is the case, then the logic is broken. In the original design
externally defined variables always took precedence over internally
defined variables. Setting an internal variable should not replace the
externally defined variable when edited using the path configuration
dialog nor should a variable defined in the settings.

On 3/8/2018 6:35 AM, Jeff Young wrote:
> The logic is broken.
>
> The first time you run KiCad, we apply the env-vars-override-kicad-
> settings rule.
>
> However, if you ever open the Configure Paths dialog and click OK, we'll
> write ALL the path definitions into the settings file -- but you only
> get a warning if you modified one of the externally set ones.
>
> After that we always overwrite the env-vars with the settings file, so
> there are no more externally-defined flags. And you therefore won't get
> a warning if you change one of them. (Nor will kicad pay any attention
> to changes made in the env vars.)
>
> You might say: only write out the changed paths, and this is fine.
> Except that everyone out there already has them all written into their
> settings files, and we probably don't want to just blast those. Beside,
> the warning says changes only take effect until the next run -- which
> means we shouldn't be saving those at all -- even if changed.
>
> Shall I reverse the logic so that we only pay attention to the settings
> file if the variable was NOT set externally?
>
> ** Changed in: kicad
> Status: Opinion => Triaged
>
> ** Changed in: kicad
> Importance: Undecided => Medium
>
> ** Changed in: kicad
> Assignee: (unassigned) => Jeff Young (jeyjey)
>
> ** Changed in: kicad
> Milestone: None => 5.0.0-rc2
>

[Jeff Young (jeyjey)]

Yeah, that’s what I thought, but thanks for confirming.

I’ll reverse the existing logic.

> On 8 Mar 2018, at 16:16, Wayne Stambaugh <email address hidden> wrote:
>
> If this is the case, then the logic is broken. In the original design
> externally defined variables always took precedence over internally
> defined variables. Setting an internal variable should not replace the
> externally defined variable when edited using the path configuration
> dialog nor should a variable defined in the settings.
>
> On 3/8/2018 6:35 AM, Jeff Young wrote:
>> The logic is broken.
>>
>> The first time you run KiCad, we apply the env-vars-override-kicad-
>> settings rule.
>>
>> However, if you ever open the Configure Paths dialog and click OK, we'll
>> write ALL the path definitions into the settings file -- but you only
>> get a warning if you modified one of the externally set ones.
>>
>> After that we always overwrite the env-vars with the settings file, so
>> there are no more externally-defined flags. And you therefore won't get
>> a warning if you change one of them. (Nor will kicad pay any attention
>> to changes made in the env vars.)
>>
>> You might say: only write out the changed paths, and this is fine.
>> Except that everyone out there already has them all written into their
>> settings files, and we probably don't want to just blast those. Beside,
>> the warning says changes only take effect until the next run -- which
>> means we shouldn't be saving those at all -- even if changed.
>>
>> Shall I reverse the logic so that we only pay attention to the settings
>> file if the variable was NOT set externally?
>>
>> ** Changed in: kicad
>> Status: Opinion => Triaged
>>
>> ** Changed in: kicad
>> Importance: Undecided => Medium
>>
>> ** Changed in: kicad
>> Assignee: (unassigned) => Jeff Young (jeyjey)
>>
>> ** Changed in: kicad
>> Milestone: None => 5.0.0-rc2
>>
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1740022
>
> Title:
> custom Environment variables are not used immediately
>
> Status in KiCad:
> Triaged
>
> Bug description:
> Hi!
>
> I'm using 64-bit KiCAD Nightly 2017-12-21 (7586afd53) on win7-64bit.
> As a library maintainer I'm using customized environment variable to
> make KiCAD use libs that I checked out somewhere on my system.
>
> I set them using the KiCAD menu entry "Preferences | Configure Paths".
> They point to checkouts of the KiCAD libs that I made below D:\KiCAD.
> I.e. ${KICAD_SYMBOL_DIR} = "D:\KiCAD\kicad-symbols", see attached
> screenshot
>
> Now I'm doing the following:
> 1. Start KiCAD, with the last project I used (sometimes I'm using a local sym-lib-table sometimes not, the global sym-lib-table is emotied, The sym-lib-table always references using environment vars, e.g. "${KICAD_SYMBOL_DIR}/Amplifier_Buffer.lib")
> 2. Opening the library editor, I see a full list of all libs in
> 3. Now I see a list of all the libs that are currently referenced by the currently used sym-lib-table, but when I try to open the libs in the "Search Tree" the nodes are empty, i.e. the libs do not expand and show NO ...

[KiCad Janitor (kicad-janitor)]

Fixed in revision 182b134872d54f9a6b8d88a90e271585d71cfc27
https://git.launchpad.net/kicad/patch/?id=182b134872d54f9a6b8d88a90e271585d71cfc27