Remapping voes - EESCHEMA symbol remappng is still struggling to convert older projects

[Art]

Today I tried to open one of the older projects (about 8 months old) It went through the dreaded remapping procedure and sure enough it didn't finish it gracefully. Trying to back trace the root of the problem brought me to the following findings.

1. The original project I was trying to open had symbol libraries "missing". Not sure what caused that but the last time I opened it, all the symbols were pulled from the proper libraries. If I open the same project with an older KiCad version I can go to the Library paths and assign proper paths to the symbol libraries, then restart the KiCad and after opening the schematics again everything opens just fine. I don't think it is really important to the problem at hand to concentrate on "why" the libraries were missing, although, like I said, last time I opened that project everything was assigned properly. What is important that the fix is simple - reassign all the paths to the proper libraries, restart and voala! (pardon my French)

2. When I try to open the same project with the new KiCad it naturally complains about libraries missing, re-maps all the symbols, then gives me a list of the symbols which were rescued (all of them) and then opens the schematic - so far everything acts as the previous case with the old KiCad. However if I go to the "Manage symbol libraries" all the proper libraries used in the project are already listed here! I can't fix the schematic by simply entering the right path to the proper libraries and then restarting KiCad! To add insult to injury, if I go and try at this point to open automatically generated "backed up" project from the back up folder, it greats me with a bunch of error messages and an empty schematics. So if you project haven't been backed up independently, you are screwed!

There is also an issue of symbols that have special characters in the name, which are no longer allowed (like "/" etc). In those cases EESchema doesn't even attempt to pull them from the rescue library, so they show as square boxes with a question mark and a partial name (or no name at all if the name started with the special symbol). So you get to exercise your brain cells and see if you can remember what was that symbol even called in the first place.

So, if you anticipate this issue you can open an older project with an older version of KiCad, make sure all the paths to the libraries are correct, then ...no you actually don't even need to save to project, just close KiCad, and open it with the new KiCad and everything gets re-mapped properly. If you open it up right away with the new version - you are stuck to re-assign all the libraries manually.

There might be an easier way to re-link all the proper symbols to the proper libraries other than doing by hand, but I haven't found it. The bottom line is, if I'm able to restore libraries manually, the remapping process should be able to do the same, since it pulls the right libraries references at the end of the process.

Application: kicad
Version: (6.0.0-rc1-dev-484-g9b50b9406), release build
Libraries:
    wxWidgets 3.0.3
    libcurl/7.54.1 OpenSSL/1.0.2l zlib/1.2.11 libssh2/1.8.0 nghttp2/1.23.1 librtmp/2.3
Platform: Windows 8 (build 9200), 64-bit edition, 64 bit, Little endian, wxMSW
Build Info:
    wxWidgets: 3.0.3 (wchar_t,wx containers,compatible with 2.8)
    Boost: 1.60.0
    OpenCASCADE Community Edition: 6.8.0
    Curl: 7.54.1
    Compiler: GCC 7.1.0 with C++ ABI 1011

Build settings:
    USE_WX_GRAPHICS_CONTEXT=OFF
    USE_WX_OVERLAY=OFF
    KICAD_SCRIPTING=ON
    KICAD_SCRIPTING_MODULES=ON
    KICAD_SCRIPTING_WXPYTHON=ON
    KICAD_SCRIPTING_ACTION_MENU=ON
    BUILD_GITHUB_PLUGIN=ON
    KICAD_USE_OCE=ON
    KICAD_USE_OCC=OFF
    KICAD_SPICE=ON

[Wayne Stambaugh (stambaughw)]

@Art, if you deleted or moved your old (v4) symbol libraries when you upgraded, this would cause the remapping to fail as you described. You have to maintain all of your old libraries in the correct paths in order for the remapping to be successful. Your safest bet is to create a project library from the cache file and make it the first symbol library in the library list of your project file. As long as the cache was not corrupted, AFAIK this will ensure 100% accurate remapping. The user documentation for this can be found at http://docs.kicad-pcb.org/stable/en/kicad.html#_upgrading_from_version_4_to_version_5

[Art (diametrix)]

@Wayne

My libraries live in my dropbox, so they never get moved when I upgrade. Like I said before, I'm not really concerned with the why the libraries were un-linked in the first place. I guess the bug description should be that when you try to upgrade a project that has symbols that are being pulled from the rescue library, after they have been remapped, there is no way to relink them to the proper library by adding the proper library to the project. The only way to do it is to go through all of the symbols manually.

This should be easily reproducible.

Why the backup didn't work is just beyond me.

[Wayne Stambaugh (stambaughw)]

@Art, The rescuer always rescues from the cache library when it cannot be found in any library. If the remapper cannot find the original library when it attempts to remap the symbol library reference, it has not choice but to fall back to the rescue library as the only guaranteed copy of the symbol. That's why you see all of your symbols referenced to the rescue library. You can remap the symbols after adding libraries to the symbol library table by using Tools->Edit Symbol Library References...

[Art (diametrix)]

@Wayne

You miss my point. I know that I can restore library links later manually. I stated so in the original bug description. Let me see if I can explain it better. Previously all you had to do is to enter correct paths to the correct libraries, restart EEschema and all the symbols would have correct links. Total time - 30 seconds. Now you go to Tools ->Edit Symbol Library and do it manually. IMPORTANT here is that all the correct libraries are already linked!!! Total time here depends on the design, but let’s say from 5-30 min. 5min>30 seconds, therefore it’s worse. Worse is opposite of better, which is what software needs to become as you develop it.

The main point here is that if the program is remapping the symbols, even though the symbol is assigned to the rescue library, the software should be able to pull it from the correct library, when that library is linked (just like the older version did)

[Wayne Stambaugh (stambaughw)]

@Art, there is a way to determine if the original v4 libraries were found during remapping. After remapping a v4 project, open the symbol library table and select the project tab. If all of the v4 libraries (the ones in the old symbol library list dialog) are not in this table, then the libraries were not found during remapping. There are a lot of reasons this could happen. When this happens, the remapping falls back to the rescue library. It is the only way to guarantee the correct symbol.

It did occur to me that you are asking for the remapping to look for the same symbol name in any new libraries in the in symbol library table. This would be exactly the same bug that the symbol library table was design to fix. I'm not going to undo this on the off chance the symbol with the same name in a different library might be the same symbol.

[Art (diametrix)]

@Wayne

I maintain that better is better. Not sure what big fix you were referring to, but if it takes me now better part of the hour to restore a project - that ain’t better

I can see that in certain cases you can run into trouble when you match symbols names automatically. However I’d rather fix one false positive later than do all of them by hand. I guess you can have a pop-up dialog or even a separate option somewhere to perform symbol search based on the symbol name, but leaving you to remap symbols manually is just not the way to do it. At the very very least you need to make a project back up that actually works, so you could go back and use older version of KiCad to restore all the missing symbols. Like I mentioned above the files in the backup folder didn’t open properly and I wasn’t able to backtrack.

[Wayne Stambaugh (stambaughw)]

On 10/06/2018 02:14 PM, Art wrote:
> @Wayne
>
> I maintain that better is better. Not sure what big fix you were
> referring to, but if it takes me now better part of the hour to restore
> a project - that ain’t better

The big fix is referring to the library ordering issue that has existed
since the beginning of the project. If you were never impacted by this
bug, you were just lucky. The next time you remap a project that remaps
all of the symbols to the rescue library, save the contents of the
message panel to a text file and attach it to this bug report so I can
see if the original libraries are being added to the project symbol
library table.

>
> I can see that in certain cases you can run into trouble when you match
> symbols names automatically. However I’d rather fix one false positive
> later than do all of them by hand. I guess you can have a pop-up dialog
> or even a separate option somewhere to perform symbol search based on
> the symbol name, but leaving you to remap symbols manually is just not
> the way to do it. At the very very least you need to make a project
> back up that actually works, so you could go back and use older version
> of KiCad to restore all the missing symbols. Like I mentioned above the
> files in the backup folder didn’t open properly and I wasn’t able to
> backtrack.
>

If possible, please include a project that exhibits this problem.
Unless someone changed something, the back up is a direct copy of the
project, symbol cache library, symbol rescue library if it exists and
all of the schematic files with a time stamp added to the file names.
You should be able to restore your project to it's original state prior
to remapping by copying the files in the backup back to the original
file names. Are you saying that the backup files are not the same as
the original files?

[Art (diametrix)]

@Wayne

About the project backup folder before remapping - I'm not sure about how it is intended to be used. Maybe I should've copied the files back to the main folder, but I just moved to that folder in the project explorer (or whatever it is called), opened the project file there and then opened the backup schematics. It gave me an error message that (The entire schematic could not be loaded. Errors occurred attempting to load hierarchical sheet schematics" And then the list of every single sheet that used to be in my project. I have only three files in the back up folder - one schematics sheet (which I'm assuming the top sheet of the hierarchical schematics) and two library files (one called cache and one called rescue). So right off the bat I can see that you are not backing up projects files properly. None of the hierarchical schematic sheets are backed up.

Another issue with remapping I noticed is that you can't cut and paste between the projects anymore. Previously you could create a new sheet in the project you want to copy schematics from, paste your schematic into the new sheet, and then copy the sheet file into the second project and then link an extra sheet to that project. Previously everything worked fine. Now it opens that new sheet with the empty squares again.

All in all, I think remapping feature broke a lot of things that previously worked and it is far from a well oiled machine.

[Art (diametrix)]

@Wayne

One more thing. Playing around with trying to link proper libraries, I added all my libraries into project specific libraries tab and removed the rescue library from there. Naturally, PCBNew wouldn't link to those new libraries and just replaced all the symbols with the empty squares. If at this point I try to restore back to the previous state by removing all the libraries I added and adding back the rescue library and exiting without saving, next time trying to open the same schematics pulls up the empty squares again. It never recovers!

>After remapping a v4 project, open the symbol library table and select the project tab. If all of
>the v4 libraries (the ones in the old symbol library list dialog) are not in this table, then the
>libraries were not found during remapping.

This doesn't seem to be true. I went back to the original state of the project (the one that got ruined by the remapping), opened it with an old version of PCBNew, re-linked all the symbols to the proper libraries (simply by linking the proper libraries and then reopening PCBNew), and then opened the same schematic with the new version of PCBNew. After the remap step, all the symbols are linked now to the proper libraries, HOWEVER there are none of the libraries listed in the project tab of the symbol libraries window. As a matter of fact both tabs look exactly the same way they did when the project got corrupted - all the proper libraries are listed in the Global Libraries tab and Project Specific Libraries tab has only the <project name>-rescue library. Which brings me back to the same question. I don't know, it is just one big convoluted mess that doesn't have any rhyme or reason.

[Rene Poeschl (poeschlr)]

I experimented quite a bit with the remap process. I documented my findings over at the forum FAQ. Maybe this helps you: https://forum.kicad.info/t/converting-kicad-version-4-projects-to-version-5-remap-a-project/13767

[Seth Hillbrand (sethh)]

Moving to incomplete. If you can provide an example that demonstrates the problem we can move it back and look at ways to resolve it.

[Launchpad Janitor (janitor)]

[Expired for KiCad because there has been no activity for 60 days.]