calibredb add cannot avoid duplicates

[xiatian]

Hi,
I tried to use calibredb add to add books from a directory, then I got an error
ValueError: U_INVALID_CHAR_FOUND and then it stopped.But if I add -d to allow duplicates, it goes well but duplicates generated.

[Kovid Goyal (kovid)]

That indicates one of your files has invalid unicode data in its
metadata. Post the full error, including the traceback. And you can
always remove duplicates after adding, using the duplicate finder
plugin.

 status incomplete

[Launchpad Janitor (janitor)]

[Expired for calibre because there has been no activity for 60 days.]

[xiatian (521xiatian)]

The Full output:
Traceback (most recent call last):
  File "site.py", line 77, in main
  File "site-packages/calibre/db/cli/main.py", line 255, in main
  File "site-packages/calibre/db/cli/main.py", line 55, in run_cmd
  File "site-packages/calibre/db/cli/cmd_add.py", line 374, in main
  File "site-packages/calibre/db/cli/cmd_add.py", line 169, in do_add
  File "site-packages/calibre/db/adding.py", line 190, in recursive_import
  File "site-packages/calibre/db/adding.py", line 149, in import_book_directory_multiple
  File "site-packages/calibre/db/cli/cmd_add.py", line 79, in add_books
  File "site-packages/calibre/db/cli/main.py", line 192, in run
  File "site-packages/calibre/db/cli/cmd_add.py", line 65, in implementation
  File "site-packages/calibre/db/cli/cmd_add.py", line 55, in add_books
  File "site-packages/calibre/db/cache.py", line 1596, in add_books
  File "site-packages/calibre/db/cache.py", line 63, in call_func_with_lock
  File "site-packages/calibre/db/cache.py", line 1536, in create_book_entry
  File "site-packages/calibre/db/cache.py", line 1521, in has_book
  File "<string>", line 5, in lower
ValueError: U_INVALID_CHAR_FOUND

[Kovid Goyal (kovid)]

That is indeed a book in your library with invalid characters in its title causing that. If you attach the metadata.db file from your calibre library folder, I can tell you more.

[xiatian (521xiatian)]

You can see the metadata.db file in the attachment. For your convenience, I also uploaded it to google drive. This is the temporal link: https://drive.google.com/open?id=1vINA-t-LVP-R_E20Te-cYCzL8SmNCof4
If this is caused by invalid characters in one of the books' title, why it can run successfully with -d (duplicate) option?

[Kovid Goyal (kovid)]

Because it is the duplicate check that works on the book titles and is failing on the title with the invalid data.

[Kovid Goyal (kovid)]

DO a search for id:3479 and edit the title and you should be fine.

[Kovid Goyal (kovid)]

Fixed in branch master. The fix will be in the next release. calibre is usually released every alternate Friday.

 status fixreleased

[xiatian (521xiatian)]

Great! I found that book and the title did contain some special characters. I deleted it from my library and now I can add books successfully!
  Thank you so much!!!

[xiatian (521xiatian)]

I'm sorry that I forgot to reply to you on this thread for several months! I came across this problem again today so I came here and you helped me fix it quickly. Really really thank you,kovid!

[Kovid Goyal (kovid)]

You're welcome :)