Encountered an anomaly during bulk update of metadata

[John Batur]

I attempt to bulk modify the date field of the metadata of numerous books at once. The date I am using is 20 Aug 1948. Once complete the date is 19 Aug 1948. If I modify one book at a time everything works fine. The failure is only when updating multiple titles.

[ownedbycats (ownedbycats)]

Did you modify the book at a time when the GMT time would've at the previous date?

[John Batur (thaikneeebubbaz)]

The date I am using is 74 years old. GMT could not have a bearing. Plus it only happens in Bulk Metadata Edit. If I edit a single file it works fine.

[ownedbycats (ownedbycats)]

Open the single MDE and hover over the pubdate field. It displays the full timestamp, adjusted for timezone.

It's possible something went wrong with this but only in bulk MDE.

[John Batur (thaikneeebubbaz)]

Hovering on the date shows the data as indicated by the display within Calibre. If the field was individually edited it shows 20 Aug etc. If bulk edited it shows 19 Aug etc. This is an error in the Bulk Edit function of Calibre.

[Charles Haley (cbhaley)]

@Kovid: this happens if the date is before the "normal" EPOCH, 1/1/1970. Dates after the epoch work. Dates before the epoch (seem to) have an incorrect conversion to UTC, resulting in a negative one-hour erroneous offset.

I suspect this is a Qt bug, but I can't prove that.

[John Batur (thaikneeebubbaz)]

I had some spare time waiting for an update on another PC to complete and confirmed the previous theory. I did not confirm an erroneous 1 hour offset but I did confirm that the dates are off by one day if set to anything before 1/1/1970.

[Charles Haley (cbhaley)]

Note that the offset of 1 hour that I see might be because I live in the UK, where BST is 1 hour ahead of GMT/UTC. Other timezones might see different offsets.

[John Batur (thaikneeebubbaz)]

I live in the US and use a VPN but regardless all Dates show as Eastern Standard Time regardless of the date used. To get the modification in date I would need to be near the International Date Line. I cannot believe this is a Operating System issue. It is a mishandling of the data presented to Calibre. I've seen this in Main Frames when I worked for IBM and we approached the year 2000.

[Kovid Goyal (kovid)]

Doesnt reproduce for me with dates in 1948 on Linux, maybe its windows
specific. The only difference between single and bulk
metadata dialogs that I can see is qt_to_dt() is called with
as_utc=False in the single but not the bulk. I will change that, to
ensure both dialogs behave the same.

[Kovid Goyal (kovid)]

Fixed in branch master. The fix will be in the next release. calibre is usually released every alternate Friday.

[Charles Haley (cbhaley)]

@kovid, the change doesn't fix it. The problem seems to be that the pythong astimezone() method produces a different answer for dates before 1970, at least on Windows.

Please find attached a test program that demonstrates the difference. It is executed with calibre-debug. It calls qt_to_dt with 2 dates, one in 1969 and one in 1971. In the 1969 case it returns a UTC date. In the 1971 case it returns a local time zone date. I am in GMT and DST is True.

The output ofthe test program on my machine is:

Fri Jul 18 00:00:00 1969 UTC+01:00 converts to 1969-07-17 23:00:00+00:00
answer is in timezone GMT Standard Time
------------------
Sun Jul 18 00:00:00 1971 UTC+01:00 converts to 1971-07-18 00:00:00+01:00
answer is in timezone GMT Daylight Time

It is hard to believe that the python library is broken, but it sure looks like that.

[Kovid Goyal (kovid)]

This is what your script outputs on my Windows system

Fri Jul 18 00:00:00 1969 UTC+05:30 converts to 1969-07-17 18:30:00+00:00
answer is in timezone India Standard Time
------------------
Sun Jul 18 00:00:00 1971 UTC+05:30 converts to 1971-07-17 19:30:00+01:00
answer is in timezone India Daylight Time

Also, the OP says it works with the single metadata edit dialog which
uses the same code, so I dont really follow how it could be wrong in one
and not the other, beyond the as_utc thing.

[Kovid Goyal (kovid)]

It's not that one is local and the other is UTC, both are local its just
there seems to be daylight savings in 71 and not 69

[Kovid Goyal (kovid)]

Incidentally in iso8601.py changing

local_tz = SafeLocalTimeZone()

to

local_tz = datetime.now().astimezone().tzinfo

fixes the difference between pre and post 1970 dates, which seems to be
an artifact of the tzlocal object from python-dateutil on windows.

[Kovid Goyal (kovid)]

I replaced the use of the dateutil timezone objects with stdlib ones, so the difference between pre 1970 and post 1970 should no longer be present on Windows.

[Charles Haley (cbhaley)]

The change does fix it for me.

[John Batur (thaikneeebubbaz)]

How do I get/apply the change!

[Charles Haley (cbhaley)]

1) As @kovid said in post #10: "Fixed in branch master. The fix will be in the next release. calibre is usually released every alternate Friday." Calibre was last released on 14 Jul, 2023.

2) To get it sooner, set up a calibre development environment. See https://manual.calibre-ebook.com/develop.html

[John Batur (thaikneeebubbaz)]

I'll wait! But I truly appreciate the response. I have an Engineer of Computer Sciences degree but have always applied it to data center size main frames. Never bothered with the small stuff. Now I am too old and too tired to get involved. The only small computer software I got involved with was OS2 if you remember when IBM developed that but that was primarily for work stations to attach to main frames. Now MVS, VM & Parallel Sysplex I understand! Since retiring I have used my Electrical Engineer and Mechanical Engineer degrees more than anything else. Mostly helping my home town thru difficulties. Make it a good one!