Calendar view, Missing entry if GIVN has special chars

[Christian Hernmarck]

My son has a name with a ì (HTML: ì) - his birthday doesn't show up in the calendar view (whatever: day. month, year etc) until I change the ì in the GIVN line in the `wt_individuals` table to a normal i.

doesn't work:
...
2 GIVN Andrìn
...

works:
...
2 GIVN Andrin
...

Reproducable... (disappears when GIVN line has ì)

The ì is no problem in the NAME line...

Didn't see problems with this entriy in the main family datasheet records (but webtrees is new for me - so maybe there are other places with this problem).

[wooc (wooc)]

What webtrees version? PHP version? MySQL version? What collation?.

On my test system I don't see this error.
webtrees: 1.3.0 svn
PHP: 5.3.8
MySql: 5.5.16
Collation: utf8_unicode_ci

[ToyGuy (toyguy)]

@Ch. Hernmarck
The bug reporting system in not a place to report support issues with webtrees. We have a very active and prompt forum to discuss these kinds of problems/issues/interpretations. If, after discussing a display problem like this, we determine it is likely a bug, we request you submit it here.

We always suggest too, prior to creating a bug report, that you test the same issue on the SVN demo site to be sure that the problem:
1) exists at all on another site, and
2) still exists in the SVN code and has not already been fixed.

I have added a similarly-named INDI to the SVN demo and I don't see any issues. (04 MAY or 05 MAY) Andrìn Kennedy
http://webtrees.net/webtrees-svn/calendar.php?ged=kennedy.ged

Please open a thread on the forum for this topic and describe how we can enter a given name, with the proper character set, and reproduce this alleged bug. Thanks in advance.
-Stephen

[Christian Hernmarck (joximu)]

Hi

ok, I'll test with SVN.
I use the latest release version 1.2.7 on a Debian 6 (mysql 5.1.61, utf8 collation, php 5.3.3).

Sory for noise here.

/Christian

[kiwi (kiwi3685-deactivatedaccount)]

I can see the problem on my test system.

[fisharebest (fisharebest)]

@kiwi - give me some clues. I've added the ì character to the given name of an individual, and I see them on the calendar as expected.

[ToyGuy (toyguy)]

And, as I pointed out, it seems to work as expected on SVN where I added a person:
http://webtrees.net/webtrees-svn/calendar.php?ged=kennedy.ged

[kiwi (kiwi3685-deactivatedaccount)]

Its hard to give more clues. I agree that there is no problem evident on the DEMO, but there is on my local system, and as far as I can see the circumstances are identical.

I also have an additional problem, unlike Christian. Even after removing the special character, that individual continues to not appear on the appropriate calendar.

I notice that calendar.php uses ob_get_clean(); Any chance that coulld have different behaviours on different versions of PHP/MySQL?

[kiwi (kiwi3685-deactivatedaccount)]

I also do not see the problem on my live site, but do on my test (local) xampp installation.

Live site: PHP 5.3.6, MySQL 5.1.52
Xampp: PHP 5.3.5, MySQL 5.0.7

The DEMO is PHP 5.3.6, MySQL 5.1.52

[kiwi (kiwi3685-deactivatedaccount)]

From my information and Christians, the common factor seems to be PHP 5.3.5 or earlier.

[kiwi (kiwi3685-deactivatedaccount)]

After further testing I'm not sure if my experience is directly related to Christians or only partly.

I can reproduce the "disappearing" calendar entry without using the special characters.

I opened the calendar at a specific day. Selected one individual from the list, and opened her INDI record in a new tab. Edited the birth (that is the date shown on the calendar), changed nothing, hit "Save", and accepted the change.
Refreshed the calendar display - and she is no longer listed!

This does NOT happen on either the DEMO or my live site.

[Christian Hernmarck (joximu)]

I digged deeper

in includes/functions/functions.php
function get_all_subrecords

there is a line:
$ct = preg_match_all('/\n1 ('.WT_REGEX_TAG.')(.*)/', $gedrec, $match, PREG_SET_ORDER|PREG_OFFSET_CAPTURE);

which searches the positions of the "1" in the record.

With "i" I got "82" for the position of the second occurence ("1 SEX ..."), with "ì" I get 84... seems to be utf8 thing.

But the "$subrec = substr($gedrec, $pos1, $pos2-$pos1);" some lines further does not care about the 2-byte character - with 84 it gives me too much back.

Now - if it's depending on the php version: with 5.3.6 the pcre-version was 8.11, where some utf8-things were fixed...
So - it's maybe the problem.

[Christian Hernmarck (joximu)]

with

$subrec = mb_substr($gedrec, $pos1, $pos2-$pos1,'ASCII');

as the substr-line - it works...

but it's a hack to force mb_substr to handle the $gedrec as ASCII and count UTF8 chars several times.

This is only ok for certain situations... (utf-8 as system default, PHP < 5.3.6, and maybe others...)

/Christian

[fisharebest (fisharebest)]

1) We do not use the mb_XXXX() functions, because they are not always available. Also, they are very slow. Instead, we have our own set of functions utf8_XXX()

2) Have you enabled the mbstring.func_overload option in your PHP configuration? You can check this in Administration -> PHP Information.

My guess is that webtrees is calling the substr() function and expecting to work on *bytes*, but your system is silently substituting the mb_substr() function, which works on *characters*.

[Christian Hernmarck (joximu)]

I also tried utf8_substr but this didn't work either.

I think the problem is the preg_match_all which counts the special chars as two bytes - but since I have no newer php version I cannot test this.
At least it seems to be a problem of handling theese utf-8 chars in the pcre routines or in the string routines...
a preg_substr would maybe solve it - or a match_all without pcre routines....

I'll try to collect more infos.

I have a "normal" Debian 6 with default apache, php (via fastcgi), mysql - nothing special.
Default locale is de_CH.UTF8

[Christian Hernmarck (joximu)]

Hi guys

again, I could investigate a bit deeper.

I installed lampp at home and compared the functions preg_match_all and strlen/substr at home (5.3.8) with the version on the server (5.3.3). There was a difference until I found a difference in the config:
To use egroupware I had to set "mbstring.func_overload" to 7 (instead of 0) - this replaces (overloads) many str functions with the mb_-pendant. After setting it back to 0 and reastarting apache I had no problem with my "Andrìn" entry in the calendar.

Now - it's not finished - I set the option again to 7, restarted apache and had a look at the calendar - everything is fine.
But - after reloading the egw-site (was logged in) - something is changed an the Andrìn entry is gone.

I'll investigate further...

/Ch

[kiwi (kiwi3685-deactivatedaccount)]

On both my local and web systems all mbstring directives are identical. But the problem exists for me only on local.

[fisharebest (fisharebest)]

<<On both my local and web systems all mbstring directives are identical>>

Identical..... and set to what value?

[kiwi (kiwi3685-deactivatedaccount)]

<<Identical..... and set to what value?>>

See attached image.

But my point was that as they are identical, I would think they are unlikely to be the cause of the problem.

[Christian Hernmarck (joximu)]

For your information:
with 1.3.1 the bug I opened here is gone on my system.
(with 1.3.0 I still can reproduce).

I think it's because of this
http://bazaar.launchpad.net/~webtrees-team/webtrees/trunk/revision/12779

Thank you for helping... :-)

Christian

[kiwi (kiwi3685-deactivatedaccount)]

Thank you, but you need to upgrade to 1.3.2 now.

[Christian Hernmarck (joximu)]

Thanx - I already did... still no problems :-)