Whitespace is too significant for factoids

[Space Hobo]

Our factoid code nicely becomes case-insensitive, but unfortunately it treats whitespace as significant. This causes the following problem::

 <@SpaceHobo> shakalaka: zoot boot
 <shakalaka> No way!
 <@SpaceHobo> shakalaka: zoot boot
 <shakalaka> I don't know what you mean, SpaceHobo.

We need to collapse whitespace down both in learn() and trigger(). The solution to this may also inform the discussion on bug#409225.

[Space Hobo (spacehobo)]

Per our discussion on IRC, we agreed that the 'key' variable from user input should be normalized in a function that is used both in learn() and in trigger(), and we should add an re.sub(r'\s+', ' ', key) call to it.

[Space Hobo (spacehobo)]

We ought to further consider normalizing away all punctuation, so that we would replace \s+ with \W+ in the substitution.

[Zen (shaboom)]

I implemented both, so that whitespace is preserved as single spaces and punctuation is removed completely. This has the advantage of not letting the presence of a comma or hyphen matter, but it does mean you can't make a unique response in cases where it might be important.

[Space Hobo (spacehobo)]

Here's a log of the discovery of this potential wrinkle:

 <@SpaceHobo> the a-team
 <shakalaka> In 1972 a crack commando unit was sent to prison by a military court for a crime they didn’t commit. These men promptly escaped from a maximum security stockade to the Los Angeles underground. Today, still wanted by the government, they survive as soldiers of fortune. If you have a problem, if no one else can help, and if you can find them, maybe you can hire the A-Team.
 <@SpaceHobo> the ateam
 <shakalaka> In 1972 a crack commando unit was sent to prison by a military court for a crime they didn’t commit. These men promptly escaped from a maximum security stockade to the Los Angeles underground. Today, still wanted by the government, they survive as soldiers of fortune. If you have a problem, if no one else can help, and if you can find them, maybe you can hire the A-Team.
 <@Zen> delicious
 <@SpaceHobo> shakalaka: delicious is pernicious
 <shakalaka> Roger that, SpaceHobo.
 <@SpaceHobo> del.icio.us
 <shakalaka> del.icio.us is pernicious
 <@Zen> haha
 <@Zen> except
 <@Zen> I would want del.icio.us to be per.nicio.us

[Space Hobo (spacehobo)]

Zen, looking at your branch, I see that you change \s+ to a single space first, then remove \W+. Since \W includes all of \s (and, in fact, the single space you replace it with), this is a little redundant. You may want to try [^\w\s]+ to kill all non-alnum non-whitespace characters.

[Zen (shaboom)]

I fixed that. Also, punctuation was handled after whitespace, so there were sometimes multiple spaces. I switched them around.

Also altered commands.py to leave trailing punctuation, since it was only aesthetic anymore. Input and output were differing in confusing ways.

We still need to decide whether stripping punctuation is really desired.

[Zen (shaboom)]

Example of why I edited commands.py:

16:45 <@SpaceHobo> shakalaka: literal f.r.o.o.-f.r.o.o.
16:45 < shakalaka> f.r.o.o.-f.r.o.o =is= <reply> ballin'

The factoid key is stored as just "froofroo", but at its earliest parsing the trailing "." was snipped. This is kind of awkward, but since the key is stored without this anyway I just left the trailing punctuation on for all other purposes:

16:58 <@SpaceHobo> shakalaka: literal f.r.o.o.-f.r.o.o.
16:58 < shakalaka> f.r.o.o.-f.r.o.o. =is= <reply> ballin'

That caused this problem:

22:20 <@Zen> shakalaka: ballin'?
22:20 < shakalaka> ballin'? is the illest

We don't want the question mark, but we DO want the apostrophe. So I hacked up commands.py some more so that when factoids are triggered or literaled, any periods, question marks, and exclamation points sitting at the end of the key get removed.

This means acronyms with periods and morse code are both kind of impossible. But it basically works.

[Space Hobo (spacehobo)]

The current branch has the following problem:

 23:00 <@SpaceHobo> "internet"
 23:00 <shakalaka> internet" is http://ecx.images-amazon.com/images/I/51K5MWCVQAL._SS500_.jpg

Since we're still stripping leading \W

[Space Hobo (spacehobo)]

http://bazaar.launchpad.net/~shaboom/yardbird/shakalaka/revision/99 fixes my complaint in #8

[xkcd (xkcd)]

There's nothing more heart-warming than the sound of a child's laughter.

[xkcd (xkcd)]

We do this with Robot9000 (the no-repeating-sentences enforcer) and it seems to work without too many unexpected collisions, although I'm not sure how we'd tell when one happened. But my guess is we'll get some weird moments plus one or two places where it's annoying that we can't have separate factoids for something (like possibly "ha ha" versus "haha"). On balance I'm in favor, at least until we see what it's like. Just leave the code so it's not too hard to uncollapse whitespace and if we need to we'll change it back.

[Nick Moffitt (nick-moffitt)]

xkcd:

  The trick here is that infobots need to *repeat* the text back to you. We are trying to normalize just the database keys, and leave the printed text as-was. Most of the movement on this merge request (https://code.launchpad.net/~shaboom/yardbird/shakalaka/+merge/9953) is about stripping trailing question marks and such without losing trailing apostrophes or other data.

We decided to leave individual spaces after normalization to keep the database keys legible when we use the Web-based admin.

[Nick Moffitt (nick-moffitt)]

xkcd:
> (like possibly "ha ha" versus "haha"). On balance I'm in favor, at
> least until we see what it's like. Just leave the code so it's not
> too hard to uncollapse whitespace and if we need to we'll change it
> back.

So if you look at the patch, we *collapse* whitespace, but *remove*
punctuation. So "ha ha" and "haha" are already two different keys, but
"ha ha" (two spaces) and "ha ha" (one space) are the same.

--
Nick Moffitt

[Nick Moffitt (nick-moffitt)]

After discussion in IRC, I've decided to merge Zen's branch, and we can later open further bugs that reference this one.