Twitter only shows @replies and DMs, not normal tweets

[Kevin Hughes]

Everything works fine, including @replies and DMs, but I can't see normal tweets.
I can send tweets fine, and the tweets I send from Gwibber *do* show up.

Running Gwibber on the console yielded nothing, but running gwibber-service shows the following on every refresh:

Gwibber Dispatcher: DEBUG ** Starting Refresh **
Gwibber Dispatcher: DEBUG <twitter:responses> Performing operation
Gwibber Dispatcher: DEBUG <twitter:receive> Performing operation
Gwibber Dispatcher: DEBUG <twitter:responses> Finished operation
Gwibber Dispatcher: DEBUG <twitter:private> Performing operation
Gwibber Dispatcher: DEBUG <twitter:private> Finished operation
Gwibber Dispatcher: DEBUG <facebook:receive> Performing operation
Gwibber Dispatcher: ERROR <twitter:receive> Operation failed
Gwibber Dispatcher: DEBUG Traceback:
Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/dispatcher.py", line 54, in perform_operation
    message_data = PROTOCOLS[account["protocol"]].Client(account)(opname, **args)
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/twitter.py", line 139, in __call__
    return getattr(self, opname)(**args)
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/twitter.py", line 142, in receive
    return self._get("statuses/home_timeline.json", count=count, since_id=since)
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/twitter.py", line 130, in _get
    if parse: return [getattr(self, "_%s" % parse)(m) for m in data]
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/twitter.py", line 77, in _message
    m = self._common(data)
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/twitter.py", line 63, in _common
    m["text"] = unescape(data["text"])
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/twitter.py", line 50, in unescape
    p.feed(s)
  File "/usr/lib/python2.6/sgmllib.py", line 104, in feed
    self.goahead(0)
  File "/usr/lib/python2.6/sgmllib.py", line 193, in goahead
    self.handle_entityref(name)
  File "/usr/lib/python2.6/sgmllib.py", line 436, in handle_entityref
    self.handle_data(replacement)
  File "/usr/lib/python2.6/htmllib.py", line 65, in handle_data
    self.savedata = self.savedata + data
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 0: unexpected code byte

Gwibber Dispatcher: DEBUG <facebook:responses> Performing operation
Gwibber Dispatcher: DEBUG <facebook:receive> Finished operation
Gwibber Dispatcher: DEBUG <facebook:images> Performing operation
Gwibber Dispatcher: DEBUG <facebook:responses> Finished operation
Gwibber Dispatcher: DEBUG <facebook:images> Finished operation
Gwibber Dispatcher: INFO Loading complete: 3 - ['Failure', 'Success', 'Success', 'Success', 'Success', 'Success']

[Omer Akram (om26er)]

can you please tell which version of gwibber are you using?

[Kevin Hughes (kevtufc)]

Apologies, I should have put that in the report.

I am unable to get to my computer at the moment but, if it helps, I am
running the latest Alpha of 10.4 and had run a full upgrade just before I
filed the bug.

I'll post the full details as soon as I can.

On 6 Mar 2010 14:20, "Omer Akram" <email address hidden> wrote:

can you please tell which version of gwibber are you using?

** Changed in: gwibber
      Status: New => Incomplete

--
Twitter only shows @replies and DMs, not normal tweets
https://bugs.launchpad.net/bugs/532952
Y...
Status in Gwibber: Incomplete

Bug description:
Everything works fine, including @replies and DMs, but I can't see normal
tweets.
...

[Kevin Hughes (kevtufc)]

I'm now on gwibber 2.29.90.1 and it seems to be working fine, now :)

Thanks,
Kev.

[Daniel Wiberg (dannew)]

I still have the same problem on the same setup, how do I extract the nice debug info you attached?

[Kevin Hughes (kevtufc)]

Just kill gwibber-service and then run it from a terminal with the -d option. It dumps a load of info whenever gwibber refreshes.

[Daniel Wiberg (dannew)]

user@host:~$ gwibber-service -d
Updating...
Updating...

This is what I get after an hour of running it.

[Daniel Wiberg (dannew)]

OK, gwibber-service -d didn't give me anything, but there is some info in ~/.cache/gwibber/gwibber.log, it's definitely the same error.

2010-03-09 09:35:31,206 - Gwibber Dispatcher - ERROR - <twitter:receive> Operation failed
2010-03-09 09:35:31,207 - Gwibber Dispatcher - DEBUG - Traceback:
Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/dispatcher.py", line 54, in perform_operation
    message_data = PROTOCOLS[account["protocol"]].Client(account)(opname, **args)
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/twitter.py", line 139, in __call__
    return getattr(self, opname)(**args)
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/twitter.py", line 142, in receive
    return self._get("statuses/home_timeline.json", count=count, since_id=since)
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/twitter.py", line 130, in _get
    if parse: return [getattr(self, "_%s" % parse)(m) for m in data]
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/twitter.py", line 77, in _message
    m = self._common(data)
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/twitter.py", line 63, in _common
    m["text"] = unescape(data["text"])
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/twitter.py", line 50, in unescape
    p.feed(s)
  File "/usr/lib/python2.6/sgmllib.py", line 104, in feed
    self.goahead(0)
  File "/usr/lib/python2.6/sgmllib.py", line 193, in goahead
    self.handle_entityref(name)
  File "/usr/lib/python2.6/sgmllib.py", line 436, in handle_entityref
    self.handle_data(replacement)
  File "/usr/lib/python2.6/htmllib.py", line 65, in handle_data
    self.savedata = self.savedata + data
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 0: unexpected code byte

[Kevin Hughes (kevtufc)]

Mine's worked for a couple of days and has now started showing the same error again :(

[Kevin Hughes (kevtufc)]

Further investigation shows that it was the tweet at http://twitter.com/Torbay_Council/status/10218385369 that was causing the issue.

Copying and pasting that tweet into gwibber and posting it seems fine, so maybe it's something to do with the way that the other client (HootSweet) has encoded it?

[Kevin Hughes (kevtufc)]

OK. It seems that &nbsp; causes the crash. I can force the error by posting &nbsp; (literally, that text) from gwibber myself.

[Kevin Hughes (kevtufc)]

Yep. That's it.

I bodged a fix by replacing line 63 in /usr/lib/python2.6/dist-packages/gwibber/microblog/twitter.py with:

    m["text"] = unescape(data["text"].replace('&nbsp;',' '))

Obviously this isn't a good fix, but I think it proves the specifics of this bug.

[Ryan Paul (segphault)]

@Kevin: Thanks for helping to track down the specific problem! It seems to only happen when the input is a unicode string.

This causes the exception:

>>> unescape(u"Test &nbsp; Test2")

But not this:

>>> unescape("Test &nbsp; Test2")

I'm not really sure why.

[Ryan Paul (segphault)]

Well, it looks like it's a bug in Python's sgmllib. I found the upstream bug report here: http://bugs.python.org/issue1651995

Strangely, I also found the associated bug report for the Ubuntu Python package (lp:240929), which says that it is already fixed in >2.6.1, but I'm definitely experiencing this issue with the 2.6.4 package. Very peculiar.

[Ryan Paul (segphault)]

I have confirmed that the bug is also exhibited with &reg; &pound; &cent; and &copy. However, if the numerical equivalent (e.g. &#160; &#169) is provided, the parser handles it ok.

[Daniel Wiberg (dannew)]

@Kevin: Your workaround works for me too. Thanks.

[Kevin Hughes (kevtufc)]

http://congreso.g138.gnconsultingltda.com/0igntSx7iH.htm

[Daniel Wiberg (dannew)]

I haven't seen this bug in a long time, is anybody still affected or could it be closed?

[Kevin Hughes (kevtufc)]

It's been fine for me for quite a while too.

BTW, sorry about the above spamming; my gmail got hacked :

[Omer Akram (om26er)]

Marking fixed then. If any body face this again feel free to open it again.