FR: recent changes should have the option to hide edits by bot

[Edward Betts]

http://openlibrary.org/recentchanges includes edits by ImportBot, there could be an option to hide these edits, like on MediaWiki.

The code for the recentchanges query is in infogami: infogami/plugins/api/code.py function 'get_recent_changes'. It builds a query and passes it to web.ctx.site.versions().

We might need to invent some new infogami query syntax, maybe something like this:

if no_bot:
    q['authors'] = {'not_in': ('/usergroup/bot', 'members')}

Or we could do a query to get all the bot usernames:

if no_bot:
    bots = withKey('/usergroup/bot').members
    q['authors'] = {'not_in': bots}

[George (george-archive)]

Good idea, Edward.

Becky - can you take please a look at a new UI for this? http://openlibrary.org/recentchanges

Tabs would work, no? Perhaps "By Humans" on the left, and open when you hit "Recent Changes", then "By Robots" on the right, closed... Also be interested to hear any ideas you have about making the changes lists more readable, if you think that's possible.

- Need a design for tabs on, tabs off, mouseovers etc (if there isn't one already).
- Might be nice to have some sort of summary statement at the top of the page?
  - "There were 10 record updates by patrons in the last hour."
  - "We imported 1,234,956 new records today."
- [CSS] FYI, on my machine it looks like the bottom of the text in each line in the table list is being cut off... Can provide a screenshot if needed.

I'm on Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.10) Gecko/2009042315 Firefox/3.0.10

** Can't for the life of me work out how to assign FR:s to people! **

[Edward Betts (edwardbetts)]

We could use http://en.wikipedia.org/wiki/Special:RecentChanges for UI ideas.

[webchick (webchick)]

Sounds good - I just assigned it to myself.

To assign a bug - click on the "Assigned to" header in the launchpad tracker table - you will see an "Assigned to" option - select a radio button (it is important to know the names of your team members) -

I know - a lot of ceremony - but it is a better than some of the other bug trackers I have used that oversimplify the process.

[webchick (webchick)]

ah - I see the culprit on the line-height - it is the css we use to truncate the book titles and user names, which can be quite lengthy -

.truncatedisplayname {
 width: 10em; /* constrain width try 20em */
    line-height:1em; /* to be adjusted */
    overflow: hidden; /* hide excess */
    position:relative; /* Fix IE missing overflow bug of r.p. span */
    background: #F9F8D0;
}

Removing the line-height fixes the problem - I may go ahead and do that on production and staging while we work out the other aspects of the UI.

[webchick (webchick)]

ok - here is a starting point for discussion -

http://www.invisible.net/openlibrary/recentchanges/

(pretend the edits on the human tab are not bots - i was having trouble finding human edits while the importbot was running)

I like the idea of keeping the comments short - perhaps with a "more" link to display lengthy comments (either as a sliding layer or as a modal window onclick) ... it keeps the page clean, but may lose a bit of the personality that the comments provide at first glance - we will have to see it in action to decide.

[George (george-archive)]

A good start!

Some quick suggestions:
- align tabs to the left (and match style)
- consider separate column for notes, and I think that truncated notes are fine, perhaps with a direct link to history page?

What are you thinking would go in Stats?

[Edward Betts (edwardbetts)]

The SQL to find recent changes is:

SELECT thing.key, version.revision, transaction.* FROM thing, version, transaction WHERE version.thing_id = thing.id AND version.transaction_id = transaction.id ORDER BY transaction.created desc LIMIT 50 OFFSET 0;

Here are the thing IDs of the bots:

/user/ImportBot = 17874995
/user/EdwardBot = 17873732
/user/RenameBot = 17873567

We can add 'author_id in (17874995, 17873732, 17873567)' to only show bot edits:

SELECT thing.key, version.revision, transaction.* FROM thing, version, transaction WHERE version.thing_id = thing.id AND version.transaction_id = transaction.id and author_id in (17874995, 17873732, 17873567) ORDER BY transaction.created desc LIMIT 50 OFFSET 0;

To hide bot edits we can add 'author_id not in (17874995, 17873732, 17873567)':

SELECT thing.key, version.revision, transaction.* FROM thing, version, transaction WHERE version.thing_id = thing.id AND version.transaction_id = transaction.id and author_id not in (17874995, 17873732, 17873567) ORDER BY transaction.created desc LIMIT 50 OFFSET 0;

[Edward Betts (edwardbetts)]

Here is my implementation of this feature. These two lines need to be in /macros/RecentChanges:

$ bot = query_param('bot', None)
$ changes = get_recent_changes(author=author, ip=ip, type=type, bot=bot, limit=limit, offset=page * limit)

To show only bot edits go to: /recentchanges?bot=only
To hide bot edits go to: /recentchanges?bot=hide

Below is a patch to infogami. In this implementation a user is considered to be a bot if the username ends in 'Bot'. This could be changed to check a usergroup or a bot flag in /type/user.

diff --git a/infogami/core/db.py b/infogami/core/db.py
index 7d9bf73..b6d98c2 100644
--- a/infogami/core/db.py
+++ b/infogami/core/db.py
@@ -52,7 +52,7 @@ def get_user_preferences(user):
     return get_version(user.key + '/preferences')

 @public
-def get_recent_changes(key=None, author=None, ip=None, type=None, limit=None, offset=None):
+def get_recent_changes(key=None, author=None, ip=None, type=None, bot=None, limit=None, offset=None):
     q = {'sort': '-created'}
     if key is not None:
         q['key'] = key
@@ -68,6 +68,8 @@ def get_recent_changes(key=None, author=None, ip=None, type=None, limit=None, of

     q['limit'] = limit or 100
     q['offset'] = offset or 0
+ if bot:
+ q['bot'] = bot
     result = web.ctx.site.versions(q)
     for r in result:
         r.thing = web.ctx.site.get(r.key, r.revision, lazy=True)
diff --git a/infogami/infobase/dbstore.py b/infogami/infobase/dbstore.py
index f21a054..86b4543 100644
--- a/infogami/infobase/dbstore.py
+++ b/infogami/infobase/dbstore.py
@@ -568,6 +568,15 @@ class DBSiteStore(common.SiteStore):
                 key = 'transaction.' + key

             where += web.reparam(' AND %s=$value' % key, locals())
+
+ if query.bot:
+ iter = self.db.query("select id from thing where key like '/user/%%Bot'")
+ bots = '(' + ','.join(str(i.id) for i in iter) + ')'
+ if query.bot == 'only':
+ where += ' AND author_id in ' + bots
+ else:
+ assert query.bot == 'hide'
+ where += ' AND author_id not in ' + bots

         sort = query.sort
         if sort and sort.startswith('-'):
diff --git a/infogami/infobase/readquery.py b/infogami/infobase/readquery.py
index 1a9c679..58c0828 100644
--- a/infogami/infobase/readquery.py
+++ b/infogami/infobase/readquery.py
@@ -233,6 +233,7 @@ def make_versions_query(store, query):
     if q.limit > 1000:
         q.limit = 1000
     q.sort = query.pop('sort', '-created')
+ q.bot = query.pop('bot', None)

     columns = ['key', 'type', 'revision', 'author', 'comment', 'machine_comment', 'ip', 'created']

[George (george-archive)]

Fantastic, Edward.

Becky - can you add to the Macro?

[Edward Betts (edwardbetts)]

It won't work unless the macro and the infogami code are updated at the same time. The recent changes page will break if only the changes to the macro are deployed.

[Edward Betts (edwardbetts)]

Any progress? Can we deploy?

[Anand Chitipothu (anandology)]

Added support for this in Infogami.

Query for all changes: {}
Query for all changes by bots: {"bot": True}
Query for all changes by humans: {"bot": False}

[webchick (webchick)]

I did some template work on that here -

http://dev.openlibrary.org/recentchanges?template_root=/user/webchick

But the plug-in changes must have broken it - I will see if I can debug -

Are we going to implement this on production?

[George (george-archive)]

Yes, we should implement this on production, and also on Upstream.

I can't view the link you've included above - "Unable to render this page"

[George (george-archive)]

Edward - Becky has noted that this code hasn't made it's way to production or upstream yet.

Can you please help port it over? It's blocking some development work we need on our new /admin screen, which you can see sans real data here:

http://upstream.openlibrary.org

Lance can help with tabs/css etc once the data's on the Recent Changes page, and Becky can get on with /admin.

Yes?

[Edward Betts (edwardbetts)]

Anand has performance objects to my implementation.

[George (george-archive)]

Great to see the tabs in place now.

I noticed that the "By Humans" tab only seems to display logged-in people's edits and no anonymous edits.

Please make "By Humans" display all edits that aren't made by a bot.

[Anand Chitipothu (anandology)]

> I noticed that the "By Humans" tab only seems to display logged-in
> people's edits and no anonymous edits.
>
> Please make "By Humans" display all edits that aren't made by a bot.

Done.

Shall we change "By Humans" to "By People"?

[George (george-archive)]

IMO, the copy's fine.

As I was testing this, I noticed that the pagination isn't working consistently. The number of rows on each page is inconsistent. Sometimes it shows 20, sometimes 5, sometimes 8. Should be 20 (or whatever the default is) at all times.

[Anand Chitipothu (anandology)]

On Mon, Dec 14, 2009 at 12:21 PM, George <email address hidden> wrote:
> IMO, the copy's fine.
>
> As I was testing this, I noticed that the pagination isn't working
> consistently. The number of rows on each page is inconsistent. Sometimes
> it shows 20, sometimes 5, sometimes 8. Should be 20 (or whatever the
> default is) at all times.

It hides the edits caused by user registrations. May be we should
create an AccountBot for creating user accounts so that all user
registrations go into Bot tab.

[George (george-archive)]

But that doesn't really make sense. Even if (from our perspective) a Bot is the thing that creates accounts, it makes more sense to say "George opened an Open Library account." as a history note under "By Humans".

Also - it's strange that we note 4 different history entries when an account is created. This needs to be fixed before we launch. By fixed, I mean bundled up into one entry, ideally listed as described above, and noted to the user's history trail too.

[Anand Chitipothu (anandology)]

Fixed. Account creation now makes one entry by the user with comment "created new account" and 3 entries by AuthorBot with comment "setup new account". Since AccountBot is marked as bot, these changes go in to the bot tab and not noticed in "By Humans" tab.

related bug: https://bugs.launchpad.net/openlibrary/+bug/445001