Feature Request: Hierarchal Checklists

[dotancohen]

I've been chasing a good hierarchical notes manager across three operating systems for about fifteen years, and Zim is probably the best one that I've found that works on my current OS (Linux). However, there is one feature that is missing to make it perfect: hierarchal checklists. This is the "killer feature" of ListPro (http://www.iliumsoft.com/site/lp/listpro.php).

Hierarchal lists would work like normal lists, but with two changes:
1) The list would have a hierarchy, similar to "sub-lists"
2) The list bullet would have a "smart checkmark" feature

The "smart checkmark" feature would work like this:
* When the user clicks a bullet, it toggles between a checked and an unchecked state. The default state for new bullets would be unchecked. A keyboard shortcut could also be assigned (F-12 seems to be unused).
* When the user changes the state of a bullet, all the bullets of the "sub lists" change to the new state of the affected bullet.
* When all the bullets of a "sub list" are checked, the parent item gets its bullet checked automatically.

See screenshot for proposed layout.

[Jaap Karssenberg (jaap.karssenberg)]

I like the screenshot. Wondering how this would fit in with the TODO list plugin, or just be an alternative.

To implement this we need a proposal on the wiki syntax to go with it. Probably a header for the start of such a list and a marker for each bullet item to give the state as well as a marker for expanded / collapsed on sub lists.

GUI could be realized either as a embedded treeview or by just embedding a checkbox widget in each line.
Embedded treeview would be great, but I have no clue if it is possible to have full fledged multiline text editing inside a treeview widget. The alternative is to really manage the list in the editor widget, but that needs a lot of code to support all the list operations and will have a long debugging time.

[dotancohen (dotancohen)]

As far as the wiki syntax goes, I would suggest using [] for unchecked list items, and [*] for checked items. This convention has several advantages:
1) They are already valid syntax items, in fact, they are already supported as automatic lists in Zim.
2) They are intuitive and easy to type when entering lists.
3) They leave open the possibility of using different check marks in the future, such as important items [!], percentages ([10%] for an item 10% complete, [50%] for an item 50% complete), and other creative uses with the remaining symbols. This could even be integrated with Styles (for instance, important items (items marked with [!] bullets) could be red and bold.

[dotancohen (dotancohen)]

Regarding the TODO plugin, I do not see any mutuat exclusivity between the two. They could both be used together (i.e., this would be another plugin).

[dotancohen (dotancohen)]

I do not think that having an embedded treeview would be the way to go. I would suggest that the indentation be handled in Zim, maybe by using bullets of differing widths. The right 16 pixels or so could be the [] or [*] indicator, and the rest would be transparent but would force indentation. This would have the added benefit of being able add / remove indentation easily via the keyboard, like in OOo, Listpro, and other implementations that have this. Shall I post an OOo document that has this feature?

[dotancohen (dotancohen)]

Regarding my last comment, scratch that!

It seems that Zim already supports indenting of lists, with Tab (and Shift-tab to unindent). So all that needs to be done to implement this feature are:
1) Have [] display as the proper graphic (a box)
2) Have [*] display as the proper graphic (a box with a checkmark)
3) Have a click on the bullet (and a keyboard shortcut) toggle it's state between [] and [*]

The "smart checkmark" feature could be implemented later. Let's get the basics out of the way, first.

[dotancohen (dotancohen)]

Upon further investigaton, it seems that Zim only supports Tab / Shft Tab indentation for checklists of the * variety (which is also the only list type in which character replacement occurs, from * to •). So add to the list of the previous comment:

4) Enable Tab / Shift Tab for [] and [*] lists.

[Jaap Karssenberg (jaap.karssenberg)]

Did a few minutes hacking in the train this weekend to see how difficult this would be. Would indeed work more or less as you describe above, except that you do not want a graphic (image) but a embedded check button widget.

[dotancohen (dotancohen)]

Seing how "•" is a unicode character, can you not use the same method for [] and [*]?

U+2610: ☐
U+2611: ☑

[Jaap Karssenberg (jaap.karssenberg)]

Yeah, of course we can do that, that would be quite easy.
But widgets are the way to go for a desktop application.
For one thing, a widget styles with your desktop theme,
and they will feel more interactive.

[Jaap Karssenberg (jaap.karssenberg)]

Attached a screenshot of proof of concept with both embedded widget and unicode characters for comparison. The widgets still have a few pixels border I need to get rid of, apart from that it seems like it might work.

[dotancohen (dotancohen)]

Is the widget necessary for the ☐ / ☑ switching on clicks? If • is already handled, how is implementing ☐ any different, other than the clicking to ☑?

In any case, I trust your implementation, I am just a bit anxious to finally get this functionality back after missing it for the past three years. Thanks!

[dotancohen (dotancohen)]

The widget implementation seems to have vertical alignment issues, can that be fixed in Zim? Is that a widget of your own creation, Jaap, or is it a standard GTK widget?

Can either the widget implementation or the Unicode implementation handle the keyboard shortcut / click for toggling state? Do either of them handle Tab/Shift-tab like the * lists do?

I must say that the widget implementation with it's shading of the checked items is much better on the eyes. I could imagine that it would be easier to scan quickly in long lists.

[Jaap Karssenberg (jaap.karssenberg)]

Ok. Conclusion about embedding widgets is that it looks good, but getting it right is difficult. So I was considering to go with unicode characters as a simple hack. However they look less good and on my windows install they do not show at all - seems the MS fonts do not cover the whole unicode range.

So I propose a third solution. I want to fake a widget by using a simple image as the bullet. Advantage of this approach is that we can control the look of the checkboxes independent from the font. An additional advantage is that the same images can be re-used when exporting to HTML (using the CSS property for the bullet image).

Any volunteer to draw two images ? One empty checkbox and one checked one. Preferably the checkmark should have some color and match the tango color scheme. Size should be something like 10x10 pixels, format PNG and background transparent. You can check how they look by inserting in zim as normal images.

[dotancohen (dotancohen)]

> Any volunteer to draw two images?

I will have them posted here shortly.

[dotancohen (dotancohen)]

Wow, that is harder than I thought. There are several proposals in there, from 9x9 to 12x12. None of them are satisfactory in my opinion, so please discuss what _is_ good in there and I will continue to work from that input.

I think that 10x10 is too small, by the way. With the checkmark I'd like to see at least 12x12, with still looks good with 10px text so long as the unchecked box is 10x10.

[Jaap Karssenberg (jaap.karssenberg)]

Attached my stab at this. Used the gtk stock icon "media stop" as the empty box and tried drawing a green checkmark. But it still shows that I just clicked a few pixels in paint :(

[dotancohen (dotancohen)]

Ouch!

I suggest that you start using any one of the graphics, and while you're a'coding I will have people more talented than us develop a graphic. I may wind up just paying someone to draw the icons, so long as I am free to release it under whatever license that I like. Which would be GPL like Zim.

I will try to have an updated, decent graphic by next week. I hope that there is code to use it by then! Thanks, Jaap!

[Jaap Karssenberg (jaap.karssenberg)]

Ok got most of it working. You can insert a checkbox, toggle it and have it indent like normal bullets. Check the snapshot at http://pardus.nl/tmp . Please test and if it does not introduce new bugs I'll merge it to trunk.

Just used the gtk-yes and gtk-no stock icons for now, which render by default as a green and a red globe respectively.

Cheers!

[dotancohen (dotancohen)]

Works:
1) [*] and [] are replaced with the correct icons.
2) Clicking icons toggles state. Yay!

Bugs:
1) Only the first line gets the icon. The next lines get "". Even replacing the "" text with [*] or [] just leaves me with the [*] and [] characters, no icon replacement.
2) No indentation on Tab. I can move to the beginning of the line and tab to indent, but with * lists one can tab _within_ the line. This is much better ofr creating hierarchies.

Feature Requests:
1) Keyboard shortcut (F12 has preference) for toggling state.
2) Smart Checking (As defined in the original report. Although I know that it won't be added now, I include it here for comleteness so that we won't have to hunt for missing items later on.)

I must add that I like the gtk-yes and gtk-no stock icons for this purpose, in fact, I move to leave them as the indicators. Thoughts?

[Jaap Karssenberg (jaap.karssenberg)]

Just verified this works under gtk 2.4 on win32 as well - seems indeed stable solution :)

Bugs:
1) Yes, reproducing bullet on the next line is still TODO. I assume we always want to initialize checkboxes on a new line with a unchecked state. Typing [], [ ] or [*] on the new line after removing the [obj] char should work though; remeber to type a white space to trigger the replacement.
2) Using <Tab> directly after the checkbox works for me, same as with bullets. Can you double check and look for error messages in the terminal ?

Open items:
1) Not sure yet about the keybinding, have to check some defaults in other applications. I was thinking just to use <Return> when standing directly behind the checkbox, but a more general keybinding could work anywhere in the line, which is much nicer.
2) Smart checking does not have to be very difficult. Just need to write some routines to find previous / next items in the list.
3) I would like to add a new requirement to integrate with the TODOList plugin. Would be nice to have TODO lists optionally use checkboxes instead of "TODO" and "DONE" tags. Together with the smart checking feature this would make it almost feature complete for tracking all kind of items.
4) Last open item is to update the manual / website to make other users aware of this feature.

Personally the yes/no icons remind me to much of a traffic light. Would rather have a real checkbox icon. Also a real checkbox will be more recognizable for users unfamiliar with this feature. I think I'll just take screenshots of checkboxes rendered in a standard Gtk theme.

[dotancohen (dotancohen)]

> Bugs:
> 1) Yes, reproducing bullet on the next line is still TODO.

Very good.

> I assume we always want to initialize checkboxes on a new line with a
> unchecked state.

I do not know if that is a valid assumption. I personally would only create unchecked items, but I cannot speak for all use cases. I know that GTK applications shy away from giving the user too many options, though, and have no problem with leaving it like this.

> Typing [], [ ] or [*] on the new line after removing the [obj] char should
> work though; remeber to type a white space to trigger the replacement.

The whitespace did it, thanks.

> 2) Using <Tab> directly after the checkbox works for me, same as with
> bullets. Can you double check and look for error messages in the terminal ?

Works now. I wonder what I was doing wrong.

In any case, could the tab indentation work from the middle of a line as well? I cannot imagine a use case where one would tab in the middle of a line, but I often want to adjust indentation from the middle of a line.

> Open items:
> 1) Not sure yet about the keybinding, have to check some defaults in
> other applications. I was thinking just to use <Return> when standing
> directly behind the checkbox, but a more general keybinding could work
> anywhere in the line, which is much nicer.

I will leave that to you to check which keybindings are available, as I do not use other GTK applications. But yes, the keybinding should work from any point in the line, not just the front.

> 2) Smart checking does not have to be very difficult. Just need to
> write some routines to find previous / next items in the list.

Good to know, thanks. From here it looks difficult :)

> 3) I would like to add a new requirement to integrate with the
> TODOList plugin. Would be nice to have TODO lists optionally
> use checkboxes instead of "TODO" and "DONE" tags. Together
> with the smart checking feature this would make it almost feature
> complete for tracking all kind of items.

That sounds good. I actually like the TODO list the way it is, as there is no other tagging ability in Zim. But you make a good argument.

> 4) Last open item is to update the manual / website to make other
> users aware of this feature.

How can I get write access? Or should I mail to you the changes / additions?

> Personally the yes/no icons remind me to much of a traffic light.

I thought of that too, which is why I liked it!

> Would
> rather have a real checkbox icon. Also a real checkbox will be more
> recognizable for users unfamiliar with this feature. I think I'll just take
> screenshots of checkboxes rendered in a standard Gtk theme.

I started with screenshots of checkboxes, but could not keep it in the 10x10 space without it looking ugly. If you can authorize me to make the icons 12x12 I will make sure that there are very nice icons soon. In fact, I am thinking that a third option would be useful as well, ☒ (U+2612, could be represented with [x] characters) and I had tried to keep that in mind when making the icons. I will make the three icon types in case you agree with me on the ☒ icon support.

A use case for ☒ would be a list of known items and an action that must be d...

[Jaap Karssenberg (jaap.karssenberg)]

> In any case, could the tab indentation work from the middle of a line as well? I cannot
> imagine a use case where one would tab in the middle of a line, but I often want to
> adjust indentation from the middle of a line.

There could be a key binding for indentation which works anywhere in the line, but we can not use Tab for that. I have some general ideas about improving indentation, but I will not work on them until after the python experiment.

>> 3) I would like to add a new requirement to integrate with the
>> TODOList plugin. Would be nice to have TODO lists optionally
>> use checkboxes instead of "TODO" and "DONE" tags. Together
>> with the smart checking feature this would make it almost feature
>> complete for tracking all kind of items.
>
> That sounds good. I actually like the TODO list the way it is, as there is no other
> tagging ability in Zim. But you make a good argument.

I'm thinking two options 1) keep TODOList as it is but treat a checked check box same as a "DONE" tag 2) ignore TODO tags completely and consider all checkboxes TODO items. Obvious this should be an user option with 1 as default.

>> 4) Last open item is to update the manual / website to make other
>> users aware of this feature.
>
> How can I get write access? Or should I mail to you the changes / additions?

Bazaar is decentralized, so you do not need access. Just branch and modify, then you can send me the resulting patch (bzr send).

> I started with screenshots of checkboxes, but could not keep it in the 10x10 space
> without it looking ugly. If you can authorize me to make the icons 12x12 I will make
> sure that there are very nice icons soon. In fact, I am thinking that a third option would
> be useful as well, ☒ (U+2612, could be represented with [x] characters) and I had tried
> to keep that in mind when making the icons. I will make the three icon types in case
> you agree with me on the ☒ icon support.

Sure, 12x12 is good for me. Keep in mind that text following the box will always align with the bottom of the image.

> A use case for ☒ would be a list of known items and an action that must be done on
> them. For instance, I am trying to decide which directories in $HOME to backup. I
> know that I must backup ~/.mozilla, so it gets a ☑ mark. I know that I will not back up
> ~/.trash, so it would get a ☒ icon. The ones that I do not yet know about (I will google
> them) remain with the ☐ icon.

We can add a third state later. Although will need to think how to toggle them. Just iterate trough states on mouse click or keybinding ? With 3 states the open state would still represent a TODO item while both other states represent closed items with positive / negative result.

[dotancohen (dotancohen)]

> There could be a key binding for indentation which works anywhere in the
> line, but we can not use Tab for that. I have some general ideas about
> improving indentation, but I will not work on them until after the python experiment

Very well, then. Might I suggest Alt-RIght and Alt-Left?

> Bazaar is decentralized, so you do not need access. Just branch and modify,
> then you can send me the resulting patch (bzr send).

I will play with that when I finish my exams in a few weeks.

> Sure, 12x12 is good for me. Keep in mind that text following the box will
> always align with the bottom of the image.

Will do.

> We can add a third state later. Although will need to think how to toggle them.

If ☐: Left Click changes to ☑, Middle Click changes to ☒
If ☑: Left Click changes to ☐, Middle Click changes to ☒
If ☒: Left Click changes to ☑, Middle Click changes to ☐

This has two advantages:
1) Intuitive
2) All states are reachable from all other states.
3) Users who do not know about ☒ and who left click only will never see ☒
4) Left Clicking always toggles between ☐ and ☑ so even if someone gets stuck, they can perform the intuitive command (left click) and toggle between the two more common states.

[Jaap Karssenberg (jaap.karssenberg)]

Ok, I'll wait for your images before proceeding.

For a triple state checkbox I think I stick to rotating all 3 states on left mouse. This also works with a single keybinding without the need for an extra combination. Only consequence is that you need to click twice to toggle from [*] via [X] to [ ], but I think that is a small price to pay.

Still pushing for a release early next month so I have my hands free to work on the other code tree. Guess this feature is set to go in with about one day more work.

Btw. Alt-Left and Alt-Right are used for page navigation, so we can not use them for anything else... I was more thinking about something like <Ctrl><Tab> while keeping <Shift><Tab> for unindent.

[dotancohen (dotancohen)]

> Ok, I'll wait for your images before proceeding.

Done, see attachment. I like the way the ☑ came out here, I was a bit creative. If these are unsatisfactory, though, I will have a designer do them.

> For a triple state checkbox I think I stick to rotating all 3 states on left mouse.

No, please! That gets confusing real quick, and is frustrating to work with long lists quickly like that.

> This also works with a single keybinding without the need for an extra
> combination. Only consequence is that you need to click twice to toggle
> from [*] via [X] to [ ], but I think that is a small price to pay.

That is a big price to pay, trust me! From experience, you want toggle behaviour!

> Still pushing for a release early next month so I have my hands free to
> work on the other code tree. Guess this feature is set to go in with about
> one day more work.

Excellent, thanks!

> Btw. Alt-Left and Alt-Right are used for page navigation, so we can not use
> them for anything else... I was more thinking about something like
> <Ctrl><Tab> while keeping <Shift><Tab> for unindent.

I have a future wish-bug for Ctrl-Tab to toggle between pages in recently used order, like Alt-Tab does for windows and like Firefox 3.1 will have for tabs. I'll file that now, but I really don't think that the Ctrl-Tab shortcut should be taken up for Indent. Is it possible to add a run-time or even a compile-time keyboard shortcuts configuration option? If so then I'll file a bug asking for that and stop complaining about keyboard shortcuts.

Note that I suggest Alt-Tab or simply Tab for indentation because that is the behaviour of other GTK applications (notably OOo). Ctrl-Tab is a navigational shortcut in at least one very popular GTK application (Firefox), so it should not be used for indentation here. If we cannot use Tab or Alt-Tab then consistency is broken with other GTK apps.

[Jaap Karssenberg (jaap.karssenberg)]

Merged to trunk - added checkbox icons

[Jaap Karssenberg (jaap.karssenberg)]

Ok, I just realized the images can be bigger than 12x12. Reason is that we render them as icons, and they get resized by the theme anyway. So drawing 24x24 and letting the theme scale down to an appropriate size looks much better. Did some Gimp'ing and came up with new design. Also added third state. Check the screenshot - will commit to trunk later.

[dotancohen (dotancohen)]

I will check it out as soon as it is committed, thanks. Should I file bugs on the implementation here, or in new bug reports?

[Jaap Karssenberg (jaap.karssenberg)]

> Should I file bugs on the implementation here, or in new bug reports?

Let's keep them in this thread until the feature is finished.

Btw. how would the "smart check" feature work for sublists that have bot [*] and [X] checked boxes ?

[dotancohen (dotancohen)]

> Let's keep them in this thread until the feature is finished.

The cursor moves to the position of the checkmarks when it is toggled via the mouse.

> Btw. how would the "smart check" feature work for sublists that
> have bot [*] and [X] checked boxes ?

It would stay unchecked. The smart check feature would ideally be activated under two circumstances:
1) An item with children has a state change. In this case, all children get changed to the same state as the parent..
2) At state change all sister items are checked. If they are all at the same state, then parent gets that state. If they are not all at the same state, then the parent gets an unchecked state (which it might already have).

These rules are recursive.

[Jaap Karssenberg (jaap.karssenberg)]

> The cursor moves to the position of the checkmarks when it is toggled via the mouse.

Yeah I know, but would need to overload a lot of C code to fix that I think, so will remain this way. Probably you would use the keybinding anyway when you are in the middle of typing a line.

>> Btw. how would the "smart check" feature work for sublists that
>> have bot [*] and [X] checked boxes ?
>
> It would stay unchecked. The smart check feature would ideally be activated under two
> circumstances:
> 1) An item with children has a state change. In this case, all children get changed to the same
> state as the parent..
> 2) At state change all sister items are checked. If they are all at the same state, then parent gets
> that state. If they are not all at the same state, then the parent gets an unchecked state (which it
> might already have).

I see one problem. If I understand correctly you propose to set all child items to the parent state even if they have been set already. Say I use a checkbox list to track TODO items. Items that are done I check with [*], although some items maybe get cancelled or whatever, so I check them [x]. Now when all items are checked I consider the parent item done so I want to check it, but I do not want to overwrite all [x] item to [*]. So I guess that toggling a item with sublist should only have consequences for [ ] sub-items.

[dotancohen (dotancohen)]

> Now when all items are checked I consider the parent item done so I
> want to check it, but I do not want to overwrite all [x] item to [*].

That's the whole point: a parent could only ever be checked (whether it be [*] or [x] marks) if all it's children are in the same state. So if an item is [*] it can be known that all subitems are [*]. One would not check a parent that has both [x] and [*] children, as that item (the parent) is neither done [*] nor canceled [x].

The situation that you mention would be possible if smart checking were made optional, which in retrospect is probably preferred. But at no time should smart checking allow an item to have a check if all it's children are not in agreement.

[Jaap Karssenberg (jaap.karssenberg)]

> But at no time should smart checking allow an item to have a check
> if all it's children are not in agreement.

I disagree. If items represent TODO items I need to be able to check them even if children are not all the same state. Else the item would stay on the TODO list for ever. I agree that it should not be possible to check an item if it has open sub-items. But with respect to smart checking I think of this as a 2 state algorithm where [ ] represents TODO and both [*] and [x] represent DONE, be it with different results. So if all sub-items are checked either with [*] or [x] the parent item cna be considered "done" or "closed" and get a check mark.

I do not want to have to disable smart checking for this, because for me that would undermine the whole reason to have this feature in the first place. If you have a list that is mixed between [*] and [x] but you can never check the parent item without loosing information, the parent should not be a checkbox in the first place but a simple bullet.

The only alternative would be a 4th state for "inconsistent" lists. But I hope you agree that adds more complexity than it merits.

[dotancohen (dotancohen)]

> If items represent TODO items I need to be able to check them even if
> children are not all the same state.

I suppose that depends upon whether or not you consider both [x] and [*] as DONE. If they are both DONE then why have two states? I know that you are trying to merge this feature with the TODO feature, but having inconsistency between parents and children is a bad idea and it destroys the purpose of smart checkmarks entirely.

> If you have a list that is mixed between [*] and [x] but you can never
> check the parent item without loosing information, the parent should
> not be a checkbox in the first place but a simple bullet.

The whole purpose of the smart checkmarks is to know that all children are in agreement. A parent should be checked iff (http://en.wikipedia.org/wiki/Iff) all it's children are in the same state. When one looks at a checked parent, he knows that all the children are checked as well without having to look at them.

> The only alternative would be a 4th state for "inconsistent" lists. But I hope
> you agree that adds more complexity than it merits.

I agree 100% that adding a fourth state for inconsistent lists adds too much complexity. When will it be shown, when all the children are checked, but some [*] and some [x]? Or will it be shown when there are some children [] as well? However, as complex as it would make this feature, I _still_ prefer it over having a parent with a checkmark that is inconsistent with one of more of it's children.

In the end, Jaap, I know that it's your baby and you are free to code it how you wish. However, allowing parents to be inconsistent with their children means that _all_ list items must be scrutinized when going through a list, as one cannot trust the parent to report on the children's states. That is bad: information should flow uphill and be trustable. Think of an army where the Head of Staff must ask each individual soldier if he is capable of performing his duty. It would be impossible to manage. The Head of Staff must be able to trust his generals. That is the whole purpose of a hierarchy. If I see an item marked [*] I must be absolutely certain that every item below it is [*] as well.

[Jaap Karssenberg (jaap.karssenberg)]

Ok, this is what I'll do. First I will fix the TODOList plugin to play nice with checkboxes. If TODOList turns out OK without smart checking it will be optional and off by default. If that is the case you can have it your way.

However my guess is that this consistency you are pressing so hard only becomes important when lists can be folded. In that case you want the parent item to show a correct "summary" of all that is below. But as long as we do not fold lists I can not imagine it would be difficult to eye-ball red crosses in a list of green checkmarks.

[dotancohen (dotancohen)]

> Ok, this is what I'll do. First I will fix the TODOList plugin to play nice with
> checkboxes. If TODOList turns out OK without smart checking it will be
> optional and off by default. If that is the case you can have it your way.

I'm not sure what you mean, but if you are saying that a checked parent means that all children have the same type of checkmark, then I thank you for it in the nicest words of the nicest language ever spoken.

> However my guess is that this consistency you are pressing so hard only
> becomes important when lists can be folded.

You are correct in assuming that I learned that behaviour from lists that had a folding option, but even in an unfolded state it is what makes the smart feature so powerful. I did not generally fold the lists even though the programs supported that.

> In that case you want the parent
> item to show a correct "summary" of all that is below.

Yes!

> But as long as we do not
> fold lists I can not imagine it would be difficult to eye-ball red crosses in a list of
> green checkmarks.

The fact that one _has_to_ eye-ball the list is the problem.

For that matter, even though I personally do not like folded lists, I could see where users may request that feature in the future. It would not be good to lock Zim out of that possibility.

Thanks, Jaap!

[Jaap Karssenberg (jaap.karssenberg)]

Fix released in version 0.27