Support page templates in new page dialog

[dotancohen]

I find that many of my Zim notes have similar format. It would be nice if Zim supported templates for new pages. I would suggest that templates be per notebook by default, with a checkbox to make them global across Zim.

[Jaap Karssenberg (jaap.karssenberg)]

Yes, this is a feature that is on my TODOList for future development. You may have noticed that we actually have two templates coded into zim already, one for new pages in general and one for calendar pages in specific. So far this is a hack and not really configurable. (You can change the wiki/_New.txt template to change the global template - but a bit of a hack.)

* A default template should be configurable per namespace
* All configured templates should be copied into the notebook for portability (but this can be hidden in the ui)
* Keep a library of global templates to choose from (include some examples e.g. books / recipes / ...)
* The "new page" dialog should allow you to choose a non-standard template from a dropdown

This work will be delayed untill the python port is available. The plan is to use the exact same template interface as for exporting with full access to all page and notebook properties from the template.

Also think about an interface for plugins to extend what can be done in a template.

[Rui Nibau (ruinibau)]

Hi jaap.

Here goes a patch that allow to choose a template when you create a new page in notebook. Not big code as everything was already available in Zim - just connecting the parts.

(1) Behavior
--------

- Templates are found in the default wiki data folder (data/templates/wiki) and in user folder (~/.local/share/zim/templates/wiki)
- Template selected by default in the combobox is the template for the namespace (if any) or 'New'.
- Selected template overwrites the default namespace template.

(2) To fix
--------

(a) Filtering available templates in combobox ? If you choose 'Calendar' template outside the calendar namespace, infos are missing

(3) To implement
------------

(a) template per namespace.

How to do that ? With a page property dialog, where the user can define template, icon, ... ?

(b) Templates stored in notebooks

You suggested to store those templates in [notebook]/_templates.

Could it be an hidden folder ([notebook]/.templates) ?

[Jaap Karssenberg (jaap.karssenberg)]

Merged in rev344 with some code cleanup.

Open issues:
1/ Make Calendar template work everywhere (default to date of today)
2/ Allow to set config per namespace (property already implemented but need proposal on how to design the UI)
3/ Add templates in notebook

@rui: I propose using "[notebook]/_templates" precisely because I don't want to hide this folder - after all the templates are actual data within the notebook.

[Jaap Karssenberg (jaap.karssenberg)]

P.S. Could you also attach some of you templates that might be useful like the one for books ? Would be nice to have some useful defaults in the list.

[Rui Nibau (ruinibau)]

> @rui: I propose using "[notebook]/_templates" precisely because I don't want to hide this folder - after all the templates are actual data within the notebook.

Yes, i see the point.

Here goes a book template, build with the fields defined in a Tellico book collection. I will build others that I have in mind in the days to come and post them here this week-end.

[Rui Nibau (ruinibau)]

Current

===============================================================================

Here's the list of templates that i have built, mainly to describe different datas :

- Book

- ComicBook

- Event

- Music

- Person

- Recipe

- Software

- Video

- Wine

Work based (more or less) on :

- Microformats (http://microformats.org/)

- Tellico collections (http://tellico-project.org/)

Questionning :

(1) Something else ?

(2) Creation date included in templates, as it the Default one ?

Ideas / improvements / questionning
 / Futur (I should put that on the wiki)
===============================================================================

Just thoughts about what it can/could be done with templates.

(1) Plugin to edit / create templates (TODO)

Clearly useful for enduser. Next step.

(2) Should the template be registered as an header in each document ? (MAYBE)

    Template: Book

- This could give the ability to apply some specific behaviors (use case ?).

- This could help to upgrade a notebook when template is updated (clearly rare use case).

(3) Should it have a special markup for 'Fields' ? (MAYBE)

I use 'bold' markup to highlight fields name. Maybe we could use a more specific markup, as it is define in ReStructuredText (http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#field-lists) :

wiki markup :

    :field name:

regex pattern :

    ^:([^:]+):

Or more restrictive :

    ^:([^:\s]+):

- It could be useful to make some exports (see below).

Problem : it conflicts with link markup (root namespace), except if we "intercept" the markup in the Parser before link treatment ('_parse_block' before '_parse_para').

But even with the bold markup, splitting for export/manipulation should be feasible :

    ^\*\*([^:\*]+)\*\*\s+:

NOTA: i'm trying to see if it could be easy to insert this new markup in the current wiki Parser.

(4) Plugin to export datas created with templates in other formats (MAYBE)

- a book page or an entire notebook as a bibliographic ref.

- a Event page as html code following hEvent microformat spec.

- etc.

[Jaap Karssenberg (jaap.karssenberg)]

On Wed, Feb 2, 2011 at 7:26 PM, Rui Nibau <email address hidden> wrote:
> (3) Should it have a special markup for 'Fields' ? (MAYBE)

I have a long time plan to add some special widget in zim for
structuring such data. Basically the template would drive having a
form on top of the page with special fields. This is a more advanced
version of just having special markup.

The main advantage of a special markup or form is the use case when
you want to search based on these fields. When we know some pages have
a special field "Author", we can make the search function intelligent
enough to search for that (effectively giving zim some basic
collection manager features).

Now I wonder if it is wise to include the current template feature in
the upcoming release. I fully see the value of having this
functionality, even without structured fields. However I'm afraid that
when we release it as is and people start to use it extensively it
will be hard to upgrade later.

Maybe it is better to disable the function for the moment and then in
the next release cycle work on special formatting etc. to release full
functionality at once. Will mean most users will have to wait a little
bit longer for this feature, but will save the trouble of manual
updating.

-- Jaap

[smu (smu)]

On Wed, Feb 02, 2011 at 06:26:05PM -0000, Rui Nibau wrote:
>
> (1) Something else ?

what about a Paper/Article (for scientific papers) template, with
entries similar to the bibtex format (author, journal, volume, year,
summary, ...)
Or maybe a more general template for bibtex entries?!

> (2) Creation date included in templates, as it the Default one ?

I would prefere that.

best regards,
 stefan

[Rui Nibau (ruinibau)]

Le jeudi 03 février 2011 à 09:37 +0000, Jaap Karssenberg a écrit :
> I have a long time plan to add some special widget in zim for
> structuring such data. (...)

Do you mean using a form for ui (quite cool), relying on special markup
in the plain text page to define which content belongs to which field ?

Example of a book page :

    Content-Type: text/x-zim-wiki
    Wiki-Format: zim 0.4
    Creation-Date: 2011-02-02T16:25:56.359000
    Template: Book
    Content-Fields: Author,Publication Date,ISBN,Summary,Review

    ====== Smoke And Mirrors ======
    Created Wednesday 02 February 2011

    :Author: Gaiman Neil
    :Publication Date: 1998
    :ISBN: 0-06-093470-0

    :Summary:

    blablabla but short

    :Review:

    blablabla but longer

Mockup of ui for that page : (see attached picture)

> Now I wonder if it is wise to include the current template feature in
> the upcoming release. (...)

That's what i thought when i wrote the field part in the previous post.
There's certainly 2 ways (no exclusive) to see what templates can do /
be used :

(a) To avoid typing redundant content (no need of fields)
(b) To structure data (need fields to friendly ui, search|sort|...
functionalities)

> Maybe it is better to disable the function for the moment and then in
> the next release cycle work on special formatting etc. to release full
> functionality at once. (...)

So, 2 options :

1. Progressive implementation :
1.1 : next release
    - Definition of a specific 'field' markup.
    - Use of that markup in the templates to mark fields.
    - Visually render that markup as bold|uppercase|whatever in zim
pageview
    - Include templates and template functionality in next release
1.2 : next release + 1
    - Use that field markup to build form ui and to structure data

Pros : user have a new functionality (a).
Cons : If user edit (delete) a field name in his documents, it will be
hard to deal with it in 1.2.

2. Wait until everything it's done :
2.1 : next release
    - Disable template functionality
    - Don't include templates
2.1 : next release + 1
    - 1.1 + 1.2

Pros : a richer functionality (a)+(b), avoiding eventual user 'data
corruption'.
Cons : user have to wait.

It's up to you Jaap.

I have a lot of structured datas with those kind of 'fields', so i know
i will have to write some code to handle it in Zim. But if you thought
about managing structured datas in Zim too, that's great and let's go
with option 2.

I will give an hand where i can.

--
Rui Nibau
Développeur web, rédacteur scientifique
email : <email address hidden>
jabber: <email address hidden>
site : http://omacronides.com
pgp : http://omacronides.com/services/pgp-public-key/

[Rui Nibau (ruinibau)]

Le jeudi 03 février 2011 à 09:53 +0000, smu a écrit :
> what about a Paper/Article (for scientific papers) template, with
> entries similar to the bibtex format (author, journal, volume, year,
> summary, ...)

'Article' template created, based on required and optional fields in
 bibTex spec. BibTex could be an output format.

> > (2) Creation date included in templates, as it the Default one ?
>
> I would prefere that.

Yes : i have updated the templates with the creation date line.

--
Rui Nibau
Développeur web, rédacteur scientifique
email : <email address hidden>
jabber: <email address hidden>
site : http://omacronides.com
pgp : http://omacronides.com/services/pgp-public-key/

[Jaap Karssenberg (jaap.karssenberg)]

On Thu, Feb 3, 2011 at 7:26 PM, Rui Nibau <email address hidden> wrote:
> Do you mean using a form for ui (quite cool), relying on special markup
> in the plain text page to define which content belongs to which field ?

Yes that is my intention. After all zim is a desktop app, so we can
use interactive widgets to improve over just pre-formatted text in the
page.

> Example of a book page :
>
>    Content-Type: text/x-zim-wiki
>    Wiki-Format: zim 0.4
>    Creation-Date: 2011-02-02T16:25:56.359000
>    Template: Book
>    Content-Fields: Author,Publication Date,ISBN,Summary,Review
>
>    ====== Smoke And Mirrors ======
>    Created Wednesday 02 February 2011
>
>    :Author: Gaiman Neil
>    :Publication Date:  1998
>    :ISBN: 0-06-093470-0
>
>    :Summary:
>
>    blablabla but short
>
>    :Review:
>
>    blablabla but longer
>
>
> Mockup of ui for that page : (see attached picture)

I really like both the syntax example and the mockup image. My own
syntax idea was to use email header style blocks, but this is much
more human readable.

One detail to be aware of though is that in this mockup the page title
+ created date will not be directly editable unless we put them in a
text areas (which will be ugly). Not completely sure how to make this
intuitive. Text below the form will not be an issue, because any form
can end with a "Comments" or "Notes" text area for arbitrary free form
text.

For the syntax I'm also still thinking where to put a definition of
the data type in each field. In particular the ability to have
specific fields link to other pages would be great (e.g. specify
somehow that "Author" should map to a page in the "Authors"
namespace). Or, to give another example, force a certain input to be
numeric or specify the field for the book cover to be an image. Such
meta-data can perfectly go in the page itself, but could also be
stored in the template and link that from the page.

I'm more or less decided to disable it for the upcoming 0.50 release
with a commitment from my side to make it full-featured in the next
release.

Regards,

Jaap

P.S. once we have such forms I would really like a plugin that can
fill in pages for books based on ISBN number - I happen to have a hand
held USB bar code scanner in my desk :)

[Jaap Karssenberg (jaap.karssenberg)]

Btw. part of what we discuss here is actually duplicate of bug #363978

[Rui Nibau (ruinibau)]

Le lundi 07 février 2011 à 20:50 +0000, Jaap Karssenberg a écrit :
> One detail to be aware of though is that in this mockup the page title
> + created date will not be directly editable unless we put them in a
> text areas (which will be ugly). (...)

How about having a field to edit the title, and just report the value on
the top of page when changed ? When datas is saved in file, we take this
field into account ; the leading header is just another way to view the
data (see attached mockup).

> For the syntax I'm also still thinking where to put a definition of
> the data type in each field. (...)

So, for forms, template should be less **how** than **what** we have to
write in page.

To deal with this kind of field-ish data in PHP, i use json files to
define the structure (because it's human readable, it can be easily
parsed with various languages - i.e PHP and javascript, and because it's
more fun to write than xml) ; if we take the book example, i could have
something like this :

    {
        "name": "Book",
        "description": "Template for books",
        "sections": // optional, to organize content
            "General": {
                "order": 0, // order in ui / page
                "label": "General" // could be localized
            },
            "Publishing": {
                "order": 1,
                "label": "Publishing"
            },
            "Classification": {
                "order": 2,
                "label": "Classification"
            }
            // ...
        },
        "fields": {
            "Author": {
                "order": 0, // order in ui / page
                "type": "tokenlist", // comma separated values
                "label": "Author", // could be localized
                "section": "General" // sectionning content
                "namespace": "Authors" // namespace in the notebook
            },
            "Cover": {
                "order": 1,
                "type": "image",
                "label": "cover",
                "section": "General"
            },
            "Publication Date": {
                "order": 1,
                "type": "date",
                "format": "%Y-%m", // formatting content
                "label": "Date",
                "section": "Publishing"
            }
            // ...
        }
    }

This template spec could be the template itsef

A few ideas :

- Types : date, string, text, tokenlist, boolean, image (?)
- sections : gtk.notebook in ui, headings in page
- When a namespace property is defined for a field, value defines a page
in the namespace
- Date type field : use calendar widget to choose date.
- Image type : use image dialog.

> P.S. once we have such forms I would really like a plugin that can
> fill in pages for books based on ISBN number - I happen to have a hand
> held USB bar code scanner in my desk :)

That's a feature that keeps me using Tellico for a long time, even if i
wanted to store my books data in plain text : web search plugins to
automatically fill an item with datas loaded from web services. Clearly
useful.

Regards,

--
Rui Nibau
Développeur web, ...

[Jaap Karssenberg (jaap.karssenberg)]

Temporarily disable this feature for 0.50 release

[Jaap Karssenberg (jaap.karssenberg)]

Btw: love the mock up ! Not sure we can implement it 1-on-1, but concept is really nice.

[Jaap Karssenberg (jaap.karssenberg)]

One question about the mock ups here. When we put a form on the top of the page, do we want the form to stay there while scrolling (so separate form from the scrolling text), or is the form part of the page and does it scroll with the rest of the page?

Reason of asking is that this determines quite a bit of the implementation.

One advantage of not scrolling the form is that it would clearly separate this as a special part of the page. Disadvantage is that the size of the form will determine the minimal window size, so large forms can not be formatted nicely.

If form is part of the page it would essentially be a special inline object. Not unlike a field list in restructured text, but with some special widgets.

[dotancohen (dotancohen)]

Whoa, it appears that I've missed much action!

Though I think the form is nice (and if it should be implemented it should not be a part of the page), but it is _not_ what this bug requests. This bug requests to fill a new page with template content. Though related, these are two separate issues.

[Jaap Karssenberg (jaap.karssenberg)]

@dotan: yes, I linked the correct report in comment #12. Reason they are now thrown together is that the main use case for having page templates seems to put in structured forms. I didn't want to release this feature without addressing that use case first. But you are right, we should keep track separately. Unfortunately LP does not have a feature to move a set of comments to another report.

[Rui Nibau (ruinibau)]

> One question about the mock ups here. When we put a form on the top of the page, do we want the form to stay there while scrolling (so separate form from the scrolling text), or is the form part of the page and does it scroll with the rest of the page?

Hard to say : i don't understand here how one part of the page can scroll and an other part can't. I was seeing the form as an alternate way of displaying / editing a page.

===== (Use|test) case 1 : form for structured data =====

Templates could be define as :

* "classic templates" : just redundant text/markup that we want to put in a document. It's the way it works now and the way the first implementation works (see http://omacronides.com/media/videos/softs/zim-templates-01.ogv).
* "form templates" : templates that defines fields and the type of data in each field (String, text, date, etc)

When we (create|edit) a page :

* An Header defines the template to use ; if it's a "classic template", just go one as it does now with the TextBuffer ; if it's a "form template", we use another object to display the page.

For pages define with "form template", we should have :

* a parser that split the page into dictionary of (field_name, field_value) pairs, based on a specific field markup
* a form object wher, foreach of this fields, we choose how to display it based on the field data type (described in the template definition) : if it's a string, gtk.Entry, if it's a "text", we need a TextBuffer dealing with a parsetree, so we can type wiki formatted content, etc.
* A dumper which rebuild the document based on the form object structure

===== (use|test) case 2 : markup for structured data =====

Another way could be :

* to define for the current parsetree a new markup for a "field" element : a strict markup that does not allow white space for example. Easy to implement and to parse.
* Display the page as usual, displaying the fields
* Build some tools or plugins which give the ability to find pages

IMO, the important thing is that content still a simple flat file that can be edit with anything as a text editor (as it is today). I clearly see how i could do that in a web application (html|javascript|css for frontend, php|whatever for backend to save|parse files) but my python skills still limited now.

I certainly have some time this week-end to explore a little more.

[smu (smu)]

what is the status of this nice feature? will it be implemented in the next time or is there already a possibility to use it?

cheers,
 stefan

[Jaap Karssenberg (jaap.karssenberg)]

It is pending the implementation of forms. Currently working on parser
redesign en custom objects. Next step is to use custom objects to implement
forms. Then I will enable this feature.

On Sat, Oct 15, 2011 at 2:36 PM, smu <email address hidden> wrote:

> what is the status of this nice feature? will it be implemented in the
> next time or is there already a possibility to use it?
>
> cheers,
> stefan
>
> --
> You received this bug notification because you are subscribed to Zim.
> https://bugs.launchpad.net/bugs/333204
>
> Title:
> Support page templates in new page dialog
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/zim/+bug/333204/+subscriptions
>

[scholi (scholi)]

Hi Guyes,
I'm facing the same need as you to have the ability to paste quickly pre-written text into a current page. I found a solution and I'd like to share it with you. Maybe this will be useful for you.

Zim give the ability to be interfaced with external commands like scripts. "Tools/Custom tools" or something like that (my zim is not in english). I then just created a small bash script as follow (I called it zim-template.sh):

#!/bin/bash

DIALOG="Xdialog"
ROOT="$2/Templates"

T="`ls $ROOT | awk '{print $0" "$0}'`"
R=`$DIALOG --no-tags --menubox "Select a template" 20 50 10 $T 2>&1`

cat $ROOT$R >> $1

===========

And bind this script in the custom tool of zim as follow:

zim-template.sh "%f" "%n"

please be sure that either zim-template is in your PATH or give to zim the full path of your script. What you need to do is create a Top-Level Page called "Templates". All sub-pages of Templates will be then available in the dialog list and it's contents will be added to the current page!

Enjoy

[scholi (scholi)]

I did'nt find an edit button. Small corrections:
I forgot the "/" in the ROOT and you can add tail-n+5 insread of cat in order to remove the useless header of the file that you don't want to be pasted!

#!/bin/bash

DIALOG="Xdialog"
ROOT="$2/Templates/"

T="`ls $ROOT | awk '{print $0" "$0}'`"
R=`$DIALOG --no-tags --menubox "Select a template" 20 50 10 $T 2>&1`

tail -n+5 $ROOT$R >> $1

[Worrier Poet (worrier-poet-deactivatedaccount)]

It's very kind of you to share this with the community. Thank you!

On 07/27/2012 09:13 AM, scholi wrote:
> I did'nt find an edit button. Small corrections:
> I forgot the "/" in the ROOT and you can add tail-n+5 insread of cat in order to remove the useless header of the file that you don't want to be pasted!
>
> #!/bin/bash
>
> DIALOG="Xdialog"
> ROOT="$2/Templates/"
>
> T="`ls $ROOT | awk '{print $0" "$0}'`"
> R=`$DIALOG --no-tags --menubox "Select a template" 20 50 10 $T 2>&1`
>
> tail -n+5 $ROOT$R >> $1
>

[DavidM (milarupa)]

Hi! Please enable this feature, even without forms and fields. (In fact, I think I would prefer them without them.)

Anyway, I'm trying scholi's custom tool and I get this error:

Failed to run application: /home/myname/notes/zim-template.sh

/home//myname/notes/zim-template.sh "/tmp/zim-myname/tmp-page-source.txt" "/home/myname/notes"
returned non-zero exit status 1

Does anyone have an idea what this means? Thanks!

[Jaap Karssenberg (jaap.karssenberg)]

Function un-hidden in release 0.61