Replace current parser through a regular one

[SirVer]

Currently, the ultisnips parser uses a combination of regular expressions for convenience and some character based parser to handle non regularities. This is quite fragile and can't handle all cases. For example snippets, that begin with \$ are not expanded correctly at the moment.

UltiSnips will need a parser that does not depend on regular expressions.

[Ryan Wooden (ryan.wooden)]

I've been thinking about this too. It shouldn't be very difficult to
write a simple lexer and parser for snippets.

Have you already started on this?

On Mon, Apr 4, 2011 at 3:22 PM, SirVer <email address hidden> wrote:
> Public bug reported:
>
> Currently, the ultisnips parser uses a combination of regular
> expressions for convenience and some character based parser to handle
> non regularities. This is quite fragile and can't handle all cases. For
> example snippets, that begin with \$ are not expanded correctly at the
> moment.
>
> UltiSnips will need a parser that does not depend on regular
> expressions.
>
> ** Affects: ultisnips
>     Importance: Medium
>         Status: Confirmed
>
> --
> You received this bug notification because you are subscribed to
> UltiSnips.
> https://bugs.launchpad.net/bugs/750577
>
> Title:
>  Replace current parser through a regular one
>

[SirVer (sirver)]

nope. I invested five minutes in a little food for thought process. The parser is not totally straigtforward to write, because it should handle things like \\\\${1:blah} correctly, so it has to parse for all text objects at the same time. But it must be able to parse for tabs alone because of recursive snippets.

But it should be doable. Feel free to jump on it if you feel like it.

[Ryan Wooden (ryan.wooden)]

I've got a working parser put together. It's almost a recursive descent parser with a very simple lexer to handle backslash escapes. As far as I can tell (and as can be seen in the test case I stuck in the file), it seems to handle everything, but it doesn't currently generate the objects used by UltiSnips. It seems bigger than it should be, so if you have any ideas on how to make it smaller/better they would be appreciated!

[SirVer (sirver)]

I'll have little time to review this in april, but I'll try to do my best.

A *real* parser will likely be a bit longer than we want to, but alas, what can one do.

[SirVer (sirver)]

I reviewed your parser.py implementation and have some comments:

1) It is generally straightforward, but not as easy to understand as I would have liked it to be. I am at a loss of proposals as to where it should be made easier though :(. A true lexer is generally not possible as we are context sensitive. So maybe a simpler solution is just not working.
2) Your current implementation handles Tabstops, mirrors and transformations as one and the same thing. This will likely not work out inside of UltiSnips. Lexing is difficult again, because $2 could be a mirror (if tabstop 2 is defined elsewhere or has been already seen) or a tabstop. Likewise ${2/blah/fasel/} is a transformation, but you only know this from the slashes.
3) Your current implementation can't parse Tabstops/Mirrors inside of tabstops if I understand correctly. Something like
${1:launchpad} ${3:http://${2:www.$1.net}}
is currently possible but will require extra work with your parser suggestion.

It is hard to foresee all cases, it most likely is easier to just plug this thing into ultisnips and tweak until all tests pass.

[SirVer (sirver)]

I added a new parser implementation in r301. It adds ~50 LOC to ultisnips and is fairly flexible. I am quite happy with the implementation, please test and report back with problems.

[SirVer (sirver)]

I wrote a quick blogpost about the new implementation and why it was needed here: http://sirver.widelands.org/blog/2011/07/24/new-parser-for-ultisnips

Glad if someone reads it :)

[SirVer (sirver)]

Release in 1.5