Ingnore hashtags and formatting in verbatim text

[Michael Entrup]

All occurrences of hashtags are recognized by RedNotebook even if they are in a verbatim text. In my opinion there is no scenario where hashtags in verbatim text are necessary. It would be nice to exclude verbatim text from the hashtag detection.

Example:
```
DecimalFormat df = new DecimalFormat("0.0##E0", DecimalFormatSymbols.getInstance(Locale.ENGLISH));
```

RedNotebook detects the hashtag ``#E0``.

[Jendrik Seipp (jendrikseipp)]

You're right, this behaviour should be changed. However, it is not easy to do that if we want to stay fast, IMO.

[elliot (e-diot)]

The hashtag slows down rednotebook, if lot of UNIX like text file content with comments (beginning with a hash) are stored. A special format specifier like <code>#!/usr/bin/sh ...</code> as used in popular wikis, that would not be parsed for hashtag cloud, could be a solution.

I'm using an older version of rednotebook, since this "problem" was introduced with the hastag cloud. No one else experiencing this problem too?? :-o

[Jendrik Seipp (jendrikseipp)]

What do you mean with "slows down RedNotebook"? What is slower?

Excluding code from the hashtag parsing is difficult, but we should
probably exclude tags beginning with "!" anyways. What other "false
positives" do you see (like #include or #!/usr/bin/sh)?

[elliot (e-diot)]

Starting up RedNoteBook is very slow. Of course this depends on the content to be loaded. Entering a search string in the search field takes again several seconds and the characters typed are displayed with delay. It seems that building the hashtag cloud produces this delay.

Well, using the application as logbook for daily UNIX tasks produces much of hashtags since comments in scripts or configuration files normally start with the hash "#". Below an excerpt from my hashtag cloud:

#defaults  #do  #dprootdir  #echo #export  #extendvg  #for  #gridcontrol  #howto  #ibm_power  #id  #ident  #if  #innodb_buffer_pool_size  #innodb_data_home_dir  #innodb_fast_shutdown  #innodb_flush_method  #innodb_force_recovery  #innodb_log_group_home_dir  #ip6forwarding  #kern  #ldflags  #listen  #ln  #local0  #log  #maxpgahead  #mklv  #mysqltuner  #oracle  #os  #partnumber2featurecode  #path  #php_admin_flag  #platform  #query_cache_limit  #query_cache_size  #r  #read  #redirection  #require  #sar  #satisfy  #script  #scripts  #security  #servername  #set  #showing  #skip  #start  #su  #sub  #support  #todo  #tsm  #tz  #user  #volume  #winserver

What about an option to completely disable the hashtag cloud function in RedNoteBook? Would be much easier to implement, as far as I can assess this. ;-)

[Michael Entrup (entrup)]

I think it would be best to add an option to disable inline hashtags. It would be nice to restore the behavior of RedNotebook 1.5, where no inline hashtags were implemented.

[Jendrik Seipp (jendrikseipp)]

I fixed the original problematic example by disallowing hashtags starting with two #'s.

I'll probably add an option to disable hashtags, but before I would like to find out what exactly works slowly for you. While searching, the cloud isn't updated, so it cannot make the search slow. I guess the slowness results from your journal being very big.

Could you please post the log that you get when you start RedNotebook in a terminal, elliot? Please also provide the files sizes of your month files.

[elliot (e-diot)]

Hello again and sorry for my late response. Below the list of data files of my journal. From my point of view, the files are not that huge. ;-)

$ ls -l
total 1668
-rw-r--r-- 1 user group 35575 Mar 12 06:57 2012-01.txt
-rw-r--r-- 1 user group 111088 Jun 20 2012 2012-02.txt
-rw-r--r-- 1 user group 324226 Jan 21 09:10 2012-03.txt
-rw-r--r-- 1 user group 151554 Jan 23 12:58 2012-04.txt
-rw-r--r-- 1 user group 127878 Jan 31 13:16 2012-05.txt
-rw-r--r-- 1 user group 99750 Jul 23 2012 2012-06.txt
-rw-r--r-- 1 user group 60864 Jan 15 06:48 2012-07.txt
-rw-r--r-- 1 user group 102837 Nov 6 10:23 2012-08.txt
-rw-r--r-- 1 user group 145841 Feb 19 10:05 2012-09.txt
-rw-r--r-- 1 user group 36913 Mar 25 14:06 2012-10.txt
-rw-r--r-- 1 user group 227149 Dec 11 16:36 2012-11.txt
-rw-r--r-- 1 user group 27308 Jan 7 12:37 2012-12.txt
-rw-r--r-- 1 user group 57121 Apr 2 14:40 2013-01.txt
-rw-r--r-- 1 user group 22784 Mar 18 09:58 2013-02.txt
-rw-r--r-- 1 user group 71554 Apr 10 14:06 2013-03.txt
-rw-r--r-- 1 user group 66366 Apr 29 11:30 2013-04.txt

I use RNB on Win7 (64-bit) and Linux (64-bit). I will copy the journal from Win7 to Linux to check, if it behaves the same.

Thanks and bye!

[Kamil Śliwak (cameel)]

I'm experiencing this slowness too. It has started when I updated to the version that automatically detects hashtags embedded directly in text. I have 4 years' worth of notes in my main journal and it takes about 2.5 MB (for detailed file sizes see attached rednotebook-journal-size.txt). Before the update the startup and search were instantaneous. Now there are about 2 seconds of delay at startup and about a second for each letter typed in the search box. The search delay accumulates so there's a lot of delay if you're trying to type a whole word.

I have looked at the code and I see that the hashtags() method gets called a lot. It's used by journal.search() to extract tags from the whole journal each time I type a letter. Profiler says that in my simple test (start the program, enter a sample phrase, wait for the search to complete and exit) all the calls to it taken together amounted to over 7 seconds. In 1.6 it's only about 0.2 s.

Maybe it would be a good idea to cache the search results? You could do it for all pages besides the one being edited. There would still be some delay at startup but eventually this could be elliminated too by persisting the cache.

[Kamil Śliwak (cameel)]

I forgot about the attachment. Here it is.

[elliot (e-diot)]

Just back to see, if there is some process fixing this issue. :-o

No update? I think this issue evolved to a big issue for RN. Would be good to read something about ongoing investigations or even findings. I really like RN and (still) use it every day...

[Jendrik Seipp (jendrikseipp)]

In addition to hashtags, all formatting should be ignored in verbatim paragraphs (see https://answers.launchpad.net/rednotebook/+question/202141).