Please support redis as db

[Michael]

Please support redis as db

[andypiper (andypiperuk)]

Can you explain why a database is needed, and why redis in particular? I think Roger removed the db dependency a few releases ago and it certainly makes mosquitto much simpler.

[Michael (hfeeki)]

I think mosquitto need to store the clientid, messages to some place. Redis is a in memory db system, it is quick and simple. So if redis is supported, that will be great!

[Roger Light (roger.light)]

Could you give more of an idea what you're trying to achieve?

[Michael (hfeeki)]

1. I just want to manages all of the clients and messages in memory, i don't want to read or write disk in most operations for clients.
2. the in-memory db system should easy to use, easy to learn.

[Roger Light (roger.light)]

I'm still not sure what you actually want to achieve. Could you explain an actual scenario where you would make use of this feature?

[Damiano Albani (damiano-albani)]

I'd like to broaden the discussion while we're on this topic of databases.
AFAIK the current persistence implementation is simply of a (periodical) dump of the memory into a flat file.
So the mosquitto server has to keep all the information in memory when running, right?
While it's perfectly fine with a few hundreds messages in the queue, that clearly doesn't scale if the server has to keep many more messages, for many more clients.

More than the use of a specific database library, I think that's this all-in-memory behavior that should be improved.
Be it with a mosquitto-specific database implementation or a more common SQLite or Redis.

[Matteo Collina (matteo-collina)]

In Mosca (https://github.com/mcollina/mosca) I support MongoDB, Redis, LevelDB (embedded) and Memory.

Even though implementing multiple DB support is not simple, I would suggest you give LevelDB a try. It is very fast.

[Damiano Albani (damiano-albani)]

Glad to hear that there exists other MQTT broker implementations.
However I'm running in a constrainted environment (OpenWrt router), so I think node.js is too heavy for my use case.

Thanks for the pointer to LevelDB. It looks very interesting and maybe even more appropriate than SQLite.

[Matteo Collina (matteo-collina)]

Depending on the resources of your OpenWrt router, you might want to give it a try.
It works very well on a Raspberry Pi.
http://oskarhane.com/raspberry-pi-install-node-js-and-npm/

LevelDB is very suited for that kind of task.

[Damiano Albani (damiano-albani)]

Roger, so what do you think of supporting various DB backends, in particular LevelDB?
Or even a home-made database implementation but which would not be memory limited.

[Roger Light (roger.light)]

Using redis/leveldb for persistence should be fairly straightforward. Using them to replace the in-memory data is much less straightforward and would require a lot of work to do properly. I used to use sqlite as the database backend, but the performance was poor due to the complex regular expressions required to do proper topic matching (see the notes at https://bitbucket.org/oojah/mosquitto/src/tip/doc/historical/old-regex.txt ). Moving to the current system produced hugely significant performance improvements. I doubt that a key/value store would be as bad but it would need to be designed carefully.

Although mosca supports redis as a persistence it does not implement topic/subscription matching very well so subscriptions will not work properly. Matteo, you should look at file mosquitto/test/broker/03-pattern-matching.py for some examples that fail in mosca.

[Roger Light (roger.light)]

Additionally, I don't believe that the regex I linked to is correct. It needs to be even more complex to work properly.

[Matteo Collina (matteo-collina)]

We do not use a regex at all: we use a data structure called 'trie', the
same thing RabbitMQ uses.
Our approach is mixed: we keep an in-memory data-structure with the list of
the 'offline' clients, then we store only the packets on disk. Here it is:
http://www.rabbitmq.com/blog/tag/trie/.

Thanks for the failing tests, we will fix them. The whole pattern matching
is so hard :/.
The fact that there is no 'official' test suite is very painful.

Il giorno giovedì 11 luglio 2013, Roger Light ha scritto:

> Additionally, I don't believe that the regex I linked to is correct. It
> needs to be even more complex to work properly.
>
> --
> You received this bug notification because you are a member of Mosquitto
> Development, which is subscribed to mosquitto.
> https://bugs.launchpad.net/bugs/931028
>
> Title:
> Please support redis as db
>
> Status in mosquitto: an mqtt message broker:
> New
>
> Bug description:
> Please support redis as db
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mosquitto/+bug/931028/+subscriptions
>
> --
> Mailing list: https://launchpad.net/~mosquitto-devel
> Post to : <email address hidden> <javascript:;>
> Unsubscribe : https://launchpad.net/~mosquitto-devel
> More help : https://help.launchpad.net/ListHelp
>

[Damiano Albani (damiano-albani)]

On Fri, Jul 12, 2013 at 9:02 AM, Matteo Collina <email address hidden>wrote:

>
> Our approach is mixed: we keep an in-memory data-structure with the list of
> the 'offline' clients, then we store only the packets on disk.

Yeah, that's the architecture I'd very much like to see in Mosquitto.
Currently the on-disk persistence feature caters for data safety, which is
nice.
But the memory usage increases too much for environments where memory is
scarce.

[Roger Light (roger.light)]

I'm putting this on the list to look at for 1.3.