Migrate from Ejabberd to Redis for OpenSRF Messaging

[Bill Erickson]

The gist is to replace Ejabberd and the XMPP message serialization with Redis (https://redis.io) as the message broker and pure JSON for message serialization.

--
Evergreen conference sessions on this topic for 2022 and 2023:

https://github.com/berick/Presentations/blob/master/Evergreen-2022/osrf-redis.md

https://github.com/berick/Presentations/blob/master/Evergreen-2023/osrf-redis-part2.md

--

Ansible Evergreen installer for the code in progress:

https://github.com/berick/evergreen-ansible-installer/tree/working/ubuntu-22.04-redis

--

Working Branches:

https://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/user/berick/lpxxx-opensrf-over-redis-v3

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/berick/lpxxx-opensrf-over-redis-v3

[Bill Erickson (berick)]

Rebased / renamed branches pushed:

https://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/user/berick/lp2017941-opensrf-on-redis-v1

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/berick/lp2017941-opensrf-on-redis-v1

[Jason Stephenson (jstephenson)]

During the 2023 Evergreen International Conference, the consensus was that switching from Ejabberd to Redis warrants bumping the OpenSRF version number to 4.0. When the OpenSRF milestone is added to this bug, it should be 4.0.

I don't know if updating the Evergreen code warrants such a drastic version bump, but it might not hurt.

[Galen Charlton (gmc)]

I took this out for a spin, and was able to get it working on Debian Buster (which yes, is Debian oldstable). Some notes on the experience:

- The minimum version of Redis needed is 6.0 because of the ACL stuff. That happened to be available in buster-backports
- At least on Buster, Redis binds only to 127.0.0.1 and ::1, so I needed to edit redis.conf to also added public.localhost and private.localhost
- It was not clear from the OpenSRF README that the Rust router is _required_. Using MONITOR via redis-cli helped point out the problem
- Once I figured out that the Rust stuff is necessary, I ran into the fact that the version of Rust and Cargo packaged for Buster is too old; I had to resort to rustup. But once I had a new enough toolchain, 'make build-opensrf-release && sudo make install-opensrf-release' from a clone of https://github.com/kcls/evergreen-universe-rs did the trick
- I noted that 167 (!) crates were pulled in.
- There was a complaint about one of the dependencies:

warning: the following packages contain code that will be rejected by a future version of Rust: traitobject v0.1.0
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`

Going up the dependency chain, I found a note by the maintainers of the websocket crate recommending that it _not_ be used for new projects in favor of either tungstenite or tokio-tungstenite.

- Without actually measuring just yet, staff client and OPAC actions felt noticeably zippier on my test VM

[Galen Charlton (gmc)]

As far as Bullseye is concerned, it packages 1.48, which I don't think is quite new enough. Also, Debian won't have the rust-all package until Bookworm.

[Bill Erickson (berick)]

Thanks Galen!

As a note of appreciation, I've ported the Websockets code from 'websockets' to 'tungstenite'.

https://github.com/kcls/evergreen-universe-rs/commit/805bfca298990c67591f1e3ec6613cbc49e7adc8

I have been testing the code so far with Ubuntu 22.04 on the assumption that it would be some time before we're ready to seriously move in this direction. I'm not too surprised the Rust bits are somewhat bleeding (lightly wounded?) edge.

I certainly have work to do on the INSTALL docs. I've been in a holding pattern waiting to see how the Rust discussion goes: http://list.evergreen-ils.org/pipermail/evergreen-dev/2023-May/000550.html. Will follow-up here with any changes based on that convo.

[Bill Erickson (berick)]

Patches pushed to the OpenSRF working branch to modify the C Router to work with Redis. This required replacing/refactoring most of the existing C Router code. I've tested a variety of multi-domain scenarios and watched for memory leaks, etc., but this much change would certainly benefit from a lot of tire kicking.

With this and the previous mods to the Websockets code, we can run the Redis branch now without the Rust additions.

I've updated my auto installer to reflect these changes:

https://github.com/berick/evergreen-ansible-installer/tree/working/ubuntu-22.04-redis

==

https://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/user/berick/lp2017941-opensrf-on-redis-v1

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/berick/lp2017941-opensrf-on-redis-v1

[Bill Erickson (berick)]

Rebased with a bit of tidying:

https://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/user/berick/lp2017941-opensrf-on-redis-v2

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/berick/lp2017941-opensrf-on-redis-v2

[Jason Stephenson (jstephenson)]

I had a look at the branches, and kicked the tires by running the Perl and PgTap tests. All tests passed, so that's good. I plan to exercise a few features in the coming days and possibly hook this up to a database with a production-sized data set.

I have a few suggestions based on the installation process:

1. The first time that I ran `osrf_control -l --reset-message-bus`, I got the following error: "Could not connect to Redis at private.localhost:6379: Connection refused." This was caused by not configuring Redis to listen on the public.localhost and private.localhost IPs configured in the hosts file. I recommend either adding an instruction to the OpenSRF README to add those IPs to the "bind" line in /etc/redis/redis.conf (my solution) or to change the instruction for /etc/hosts to use 127.0.0.1 for both the public.localhost and private.localhost domains.

2. More help on finding the passwords in the redis-accounts.txt file would be useful. I had to guess which lines they were on, and though it seemed kind of obvious, it may not be obvious to others.

3. Renaming 'redis-accounts.example.txt' to 'redis-accounts.txt.example' to be congruent with the naming of the other example files. (I tried using a for loop in bash to rename the examples, but this one slipped through.)

4. The opensrf_core.xml.example file still says "Jabber" and should be changed to "Redis."

5. A couple of junk files are left behind in the SYSCONFDIR. One is named "password" and the other looks like a password UUID. Both files are empty. They should be cleaned up by the process that sets the passwords.

I noticed 1 difference on the Evergreen side of things. When running 'make check' from ~/Evergreen/, make will attempt another compile of the code. This compile fails in the Redis branch, but succeeds in main. This is minor because 'make check' is supposed to be run from ~/Evergreen/Open-ILS/src/perlmods/, but it might be worth looking into why the compilation fails with 'make check' versus make.

This looks like a promising start. Thanks, Bill!

[Bill Erickson (berick)]

Thanks, Jason!

Patch pushed to the EG branch to fix the 'make check' issues.

Will look at other items shortly.

[Bill Erickson (berick)]

1. Going forward, sharing 127.0.0.1 would be simplest. We'll need an upgrade note regardless. I've not made any changes on this one yet.

2. I added some more verbiage to INSTALL.

3. Done.

4. Done.

5. I'm not seeing these junk files in /openils/conf/ or in either of the repositories after building.

[Bill Erickson (berick)]

Rebased branches pushed:

https://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/user/berick/lp2017941-opensrf-on-redis-v3

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/berick/lp2017941-opensrf-on-redis-v3

---

I also stepped through and upgrade from Evergreen/OpenSRF main to the Redis branches and documented the process in a new release notes doc (linked from Github for auto-formatting):

https://github.com/berick/Evergreen/blob/user/berick/lp2017941-opensrf-on-redis-v3/docs/RELEASE_NOTES_NEXT/Administration/redis-upgrade.adoc

Upgrade notes will likely need a little love and proofreading, but cover the basics.

[Bill Erickson (berick)]

Migrating to collab branches to accommodate testing:

https://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/collab/berick/lp2017941-opensrf-on-redis-v3

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/collab/berick/lp2017941-opensrf-on-redis-v3

[Bill Erickson (berick)]

Opened bug #2041431 to track EG 3.12-specific changes.

[Bill Erickson (berick)]

Staged a patch here to add usernames to bus addresses for extended domain-level differentiation (i.e. honor router_name)

https://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/collab/berick/lp2017941-opensrf-on-redis-v3-bus-addr-gets-username

Will merge to main collab branch once reviewed.

[Bill Erickson (berick)]

After further testing, I have merged the commit from lp2017941-opensrf-on-redis-v3-bus-addr-gets-username into the main opensrf branch (collab/berick/lp2017941-opensrf-on-redis-v3) for ease of review and testing.

[Jane Sandberg (sandbergja)]

Thanks, Bill, Galen, and Jason. This is a great step toward a more maintainable/understandable stack. I pushed the Evergreen parts to main for inclusion in 3.12.

The OpenSRF parts still need to be pushed.

[Jane Sandberg (sandbergja)]

Just adding a note that I ran the OpenSRF branch in a qa server that was exposed to the big scary internet for a few weeks. With the top two commits 9e89bb and 2d5d55, it worked very well (no crashes due to bad input from bots, nice performance, no unexpected issues). Thanks for your continued work on this, Bill!

[Jason Stephenson (jstephenson)]

Using the latest collab/berick/lp2017941-opensrf-on-redis-v3, I got the following on a VM where services were not running.

osrf_control -l --diagnostic
[auth] WRONGPASS invalid username-password pair, at /usr/share/perl5/Redis.pm line 311.

After starting services and trying again, things worked as normal. We may want to consider resetting the passwords any time that osrf_control runs, not just when starting services.

I was trying to check if services were running on a system that had been restarted a few days ago, and I couldn't remember if I had started services or not.

[Bill Erickson (berick)]

Makes sense, Jason. Should be a simple osrf_control script change. Will circle back to this soon unless someone else grabs it.

[Jason Stephenson (jstephenson)]

Bill, I'll take a stab at it.

[Jason Stephenson (jstephenson)]

That wasn't too hard. I pushed a commit to the collab/berick/lp2017941-opensrf-on-redis-v3 branch that works for me.

osrf_control (opensrf-perl.pl) now does a forced message bus reset when the --diagnostic option is used.

[Bill Erickson (berick)]

Thumbs-up to patch.

[Bill Erickson (berick)]

For reference:

https://www.linuxfoundation.org/press/linux-foundation-launches-open-source-valkey-community

https://github.com/valkey-io/valkey

Building Valkey is currently a manual process, but work is under way for packages, etc. Confirmed it acts as a drop-in replacement for Redis.

[Blake GH (bmagic)]

All,

Just noting here that we upgraded a production machine to use this code. As sort of a proof of concept, ready to port back to ejabberd if needs be. We're over a week now on production with no issues. Granted it's a small production machine with little usage (which is why we chose it) but it's running perfectly.

Incidentally, it's also running PG15.

FWIW