Some constraints on why we have an oplog, and why we have to be a bit careful as to how we size it.

1) If you ever want to go HA, you need to start mongo with --replicaSet and have an oplog. AIUI it isn't possible to change this after the database has been defined, so if we wanted to start without it, then we would need to spend time creating a new database, and copying the data over there.
2) It isn't possible to resize the oplog after the fact. Once it has been allocated, mongo doesn't provide tools to change its size.
3) Backups are much more reliable if you have the oplog, because it is possible to actually snapshot the database without stopping it, because it can monitor the oplog and make sure all applied changes are recorded in the backup.

I do agree that the goals of mongo's defaults don't quite match what we actually need out of it. I would like to understand what our actual consumption is likely to be. (How many operations would fit in a 1GB oplog?)
Mongo's defaults are intended to allow replicas to get out of date on a "busy" db by a day, and still be able to get back into sync without having to start from scratch again.
We've generally operated on the assumption that our dataset size isn't enormous (1-10GB would be quite large for a Juju DB), thus we don't really need 50GB for just the oplog.

I feel like we could *probably* get away with a 1GB default oplog size.

I don't think we want to remove replicaset entirely. It makes it far easier to go HA when we need it, and it makes the backup story *much* better. (You can do a safe backup without taking the system offline.)

As for "noop" writes, the problem (AIUI) is in the transaction model we have, it isn't easy to realize that a transaction is a no op and then just not apply it. We've generally used the model of "set the value of X to Y", and allowed that if the value is already Y, thats fine. Even if the transaction was "if the value is not Y, set it to Y", AIUI that still creates an entry in the transaction log (it is a failed transaction, but it still needed to be recorded so that it could be tested for whether it applied.)

I have the feeling  that for IP addresses, we aren't particularly concerned with concurrency (last write always wins, because we want to just have whatever the current value is), and if we are polling it, even if we missed one update we would get it right later, and we are unlikely to change IP address and then quickly change back to the previous address. So likely reading the database, and then doing the check of "should this be changed" only in memory and only if we think it might change something then doing the transaction in the database.

I've seen comments in the past that we actually were doing that, but we have significant evidence to the contrary. So maybe what check we have is faulty somehow.

Anyway, if we can be more careful about not creating no-op transactions that probably also helps shrink the bounds of what oplog size we need. Since there will be fewer 'things changing in mongo' that the oplog is tracking.

Could we go below 1GB for the oplog? Maybe, but I'd like us to do a lot of careful analysis if we wanted to do so. Going with the standard minimum bound seems like a way to move forward and still allow a good margin of error. (The biggest problem being that if we get it wrong, mongo does *not* like to change the size after the fact, so it is hard to ever grow the value.)