@Colin, Sure it definitely needs to be improved. As I said above, this is just a cheap way to get us going, and the proper fix is a transactional storage on disk.
@Mark, Yes, we can prune based on number of changes as well. It's not expensive. That's a good trick to get us going farther.
@John, 2MB / (10 tasks * 180 changes) is in the ballpark of 1kb per task. That's not unrealistic I think. Also worth noting it's not just plain data.. there will be a small amount of additional garbage created per entry when processing it.
@Colin, Sure it definitely needs to be improved. As I said above, this is just a cheap way to get us going, and the proper fix is a transactional storage on disk.
@Mark, Yes, we can prune based on number of changes as well. It's not expensive. That's a good trick to get us going farther.
@John, 2MB / (10 tasks * 180 changes) is in the ballpark of 1kb per task. That's not unrealistic I think. Also worth noting it's not just plain data.. there will be a small amount of additional garbage created per entry when processing it.