So all of this ram is being used and is only accessed sporadically if at all. This is not good.
Sounds like you could implement the entire thing on a micro db instance (redis or a regular db) with no raft or any other custom implementation or messing.
they store the index of files only in memory. and have the entire build time to fetch build-1 images to get ready for the diff.
it's much easier than most use cases