I think I know the reason for the high memory usage, it's the Ubuntu font. Try to change the fonts in gnome-tweaks → Fonts. Change the Interface Text, Document Text and Legacy Window Titles to Sans Regular and Sans Bold, respectively. This is the default sans font.
I didn't try to pinpoint which option exactly affects the shell.
I think I know the reason for the high memory usage, it's the Ubuntu font. Try to change the fonts in gnome-tweaks → Fonts. Change the Interface Text, Document Text and Legacy Window Titles to Sans Regular and Sans Bold, respectively. This is the default sans font.
I didn't try to pinpoint which option exactly affects the shell.