How can Wordpress be made more memory efficient?

Let's assume this is a site that has dynamic content pertaining to the user, or content that changes very often, so varnish is out. Let's also assume that there are no plugins and the theme is vanilla.

Can Wordpress be tuned so it uses less memory each request? It seems to include all libraries and functions by default. Is it all or nothing, or can it load say the media functions only if it really needs them without being lobotomized?

I was going to ask this on StackOverflow, but I'm pretty sure they would have pointed me here. Maybe ServerFault?