We depend heavily on the varnish cache to run iusethis and our other sites. These tips from Kristian@linpro are a must read:
ESI is cool. And requires more session workspace. If you don’t increase the value of sess_workspace (default: 16kb. Advised value: 32kb or more), you are likely to see assert errors in your syslog.
(Hint: If you are going wtf is ESI, you really want to find out more :)