Someone commented at my previous question and said it's possible to do this with Varnish:

Instead of :

Client Request > Varnish LB > Backend > Varnish LB > Client

I want to have (Direct reply from backend to client, instead of going through the LB) :

Client Request > Varnish LB > Backend > Client

This is not working :

sub vcl_pass {
  if (req.http.host ~ "^(www.)?example.com$") {
     set req.backend = baz;
     return (pass);
}
}