I have nginx on SERVER1 that is acting as load balancing between SERVER1 and SERVER2

in SERVER1 I have the upstreams for the load balancing defined as :

upstream de.server.com { # similar upstreams defined also for other languages
    # SELF SERVER1
    server 127.0.0.1:8082 weight=3 max_fails=3 fail_timeout=2;
    # other SERVER2
    server otherserverip:8082 max_fails=3 fail_timeout=2;
}

The load balancing config on SERVER1 is this one:

server {
    listen   80;
    server_name ~^(?<LANG>de|es|fr)\.server\.com;
    location / {
        proxy_pass   http://$LANG.server.com;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

        # trying to pass a variable in the header to SERVER2
        proxy_set_header        Is-From-Load-Balancer 1;        
    }
}

Then in server 2 I have:

server {
    listen   8082;
    server_name localhost;
    root   /var/www/server.com/public;
    # test output values
    add_header testloadbalancer $http_is_from_load_balancer;
    add_header testloadbalancer2 not_load_bal;

    ## other stuff here to process the request
}

I can see the "testloadbalancer" in the response header is set to 1 when the request is coming from the load balancing, it is not present when from a direct access: SERVER2:8082 .

I would like to bounce back to the SERVER1 all the direct requests that are sent to SERVER2, but keep the ones from the load balancing. So this should forbid direct access to SERVER2:8082 and redirect to SERVER1:80 .