One of my customers of the company I work for has made a change to their Internet connection, and now we can't connect to them any more from our LAN.
To help me troubleshoot this issue, the network guy on the customer's site has configured their firewall so that a HTTPS connection to their public IP address is open to any IP. I should put https://<customer's IP> in my browser and get a web page. Well, it works from any network I've tried (even from my smartphone), just not from my company's LAN.
I thought it may be an issue with our firewall (though I checked its rules and it allows outbound TCP port 443 to anywhere), so I just connected a PC directly to the network connection of our provider, bypassing out firewall completely, and still it didn't work (everything else worked).
So I asked for help to our Internet provider's customer service, and they asked me to do a tracert to our customer's IP. The tracert is successful, as the final hop shown in the output is the host I want to reach. So they said there's no problem. :(
I also tried telnet <customer's IP> 443 and that works as well: I get a blank page with the cursor blinking (I've tried using another random port and that gives me an error message, as it should).
Still, from any browser of any PC in my LAN I can't open that URL.
I tried checking the network traffic with Wireshark: I see the packages going through and answers coming back, thought the packets I see passing are far less than they are if I successfully connect to another HTTPS website. See the attached screenshot: I had to blur the IPs, anyway the longer string is my PC's local IP address, the shorter one is the customer's public IP.
I don't know what else to try. This is the only IP doing this... Any idea what could I try to find a solution to this issue?
Thanks, let me know if you need further details.
Edit: when I say "it doesn't work" I mean: the page doesn't open, the browser keeps loading for a long time and eventually shows an error saying that the page cannot be opened. I'm not in my office now so I can't paste the exact message, but it's the usual message you get when the browser reaches its timeout.
When I say "it works", I mean the browser loads and shows a webpage (it's the logon page for the customers' firewall admin interface: so there's the firewall brand's logo and there are fields to enter a user id and a password).
Update 13/09/2012: tried again to connect to the customer's network through our Internet connection without a firewall. This is what I did:
Run a Kubuntu 12.04 live distro on a spare laptop;
Updated all the packages I could and installed WireShark;
Attached it to my LAN and verified that I couldn't open https://<customer's IP>. Verified that the Wireshark trace for this attempt was the same as the one I've already posted;
Verified that I could connect to another customer's host using rdesktop (it worked);
Tried to rdesktop to <customer's IP>, here's the output:
kubuntu@kubuntu:/etc$ rdesktop <customer's IP>
Autoselected keyboard map en-us
ERROR: recv: Connection reset by peer
Disconnected the laptop from the LAN;
Disconnected the firewall from the Extranet connection, connected the laptop instead. Set its network configuration so that I could access the Internet;
Verified that I could connect to other websites in http and https and in RDP to other customers' hosts - it all worked as expected;
Verified that I could still traceroute to <customer's IP>: I could;
Verified that I still couldn't open https://<customer's IP> (same exact result as before);
Checked the WireShark trace for this attempt and noticed a different behaviour: I could see packets going out to the customer's IP, but no replies at all;
Tried to run rdesktop again, with a slightly different result:
kubuntu@kubuntu:/etc/network$ rdesktop <customer's IP>
Autoselected keyboard map en-us
ERROR: <customer's IP>: unable to connect
Finally gave up, put everything back as it was before, turned off the laptop and lost the WireShark traces I had saved. :( I still remember them very well though. :)
Can you get anything out of it? Thank you very much.
Update 12/09/2012 n.2: I followed the suggestion by MadHatter in the comments. From inside the firewall, this is what I get:
user@ubuntu-mantis:~$ openssl s_client -connect <customer's IP>:443
CONNECTED(00000003)
If I now type GET / the output pauses for several seconds and then I get:
write:errno=104
I'm going to try the same, but bypassing the firewall, as soon as I can.
Thanks.
Update 12/09/2012 n.3: So, I think ISA Server is altering the results of my tests... I tried installing Wireshark directly on the firewall and monitoring the packets on the Extranet network card. When the destination is the customer's IP, whatever service I try to connect to (HTTPS, RDP or SAProuter), I can only see outbound packets and no response packets whatsoever from their side.
It looks like ISA Server is "faking" the remote server's replies, that's why I get a connection using telnet or the openSSL client.
This is the wireshark trace from inside our LAN:
But this is the trace on the Extranet network card:
This makes a bit more sense... I'll send this info to the customer's tech and see if he can make anything out of it.
Thanks to all that took the time to read my question and post suggestions. I'll update this post again.
