I've been bashing my head against a wall with this one all day and I would greatly appreciate a few more eyes on the problem at hand.
We have an in-house SVN Server that contains all live and development code for our website. Our live server can connect to this and get updates from the repository.
This was all working fine until we migrated the SVN Server from a physical machine to a vSphere VM. Now, for some reason that continues to fathom me, we can no longer connect to the SVN Server.
The SVN Server runs CentOS 6.2, Apache and SVN 1.7.2. SELinux is well and trully disabled and the problem remains when iptables is stopped.
Our production server does run an older version of CentOS and SVN but the same system worked previously so I don't think that this is the issue.
Of note, if I have iptables enabled, using service iptables status, I can see a single packet coming in and being accepted but the production server simply hangs on any svn command. If I give up waiting and do a CTRL-C to break the process I get a "could not connect to server".
To me it appears to be something to do with the SVN Server rejecting external connections but I have no idea how this would happen.
Any thoughts on what I can try from here?
Thanks, Rob
Edit: Network topology
Production server sits externally to our in-house SVN server. Our IPCop (?) firewall allows connections from it (and it alone) on port 80 and passes the connection to the SVN Server. The hardware is all pretty decent and I don't doubt that its doing its job correctly, especially as iptables is seeing the new connections.
subversion.conf (in /etc/httpd/conf.d)
LoadModule dav_svn_module modules/mod_dav_svn.so
<Location /repos>
DAV svn
SVNPath /var/svn/repos
<LimitExcept PROPFIND OPTIONS REPORT>
AuthType Basic
AuthName "SVN Server"
AuthUserFile /var/svn/svn-auth
Require valid-user
</LimitExcept>
</Location>
ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:5F:C8:3A
inet addr:172.16.0.14 Bcast:172.16.0.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe5f:c83a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:32317 errors:0 dropped:0 overruns:0 frame:0
TX packets:632 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2544036 (2.4 MiB) TX bytes:143207 (139.8 KiB)
netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 1484/mysqld
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1135/rpcbind
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1351/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1230/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1575/master
tcp 0 0 0.0.0.0:58401 0.0.0.0:* LISTEN 1153/rpc.statd
tcp 0 0 0.0.0.0:5672 0.0.0.0:* LISTEN 1626/qpidd
tcp 0 0 :::139 :::* LISTEN 1678/smbd
tcp 0 0 :::111 :::* LISTEN 1135/rpcbind
tcp 0 0 :::80 :::* LISTEN 1615/httpd
tcp 0 0 :::22 :::* LISTEN 1351/sshd
tcp 0 0 ::1:631 :::* LISTEN 1230/cupsd
tcp 0 0 ::1:25 :::* LISTEN 1575/master
tcp 0 0 :::445 :::* LISTEN 1678/smbd
tcp 0 0 :::56799 :::* LISTEN 1153/rpc.statd
iptables --list -v -n (when iptables is stopped)
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
iptables --list -v -n (when iptables is running, after one attempted svn connection)
Chain INPUT (policy ACCEPT 68 packets, 6561 bytes)
pkts bytes target prot opt in out source destination
19 1304 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:80
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 17 packets, 1612 bytes)
pkts bytes target prot opt in out source destination
tcpdump
17:08:18.455114 IP 'production server'.43255 > 'svn server'.local.http: Flags [S], seq 3200354543, win 5840, options [mss 1380,sackOK,TS val 2011458346 ecr 0,nop,wscale 7], length 0
17:08:18.455169 IP 'svn server'.local.http > 'production server'.43255: Flags [S.], seq 629885453, ack 3200354544, win 14480, options [mss 1460,sackOK,TS val 816478 ecr 2011449346,nop,wscale 7], length 0
17:08:19.655317 IP 'svn server'.local.http > 'production server'k.43255: Flags [S.], seq 629885453, ack 3200354544, win 14480, options [mss 1460,sackOK,TS val 817679 ecr 2011449346,nop,wscale 7], length 0
