Probably just watch this video: http://screencast.com/t/OWE1OWVkO As you see, the delay between a connection being initiated (via telnet or firefox) and my program first getting word of it.

Here's the code that waits for the connection

    public IDLServer(System.Net.IPAddress addr,int port)
    {
        Listener = new TcpListener(addr, port);

        Listener.Server.NoDelay = true;//I added this just for testing, it has no impact

        Listener.Start();

        ConnectionThread = new Thread(ConnectionListener);
        ConnectionThread.Start();


    }

    private void ConnectionListener()
    {
        while (Running)
        {
            while (Listener.Pending() == false) { System.Threading.Thread.Sleep(1); }//this is the part with the lag
            Console.WriteLine("Client available");//from this point on everything runs perfectly fast 
            TcpClient cl = Listener.AcceptTcpClient(); 

            Thread proct = new Thread(new ParameterizedThreadStart(InstanceHandler));
            proct.Start(cl);


        }

    }

(I was having some trouble getting the code into a code block)

I've tried a couple different things, could it be I'm using TcpClient/Listener instead of a raw Socket object? It's not a mandatory TCP overhead I know, and I've tried running everything in the same thread, etc.