WCF fails every 10 calls in a silent and infuriating way

So I have a simple test case WCF service. The idea is that one process calls another and registers for callbacks, and the other then calls the original caller when events occur. WCF native callbacks don't work properly, or for this application, so don't bother suggesting that. I have a very simple test program that exhibits the behavior. Every 10 tries, it will pause for a while (the timeout period) and the recover after it. It recovers without firing any channel events - open/close/whatever. I have to be missing something obviously, but what...

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ServiceModel;
using System.Threading;

namespace WCFTest
{
 [ServiceContract(Namespace = "http://WCF.WTF")]
 public interface IServerEvents
 {
  [OperationContract(IsOneWay = true)]
  void Heartbeat();
 }

 [ServiceContract(Namespace = "http://WCF.WTF")]
 public interface ICallbackEvents
 {
  [OperationContract(IsOneWay = true)]
  void HeartbeatAck();
 }

 [ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple, InstanceContextMode =     InstanceContextMode.Single)]
 [CallbackBehavior(IncludeExceptionDetailInFaults = true)]
 class Program : IServerEvents, ICallbackEvents
 {
  static AutoResetEvent CalledBack = new AutoResetEvent(false);
  static ChannelFactory<IServerEvents> ServerChannelFactory;
  static ChannelFactory<ICallbackEvents> ClientChannelFactory;
  static ServiceHost ServerHost;
  static ServiceHost ClientHost;

  static int Timeout = 5;

  private Program()
  {
  }

  static void Main(string[] args)
  {
   NetTcpBinding binding = new NetTcpBinding(SecurityMode.None)
    {
     OpenTimeout = new TimeSpan(0, 0, Timeout),
     SendTimeout = new TimeSpan(0, 0, Timeout),
     ReceiveTimeout = new TimeSpan(0, 0, Timeout),
     MaxConnections = 50,
     ListenBacklog = 50     
    };
   Uri serverUri = new Uri("net.tcp://localhost:3123/WTF");
   ServerChannelFactory = new ChannelFactory<IServerEvents>(binding, new     EndpointAddress(serverUri));
   ClientChannelFactory = new ChannelFactory<ICallbackEvents>(binding, new     EndpointAddress("net.tcp://localhost:3123/WTF/Client"));
   ServerChannelFactory.Closing += new EventHandler((s,x) =>     Console.WriteLine("SClosing"));
   ServerChannelFactory.Opening += new EventHandler((s, x) => Console.WriteLine("SOpening"));
   ServerChannelFactory.Faulted += new EventHandler((s,x)=> Console.WriteLine("SFaulted"));
   ClientChannelFactory.Closing += new EventHandler((s, x) => Console.WriteLine("CClosing"));
   ClientChannelFactory.Opening += new EventHandler((s, x) => Console.WriteLine("COpening"));
   ClientChannelFactory.Faulted += new EventHandler((s, x) => Console.WriteLine("CFaulted"));

   ServerHost = StartServer("/Server", new Program(), typeof(IServerEvents));
   ClientHost = StartServer("/Client", new Program(), typeof(ICallbackEvents));

   while (true)
   {
    Thread.Sleep(100);
    try
    {
     new Program().Heartbeat();
     if (!CalledBack.WaitOne(2500, true))
     {
      throw new TimeoutException("Epic fail.");
     }
    }
    catch (Exception x)
    {
     Console.WriteLine("Failed heartbeat.\n{0}", x);
    }
   }
  }

  public void Heartbeat()
  {
   Console.Write(".");
   Console.Out.Flush();
   try
   {
    ClientChannelFactory.CreateChannel().HeartbeatAck();
   }
   catch (Exception x)
   {
    Console.WriteLine("Couldn't ACK heartbeat.\n{0}", x);
   }
  }

  public void HeartbeatAck()
  {
   Console.Write("!");
   Console.Out.Flush();
   CalledBack.Set();
  }

  private static ServiceHost StartServer<T>(string fragment, T remoteObject, Type interfaceType)
  {开发者_开发技巧
   ServiceHost retHost = null;
   using (AutoResetEvent revent = new AutoResetEvent(false))
   {
bool hostOk = false;
// The service host has to be started on a non-sync-context thread or bad things (tm) will happen.
ThreadPool.QueueUserWorkItem((oo) =>
{
 try
 {
  retHost = new ServiceHost(remoteObject, new Uri("net.tcp://localhost:3123/WTF"));
  var binding = new NetTcpBinding(SecurityMode.None)
      {
       OpenTimeout = new TimeSpan(0, 0, Timeout),
       SendTimeout = new TimeSpan(0, 0, Timeout),
       ReceiveTimeout = new TimeSpan(0, 0, Timeout),
       MaxConnections = 50,
       ListenBacklog = 50           
      };
  retHost.AddServiceEndpoint(interfaceType, binding, fragment);
      retHost.Open();
      hostOk = true;
     }
     catch (Exception xc)
     {
      Console.WriteLine("Couldn't start WCF Service Host!\n{0}", xc);
     }
     finally
     {
      try { revent.Set(); }
      catch { }
     }
    });
    revent.WaitOne(5000, true);
    return hostOk ? retHost : null;
   }
  }
 }
}

---

[This isn't an answer, but too long to fit as a comment.]

The test case differs from the original code. In cases where you aren't really sure where a problem exists, its quite unhelpful to change the code base as it skew's the results.

Instead, mock out the client and the server so you can unit test the two seperately. WcfTestClient acts a client quite well.

Due to the nature of services, I recommend keeping near 0 logic in the primary layer and instead bounce the implementations into a re-useable DLL (a mating of MVC and n-tier). This makes it super easy to lay a new front end on the WCF service (like a console app) for debugging purposes.

---

So I think the fundamental problem here is bad documentation and bad runtime reporting. While it was not easy to confirm, there seems to be some limit (not MaxConnections/ListenBackLog) of 10 active connections. Since Dispose is called whenever, the ChannelFactory will release it's connection whenever. As a result, 10 connections get used up and it pauses (while making a socket request, NOT while doing "real WCF" activities that would be subject to specified timeouts) until Dispose runs. So this means you can't store Open channel factories "long term" in WCF, which I think is bad, especially since you can't open and close factories multiple times. So the best you can do is store Bindings/EndpointAddresses.