WebClient.DownloadString() + REST + HTTPS + Win2003 = hung thread

I've been battling with this issue for over 2 weeks now and have gotten nowhere.

First the environment: Windows 2003 R2 SP2 / SharePoint 2007 SP1 / .NET 3.5.

Basically, we are making web service calls to gather data from a remote API. The API has several endpoints for REST and several for SOAP. The endpoints are HTTPS endpoints with digest authentication. When we make calls with the SOAP endpoints, everything seems to work just fine. But then we try to make a call using REST and the thread hangs then dies a horrible death when IIS decides that the thread isn't responding anymore and kills it. At first, we thought this was an SSL issue (and it still might be) because we don't see any issues when using the HTTP endpoints (route to the same API just not SSL).

Below is the code we're using to make the REST call:

private void Process(HttpContext context, String url, String restParam)
{
    ServicePointManager.ServerCertificateValidationCallback += new Sy开发者_运维问答stem.Net.Security.RemoteCertificateValidationCallback(validateCertificate);

    WriteLogMessage("Start Process");
    String pattern = "{0}{1}";
    String address = String.Format(pattern, url, restParam);

    WriteLogMessage("ADDRESS is" + address);
    LSWebClient client = new LSWebClient();
    client.Timeout = 600000;

    WriteLogMessage("TIMEOUT (client.Timeout) is " + client.Timeout.ToString());
    client.Credentials = new NetworkCredential(XYZConfigurationSettings.APIUserName, XYZConfigurationSettings.APIPassword);

    try {
        String result = client.DownloadString(address);
        WriteLogMessage("End Process. RESULT length is " + (result != null ? result.Length : 0));
        context.Response.Write(result);
    } 
    catch (Exception ex) 
    {
        WriteLogMessage("EXCEPTION!!! Message----" + ex.Message + "---- StackTrace ----" + ex.StackTrace + "");
    }
}

private bool validateCertificate(object sender, X509Certificate cert, X509Chain chain, System.Net.Security.SslPolicyErrors error)
{
    WriteLogMessage("bypassAllCertificateStuff");
    return true;
}

So, crappy code aside, we put in a few things here to try to get around what we thought was an SSL Certificate issue. (setting the request timeout to 10 minutes, using custom certificate validation, etc...) However, none of this seems to fix the issue.

Here's the result of our logging:

2/28/2011 3:35:28 PM: Start
2/28/2011 3:35:28 PM: Start Process
2/28/2011 3:35:28 PM: ADDRESS ishttps://<host>/ws/rs/v1/taxonomy/TA/root/
2/28/2011 3:35:28 PM: TIMEOUT (client.Timeout) is 600000
2/28/2011 3:35:50 PM: CheckValidationResult
2/28/2011 3:35:50 PM: bypassAllCertificateStuff
2/28/2011 3:41:51 PM: EXCEPTION!!! Message ----Thread was being aborted.---- StackTrace ----   at System.Net.Connection.CompleteStartConnection(Boolean async, HttpWebRequest httpWebRequest)
   at System.Net.Connection.CompleteStartRequest(Boolean onSubmitThread, HttpWebRequest request, TriState needReConnect)
   at System.Net.Connection.SubmitRequest(HttpWebRequest request)
   at System.Net.ServicePoint.SubmitRequest(HttpWebRequest request, String connName)
   at System.Net.HttpWebRequest.SubmitRequest(ServicePoint servicePoint)
   at System.Net.HttpWebRequest.GetResponse()
   at System.Net.WebClient.GetWebResponse(WebRequest request)
   at System.Net.WebClient.DownloadBits(WebRequest request, Stream writeStream, CompletionDelegate completionDelegate, AsyncOperation asyncOp)
   at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
   at System.Net.WebClient.DownloadString(Uri address)
   at System.Net.WebClient.DownloadString(String address)
   at XYZ.DAO.Handlers.RestServiceHandler.Process(HttpContext context, String url, String restParam)
   at XYZ.DAO.Handlers.RestServiceHandler.ProcessRequest(HttpContext context)----

I have attempted to use my browser to view the return data, but the browser is IE6, which doesn't support SSL. However, I can see (in Fiddler / Charles proxy) that it does attempt to make the request and receives a 401 error but since I can not see server traffic using these programs I can not tell at exactly what step the error is happening.

To make matters worse, I can not reproduce this issue on any other server I have (note: they are all Windows 2008 servers).

So, in summary, here's what I've found:

SOAP - work

REST - no work

Win2008 - work

Win2003 - no work

HTTP - work

HTTPS - no work

If anyone has any insight or any other debugging / information gathering that I haven't tried I would be extremely greatful.

---

You should be able to get a bunch more tracing information if you add the following to your client .config file.

<system.diagnostics>
    <sources>
        <source name="System.Net" switchValue="Information, ActivityTracing">
            <listeners>
                <add name="System.Net"
                     type="System.Diagnostics.TextWriterTraceListener"
                     initializeData="System.Net.trace.log" />
            </listeners>
        </source>
    </sources>
</system.diagnostics>

---

I've found what was causing the web service call to hang - the issue was that the service we were calling was using replay attack protection along with digest security:

1. Our server would send an initial request sans security header
2. The request was responded to with a standard 401 challenge providing a nonce for use. (That nonce expires after 10 seconds after the challenge)
3. Our server then took 30 seconds to generate a second response using this nonce
4. So the remote server would then find the expired nonce and again issue another 401 challenge.

The cycle would continue until the local server's thread was terminated. However, why our local server is taking 30 $@#%! seconds to generate a security header is beyond me. I inspected the logs that were provided through the diagnostics above, but none of it was much help. I'm going to chalk it up to the server being overloaded and not having enough memory to process it's way out of a wet paper bag.