CURL response different than response to request sent from browser_问答_开发者

Attempting to submit a form with CURL, both via PHP and the command line. The response from the server consists of null content (the headers posted below).

When the same URL is submitted via a browser, the response consists of a proper webapge.

Have tried submitting the CURL request parameters via POST and GET via each of the following command line curl flags "-d" "-F" and "-G".

If the query string parameters are posted with "-d" flag, resulting header is:

HTTP/1.1 302 Moved Temporarily

Date: Thu, 02 Jun 2011 21:41:54 GMT

Server: Apache

Set-Cookie: JSESSIONID=DC5F435A96A353289F58593D54B89570; Path=/XXXXXXX

P3P: CP="CAO PSA OUR"

Location: http://www.XXXXXXXX.com/

Content-Length: 0

Connection: close

Content-Type: text/html;charset=UTF-8

Set-Cookie: XXXXXXXXXXXXXXXX=1318103232.20480.0000; path=/

If the query string parameters are posted with "-F" flag, the resulting header is:

HTTP/1.1 100 Continue

HTTP/1.1 500 Internal Server Error

Date: Thu, 02 Jun 2011 21:52:54 GMT

Server: Apache

Content-Length: 1677

Connection: close

Content-Type: text/html;charset=utf-8

Set-Cookie: XXXXXXXXXXXXXX=1318103232.20480.0000; path=/

Vary: Accept-Encoding

<html><head><title>Apache Tomcat/5.5.26 - Error report</开发者_开发百科title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error () that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.servlet.ServletException: Servlet execution threw an exception<br>
</pre></p><p><b>root cause</b> <pre>java.lang.NoClassDefFoundError: com/oreilly/servlet/multipart/MultipartParser<br>
    com.corsis.tuesday.servlet.mp.MPRequest.<init>(MPRequest.java:27)<br>
    com.corsis.tuesday.servlet.mp.MPRequest.<init>(MPRequest.java:21)<br>
    com.corsis.tuesday.servlet.TuesdayServlet.doPost(TuesdayServlet.java:494)<br>
    javax.servlet.http.HttpServlet.service(HttpServlet.java:710)<br>
    javax.servlet.http.HttpServlet.service(HttpServlet.java:803)<br>
</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/5.5.26 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/5.5.26</h3></body></html>

Questions:

What might cause a server to respond different depending on the nature of the CURL request.
How to successfully submit request via CURL?

HTTP/1.1 100 Continue

I had problems associated with this header before. Some servers simply do not understand it. Try this option to override Expect header.

curl_setopt( $curl_handle, CURLOPT_HTTPHEADER, array( 'Expect:' ) );

To add to what Richard said, I have seen cases where servers check the User-Agent string and behave differently based on its value.

I have just had an experience with this and what fixed it was surprising. In my situation I was logging into a server so I could upload a file, have the server do work on it, and then download the new file. I did this in Chrome first and used the dev tools to capture over 100 HTTP requests in this simple transaction. Most are simply grabbing resources I don't need if I am trying to do all of this from the command line, so I filtered out only the ones I knew at a minimum I should need.

Initially this boiled down to a GET to set the cookie and log in with a username and password, a POST to upload the file, a POST to execute the work on the file, and a GET to retrieve the new file. I could not get the first POST to actually work though. The response from that POST is supposed to be information containing the upload ID, time uploaded, etc, but instead I was getting empty JSON lists even though the status was 200 OK.

I used CURL to spoof the requests from the browser exactly (copying the User-Agent, overriding Expect, etc) and was still getting nothing. Then I started arbitrarily adding in some of the requests that I captured from Chrome between the first GET and POST, and low and behold after adding in a GET request for the JSON history before the POST the POST actually returned what it was supposed to.

TL;DR Some websites require more requests after the initial log in before you can POST. I would try to capture a successful exchange between the server and browser and look at all of the requests. Some requests might not be as superfluous as the seem.