Curl Errors While Pulling from Twitter

I'm trying to pull some data from twitter via PHP. I'm using the tmhOAuth plugin, which can be found here. https://github.com/themattharris/tmhOAuth/

I wrote my code based off the example file "streaming.php", which can also be found on the above github page. Here is my code:

    require 'tmhOAuth.php';

    $tmhOAuth = new tmhOAuth(array(
            'consumer_key' => 'xxxhiddenxxx',
            'consumer_secret' => 'xxxhiddenxxx',
            'user_token' => 'xxxhiddenxxx',
            'user_secret' => 'xxxhiddenxxx'
    ));

    $method = 'http://stream.twitter.com/1/statuses/filter.json';

    $params = array(
            'follow' => '1307392917',
            'count'  => '5'
    );
    $tmhOAuth->streaming_request('POST', $method, $params, 'my_streaming_callback');
    $tmhOAuth->pr($tmhOAuth);

That was not printing out any of the twitter data I wanted to pull, and was only showing the debug information that the pr() command writes.

While trying to debug why I wasn't getting any data, I went in and added a line to tmhOAuth.php so that I could see what error cURL was giving. I did this by using

 echo curl_error($C);

The error that cURL outputed was :

transfer closed with outstanding read data remaining

I've done some research on that error, but I can't find anything that helps. There were a couple things that I found regarding content-length, but when I dug into the code I saw that the author of tmhOAuth had already addressed those issues (and commenting out his fixes didn't help).

Any help?

Update 1 Here is the response info gathered using curl_getinfo:

//Removed - an updated version is below

Update 2 Thanks to the comments below I realized that twitter was sending me data with transfer-encoding: chunked. I put this line into tmhOAuth.php to force out chunked data:

curl_setopt($c, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_0);

That worked, somewhat. I'm no longer getting any cURL errors, but my WRITEFUNCTION callback is still never getting called - so I'm never getting any actual data. Here's the output of my cURL object again:

    [response] => Array
        (
            [content-length] => 0
            [headers] => Array
                (
                    [content_type] => text/html; charset=iso-8859-1
                    [server] => Jetty(6.1.25)
                )

            [code] => 416
            [response] => 1
            [info] => Array
                (
                    [url] => http://stream.twitter.com/1/statuses/filter.json
                    [content_type] => text/html; charset=iso-8859-1
         开发者_JAVA技巧           [http_code] => 416
                    [header_size] => 116
                    [request_size] => 532
                    [filetime] => -1
                    [ssl_verify_result] => 0
                    [redirect_count] => 0
                    [total_time] => 0.118553
                    [namelookup_time] => 0.043927
                    [connect_time] => 0.070477
                    [pretransfer_time] => 0.07049
                    [size_upload] => 25
                    [size_download] => 0
                    [speed_download] => 0
                    [speed_upload] => 210
                    [download_content_length] => -1
                    [upload_content_length] => -1
                    [starttransfer_time] => 0.118384
                    [redirect_time] => 0
                    [request_header] => POST /1/statuses/filter.json HTTP/1.0
User-Agent: themattharris' HTTP Client
Host: stream.twitter.com
Accept: */*
Authorization: OAuth oauth_consumer_key="xxxhiddenxxx", oauth_nonce="xxxhidden", oauth_signature="xxxhidden", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1308226585", oauth_token="xxxhiddenxxx", oauth_version="1.0"
Content-Length: 25
Content-Type: application/x-www-form-urlencoded


                )

        )

)

Update 3: Couple things I've figured out so far... I removed the 'count' parameters from my POST request, and now the page seems to take forever. I figured this meant it was just downloading tons and tons of data, so I put a break into the streaming callback function, setup so that it kills the page after 5 loops.

I did this, and let it sit for quite awhile. After about 5 minutes, the page finished loading, and showed me what data I had gathered. It looked like I had gotten no data each time it ran through - only an end of line character. So, it's taking a minute for every piece of data I am downloading, and even then the only data that shows is an end of line character. Weird? Is this a twitter issue or a cURL issue?

---

I tried with the token api but never got something good, so this is the script I found here :

<?php
/**
 * API Streaming for Twitter.
 *
 * @author  Loïc Gerbaud <gerbaudloic@gmail.com>
 * @version 0.1 "itjustworks"
 */

define('TWITTER_LOGIN','login'); //login twitter
define('TWITTER_PASSWORD','myp4ssw0rd');    //password twitter
$sTrackingList = 504443371;//read my account but could be keywords

// ?
while(1){
 echo 'Connexion ';
 read_the_stream($sTrackingList);
 echo 'Deconnexion ';
}

/**read the stream
 *
 */
function read_the_stream($sTrackingList){
 $ch = curl_init();
 curl_setopt($ch,CURLOPT_URL,'https://stream.twitter.com/1/statuses/filter.json');
 curl_setopt($ch,CURLOPT_USERPWD,TWITTER_LOGIN.':'.TWITTER_PASSWORD);//Le couple login:password
 curl_setopt($ch, CURLOPT_NOBODY, 0);
 curl_setopt($ch, CURLOPT_HEADER, 0);
 curl_setopt($ch, CURLOPT_USERAGENT, '');
 curl_setopt($ch, CURLOPT_FOLLOWLOCATION,1);
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
 curl_setopt($ch, CURLOPT_HTTPHEADER, array('X-Twitter-Client: ItsMe','X-Twitter-Client-Version: 0.1','X-Twitter-Client-URL: http://blog.loicg.net/'));
 curl_setopt($ch, CURLOPT_POST, true);
 curl_setopt($ch, CURLOPT_POSTFIELDS,"follow=".$sTrackingList);//read the doc for your request
 curl_setopt($ch, CURLOPT_WRITEFUNCTION, 'write_callback');//function callback

 curl_exec($ch);
 curl_close($ch);
}

/** a demo with a writting log or put in MySQL
 */


 function write_callback($ch, $data) {
 if(strlen($data)>2){
 $oData = json_decode($data);
 if(isset($oData->text)){
 file_put_contents('log',$oData->text."\n",FILE_APPEND);
 }
 }
 return strlen($data);
} 
?>

run this script in your browser (you can close it after), update your twitter account and check the .log

---

After about 5 minutes, the page finished loading

Are you running streaming.php in the browser? If so, you have to run it via ssl, otherwise it doesn't work. I have a server chron job pointing to the file but you can do it also with the terminal:

 php /path/to/here/streaming.php

For view the data you are getting, you can store it into a database or log:

function my_streaming_callback($data, $length, $metrics) {  
    $ddf = fopen('/twitter/mydata.log','a');
    fwrite($ddf,$data);
    fclose($ddf);
}