HTTP Vs Vanilla sockets for uploading large binary file (50-200 MB) from Android device over a single hop Wi-Fi Network

Is there a substantial overhead of using HTTP over plain sockets (Java on Android) to send a large (50-200 MB) file [file is on the SD card] from an Android device to a Linux server over a Wi-Fi network.

In my current prototype I'm using CherryPy-3.2.0 to implement my HTTP server. I'm running Android 2.3.3 on a Nexus one as my client.

Currently it's taking around ~100 seconds** (on slower network 18 Mbps*) and ~50 seconds (on a faster 54 Mbps*) Wi-Fi network to upload a 50 MB binary file.

NOTE:

*I'm using WifiInfo.getLinkSpeed() to measure the network link speed

** This is the time difference before and after the HTTPClient.execute(postRequest)

Any other ideas regarding other expensive operations that may have a substantial part in the total time apart from the network and how to reduce this time would be appreciated.

Thanks.

EDIT - HTTP post code on Android

private void doHttpPost(String fileName) throws Exception{

    HttpParams httpParameters = new BasicHttpParams();

    // Set the timeout in milliseconds until a connection is established.
    int timeoutConnection = 9000000;
    HttpConnectionParams.setConnectionTimeout(httpParameters, timeoutConnection);
    // Set the default socket timeout (SO_TIMEOUT) 
    // in milliseconds which is the timeout for waiting for data.
    int timeoutSocket = 9000000;
    HttpConnectionParams.setSoTimeout(httpParameters, timeoutSocket);

    HttpClient client = new DefaultHttpClient(httpParameters);

    client.getParams().setParameter(ClientPNames.COOKIE_POLICY, CookiePolicy.RFC_2109);

    HttpPost postRequest = new HttpPost();
    postRequest.setURI(new URI("http://192.168.1.107:9999/upload/"));

    MultipartEntity multiPartEntity = new MultipartEntity();
    multiPartEntity.addPart("myFile", new FileBody(new File(fileName)));
    postRequest.setEntity(multiPartEntity);

    long before 开发者_如何学编程= TrafficStats.getTotalTxBytes();
    long start = System.currentTimeMillis();
    HttpResponse response = client.execute(postRequest);
    long end = System.currentTimeMillis(); 
    long after = TrafficStats.getTotalTxBytes();

    Log.d(LOG_TAG, "HTTP Post Execution took " + (end - start) + " ms.");


    if( before != TrafficStats.UNSUPPORTED && after != TrafficStats.UNSUPPORTED)
        Log.d(LOG_TAG, (after-before) + " bytes transmitted to the server");
    else
        Log.d(LOG_TAG, "This device doesnot support Network Traffic Stats");

    HttpEntity responseEntity = response.getEntity();


    if (responseEntity != null) {
        responseEntity.consumeContent();
        Log.d(LOG_TAG, "HTTP Post Response " + response.getEntity().getContent().toString() );
    }

    client.getConnectionManager().shutdown(); 

}

EDIT 2: Based on the results reported by this tool it looks like the SD card read speed is not an issue. So it may either be the HttpClient library or something else.

---

Overhead on HTTP connection comes from the headers that it sends along with your data (which is basically a constant). So the more data you send, the less the headers 'hurt you'. However, the much more important aspect to consider is encoding.

For example, if you are sending non-ASCII data, paired with a mime type of application/x-www-form-urlencoded you run the risk of exploding the input size because non-ASCII characters must be escaped.

From the spec:

The content type "application/x-www-form-urlencoded" is inefficient for sending large quantities of binary data or text containing non-ASCII characters. The content type "multipart/form-data" should be used for submitting forms that contain files, non-ASCII data, and binary data.

The alternative is multipart/form-data which efficient for binary data. So, make sure your application is using this MIME type (you can even probably check this on your server logs).

Another method which can considerably reduce your upload time is compression. If you are uploading data which isn't already compressed (most image and video formats are already compressed) try adding gzip compression to your uploads. Another post shows the details of setting this up in android.

If your data is of a specific format (say an image), you can look into lossless compression algorithms for your type of data (png for images, FLAC for audio, etc.). Compression always comes at the price of CPU (battery), so keep that in mind.

Remember:

Don't optimize something until you know its the bottleneck. Maybe your server's connection is slow, maybe you can't read from the android file system fast enough to push your data to the network. Run some tests and see what works.

If it were me, I would not implement the straight tcp approach. Just my 2 cents, good luck!

---

No there is no significant overhead associated with using HTTP over raw sockets. However, it really depends on how you're using HttpClient to send this file. Are you properly buffering between the file system and HttpClient? The latency might not be the network, but reading the file from the filesystem. In fact you increased the raw link speed by 3x and only saw a reduction of 2x. That probably means there is some latency else where in your code or the server or filesystem. You might try uploading a file from a desktop client to make sure it's not the server causing the latency. Then look at the filesystem through put. If that all checks out then look at the code you've written using HttpClient and see if that could be optimized.

---

Note also in CherryPy 3.2 that the system for handling request bodies has been completely reworked, and you are much more free to implement varying handlers based on the media type of the request. By default, CherryPy will read your uploaded bytes into a temporary file; I assume your code then copies that to a more permanent location, which might be overhead that isn't useful to you (although there are good security reasons to use a temporary file). See also this question for discussion on renaming temp files.

You can override that behavior; make a subclass of _cpreqbody.Part with a make_file function that does what you want, then, in a Tool, replace cherrypy.request.body.part_class for that URI. Then post your code on http://tools.cherrypy.org so everyone can benefit :)