Qt quncompress gzip data

I stumble upon a problem, and can't find a solution.

So what I want to do is uncompress data in qt, using qUncompress(QByteArray), send from www in gzip format. I used wireshark to determine that this is valid gzip stream, also tested with zip/rar and both can uncompress it.

Code so far, is like this:

    static const char dat[40] = {
         0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0xaa, 0x2e, 0x2e, 0x49, 0x2c, 0x29,
         0x2d, 0xb6, 0x4a, 0x4b, 0xcc, 0x29, 0x4e, 0xad, 0x05, 0x00, 0x00, 0x00, 0xff, 0xff, 0x03, 0x00,
         0x2a, 0x63, 0x18, 0xc5, 0x0e, 0x00, 0x00, 0x00
    };
//this data contains string: {status:false}, in gzip format
QByteArray data;
           data.append( dat, sizeof(dat) );

unsigned int size = 14; //expected uncompresed size, reconstruct it BigEndianes

//prepand expected uncompressed size, last 4 byte in dat 0x0e = 14
QByteArray dataPlusSize;

dataPlusSize.append( (unsigned int)((size >> 24) & 0xFF));
dataPlusSize.append( (unsigned int)((size >> 16) & 0xFF));
dataPlusSize.append( (unsigned int)((size >> 8) & 0xFF));
dataPlusSize.append( (unsigned int)((size >> 0) & 0xFF));

QByteArray uncomp = qUncompress( dataPlusSize );
qDebug() << uncomp;

And uncompression fails with: qUncompress: Z_DATA_ERROR: Input data is corrupted.

AFAIK gzip consist of 10 byte header, DEFLATE peyload, 12 byte trailer ( 8 byte CRC32 + 4 byte ISIZE - uncompresed data size ). Striping header and trailer should leave me with DEFLATE data stream, qUncompress yields same error.

I checked with data string compressed in PHP, like this:

$stringData = gzcompress( "{status:false}", 1);

and qUncompress uncompress that data.(I didn't see and gzip header though i.e. ID1 = 0x1f, ID2 = 0x8b ) I checked above code with debug, and error o开发者_Go百科ccurs at:

        if (
        #endif
            ((BITS(8) << 8) + (hold >> 8)) % 31) { //here is error, WHY? long unsigned int hold = 35615
            strm->msg = (char *)"incorrect header check";
            state->mode = BAD;
            break;
        } 

inflate.c line 610.

I know that qUncompress is simply a wrapper to zlib, so I suppose it should handle gzip without any problem. Any comments are more then welcome.

Best regards

---

Here is my contribution... I've developed a class (QCompressor), based on zlib for easily compress/decompress QByteArray using GZIP.

qcompressor.h:

#ifndef QCOMPRESSOR_H
#define QCOMPRESSOR_H

#include <zlib.h>
#include <QByteArray>

#define GZIP_WINDOWS_BIT 15 + 16
#define GZIP_CHUNK_SIZE 32 * 1024

class QCompressor
{
public:
    static bool gzipCompress(QByteArray input, QByteArray &output, int level = -1);
    static bool gzipDecompress(QByteArray input, QByteArray &output);
};

#endif // QCOMPRESSOR_H

qcompressor.cpp:

#include "qcompressor.h"

/**
 * @brief Compresses the given buffer using the standard GZIP algorithm
 * @param input The buffer to be compressed
 * @param output The result of the compression
 * @param level The compression level to be used (@c 0 = no compression, @c 9 = max, @c -1 = default)
 * @return @c true if the compression was successful, @c false otherwise
 */
bool QCompressor::gzipCompress(QByteArray input, QByteArray &output, int level)
{
    // Prepare output
    output.clear();

    // Is there something to do?
    if(input.length())
    {
        // Declare vars
        int flush = 0;

        // Prepare deflater status
        z_stream strm;
        strm.zalloc = Z_NULL;
        strm.zfree = Z_NULL;
        strm.opaque = Z_NULL;
        strm.avail_in = 0;
        strm.next_in = Z_NULL;

        // Initialize deflater
        int ret = deflateInit2(&strm, qMax(-1, qMin(9, level)), Z_DEFLATED, GZIP_WINDOWS_BIT, 8, Z_DEFAULT_STRATEGY);

        if (ret != Z_OK)
            return(false);

        // Prepare output
        output.clear();

        // Extract pointer to input data
        char *input_data = input.data();
        int input_data_left = input.length();

        // Compress data until available
        do {
            // Determine current chunk size
            int chunk_size = qMin(GZIP_CHUNK_SIZE, input_data_left);

            // Set deflater references
            strm.next_in = (unsigned char*)input_data;
            strm.avail_in = chunk_size;

            // Update interval variables
            input_data += chunk_size;
            input_data_left -= chunk_size;

            // Determine if it is the last chunk
            flush = (input_data_left <= 0 ? Z_FINISH : Z_NO_FLUSH);

            // Deflate chunk and cumulate output
            do {

                // Declare vars
                char out[GZIP_CHUNK_SIZE];

                // Set deflater references
                strm.next_out = (unsigned char*)out;
                strm.avail_out = GZIP_CHUNK_SIZE;

                // Try to deflate chunk
                ret = deflate(&strm, flush);

                // Check errors
                if(ret == Z_STREAM_ERROR)
                {
                    // Clean-up
                    deflateEnd(&strm);

                    // Return
                    return(false);
                }

                // Determine compressed size
                int have = (GZIP_CHUNK_SIZE - strm.avail_out);

                // Cumulate result
                if(have > 0)
                    output.append((char*)out, have);

            } while (strm.avail_out == 0);

        } while (flush != Z_FINISH);

        // Clean-up
        (void)deflateEnd(&strm);

        // Return
        return(ret == Z_STREAM_END);
    }
    else
        return(true);
}

/**
 * @brief Decompresses the given buffer using the standard GZIP algorithm
 * @param input The buffer to be decompressed
 * @param output The result of the decompression
 * @return @c true if the decompression was successfull, @c false otherwise
 */
bool QCompressor::gzipDecompress(QByteArray input, QByteArray &output)
{
    // Prepare output
    output.clear();

    // Is there something to do?
    if(input.length() > 0)
    {
        // Prepare inflater status
        z_stream strm;
        strm.zalloc = Z_NULL;
        strm.zfree = Z_NULL;
        strm.opaque = Z_NULL;
        strm.avail_in = 0;
        strm.next_in = Z_NULL;

        // Initialize inflater
        int ret = inflateInit2(&strm, GZIP_WINDOWS_BIT);

        if (ret != Z_OK)
            return(false);

        // Extract pointer to input data
        char *input_data = input.data();
        int input_data_left = input.length();

        // Decompress data until available
        do {
            // Determine current chunk size
            int chunk_size = qMin(GZIP_CHUNK_SIZE, input_data_left);

            // Check for termination
            if(chunk_size <= 0)
                break;

            // Set inflater references
            strm.next_in = (unsigned char*)input_data;
            strm.avail_in = chunk_size;

            // Update interval variables
            input_data += chunk_size;
            input_data_left -= chunk_size;

            // Inflate chunk and cumulate output
            do {

                // Declare vars
                char out[GZIP_CHUNK_SIZE];

                // Set inflater references
                strm.next_out = (unsigned char*)out;
                strm.avail_out = GZIP_CHUNK_SIZE;

                // Try to inflate chunk
                ret = inflate(&strm, Z_NO_FLUSH);

                switch (ret) {
                case Z_NEED_DICT:
                    ret = Z_DATA_ERROR;
                case Z_DATA_ERROR:
                case Z_MEM_ERROR:
                case Z_STREAM_ERROR:
                    // Clean-up
                    inflateEnd(&strm);

                    // Return
                    return(false);
                }

                // Determine decompressed size
                int have = (GZIP_CHUNK_SIZE - strm.avail_out);

                // Cumulate result
                if(have > 0)
                    output.append((char*)out, have);

            } while (strm.avail_out == 0);

        } while (ret != Z_STREAM_END);

        // Clean-up
        inflateEnd(&strm);

        // Return
        return (ret == Z_STREAM_END);
    }
    else
        return(true);
}

and here the main() of my test program:

#include <QDebug>
#include "qcompressor.h"

int main(int argc, char *argv[])
{
    Q_UNUSED(argc);
    Q_UNUSED(argv);

    QString initialPlainText = "This is a test program for verifying that the QCompressor class works fine!";

    qDebug() << "Initial plain text is: " << initialPlainText;

    QByteArray compressed;

    if(QCompressor::gzipCompress(initialPlainText.toLatin1(), compressed))
    {
        qDebug() << "Compressed text length is:" << compressed.length();

        QByteArray decompressed;

        if(QCompressor::gzipDecompress(compressed, decompressed))
        {
            qDebug() << "Decompressed text is: " << QString::fromLatin1(decompressed);
        }
        else
            qDebug() << "Can't decompress";
    }
    else
        qDebug() << "Can't compress";
}

For let this work, you need to add a line LIBS += -lz to your .pro file for linking against zlib.

---

Directly using zlib is not that hard.

I've been doing it like this:

QByteArray gUncompress(const QByteArray &data)
{
    if (data.size() <= 4) {
        qWarning("gUncompress: Input data is truncated");
        return QByteArray();
    }

    QByteArray result;

    int ret;
    z_stream strm;
    static const int CHUNK_SIZE = 1024;
    char out[CHUNK_SIZE];

    /* allocate inflate state */
    strm.zalloc = Z_NULL;
    strm.zfree = Z_NULL;
    strm.opaque = Z_NULL;
    strm.avail_in = data.size();
    strm.next_in = (Bytef*)(data.data());

    ret = inflateInit2(&strm, 15 +  32); // gzip decoding
    if (ret != Z_OK)
        return QByteArray();

    // run inflate()
    do {
        strm.avail_out = CHUNK_SIZE;
        strm.next_out = (Bytef*)(out);

        ret = inflate(&strm, Z_NO_FLUSH);
        Q_ASSERT(ret != Z_STREAM_ERROR);  // state not clobbered

        switch (ret) {
        case Z_NEED_DICT:
            ret = Z_DATA_ERROR;     // and fall through
        case Z_DATA_ERROR:
        case Z_MEM_ERROR:
            (void)inflateEnd(&strm);
            return QByteArray();
        }

        result.append(out, CHUNK_SIZE - strm.avail_out);
    } while (strm.avail_out == 0);

    // clean up and return
    inflateEnd(&strm);
    return result;
}

The code is monstly copied from the zlib code example page. You will need to include <zlib.h>

---

You also forgot dataPlusSize.append(data);. However, that won't solve your problem. The problem is that while gzip and zlib have the same compressed data format, their headers and trailers are different. See: http://www.zlib.net/zlib_faq.html#faq18

qUncompress uses the zlib uncompress, so it can only handle the zlib format, not the gzip format. It would need to call the gzXXXX functions to handle the gzip format.

The reason that qUncompress can handle output from PHP's gzcompress is that gzcompress compresses the given string using the ZLIB data format. See: http://php.net/manual/en/function.gzcompress.php

As CiscoIPPhone mentioned, you'll need to write your own to functions to handle gzip data.

---

Inspired from this answer and this answer, I have written further simplified C++ code. It's independent of Qt.

namespace Zlib
{
  using std::string;

#define GZIP(FUNCTION, ...) \
  static const size_t CHUNK_SIZE = 1000000; \
  ::z_stream stream = {}; \
  VERIFY(FUNCTION##Init2(&stream, __VA_ARGS__) == Z_OK, ""); \
  FINALLY_CALL(:FUNCTION##End(&stream)); \
  \
  string output; \
  for(int flush = Z_NO_FLUSH; flush == Z_NO_FLUSH; stream.avail_out = 0) \
  { \
    const auto pending = input.size() - stream.total_in; \
    stream.avail_in = std::min(CHUNK_SIZE, pending); \
    stream.next_in = reinterpret_cast<Bytef*>(const_cast<char*>(input.data()) + stream.total_in); \
    \
    while(stream.avail_out == 0) \
    { \
      stream.avail_out = CHUNK_SIZE; \
      output.resize(stream.total_out + stream.avail_out); \
      stream.next_out = reinterpret_cast<Bytef*>(&output[0] + stream.total_out); \
      \
      flush = (stream.avail_in == pending)? Z_FINISH : Z_NO_FLUSH; \
      FUNCTION(&stream, flush); \
    } \
  } \
  output.resize(stream.total_out); \
  return output


  string
  StringToGzip (const string& input,
                const int level = Z_DEFAULT_COMPRESSION)
  {
    GZIP(::deflate, level, Z_DEFLATED, 15 | 16, 8, Z_DEFAULT_STRATEGY);
  }

  string
  GzipToString (const string& input)
  {
    GZIP(::inflate, 15 | 16);
  }
}