How would I convert this crypto from C# to C

This is the C# code I use:

public void Decrypt(byte[] @in, byte[] @out, int size)
{
    lock (this)
    {
        for (ushort i = 0; i < size; i++)
        {
            if (_server)
            {
                @out[i] = (byte)(@in[i] ^ 0xAB);
                @out[i] = (byte)((@out[i] << 4) | (@out[i] >> 4));
                @out[i] = (byte)(ConquerKeys.Key2[_inCounter >> 8] ^ @out[i]);
                @out[i] = (byte)(ConquerKeys.Key1[_inCounter & 0xFF] ^ @out[i]);
            }
            else
            {
                @out[i] = (byte)(ConquerKeys.Key1[_inCounter & 0xFF] ^ @in[i]);
                @out[i] = (byte)(ConquerKeys.Key2[_inCounter >> 8] ^ @out[i]);
                @out[i] = (byte)((@out[i] << 4) | (@out[i] >> 4));
                @out[i] = (byte)(@out[i] ^ 0xAB);
            }
            _inCounter = (ushort)(_inCounter + 1);
        }
    }
}

and this is how I converted it to work in C.

char* decrypt(char* in, int size, int server)
{
    char out[size];
    memset(out, 0, size开发者_如何学编程);
    for (int i = 0; i < size; i++)
    {
        if (server == 1)
        {
            out[i] = in[i] ^ 0xAB;
            out[i] = out[i] << 4 | out[i] >> 4;
            out[i] = Key2[incounter >> 8] ^ out[i];
            out[i] = Key1[incounter & 0xFF] ^ in[i];
        }
        else if (server == 0)
        {
            out[i] = Key1[incounter & 0xFF] ^ in[i];
            out[i] = Key2[incounter >> 8] ^ out[i];
            out[i] = out[i] << 4 | out[i] >> 4;
            out[i] = out[i] ^ 0xAB;
        }
        incounter++;
    }
    return out;
}

However for some reason the C one does not work.

Link for the full C# file

Link for the full C file

Link for the C implementation

---

There was a translation error.

The C# line:

@out[i] = (byte)(ConquerKeys.Key1[_inCounter & 0xFF] ^ @out[i]);

Became:

out[i] = Key1[incounter & 0xFF] ^ in[i];

The value on the right of the xor (^) is from the wrong array.

Additionally, you are returning a stack-allocated variable, which will cause all sorts of problem.

Change:

char out[size];
memset(out, 0, size);

to:

char *out = (char*)calloc(size, sizeof(char));

---

The most glaring error I see is that you are returning a pointer to a stack-allocated array, which is going to get stomped by the next function call after decrypt() returns. You need to malloc() that buffer or pass in a pointer to a writable buffer.

---

You are returning a reference to a local variable which is illegal. Either let the caller pass in an array or use malloc() to create an array inside the method.

---

I also suggest turning char into unsigned char since it is more portable. If your platform assumes char is the same as signed char, the arithmetic (bit shifts, etc) will not work right.
So just specify unsigned char explicitly (use a typedef or include <stdint.h> if unsigned char seems too long-winded for you).