When I look at this description, I have a feeling, that this xdr integer is just a big-endian "standard" integer, but it's expressed in the most obfuscated way. Two's complement notation is better know as U2, and it's what we are using on today's processors. The byte order indicates that it's a big-endian notation.
So, answering your question, you should inverse elements in your array (0 <--> 3, 1 <-->2), as they are encoded in little-endian. Just to make sure, you should first check BitConverter.IsLittleEndian
to see on what machine you are running.