[ACCEPTED]-Protobuf checksum (crc)-crc

Accepted answer

Score: 10

Protobuf streams are appendable. If you 20 know a field number that doesn't exist in the data, you can simply 19 append data against that field. If you are 18 intending to add 1 or 2 bytes of CRC data, then 17 a "varint" is probably your best bet (note 16 that "varint" is a 7 bit encoding format 15 with the 8th bit a continuation marker, so 14 you probably want to use 7, 14 or 21 bits 13 or actual CRC data), then you can just append:

the chosen field number, left-shifted 3 bits, then varint encoded
the CRC data, varint encoded

However! The 12 wrinkle in this is that the decoder will 11 still often interpret and store this data, meaning 10 that if you serialize it, it will include 9 this data in the output.

The other approach, which 8 avoids this, would be to encapsulate the protobuf data 7 in some framing mechanism of your own devising. For 6 example, you could choose to do:

4 bytes to represent the protobuf payload length, "n"
"n" bytes of the protobuf payload
2 bytes of CRC data calculated over the "n" bytes

I'd probably 5 go with the second option. Note that you 4 could choose "varint" encoding rather than 3 fixed length encoding for the length prefix 2 if you want. Probably not worth it for the 1 CRC, though, since that will be fixed length.

Score: 1

Crc should be saved before. This makes deserialization 2 from stream trivial by using Seek (to skip header).

Here 1 is simplest implementation:

// serialize
using (var file = File.Create("test.bin"))
using (var mem = new MemoryStream())
{
    Serializer.Serialize(mem, obj); // serialize obj into memory first
    // ... calculate crc
    file.Write(new byte[] { crc }, 0, 1);
    mem.WriteTo(file);
}

// deserialize
using (var file = File.OpenRead("test.bin"))
{
    var crc = file.ReadByte();
    // ... calculate and check crc
    file.Seek(1, SeekOrigin.Begin);
    Serializer.Deserialize<ObjType>(file);
}

Source: stackoverflow.com

[ACCEPTED]-Protobuf checksum (crc)-crc

How could I guess a checksum algorithm?

CRC16 checksum: HCS08 vs. Kermit vs. XMODEM

_mm_crc32_u64 poorly defined

Calculating FCS(CRC) for HDLC frame

Verification of a CRC checksum against zero

Which protobuf optimization?

Scala hex string to bytes

How to include custom directory when building gRPC dotnet core project with MSBuild

protobuf-net enum serialization

Self referencing fields in protobuf messages