Is it possible to serialize and deserialize a class in C++?
I've been using Java for 3 years now, and serialization / deserialization is fairly trivial in that language. Does C++ have similar features? Are there native libraries that handle serialization?
An example would be helpful.
This question is related to
c++
serialization
You can check the amef protocol, an example of C++ encoding in amef would be like,
//Create a new AMEF object
AMEFObject *object = new AMEFObject();
//Add a child string object
object->addPacket("This is the Automated Message Exchange Format Object property!!","adasd");
//Add a child integer object
object->addPacket(21213);
//Add a child boolean object
object->addPacket(true);
AMEFObject *object2 = new AMEFObject();
string j = "This is the property of a nested Automated Message Exchange Format Object";
object2->addPacket(j);
object2->addPacket(134123);
object2->addPacket(false);
//Add a child character object
object2->addPacket('d');
//Add a child AMEF Object
object->addPacket(object2);
//Encode the AMEF obejct
string str = new AMEFEncoder()->encode(object,false);
Decoding in java would be like,
string arr = amef encoded byte array value;
AMEFDecoder decoder = new AMEFDecoder()
AMEFObject object1 = AMEFDecoder.decode(arr,true);
The Protocol implementation has codecs for both C++ and Java, the interesting part is it can retain object class representation in the form of name value pairs, I required a similar protocol in my last project, when i incidentally stumbled upon this protocol, i had actually modified the base library according to my requirements. Hope this helps you.
Boost is a good suggestion. But if you would like to roll your own, it's not so hard.
Basically you just need a way to build up a graph of objects and then output them to some structured storage format (JSON, XML, YAML, whatever). Building up the graph is as simple as utilizing a marking recursive decent object algorithm and then outputting all the marked objects.
I wrote an article describing a rudimentary (but still powerful) serialization system. You may find it interesting: Using SQLite as an On-disk File Format, Part 2.
I recommend Google protocol buffers. I had the chance to test drive the library on a new project and it's remarkably easy to use. The library is heavily optimized for performance.
Protobuf is different than other serialization solutions mentioned here in the sense that it does not serialize your objects, but rather generates code for objects that are serialization according to your specification.
If you want simple and best performance and don't care about backward data compatibility, try HPS, it's lightweight, much faster than Boost, etc, and much easier to use than Protobuf, etc.
Example:
std::vector<int> data({22, 333, -4444});
std::string serialized = hps::serialize_to_string(data);
auto parsed = hps::parse_from_string<std::vector<int>>(serialized);
Boost is a good suggestion. But if you would like to roll your own, it's not so hard.
Basically you just need a way to build up a graph of objects and then output them to some structured storage format (JSON, XML, YAML, whatever). Building up the graph is as simple as utilizing a marking recursive decent object algorithm and then outputting all the marked objects.
I wrote an article describing a rudimentary (but still powerful) serialization system. You may find it interesting: Using SQLite as an On-disk File Format, Part 2.
Sweet Persist is another one.
It is possible to serialize to and from streams in XML, JSON, Lua, and binary formats.
You can check the amef protocol, an example of C++ encoding in amef would be like,
//Create a new AMEF object
AMEFObject *object = new AMEFObject();
//Add a child string object
object->addPacket("This is the Automated Message Exchange Format Object property!!","adasd");
//Add a child integer object
object->addPacket(21213);
//Add a child boolean object
object->addPacket(true);
AMEFObject *object2 = new AMEFObject();
string j = "This is the property of a nested Automated Message Exchange Format Object";
object2->addPacket(j);
object2->addPacket(134123);
object2->addPacket(false);
//Add a child character object
object2->addPacket('d');
//Add a child AMEF Object
object->addPacket(object2);
//Encode the AMEF obejct
string str = new AMEFEncoder()->encode(object,false);
Decoding in java would be like,
string arr = amef encoded byte array value;
AMEFDecoder decoder = new AMEFDecoder()
AMEFObject object1 = AMEFDecoder.decode(arr,true);
The Protocol implementation has codecs for both C++ and Java, the interesting part is it can retain object class representation in the form of name value pairs, I required a similar protocol in my last project, when i incidentally stumbled upon this protocol, i had actually modified the base library according to my requirements. Hope this helps you.
I recommend Google protocol buffers. I had the chance to test drive the library on a new project and it's remarkably easy to use. The library is heavily optimized for performance.
Protobuf is different than other serialization solutions mentioned here in the sense that it does not serialize your objects, but rather generates code for objects that are serialization according to your specification.
I'm using the following template to implement serialization:
template <class T, class Mode = void> struct Serializer
{
template <class OutputCharIterator>
static void serializeImpl(const T &object, OutputCharIterator &&it)
{
object.template serializeThis<Mode>(it);
}
template <class InputCharIterator>
static T deserializeImpl(InputCharIterator &&it, InputCharIterator &&end)
{
return T::template deserializeFrom<Mode>(it, end);
}
};
template <class Mode = void, class T, class OutputCharIterator>
void serialize(const T &object, OutputCharIterator &&it)
{
Serializer<T, Mode>::serializeImpl(object, it);
}
template <class T, class Mode = void, class InputCharIterator>
T deserialize(InputCharIterator &&it, InputCharIterator &&end)
{
return Serializer<T, Mode>::deserializeImpl(it, end);
}
template <class Mode = void, class T, class InputCharIterator>
void deserialize(T &result, InputCharIterator &&it, InputCharIterator &&end)
{
result = Serializer<T, Mode>::deserializeImpl(it, end);
}
Here T is the type you want to serialize Mode is a dummy type to differentiate between different kinds of serialization, eg. the same integer can be serialized as little endian, big endian, varint, etc.
By default the Serializer delegates the task to the object being serialized. For built in types you should make a template specialization of the Serializer.
Convenience function templates are also provided.
For example little endian serialization of unsigned integers:
struct LittleEndianMode
{
};
template <class T>
struct Serializer<
T, std::enable_if_t<std::is_unsigned<T>::value, LittleEndianMode>>
{
template <class InputCharIterator>
static T deserializeImpl(InputCharIterator &&it, InputCharIterator &&end)
{
T res = 0;
for (size_t i = 0; i < sizeof(T); i++)
{
if (it == end) break;
res |= static_cast<T>(*it) << (CHAR_BIT * i);
it++;
}
return res;
}
template <class OutputCharIterator>
static void serializeImpl(T number, OutputCharIterator &&it)
{
for (size_t i = 0; i < sizeof(T); i++)
{
*it = (number >> (CHAR_BIT * i)) & 0xFF;
it++;
}
}
};
Then to serialize:
std::vector<char> serialized;
uint32_t val = 42;
serialize<LittleEndianMode>(val, std::back_inserter(serialized));
To deserialize:
uint32_t val;
deserialize(val, serialized.begin(), serialized.end());
Due to the abstract iterator logic, it should work with any iterator (eg. stream iterators), pointer, etc.
I realize this is an old post but it's one of the first that comes up when searching for c++ serialization.
I encourage anyone who has access to C++11 to take a look at cereal, a C++11 header only library for serialization that supports binary, JSON, and XML out of the box. cereal was designed to be easy to extend and use and has a similar syntax to Boost.
Here is a simple serializer library I knocked up. It's header only, c11 and has examples for serializing basic types. Here's one for a map to class.
https://github.com/goblinhack/simple-c-plus-plus-serializer
#include "c_plus_plus_serializer.h"
class Custom {
public:
int a;
std::string b;
std::vector c;
friend std::ostream& operator<<(std::ostream &out,
Bits my)
{
out << bits(my.t.a) << bits(my.t.b) << bits(my.t.c);
return (out);
}
friend std::istream& operator>>(std::istream &in,
Bits my)
{
in >> bits(my.t.a) >> bits(my.t.b) >> bits(my.t.c);
return (in);
}
friend std::ostream& operator<<(std::ostream &out,
class Custom &my)
{
out << "a:" << my.a << " b:" << my.b;
out << " c:[" << my.c.size() << " elems]:";
for (auto v : my.c) {
out << v << " ";
}
out << std::endl;
return (out);
}
};
static void save_map_key_string_value_custom (const std::string filename)
{
std::cout << "save to " << filename << std::endl;
std::ofstream out(filename, std::ios::binary );
std::map< std::string, class Custom > m;
auto c1 = Custom();
c1.a = 1;
c1.b = "hello";
std::initializer_list L1 = {"vec-elem1", "vec-elem2"};
std::vector l1(L1);
c1.c = l1;
auto c2 = Custom();
c2.a = 2;
c2.b = "there";
std::initializer_list L2 = {"vec-elem3", "vec-elem4"};
std::vector l2(L2);
c2.c = l2;
m.insert(std::make_pair(std::string("key1"), c1));
m.insert(std::make_pair(std::string("key2"), c2));
out << bits(m);
}
static void load_map_key_string_value_custom (const std::string filename)
{
std::cout << "read from " << filename << std::endl;
std::ifstream in(filename);
std::map< std::string, class Custom > m;
in >> bits(m);
std::cout << std::endl;
std::cout << "m = " << m.size() << " list-elems { " << std::endl;
for (auto i : m) {
std::cout << " [" << i.first << "] = " << i.second;
}
std::cout << "}" << std::endl;
}
void map_custom_class_example (void)
{
std::cout << "map key string, value class" << std::endl;
std::cout << "============================" << std::endl;
save_map_key_string_value_custom(std::string("map_of_custom_class.bin"));
load_map_key_string_value_custom(std::string("map_of_custom_class.bin"));
std::cout << std::endl;
}
Output:
map key string, value class
============================
save to map_of_custom_class.bin
read from map_of_custom_class.bin
m = 2 list-elems {
[key1] = a:1 b:hello c:[2 elems]:vec-elem1 vec-elem2
[key2] = a:2 b:there c:[2 elems]:vec-elem3 vec-elem4
}
Sweet Persist is another one.
It is possible to serialize to and from streams in XML, JSON, Lua, and binary formats.
I'm using the following template to implement serialization:
template <class T, class Mode = void> struct Serializer
{
template <class OutputCharIterator>
static void serializeImpl(const T &object, OutputCharIterator &&it)
{
object.template serializeThis<Mode>(it);
}
template <class InputCharIterator>
static T deserializeImpl(InputCharIterator &&it, InputCharIterator &&end)
{
return T::template deserializeFrom<Mode>(it, end);
}
};
template <class Mode = void, class T, class OutputCharIterator>
void serialize(const T &object, OutputCharIterator &&it)
{
Serializer<T, Mode>::serializeImpl(object, it);
}
template <class T, class Mode = void, class InputCharIterator>
T deserialize(InputCharIterator &&it, InputCharIterator &&end)
{
return Serializer<T, Mode>::deserializeImpl(it, end);
}
template <class Mode = void, class T, class InputCharIterator>
void deserialize(T &result, InputCharIterator &&it, InputCharIterator &&end)
{
result = Serializer<T, Mode>::deserializeImpl(it, end);
}
Here T is the type you want to serialize Mode is a dummy type to differentiate between different kinds of serialization, eg. the same integer can be serialized as little endian, big endian, varint, etc.
By default the Serializer delegates the task to the object being serialized. For built in types you should make a template specialization of the Serializer.
Convenience function templates are also provided.
For example little endian serialization of unsigned integers:
struct LittleEndianMode
{
};
template <class T>
struct Serializer<
T, std::enable_if_t<std::is_unsigned<T>::value, LittleEndianMode>>
{
template <class InputCharIterator>
static T deserializeImpl(InputCharIterator &&it, InputCharIterator &&end)
{
T res = 0;
for (size_t i = 0; i < sizeof(T); i++)
{
if (it == end) break;
res |= static_cast<T>(*it) << (CHAR_BIT * i);
it++;
}
return res;
}
template <class OutputCharIterator>
static void serializeImpl(T number, OutputCharIterator &&it)
{
for (size_t i = 0; i < sizeof(T); i++)
{
*it = (number >> (CHAR_BIT * i)) & 0xFF;
it++;
}
}
};
Then to serialize:
std::vector<char> serialized;
uint32_t val = 42;
serialize<LittleEndianMode>(val, std::back_inserter(serialized));
To deserialize:
uint32_t val;
deserialize(val, serialized.begin(), serialized.end());
Due to the abstract iterator logic, it should work with any iterator (eg. stream iterators), pointer, etc.
I suggest looking into Abstract factories which is often used as a basis for serialization
I have answered in another SO question about C++ factories. Please see there if a flexible factory is of interest. I try to describe an old way from ET++ to use macros which has worked great for me.
ET++ was a project to port old MacApp to C++ and X11. In the effort of it Eric Gamma etc started to think about Design Patterns. ET++ contained automatic ways for serialization and introspection at runtime.
As far as "built-in" libraries go, the << and >> have been reserved specifically for serialization.
You should override << to output your object to some serialization context (usually an iostream) and >> to read data back from that context. Each object is responsible for outputting its aggregated child objects.
This method works fine so long as your object graph contains no cycles.
If it does, then you will have to use a library to deal with those cycles.
I recommend using boost serialization as described by other posters. Here is a good detailed tutorial on how to use it which complements the boost tutorials nicely: http://www.ocoudert.com/blog/2011/07/09/a-practical-guide-to-c-serialization/
If you want simple and best performance and don't care about backward data compatibility, try HPS, it's lightweight, much faster than Boost, etc, and much easier to use than Protobuf, etc.
Example:
std::vector<int> data({22, 333, -4444});
std::string serialized = hps::serialize_to_string(data);
auto parsed = hps::parse_from_string<std::vector<int>>(serialized);
I recommend Google protocol buffers. I had the chance to test drive the library on a new project and it's remarkably easy to use. The library is heavily optimized for performance.
Protobuf is different than other serialization solutions mentioned here in the sense that it does not serialize your objects, but rather generates code for objects that are serialization according to your specification.
I recommend using boost serialization as described by other posters. Here is a good detailed tutorial on how to use it which complements the boost tutorials nicely: http://www.ocoudert.com/blog/2011/07/09/a-practical-guide-to-c-serialization/
As far as "built-in" libraries go, the << and >> have been reserved specifically for serialization.
You should override << to output your object to some serialization context (usually an iostream) and >> to read data back from that context. Each object is responsible for outputting its aggregated child objects.
This method works fine so long as your object graph contains no cycles.
If it does, then you will have to use a library to deal with those cycles.
Boost::serialization is a great option, but I've encountered a new project: Cereal which I find much more elegant! I highly suggest investigating it.
As far as "built-in" libraries go, the << and >> have been reserved specifically for serialization.
You should override << to output your object to some serialization context (usually an iostream) and >> to read data back from that context. Each object is responsible for outputting its aggregated child objects.
This method works fine so long as your object graph contains no cycles.
If it does, then you will have to use a library to deal with those cycles.
I recommend Google protocol buffers. I had the chance to test drive the library on a new project and it's remarkably easy to use. The library is heavily optimized for performance.
Protobuf is different than other serialization solutions mentioned here in the sense that it does not serialize your objects, but rather generates code for objects that are serialization according to your specification.
Boost is a good suggestion. But if you would like to roll your own, it's not so hard.
Basically you just need a way to build up a graph of objects and then output them to some structured storage format (JSON, XML, YAML, whatever). Building up the graph is as simple as utilizing a marking recursive decent object algorithm and then outputting all the marked objects.
I wrote an article describing a rudimentary (but still powerful) serialization system. You may find it interesting: Using SQLite as an On-disk File Format, Part 2.
Boost::serialization is a great option, but I've encountered a new project: Cereal which I find much more elegant! I highly suggest investigating it.
I realize this is an old post but it's one of the first that comes up when searching for c++ serialization.
I encourage anyone who has access to C++11 to take a look at cereal, a C++11 header only library for serialization that supports binary, JSON, and XML out of the box. cereal was designed to be easy to extend and use and has a similar syntax to Boost.
I suggest looking into Abstract factories which is often used as a basis for serialization
I have answered in another SO question about C++ factories. Please see there if a flexible factory is of interest. I try to describe an old way from ET++ to use macros which has worked great for me.
ET++ was a project to port old MacApp to C++ and X11. In the effort of it Eric Gamma etc started to think about Design Patterns. ET++ contained automatic ways for serialization and introspection at runtime.
Here is a simple serializer library I knocked up. It's header only, c11 and has examples for serializing basic types. Here's one for a map to class.
https://github.com/goblinhack/simple-c-plus-plus-serializer
#include "c_plus_plus_serializer.h"
class Custom {
public:
int a;
std::string b;
std::vector c;
friend std::ostream& operator<<(std::ostream &out,
Bits my)
{
out << bits(my.t.a) << bits(my.t.b) << bits(my.t.c);
return (out);
}
friend std::istream& operator>>(std::istream &in,
Bits my)
{
in >> bits(my.t.a) >> bits(my.t.b) >> bits(my.t.c);
return (in);
}
friend std::ostream& operator<<(std::ostream &out,
class Custom &my)
{
out << "a:" << my.a << " b:" << my.b;
out << " c:[" << my.c.size() << " elems]:";
for (auto v : my.c) {
out << v << " ";
}
out << std::endl;
return (out);
}
};
static void save_map_key_string_value_custom (const std::string filename)
{
std::cout << "save to " << filename << std::endl;
std::ofstream out(filename, std::ios::binary );
std::map< std::string, class Custom > m;
auto c1 = Custom();
c1.a = 1;
c1.b = "hello";
std::initializer_list L1 = {"vec-elem1", "vec-elem2"};
std::vector l1(L1);
c1.c = l1;
auto c2 = Custom();
c2.a = 2;
c2.b = "there";
std::initializer_list L2 = {"vec-elem3", "vec-elem4"};
std::vector l2(L2);
c2.c = l2;
m.insert(std::make_pair(std::string("key1"), c1));
m.insert(std::make_pair(std::string("key2"), c2));
out << bits(m);
}
static void load_map_key_string_value_custom (const std::string filename)
{
std::cout << "read from " << filename << std::endl;
std::ifstream in(filename);
std::map< std::string, class Custom > m;
in >> bits(m);
std::cout << std::endl;
std::cout << "m = " << m.size() << " list-elems { " << std::endl;
for (auto i : m) {
std::cout << " [" << i.first << "] = " << i.second;
}
std::cout << "}" << std::endl;
}
void map_custom_class_example (void)
{
std::cout << "map key string, value class" << std::endl;
std::cout << "============================" << std::endl;
save_map_key_string_value_custom(std::string("map_of_custom_class.bin"));
load_map_key_string_value_custom(std::string("map_of_custom_class.bin"));
std::cout << std::endl;
}
Output:
map key string, value class
============================
save to map_of_custom_class.bin
read from map_of_custom_class.bin
m = 2 list-elems {
[key1] = a:1 b:hello c:[2 elems]:vec-elem1 vec-elem2
[key2] = a:2 b:there c:[2 elems]:vec-elem3 vec-elem4
}
Source: Stackoverflow.com