[java] Java: JSON -> Protobuf & back conversion

I have an existing system, which is using protobuf-based communication protocol between GUI and server. Now I would like to add some persistence, but at the moment protobuf messages are straight converted to a third-party custom objects.

Is there a way to convert proto messages to json, which could be then persisted to database.

N.B.: I don't much like an idea of writing binary protobuf to database, because it can one day become not backward-compatible with newer versions and break the system that way.

This question is related to java json serialization persistence protocol-buffers

The answer is


Here is my utility class, you may use:

package <removed>;
import com.google.protobuf.Message;
import com.google.protobuf.MessageOrBuilder;
import com.google.protobuf.util.JsonFormat;
/**
 * Author @espresso stackoverflow.
 * Sample use:
 *      Model.Person reqObj = ProtoUtil.toProto(reqJson, Model.Person.getDefaultInstance());
        Model.Person res = personSvc.update(reqObj);
        final String resJson = ProtoUtil.toJson(res);
 **/
public class ProtoUtil {
    public static <T extends Message> String toJson(T obj){
        try{
            return JsonFormat.printer().print(obj);
        }catch(Exception e){
            throw new RuntimeException("Error converting Proto to json", e);
        }
    }
   public static <T extends MessageOrBuilder> T toProto(String protoJsonStr, T message){
        try{
            Message.Builder builder = message.getDefaultInstanceForType().toBuilder();
            JsonFormat.parser().ignoringUnknownFields().merge(protoJsonStr,builder);
            T out = (T) builder.build();
            return out;
        }catch(Exception e){
            throw new RuntimeException(("Error converting Json to proto", e);
        }
    }
}

Try JsonFormat.printer().print(MessageOrBuilder), it looks good for proto3. Yet, it is unclear how to convert the actual protobuf message (which is provided as the java package of my choice defined in the .proto file) to a com.google.protbuf.Message object.


Adding to Ophirs answer, JsonFormat is available even before protobuf 3.0. However, the way to do it differs a little bit.

In Protobuf 3.0+, the JsonFormat class is a singleton and therefore do something like the below

String jsonString = "";
JsonFormat.parser().ignoringUnknownFields().merge(jsonString,yourObjectBuilder);

In Protobuf 2.5+, the below should work

String jsonString = "";
JsonFormat jsonFormat = new JsonFormat();
jsonString = jsonFormat.printToString(yourProtobufMessage);

Here is a link to a tutorial I wrote that uses the JsonFormat class in a TypeAdapter that can be registered to a GsonBuilder object. You can then use Gson's toJson and fromJson methods to convert the proto data to Java and back.

Replying to jean . If we have the protobuf data in a file and want to parse it into a protobuf message object, use the merge method TextFormat class. See the below snippet:

// Let your proto text data be in a file MessageDataAsProto.prototxt
// Read it into string  
String protoDataAsString = FileUtils.readFileToString(new File("MessageDataAsProto.prototxt"));

// Create an object of the message builder
MyMessage.Builder myMsgBuilder = MyMessage.newBuilder();

// Use text format to parse the data into the message builder
TextFormat.merge(protoDataAsString, ExtensionRegistry.getEmptyRegistry(), myMsgBuilder);

// Build the message and return
return myMsgBuilder.build();

As mentioned in an answer to a similar question, since v3.1.0 this is a supported feature of ProtocolBuffers. For Java, include the extension module com.google.protobuf:protobuf-java-util and use JsonFormat like so:

JsonFormat.parser().ignoringUnknownFields().merge(json, yourObjectBuilder);
YourObject value = yourObjectBuilder.build();

For protobuf 2.5, use the dependency:

"com.googlecode.protobuf-java-format" % "protobuf-java-format" % "1.2"

Then use the code:

com.googlecode.protobuf.format.JsonFormat.merge(json, builder)
com.googlecode.protobuf.format.JsonFormat.printToString(proto)

Well, there is no shortcut to do it as per my findings, but somehow you
an achieve it in few simple steps

First you have to declare a bean of type 'ProtobufJsonFormatHttpMessageConverter'

@Bean  
@Primary  
public ProtobufJsonFormatHttpMessageConverter protobufHttpMessageConverter() {  
  return new ProtobufJsonFormatHttpMessageConverter(JsonFormat.parser(), JsonFormat.printer());  
}  

Then you can just write an Utility class like ResponseBuilder, because it can parse the request by default but without these changes it can not produce Json response. and then you can write few methods to convert the response types to its related object type.

public static <T> T build(Message message, Class<T> type) {
  Printer printer = JsonFormat.printer();
  Gson gson = new Gson();
  try {
    return gson.fromJson(printer.print(message), type);
  } catch (JsonSyntaxException | InvalidProtocolBufferException e) {
    throw new ApiException(HttpStatus.INTERNAL_SERVER_ERROR, "Response   conversion Error", e);
  }
}

Then you can call this method from your controller class as last line like -

return ResponseBuilder.build(<returned_service_object>, <Type>);

Hope this will help you to implement protobuf in json format.


I don't much like an idea of writing binary protobuf to database, because it can one day become not backward-compatible with newer versions and break the system that way.

Converting protobuf to JSON for storage and then back to protobuf on load is much more likely to create compatibility problems, because:

  • If the process which performs the conversion is not built with the latest version of the protobuf schema, then converting will silently drop any fields that the process doesn't know about. This is true both of the storing and loading ends.
  • Even with the most recent schema available, JSON <-> Protobuf conversion may be lossy in the presence of imprecise floating-point values and similar corner cases.
  • Protobufs actually have (slightly) stronger backwards-compatibility guarantees than JSON. Like with JSON, if you add a new field, old clients will ignore it. Unlike with JSON, Protobufs allow declaring a default value, which can make it somewhat easier for new clients to deal with old data that is otherwise missing the field. This is only a slight advantage, but otherwise Protobuf and JSON have equivalent backwards-compatibility properties, therefore you are not gaining any backwards-compatibility advantages from storing in JSON.

With all that said, there are many libraries out there for converting protobufs to JSON, usually built on the Protobuf reflection interface (not to be confused with the Java reflection interface; Protobuf reflection is offered by the com.google.protobuf.Message interface).


Generics Solution

Here's a generic version of Json converter

package com.github.platform.util;

import java.io.IOException;
import java.lang.reflect.InvocationTargetException;
import com.google.protobuf.AbstractMessage.Builder;
import com.google.protobuf.Message;
import com.google.protobuf.MessageOrBuilder;
import com.google.protobuf.util.JsonFormat;

/**
 * Generic ProtoJsonUtil to be used to serialize and deserialize Proto to json
 * 
 * @author [email protected]
 *
 */
public final class ProtoJsonUtil {

  /**
   * Makes a Json from a given message or builder
   * 
   * @param messageOrBuilder is the instance
   * @return The string representation
   * @throws IOException if any error occurs
   */
  public static String toJson(MessageOrBuilder messageOrBuilder) throws IOException {
    return JsonFormat.printer().print(messageOrBuilder);
  }

  /**
   * Makes a new instance of message based on the json and the class
   * @param <T> is the class type
   * @param json is the json instance
   * @param clazz is the class instance
   * @return An instance of T based on the json values
   * @throws IOException if any error occurs
   */
  @SuppressWarnings({"unchecked", "rawtypes"})
  public static <T extends Message> T fromJson(String json, Class<T> clazz) throws IOException {
    // https://stackoverflow.com/questions/27642021/calling-parsefrom-method-for-generic-protobuffer-class-in-java/33701202#33701202
    Builder builder = null;
    try {
      // Since we are dealing with a Message type, we can call newBuilder()
      builder = (Builder) clazz.getMethod("newBuilder").invoke(null);

    } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException
        | NoSuchMethodException | SecurityException e) {
      return null;
    }

    // The instance is placed into the builder values
    JsonFormat.parser().ignoringUnknownFields().merge(json, builder);

    // the instance will be from the build
    return (T) builder.build();
  }
}

Using it is as simple as follows:

Message instance

GetAllGreetings.Builder allGreetingsBuilder = GetAllGreetings.newBuilder();

allGreetingsBuilder.addGreeting(makeNewGreeting("Marcello", "Hi %s, how are you", Language.EN))
        .addGreeting(makeNewGreeting("John", "Today is hot, %s, get some ice", Language.ES))
        .addGreeting(makeNewGreeting("Mary", "%s, summer is here! Let's go surfing!", Language.PT));

GetAllGreetings allGreetings = allGreetingsBuilder.build();

To Json Generic

String json = ProtoJsonUtil.toJson(allGreetingsLoaded);
log.info("Json format: " + json);

From Json Generic

GetAllGreetings parsed = ProtoJsonUtil.fromJson(json, GetAllGreetings.class);
log.info("The Proto deserialized from Json " + parsed);

We are currently using protobuf-java-format to convert our Protobuf messages (anything subclass of Message) into a JSON format to send over our web API.

Simply do:

  JsonFormat.printToString(protoMessage)

Examples related to java

Under what circumstances can I call findViewById with an Options Menu / Action Bar item? How much should a function trust another function How to implement a simple scenario the OO way Two constructors How do I get some variable from another class in Java? this in equals method How to split a string in two and store it in a field How to do perspective fixing? String index out of range: 4 My eclipse won't open, i download the bundle pack it keeps saying error log

Examples related to json

Use NSInteger as array index Uncaught SyntaxError: Unexpected end of JSON input at JSON.parse (<anonymous>) HTTP POST with Json on Body - Flutter/Dart Importing json file in TypeScript json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 190) Angular 5 Service to read local .json file How to import JSON File into a TypeScript file? Use Async/Await with Axios in React.js Uncaught SyntaxError: Unexpected token u in JSON at position 0 how to remove json object key and value.?

Examples related to serialization

laravel Unable to prepare route ... for serialization. Uses Closure TypeError: Object of type 'bytes' is not JSON serializable Best way to save a trained model in PyTorch? Convert Dictionary to JSON in Swift Java: JSON -> Protobuf & back conversion Understanding passport serialize deserialize How to generate serial version UID in Intellij Parcelable encountered IOException writing serializable object getactivity() Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects Cannot deserialize the JSON array (e.g. [1,2,3]) into type ' ' because type requires JSON object (e.g. {"name":"value"}) to deserialize correctly

Examples related to persistence

Java: JSON -> Protobuf & back conversion What is Persistence Context? Name attribute in @Entity and @Table The import javax.persistence cannot be resolved Confusion: @NotNull vs. @Column(nullable = false) with JPA and Hibernate Correct use of flush() in JPA/Hibernate When to use EntityManager.find() vs EntityManager.getReference() with JPA No Persistence provider for EntityManager named Setting a JPA timestamp column to be generated by the database? Hibernate: "Field 'id' doesn't have a default value"

Examples related to protocol-buffers

How to define an optional field in protobuf 3 ImportError: No module named google.protobuf Java: JSON -> Protobuf & back conversion Installing Google Protocol Buffers on mac Biggest differences of Thrift vs Protocol Buffers?