By yjzhang


2015-08-04 05:23:26 8 Comments

I'm recently using gRPC with proto3, and I've noticed that required and optional has been removed in new syntax.

Would anyone kindly explain why required/optional are removed in proto3? Such kind of constraints just seem necessary to make definition robust.

syntax proto2:

message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
  optional int32 result_per_page = 3;
}

syntax proto3:

syntax = "proto3";
message SearchRequest {
  string query = 1;
  int32 page_number = 2;
  int32 result_per_page = 3;
}

2 comments

@maiyang 2018-09-17 02:56:05

You can find the explanation in this protobuf Github issue:

We dropped required fields in proto3 because required fields are generally considered harmful and violating protobuf's compatibility semantics. The whole idea of using protobuf is that it allows you to add/remove fields from your protocol definition while still being fully forward/backward compatible with newer/older binaries. Required fields break this though. You can never safely add a required field to a .proto definition, nor can you safely remove an existing required field because both of these actions break wire compatibility. For example, if you add a required field to a .proto definition, binaries built with the new definition won't be able to parse data serialized using the old definition because the required field is not present in old data. In a complex system where .proto definitions are shared widely across many different components of the system, adding/removing required fields could easily bring down multiple parts of the system. We have seen production issues caused by this multiple times and it's pretty much banned everywhere inside Google for anyone to add/remove required fields. For this reason we completely removed required fields in proto3.

After the removal of "required", "optional" is just redundant so we removed "optional" as well.

@Shmuel H. 2019-01-03 07:07:00

I don't get it; what's the difference between dropping a message after deserializing and on deserialization? it will get dropped by the older client since it does not contains a field that is needed (e.g. id).

@Cruncher 2019-07-03 15:27:05

I'm inclined to agree with @ShmuelH. required fields are going to be a part of an api one way or another. Well that's supported automatically through syntax given to both parties, or hidden in the backend, it's still there. May as well make it visible in the api definition

@manikawnth 2019-07-19 14:40:02

The problem is you're confusing gprc with protobuf. required fields are going to be a part of an api one way or another makes sense only for gprc api and not for protobuf in general. API messaging is not the only goal for Ser-Des.

@patrickbarker 2019-09-06 00:41:48

I totally agree with @ShmuelH. fields are required in an API one way or another and its useful for the customer to know this. This makes me think we just haven't gotten versioning right yet.

@Timmmm 2019-12-04 11:05:28

Another vote for @ShmuelH. If you change your API in a backwards-incompatible way (adding a required field), then surely you want your parser to detect that? Version your APIs! You can even do it completely in Protobuf if you want, using oneof { MessageV1, MessageV2, etc. }.

@Eric Anderson 2015-08-04 16:49:45

The usefulness of required has been at the heart of many a debate and flame war. Large camps have existed on both sides. One camp liked guaranteeing a value was present and was willing to live with its limitations but the other camp felt required dangerous or unhelpful as it can't be safely added nor removed.

Let me explain more of the reasoning why required fields should be used sparingly. If you are already using a proto, you can't add a required field because old application's won't be providing that field and applications in general don't handle the failure well. You can make sure that all old applications are upgraded first, but it can be easy to make a mistake and it doesn't help if you are storing the protos in any datastore (even short-lived, like memcached). The same sort of situation applies when removing a required field.

Many required fields were "obviously" required until... they weren't. Let's say you have an id field for a Get method. That is obviously required. Except, later you might need to change the id from int to string, or int32 to int64. That requires adding a new muchBetterId field, and now you are left with the old id field that must be specified, but eventually is completely ignored.

When those two problems are combined, the number of beneficial required fields becomes limited and the camps argue over whether it still has value. The opponents of required weren't necessarily against the idea, but its current form. Some suggested developing a more expressive validation library that could check required along with something more advanced like name.length > 10, while also making sure to have a better failure model.

Proto3 overall seems to favor simplicity, and required removal is simpler. But maybe more convincing, removing required made sense for proto3 when combined with other features, like removal of field presence for primitives and removal of overriding default values.

I'm not a protobuf developer and am in no way authoritative on the subject, but I still hope that the explanation is useful.

@Kenton Varda 2015-08-06 02:25:51

Yep. See also this extended explanation of things that can go horribly wrong with required fields: capnproto.org/…

@YavgenyP 2017-01-10 11:46:05

I dont know if using 'required' can cause any trouble, but removal of 'optional' and 'has_field' most certainly does. Even at the simplest case of a bool which has a meaning for both its possible values, this forces you to re-invent the wheel or to add more params to your api, instead of just relying on the infrastructure layer

@Eric Anderson 2017-01-11 00:31:02

Optional isn't removed; everything is optional in proto3. But yes, field visibility (has_field) has been removed for primitives. If you need field visibility use wrappers.proto which has messages like StringValue. Since they are messages, has_field is available. This is effectively "boxing" which is common in many languages.

@Vagrant 2017-01-24 01:12:35

On the contrary, it seems like "optional" was removed in proto3. Every field exists, and is filled in with a default value. You have no way of knowing if the primitive field was filled in by the user, or by default. Message fields, which are basically pointers, are optional, in that they can have a null value.

@Eric Anderson 2017-01-31 22:31:57

Proto2 would default to 0, if not overridden, for primitives when calling the getter. This behaves the same in proto2 and proto3 (in most languages). The optional keyword was removed in proto3, but it behaves like optional in proto2.

@Ivan Vučica 2017-03-28 10:56:59

@Vagrant "Every field exists" -- but consider: does it exist when serialized?

@Fabian Hertwig 2017-05-02 13:43:00

int32, uint32, int64, uint64, and bool are all compatible – this means you can change a field from one of these types to another without breaking forwards- or backwards-compatibility. If a number is parsed from the wire which doesn't fit in the corresponding type, you will get the same effect as if you had cast the number to that type in C++ (e.g. if a 64-bit number is read as an int32, it will be truncated to 32 bits). developers.google.com/protocol-buffers/docs/proto#updating

@Eric Anderson 2017-05-26 16:49:53

@FabianHertwig, int32, uint32, int64, uint64, and bool are wire-compatible, but they aren't necessarily API-compatible. Changing the .proto may cause applications to fail to compile or break at runtime. Thus, unless you have tight control on all the usages and there are few of them, it may be better to introduce a new field instead of changing an existing one.

@Goswin von Brederlow 2017-07-18 13:05:49

I would love to mark a field as required when creating a new message but not when parsing a string. Say I add a new field then I want to make sure all the code actually sets that new field. But messages from old code or storage can get the default value.

@the0ther 2018-02-06 19:53:16

i feel like protobuf is a language designed expressly to start flame wars

@Holoceo 2018-11-09 08:05:24

Seems like most people don't want to version their API's. It's easier for them to make everything optional for "backward compatibility".

@puppylpg 2019-11-24 06:16:18

Yeah, it seems that the underlying protocol shouldn't have these validation, which should happen on server side.

Related Questions

Sponsored Content

7 Answered Questions

[SOLVED] How to define an optional field in protobuf 3

13 Answered Questions

[SOLVED] Installing Google Protocol Buffers on mac

1 Answered Questions

[SOLVED] Integrating gorm.Model fields into protobuf definitions

1 Answered Questions

[SOLVED] Intialise protocol buffers in single expression

14 Answered Questions

[SOLVED] Biggest differences of Thrift vs Protocol Buffers?

1 Answered Questions

[SOLVED] Confusion creating swagger.json file from proto file

1 Answered Questions

3 Answered Questions

1 Answered Questions

[SOLVED] why proto buffer no date data type?

Sponsored Content