Switching My gRPC Project to Use Buf
Updated: Thursday, December 16, 2021
Do you work with Protobufs and gRPC in your projects or at work? Have you ever been frustrated with the tooling, boilerplate setup, or wondered if you were doing things the right way ? Same. Good news! There is a better way.
In this post I’ll go over some of my frustrations with the Protobuf and gRPC ecosystems and how I switched Flipt over to use Buf and why you should too. At the end of this post you will have a better understanding of how Buf can help you better organize and work with your Protobufs and save you time generating code.
note: I’m not affiliated with the Buf team. I just like the tooling.
If you’re not familiar with Protobufs or gRPC, I’d first recommend reading up on them and the problems they intend to solve before going further. There’s also an architecture page on how/why I use gRPC and gRPC Gateway in Flipt for more info on my specific use case.
After working with Protobuf and gRPC over the years, I’ve come to dislike the whole process of generating code from these
.proto files whenever a change is made to an existing API or a new API is added.
Some of my gripes include:
The actual commands to invoke
protocand the multitude of options (with varying degrees of documentation) have always seemed archaic to me.
protoc -I ./proto \ --go_out ./proto --go_opt paths=source_relative \ --go-grpc_out ./proto --go-grpc_opt paths=source_relative \ --grpc-gateway_out ./proto --grpc-gateway_opt paths=source_relative \ ./proto/helloworld/hello_world.proto
Who designed this CLI? Even better, each
protoc-*-plugincan have a different set of options, so if you want to generate clients in a multitude of languages, you’ll have to (usually) do a lot of trial and error to find the correct commands to execute.
You have to know about the rules of adding/updating/deprecating fields in your
.protofiles because they aren’t enforced until you generate the code and try to use it. Nothing like getting to your pre-production or even production environments and finding out your new field is not backward compatible with the data from the server because someone didn’t use
reserved. These types of breaking changes are also hard to spot in code-review unless you know what you are looking for.
Finally, depending on any other
.protofile from your organization (such as a common library or set of domain types) means that you had to physically copy or clone that file into your own project or tell
protocwhere to find it on your machine.
While this seems obvious that this would be the case, it’s a huge hassle, especially when working in a large organization across many different repositories with many developers to require everyone to copy code locally on their machine and run a verbose encantation each time they want to generate code.
And yes I’ve tried Bazel and don’t even get me started…
Level Up With Go
Thanks for reading! While you're here, sign up to receive a free sample chapter of my upcoming guide: Level Up With Go.
No spam. I promise.
Buf to the Rescue
I recently stumbled upon Buf and I’ll admit that I was skeptical at first, however, after using it for about a week I can say that I would recommend it to anyone who works with Protobufs in their projects.
But what is Buf? Per their documentation:
Buf is building tooling to make Protobuf reliable and easy to use for service owners and clients, while keeping it the obvious choice on the technical merits.
buf CLI is a tool that you install locally on your machine which fixes most of my gripes with the existing proto tools. It includes commands for:
- Linting your
.protofiles to ensure they match your organization or personal as well as industry standards.
- Detecting breaking changes against a base branch (i.e.
main) via a set of rules that enforce backward compatibility.
- Code generation that invokes
protocbehind the scenes, but uses a sane configurable template (YAML file) that can be checked in and managed as code. This was the big one for me as it means that I no longer have to think about
You still need to have
protoc and the necessary plugins installed on your machine however, for example, if you want to generate Go GRPC clients you’d still need to first ensure you have Go installed as well as the
protoc-gen-go-grpc plugins via:
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
In my opinion, it would be great if
buf would install these plugins for you if they didn’t exist on your machine, depending on which language you are trying to generate code for, somewhat like how
bazel does.. just with less fuss.
buf installed and configured properly, I was able to simplify my Makefile for the Protobuf generation step from a mess of
protoc arguments to one command:
buf generate. Win.
I also found that my
.proto file failed many of the linting rules that
buf lint uses by default. In the meantime, I simply added those failing rules to be ignored so that I can fix them over time to ensure that I don’t break any existing client code.
My final complaint with the traditional Protobuf ecosystem was that depending on any shared
.protos required you to physically copy those files to your machine where
protoc could find them. In my case, I am generating openAPI v2 documentation for my REST API straight from my
.proto file with the use of the
Before Buf, this required me to keep a copy of two files options/annotations.proto and options/openapiv2.proto within my project in order to generate the docs successfully. Now, with the help of the Buf Schema Repository, I can delete those files locally and simply add the required dependencies in my
deps: - buf.build/googleapis/googleapis - buf.build/grpc-ecosystem/grpc-gateway
buf can get these files directly from the BSR instead of them having to exist on my machine. This will be a game-changer for organizations or large projects that have multiple shared proto dependences! Now you can simply publish a module that contains your
.proto files to the BSR and your other projects can depend on them directly if they are also using
buf. Best of all, these modules are versioned automatically, so different consumers can depend on different versions independently!
Simplifying Client Generation
Another part of my workflow that I was able to simplify with
buf was generating client code in other languages, namely Ruby. As Flipt exposes both a REST API (via gRPC gateway) and a gRPC API, I wanted to be able to generate gRPC clients in at least a couple of languages for consumers to use. I chose Ruby and Go since those are the languages I know the best. These two clients are published to separate repositories on GitHub, flipt-grpc-ruby and flipt-grpc-go respectively.
I had previously captured the client code generation in a script, which again was mainly just a bunch of
protoc-*-plugin options, however, it had the added step of installing the
grpc-tools RubyGem if it wasn’t already installed. Honestly, though, I disliked having to ensure I had an up-to-date version of Ruby and the
grpc-tools gem on my machine just to generate this Ruby client once in a while.
Thankfully, I was able to replace all of the above with a separate buf.public.gen.yaml configuration to generate these clients. Now all I have to do is run
buf generate --template=buf.public.gen.yaml to generate these clients, without having to script anything. The best part is I don’t even need to worry about ensuring that I have Ruby or the
grpc-tools gem installed locally, thanks to Buf’s experimental Remote Generation!
I wanted to see how this is done, so I fired up my favorite proxy and captured the output. Sure enough, it seems that the
buf client makes a request to
https://api.buf.build/ with the contents of my
.proto file and returns generated Ruby code that it then saves locally! Wild.
So as you can see I’ve pretty much become a Buf convert. I think it smooths most of the rough edges around working with Protobufs and gRPC and that it’s going to be the main way that developers interact with these ecosystems in the future. Buf has great documentation and an awesome tutorial that helped me get going without much headache.
I’m looking forward to seeing what the Buf team releases next and how it helps shape the adoption of Protobufs and gRPC in the future.
Have you or your team tried moving your projects to use Buf? How did it go? Have you run into any issues in the process? Let me know on Twitter.