In this part, we’ll explore the fundamentals of RPC, why gRPC is a game-changer for modern microservices, and how it leverages HTTP/2 and Protocol Buffers to deliver efficient, scalable, and secure communication.

This story is free for everyone: Free Link
Part 7: Link

Table of Contents

  • What is RPC?
  • Why Use RPC?
  • What Actually Happens Under the Hood?
  • Challenges in Traditional RPC
  • Enter gRPC
  • How gRPC Exploits HTTP/2
  • What are Protocol Buffers?
  • Bringing It Together: gRPC + HTTP/2 + Protobuf
  • Example: Creating a Greet Service
  • Streaming in gRPC
  • TLS Security and Authentication
  • gRPC Interceptors
  • gRPC Gateway for REST/JSON Compatibility
  • Till Next Time
  • References

Prerequisites

This is an advanced concept and requires some basic knowledge of the following

  • Go programming
  • Familiarity with networking concepts (e.g., TCP, HTTP)

What is RPC?

Remote Procedure Call (RPC) is a protocol or conceptual model that allows a program to execute a function or procedure located on another machine (or process) as if it were a local function call. Imagine you’re writing a simple function like this:

result := Add(2, 3)

Now imagine that the Add function actually lives on a different server across the network, possibly in another country. RPC allows you to invoke Add(2, 3) as if it were a normal function, while behind the scenes the call is sent across the network, executed remotely, and the result is sent back.

The core idea of RPC is to abstract the complexity of network communication. Instead of writing manual code to:

  • create a network socket
  • serialize data
  • send a request
  • listen for a response
  • deserialize the response

…you simply call a function, and the RPC system handles all those details transparently.

Why Use RPC?

The main reason developers use RPC is to enable distributed computing in an intuitive way.

In modern systems — especially microservices and IoT platforms — functionality is spread across multiple services or nodes. You might have:

  • A service for authentication
  • Another for processing data
  • Another for storing metrics

These services need to talk to each other. RPC allows them to invoke operations on each other without managing the low-level networking or serialization mechanics.

Unlike REST APIs which focus on resources and HTTP semantics, RPC focuses on actions — calling remote procedures. This makes it a natural fit for system-to-system communication.

So What Actually Happens Under the Hood?

Let’s now lift the hood and look at what really happens during an RPC call. When you write:

result := Add(2, 3)

In RPC, this would typically result in:

  1. Stub Generation: The Add function is replaced by a client stub, a piece of code generated by the RPC framework.
  2. Serialization: The arguments (2, 3) are serialized into a binary or JSON format.
  3. Network Transport: The serialized request is sent over a socket, typically TCP or HTTP.
  4. Server Handling: The remote server receives the request, deserializes it, invokes the real Add function, and gets a result.
  5. Serialization (again): The result is serialized.
  6. Response: It’s sent back to the client over the network.
  7. Deserialization: The client stub deserializes the result and gives it back as result.

Challenges in Traditional RPC

While the basic idea is powerful, traditional RPC implementations have faced many issues, especially in distributed and high-performance systems:

  • Tight coupling: Client and server must agree on interfaces, but often there’s no standard way to define these contracts.
  • Inflexible serialization: Many old RPC systems use verbose or inefficient formats like XML (e.g., SOAP).
  • Insecure transport: Early systems used custom TCP layers without built-in encryption.
  • No streaming: You can’t easily send/receive data continuously in one connection.
  • Poor error handling: It’s hard to standardize how timeouts, retries, and failures are reported.

Enter gRPC

To solve these problems, Google developed gRPC, which is an open-source RPC framework with the following key improvements:

  • Contract-first development using Protocol Buffers (.proto files)
  • Strongly typed code generated in many languages
  • HTTP/2-based transport for high performance and multiplexing
  • Support for streaming (client, server, and bidirectional)
  • Efficient binary serialization with Protocol Buffers
  • Built-in authentication, TLS support
  • Interoperability with multiple languages (Go, Java, Python, C++, etc.)

gRPC retains the familiar “call a function remotely” concept but wraps it in a modern, scalable, and efficient architecture.

How gRPC Exploits HTTP/2 (optional)

To understand why gRPC uses HTTP/2, you need to first understand the limitations of HTTP/1.1, which is what most traditional REST APIs are built on.

HTTP/1.1 allows only one request per TCP connection at a time. If you want to make multiple concurrent requests, the client must open multiple TCP connections. This leads to inefficiencies like head-of-line blocking, where a slow request blocks others behind it on the same connection. Also, HTTP/1.1 headers are textual and repetitive, adding unnecessary size to every request and response. These limitations were tolerable for web browsers but are significant bottlenecks in high-performance microservices and IoT communication.

Enter HTTP/2, a major evolution of the protocol that fundamentally changes how data is transmitted between client and server. Unlike HTTP/1.1, HTTP/2 uses a single multiplexed connection that can carry multiple independent streams of data concurrently. This means thatmany RPC calls can share the same connection without blocking each other. Each request/response pair is broken down into small frames, which are interleaved and then reassembled, enabling efficient use of bandwidth.

gRPC leverages these features natively and aggressively. Each gRPC method invocation becomes a separate HTTP/2 stream within the same TCP connection. This eliminates the need to repeatedly open and close sockets, reducing latency and increasing throughput. Moreover, because streams are independent, a slow RPC doesn’t stall others, enabling true concurrency over a single persistent connection.

In addition, HTTP/2 supports header compression using HPACK, which reduces the overhead caused by large and redundant headers. While REST APIs often include verbose headers like "Content-Type: application/json" and repeated auth tokens, gRPC compresses and minimizes such metadata efficiently.

Another powerful feature of HTTP/2, and one heavily exploited by gRPC, is bidirectional streaming. In HTTP/1.1, the client sends a request and then waits for the full response. But in HTTP/2, client and server can both send and receive data simultaneously, making long-lived, streaming connections possible. This is foundational to gRPC’s ability to support streaming RPCs, including:

  • Client streaming: the client sends a sequence of messages.
  • Server streaming: the server sends a sequence of messages.
  • Bidirectional streaming: both client and server send and receive streams in real time.

Because of HTTP/2, gRPC doesn’t need to fall back on hacks like WebSockets or polling to simulate real-time communication. It’s built in from the transport layer up.

Finally, HTTP/2 includes native support for TLS, which gRPC integrates tightly with. This means gRPC can secure its communications without requiring extra layers or protocol negotiation, making it suitable for production environments and zero-trust networks.

What are Protocol Buffers?

Now let’s explore Protocol Buffers (commonly called protobufs) — the data serialization format at the heart of gRPC. If HTTP/2 is the rocket engine that powers gRPC’s transport layer, Protocol Buffers are the fuel, providing compact, efficient, and strongly typed message encoding.

Protocol Buffers are a language-neutral, platform-neutral, extensible mechanism developed by Google to serialize structured data. Unlike JSON or XML, which are human-readable but bulky and slow to parse, Protocol Buffers are designed to be machine-efficient, offering small message sizes and blazing-fast serialization/deserialization speeds.

Here’s the fundamental idea: you define your data schema once using a .proto file, and from that, the protobuf compiler (protoc) generates code in your target language (Go, Python, Java, etc.). This generated code includes:

  • Struct-like message definitions
  • Accessor methods
  • Validation and default handling
  • Serialization/deserialization logic

Take the following .proto definition:

message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;
}

This simple definition declares a message type with three fields. What makes Protocol Buffers efficient is how these messages are encoded in binary form.

Each field in a protobuf message is encoded using a tag-value format. The number (= 1, = 2, etc.) is a unique field identifier. These numbers, not the field names, are used in the binary representation, which is part of what makes protobuf messages so compact.

When serialized, the above Person message might look like this in raw binary:

0a 07 4a 6f 68 6e 20 44 6f 65 10 01 1a 0f 6a 6f 68 6e 40 65 78 61 6d 70 6c 65 2e 63 6f 6d

This seems like gibberish, but in protobuf’s world, it’s elegant and efficient. Every byte has a purpose. The initial byte is a combination of the field number and the wire type (a compact encoding that indicates how the data should be parsed), followed by the actual data. Integers are encoded using a technique called varint, which uses fewer bytes for smaller numbers. Strings are length-prefixed, so the parser knows exactly how many bytes to read.

This binary format gives Protocol Buffers two enormous advantages over formats like JSON:

  1. Size: Protobuf messages are significantly smaller than equivalent JSON or XML documents. That matters a lot when you’re sending thousands or millions of messages per second in a distributed system or over low-bandwidth links like in IoT.
  2. Speed: Parsing protobuf data is much faster than parsing text formats because it’s closer to how computers operate — there’s no need to tokenize or look up field names. Just read the tag, interpret the type, grab the bytes, and move on.

There’s also versioning baked into the format. You can safely add new fields to a message and older clients will simply ignore them. You can even deprecate fields by reserving their tag numbers to avoid future reuse. This makes evolving an API over time much safer and more controlled compared to brittle JSON-based APIs.

Protocol Buffers also support nested messages, enums, default values, and even custom options. This makes them incredibly expressive while still being lean and performant.

Finally, the protobuf system is self-documenting. The .proto file acts as both a schema and a contract. Tools can introspect it, generate client/server code, generate documentation, and more. In a sense, it’s the perfect combination of IDL (Interface Definition Language) and data schema.

Bringing It Together: gRPC + HTTP/2 + Protobuf

When you use gRPC, you’re stacking these technologies to achieve an ideal RPC system:

  • HTTP/2 is the transport layer: multiplexed, fast, secure, and stream-capable.
  • Protobuf is the serialization format: compact, typed, and efficient.
  • gRPC itself is the abstraction that allows you to define services, auto-generate code, and communicate across languages with minimal boilerplate.

When you define a gRPC service:

  1. The system automatically:
  2. Generates client/server stubs in your chosen language.
  3. Uses Protocol Buffers to encode the HelloRequest and HelloReply into compact binary.
  4. Sends them over an HTTP/2 stream, with full streaming support and multiplexing.
  5. Receives, decodes, and handles the request on the server side, and sends the response back — efficiently, securely, and with full error-handling support.

The entire operation is efficient, observable, and scalable, which is why gRPC is rapidly becoming a core building block in modern cloud-native platforms.

Example: Creating a Greet service

Installations:

To get started, we need to first install protoc. protoc is the Protocol Buffers compiler provided by Google. It reads your .proto files and generates code for your target language (Go, in our case), including types and service interfaces.

Run# the following to install it:

# For MAC
brew install protobuf 

# For Linux (use WSL for Windows)
sudo apt install -y protobuf-compiler

protoc --version
# Output should be something like: libprotoc 3.21.12

You also need two plugins for Go:

  1. protoc-gen-go: Generates Go types from .proto files.
  2. protoc-gen-go-grpc: Generates Go gRPC service interfaces.

Install both with:

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Ensure these binaries are in your system $PATH, usually:

export PATH="$PATH:$(go env GOPATH)/bin"

Defining a gRPC Greet Service in .proto

Now let’s create the proto file for our greet service

Create a folder structure:

mkdir -p grpc-greet/greet
cd grpc-greet

Now create the file: greet/greet.proto

syntax = "proto3";

package greet;

option go_package = "grpc-greet/greet;greet";

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse);
}

message GreetRequest {
  string first_name = 1;
  string last_name = 2;
}

message GreetResponse {
  string result = 1;
}

Explanation:

  • syntax = "proto3": Uses the latest Protobuf syntax.
  • package greet: Declares the package name in the .proto world.
  • option go_package: Maps the proto package to a Go module and package path.
  • service GreetService: Declares an RPC service with a method Greet.
  • rpc Greet(...): The actual RPC method accepting a GreetRequest and returning a GreetResponse.

Generate Go Code from .proto

From the project root (where greet folder lives), run:

protoc --go_out=. --go-grpc_out=. greet/greet.proto

This will generate:

  • greet/greet.pb.go: Data structures and serialization logic.
  • greet/greet_grpc.pb.go: gRPC service interfaces.

We will use these in our main go code.

Implementing the gRPC Server

Create a file server/main.go

package main

import (
 "context"
 "fmt"
 "log"
 "net"

 "google.golang.org/grpc"
 "grpc-greet/greet" // adjust import path to your module name
)

type server struct {
 greet.UnimplementedGreetServiceServer
}

// Greet implements the unary RPC call
func (s *server) Greet(ctx context.Context, req *greet.GreetRequest) (*greet.GreetResponse, error) {
 first := req.GetFirstName()
 last := req.GetLastName()
 message := fmt.Sprintf("Hello, %s %s!", first, last)
 return &greet.GreetResponse{Result: message}, nil
}

func main() {
 listener, err := net.Listen("tcp", ":50051")
 if err != nil {
  log.Fatalf("Failed to listen: %v", err)
 }

 grpcServer := grpc.NewServer()
 greet.RegisterGreetServiceServer(grpcServer, &server{})

 log.Println("Server is running on port 50051...")
 if err := grpcServer.Serve(listener); err != nil {
  log.Fatalf("Failed to serve: %v", err)
 }
}
  • The server struct implements the generated gRPC interface.
  • The Greet method extracts fields from the request and returns a greeting string.
  • The main function starts a gRPC server on TCP port 50051.

Implementing the gRPC Client

Create a file client/main.go

package main

import (
 "context"
 "fmt"
 "log"
 "time"

 "google.golang.org/grpc"
 "grpc-greet/greet" // adjust to your module name
)

func main() {
 conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
 if err != nil {
  log.Fatalf("Could not connect: %v", err)
 }
 defer conn.Close()

 client := greet.NewGreetServiceClient(conn)

 ctx, cancel := context.WithTimeout(context.Background(), time.Second)
 defer cancel()

 req := &greet.GreetRequest{
  FirstName: "Ben",
  LastName:  "Meehan",
 }

 res, err := client.Greet(ctx, req)
 if err != nil {
  log.Fatalf("Error calling Greet: %v", err)
 }

 fmt.Println("Response from server:", res.GetResult())
}
  • Creates a client stub using NewGreetServiceClient.
  • Sends a GreetRequest to the server and prints the response.
  • Uses a context with a timeout (best practice).

Running It All

1. Start the Server:

go run server/main.go

You should see:

Server is running on port 50051...

2. Run the Client (in another terminal):

go run client/main.go

Expected Output:

Response from server: Hello, Ben Meehan

Streaming in gRPC

Unlike typical REST endpoints that follow a strict request-response model, gRPC supports streaming — allowing messages to be continuously sent or received over a single RPC call.

Server Streaming

In server streaming, the client sends a single request and receives a stream of responses.

Use Case:

  • A weather client asking for hourly forecasts.
  • A dashboard client that wants live logs or event streams.

.proto Example:

rpc StreamGreetings(GreetRequest) returns (stream GreetResponse);

Here, GreetRequest is sent once, and the server replies with multiple GreetResponse messages.

Server-side (conceptually):

for i := 0; i < 10; i++ {
  res := &greet.GreetResponse{Result: fmt.Sprintf("Hello #%d", i)}
  stream.Send(res)
  time.Sleep(time.Second)
}

Client-side:

stream, _ := client.StreamGreetings(ctx, req)
for {
  res, err := stream.Recv()
  if err == io.EOF {
    break
  }
  log.Println(res.GetResult())
}

Client Streaming

In client streaming, the client sends multiple messages, and the server replies once.

Use Case:

  • A client uploading chunks of a file.
  • Aggregating metrics on the server side.

.proto Example:

rpc UploadGreetings(stream GreetRequest) returns (GreetResponse);

Client-side:

stream := client.UploadGreetings(ctx)
for _, name := range names {
  stream.Send(&greet.GreetRequest{FirstName: name})
}
res, _ := stream.CloseAndRecv()

Server-side:

var names []string
for {
  req, err := stream.Recv()
  if err == io.EOF {
    result := strings.Join(names, ", ")
    return &greet.GreetResponse{Result: "Hello to: " + result}, nil
  }
  names = append(names, req.GetFirstName())
}

Bidirectional Streaming

Both client and server send a stream of messages independently — like a WebSocket connection.

Use Case:

  • Real-time chat.
  • Multiplayer gaming.
  • Live sensor data ingestion and feedback.

.proto Example:

rpc Chat(stream GreetRequest) returns (stream GreetResponse);

Server-side:

for {
  req, err := stream.Recv()
  if err == io.EOF {
    return nil
  }
  res := &greet.GreetResponse{Result: "Hello " + req.GetFirstName()}
  stream.Send(res)
}

Client-side:

stream, _ := client.Chat(ctx)
go func() {
  for _, name := range names {
    stream.Send(&greet.GreetRequest{FirstName: name})
  }
  stream.CloseSend()
}()

for {
  res, err := stream.Recv()
  if err == io.EOF {
    break
  }
  log.Println(res.GetResult())
}

TLS Security and Authentication

By default, gRPC is designed to use TLS (Transport Layer Security) for secure communication, especially because it rides on HTTP/2.

TLS with Self-signed Certs

You configure TLS at the server level:

creds, err := credentials.NewServerTLSFromFile("server.crt", "server.key")
grpcServer := grpc.NewServer(grpc.Creds(creds))

And the client must use:

creds, err := credentials.NewClientTLSFromFile("ca.crt", "")
conn, err := grpc.Dial("localhost:50051", grpc.WithTransportCredentials(creds))

Mutual TLS (mTLS)

In mTLS, both client and server authenticate each other using certificates. This is ideal for internal microservices or IoT agents where both ends must be verified.

You configure client and server with certs and require verification of peer identities. The x509 package is typically used to validate certificates.

Authentication via Metadata (Bearer Tokens, JWT)

You can send auth tokens using gRPC metadata

md := metadata.Pairs("authorization", "Bearer <token>")
ctx := metadata.NewOutgoingContext(context.Background(), md)
client.Greet(ctx, req)

On the server side, intercept metadata using:

md, ok := metadata.FromIncomingContext(ctx)
token := md["authorization"]

You can then validate the token using JWT libraries or a custom auth provider.

gRPC Interceptors

In gRPC, interceptors are middleware functions that can wrap RPC invocations — both unary and streaming.

Use Cases:

  • Logging
  • Authentication
  • Rate limiting
  • Tracing

Unary Logging Interceptor:

func UnaryLogger(
  ctx context.Context,
  req interface{},
  info *grpc.UnaryServerInfo,
  handler grpc.UnaryHandler,
) (resp interface{}, err error) {
  log.Printf("Incoming RPC: %s", info.FullMethod)
  return handler(ctx, req)
}

grpcServer := grpc.NewServer(grpc.UnaryInterceptor(UnaryLogger))

Streaming interceptors are similar but operate on stream interfaces.

gRPC Gateway for REST/JSON Compatibility

Many times, clients can’t use gRPC (e.g., browsers, mobile apps). gRPC-Gateway is a plugin that turns your gRPC service into a RESTful JSON HTTP API.

How It Works:

  • You define HTTP mappings in your .proto file using annotations:
import "google/api/annotations.proto";

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {
    option (google.api.http) = {
      post: "/v1/greet"
      body: "*"
    };
  }
}
  • Then generate gateway code using:
protoc -I . --go_out . --go-grpc_out . \
  --grpc-gateway_out . greet.proto
  • Finally, set up a gateway HTTP server that translates HTTP/JSON requests to gRPC:
mux := runtime.NewServeMux()
greet.RegisterGreetServiceHandlerFromEndpoint(ctx, mux, "localhost:50051", opts)
http.ListenAndServe(":8080", mux)

Clients can now make POST /v1/greet requests with JSON bodies like:

{
  "first_name": "Ben",
  "last_name": "Meehan"
}

And the server responds using the gRPC service under the hood. The response is again converted back to a HTTP JSON response and sent back to the client.

Till Next Time

gRPC is a deep and powerful ecosystem — one that brings high-performance, strongly-typed, and scalable communication to modern distributed systems. In this part, we’ve journeyed through the essentials of gRPC in Go: from installing protoc and defining your first .proto service, to building unary RPCs, and then expanding into server, client, and bidirectional streaming.

We also covered critical real-world features like securing communication with TLS and mTLS, authenticating clients via metadata, intercepting calls for logging or metrics, and bridging the gRPC-REST gap using gRPC Gateway.

But as comprehensive as this overview might seem, I could never do full justice to the depth and flexibility that gRPC offers in a single guide. Each section, whether it’s streaming, authentication, or gateway integration — has layers of nuance and best practices depending on your use case.

So take this course as your launchpad, and explore the official documentation, try out new patterns, read real-world case studies, and most importantly: build. Because mastery in gRPC, like anything in Go, comes from writing and refining real code in real systems.

Also, look at an alternative to gRPC, Apache Thrift. It was originally developed at Facebook and later donated to the Apache Software Foundation. It lacks some features compared to gRPC but still worth having a look.

Happy coding!

References:

🔗 gRPC Overview
 https://grpc.io/docs/
 The main portal for understanding gRPC concepts, supported languages, and use cases.

🔗 Protocol Buffers Language Guide (proto3)
 https://protobuf.dev/programming-guides/proto3/
 Explains how to write .proto files using proto3 syntax.

🔗 gRPC-Gateway Project
 https://github.com/grpc-ecosystem/grpc-gateway
 Bridges gRPC and REST, allowing you to expose your gRPC service over HTTP/JSON.

🔗 Buf (Better Protobuf tooling)
 https://buf.build/
 For linting, breaking change detection, and API consistency with Protobuf.

🔗 gRPC Go Examples
 https://github.com/grpc/grpc-go/tree/master/examples
 Official examples for server, client, TLS, interceptors, and streaming.