Grpc Microservices

gRPC Microservices

A comprehensive skill for building high-performance, type-safe microservices using gRPC and Protocol Buffers. This skill covers service design, all streaming patterns, interceptors, load balancing, error handling, and production deployment patterns for distributed systems.

When to Use This Skill

Use this skill when:

• Building microservices that require high-performance, low-latency communication
• Implementing real-time data streaming between services
• Designing type-safe APIs with strong contracts using Protocol Buffers
• Creating polyglot systems where services are written in different languages
• Building distributed systems requiring bidirectional streaming
• Implementing service meshes with advanced routing and observability
• Designing APIs that need to evolve with backward/forward compatibility
• Creating internal APIs where performance and type safety are critical
• Building event-driven architectures with streaming data pipelines
• Implementing client-server systems with push capabilities (server streaming)
• Designing systems requiring efficient binary serialization
• Building microservices requiring automatic code generation for multiple languages

Core Concepts

gRPC Fundamentals

gRPC is a modern open-source RPC framework that can run anywhere. It enables client and server applications to communicate transparently and makes it easier to build connected systems.

Key Characteristics:

• HTTP/2 based: Multiplexing, server push, header compression
• Protocol Buffers: Efficient binary serialization format
• Streaming: Bidirectional streaming support built-in
• Code Generation: Auto-generate client/server code in 10+ languages
• Deadlines/Timeouts: First-class timeout support
• Cancellation: Propagate cancellation across services
• Interceptors: Middleware pattern for cross-cutting concerns

Protocol Buffers (protobuf)

Protocol Buffers is a language-neutral, platform-neutral extensible mechanism for serializing structured data.

Advantages:

• Compact: 3-10x smaller than JSON
• Fast: 20-100x faster to serialize/deserialize than JSON
• Type-safe: Strongly typed schema with validation
• Backward/Forward Compatible: Evolve schemas safely
• Language Support: Official support for 10+ languages
• Self-documenting: Schema serves as documentation

Basic Syntax:

syntax = "proto3";

message User {
  int32 id = 1;
  string name = 2;
  string email = 3;
}

Service Definitions

gRPC services are defined in .proto files and specify available methods and their input/output types.

Basic Service:

service UserService {
  rpc GetUser(GetUserRequest) returns (GetUserResponse);
  rpc CreateUser(CreateUserRequest) returns (CreateUserResponse);
}

Four Types of RPC Methods

1. Unary RPC (Request-Response)

Simple request-response pattern, like a traditional REST API call.

rpc GetUser(GetUserRequest) returns (GetUserResponse);

Use Cases:

• CRUD operations
• Simple queries
• Synchronous operations
• Traditional request-response patterns

2. Server Streaming RPC

Client sends one request, server returns a stream of responses.

rpc ListUsers(ListUsersRequest) returns (stream User);

Use Cases:

• Paginated results
• Real-time updates
• Server-side event push
• Large dataset downloads

3. Client Streaming RPC

Client sends a stream of requests, server returns one response.

rpc CreateUsers(stream CreateUserRequest) returns (CreateUsersResponse);

Use Cases:

• Bulk uploads
• Batch processing
• Client-side aggregation
• File uploads in chunks

4. Bidirectional Streaming RPC

Both client and server send streams of messages independently.

rpc Chat(stream ChatMessage) returns (stream ChatMessage);

Use Cases:

• Real-time chat applications
• Live collaboration
• Gaming (real-time state sync)
• IoT bidirectional communication

Protobuf Schema Design

Message Design Best Practices

1. Use Explicit Field Numbers

Field numbers are critical for backward compatibility and should never be reused.

message User {
  int32 id = 1;           // Never change this number
  string name = 2;        // Never change this number
  string email = 3;       // Never change this number
  // int32 age = 4;       // DEPRECATED - don't reuse 4
  string phone = 5;       // New field - use next available
}

2. Use Enumerations for Fixed Sets

enum UserRole {
  USER_ROLE_UNSPECIFIED = 0;  // Always have a zero value
  USER_ROLE_ADMIN = 1;
  USER_ROLE_MODERATOR = 2;
  USER_ROLE_MEMBER = 3;
}

message User {
  int32 id = 1;
  string name = 2;
  UserRole role = 3;
}

3. Use Nested Messages for Complex Types

message User {
  int32 id = 1;
  string name = 2;

  message Address {
    string street = 1;
    string city = 2;
    string state = 3;
    string zip = 4;
  }

  Address address = 3;
  repeated Address additional_addresses = 4;
}

4. Use `repeated` for Arrays

message UserList {
  repeated User users = 1;
}

message User {
  int32 id = 1;
  string name = 2;
  repeated string tags = 3;
}

5. Use `oneof` for Union Types

message SearchRequest {
  string query = 1;

  oneof filter {
    string category = 2;
    int32 user_id = 3;
    string tag = 4;
  }
}

6. Use `google.protobuf` Well-Known Types

import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";
import "google/protobuf/empty.proto";
import "google/protobuf/wrappers.proto";

message Event {
  string id = 1;
  string name = 2;
  google.protobuf.Timestamp created_at = 3;
  google.protobuf.Duration duration = 4;
  google.protobuf.Int32Value optional_count = 5;
}

Service Design Patterns

1. Resource-Oriented Design

Follow RESTful principles adapted for RPC:

service UserService {
  // Get single resource
  rpc GetUser(GetUserRequest) returns (User);

  // List resources
  rpc ListUsers(ListUsersRequest) returns (ListUsersResponse);

  // Create resource
  rpc CreateUser(CreateUserRequest) returns (User);

  // Update resource
  rpc UpdateUser(UpdateUserRequest) returns (User);

  // Delete resource
  rpc DeleteUser(DeleteUserRequest) returns (google.protobuf.Empty);
}

2. Pagination Pattern

message ListUsersRequest {
  int32 page_size = 1;
  string page_token = 2;
  string filter = 3;
}

message ListUsersResponse {
  repeated User users = 1;
  string next_page_token = 2;
  int32 total_count = 3;
}

3. Batch Operations Pattern

message BatchGetUsersRequest {
  repeated int32 user_ids = 1;
}

message BatchGetUsersResponse {
  map<int32, User> users = 1;
  repeated int32 not_found = 2;
}

4. Long-Running Operations Pattern

import "google/longrunning/operations.proto";

service BatchJobService {
  rpc ProcessBatch(BatchRequest) returns (google.longrunning.Operation);
  rpc GetOperation(GetOperationRequest) returns (google.longrunning.Operation);
}

Streaming Patterns

Server Streaming Patterns

1. Pagination Streaming

Stream large result sets efficiently:

service ProductService {
  rpc SearchProducts(SearchRequest) returns (stream Product);
}

message SearchRequest {
  string query = 1;
  int32 limit = 2;
}

Implementation (Go):

func (s *server) SearchProducts(req *pb.SearchRequest, stream pb.ProductService_SearchProductsServer) error {
    products := s.db.Search(req.Query, req.Limit)

    for _, product := range products {
        if err := stream.Send(&product); err != nil {
            return err
        }
    }

    return nil
}

2. Real-Time Updates

Push updates to clients as they occur:

service EventService {
  rpc SubscribeToEvents(SubscribeRequest) returns (stream Event);
}