Introduction: Understanding the Paradigm Shift
GraphQL and gRPC represent two fundamentally different approaches to API design. GraphQL gives clients the power to query exactly the data they need using a flexible, text-based query language over HTTP. gRPC, on the other hand, is a high-performance RPC framework built on Protocol Buffers (protobuf) and HTTP/2, emphasizing strongly-typed contracts, binary serialization, and bidirectional streaming.
Migrating from GraphQL to gRPC is not merely swapping one protocol for another — it is a deliberate architectural shift from a client-driven query model to a server-defined contract model. This tutorial walks you through the complete process, from auditing your existing schema to decommissioning your GraphQL endpoint, with practical code examples at every stage.
Why Migrate from GraphQL to gRPC?
🚀 Deploy your AI agent in 10 minutes
Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.
Try it free →Understanding the motivations behind such a migration helps you justify the effort to your team and stakeholders. Here are the primary drivers:
- Performance and efficiency: gRPC uses binary protobuf encoding over HTTP/2 multiplexed connections, dramatically reducing payload size and eliminating the overhead of JSON parsing. For microservices communicating at high frequency, this matters enormously.
- Strict contract enforcement: GraphQL schemas can evolve loosely, but protobuf definitions are rigorously typed and versioned. Breaking changes are explicit, making multi-team coordination safer.
- Streaming capabilities: gRPC natively supports client-streaming, server-streaming, and bidirectional streaming RPCs. GraphQL subscriptions offer a partial equivalent, but they lack the same level of transport-level efficiency and backpressure handling.
- Reduced attack surface: GraphQL's flexible query model can be abused via deeply nested queries, introspection abuse, and query-based denial-of-service attacks. gRPC's fixed method signatures eliminate these classes of vulnerability.
- Cost control at scale: GraphQL resolvers often trigger unpredictable database cascades. gRPC's predefined message shapes make resource consumption predictable and cacheable.
Step-by-Step Migration Guide
The migration follows a strangler fig pattern: you build the new gRPC system alongside the existing GraphQL API, gradually shifting traffic until the old system can be safely removed. Let's walk through each step.
Step 1: Audit Your Existing GraphQL Schema
Before writing a single protobuf file, you need a complete inventory of what your GraphQL API exposes. Export your schema and list every query, mutation, subscription, type, enum, and interface. Pay special attention to:
- Custom scalars (they need explicit protobuf representations)
- Union types (protobuf uses
oneoffields instead) - Nested fragment patterns that clients commonly use
- Resolver-level authorization logic that must be replicated
Here is an example GraphQL schema we will use throughout this tutorial:
# schema.graphql
type Query {
user(id: ID!): User
users(ids: [ID!]!): [User!]!
searchUsers(query: String!, limit: Int): [User!]!
}
type Mutation {
createUser(input: CreateUserInput!): User!
updateUser(id: ID!, input: UpdateUserInput!): User!
deleteUser(id: ID!): Boolean!
}
type Subscription {
userUpdated(id: ID!): User!
}
type User {
id: ID!
name: String!
email: String!
role: UserRole!
createdAt: String!
metadata: JSON
}
enum UserRole {
ADMIN
EDITOR
VIEWER
}
input CreateUserInput {
name: String!
email: String!
role: UserRole!
metadata: JSON
}
input UpdateUserInput {
name: String
email: String
role: UserRole
metadata: JSON
}
scalar JSON
Step 2: Design Your Protobuf Contracts
Now translate the GraphQL schema into protobuf definitions. The key mapping principles are:
- GraphQL queries and mutations become gRPC unary RPCs
- GraphQL subscriptions become server-streaming RPCs
- GraphQL types become protobuf messages
- GraphQL enums become protobuf enums
- GraphQL unions become messages with oneof fields
- GraphQL custom scalars (like JSON) become bytes or google.protobuf.Struct
Create a new directory for your protobuf files. Here is the complete translation of our example schema:
// proto/user_service.proto
syntax = "proto3";
package user.service.v1;
import "google/protobuf/struct.proto";
import "google/protobuf/timestamp.proto";
import "google/protobuf/empty.proto";
// --- Enums ---
enum UserRole {
USER_ROLE_UNSPECIFIED = 0; // Always start with a zero-value default
USER_ROLE_ADMIN = 1;
USER_ROLE_EDITOR = 2;
USER_ROLE_VIEWER = 3;
}
// --- Core Messages ---
message User {
string id = 1;
string name = 2;
string email = 3;
UserRole role = 4;
google.protobuf.Timestamp created_at = 5;
google.protobuf.Struct metadata = 6; // Replaces the JSON scalar
}
// --- Request Messages (mapping GraphQL inputs) ---
message GetUserRequest {
string id = 1;
}
message GetUsersRequest {
repeated string ids = 1;
}
message SearchUsersRequest {
string query = 1;
int32 limit = 2; // Use int32, not Int (GraphQL's Int is a 32-bit integer)
}
message CreateUserRequest {
string name = 1;
string email = 2;
UserRole role = 3;
google.protobuf.Struct metadata = 4;
}
message UpdateUserRequest {
string id = 1;
optional string name = 2; // optional for partial updates
optional string email = 3;
optional UserRole role = 4;
optional google.protobuf.Struct metadata = 5;
}
message DeleteUserRequest {
string id = 1;
}
// --- Response Messages ---
message DeleteUserResponse {
bool success = 1;
}
message UserUpdateEvent {
User user = 1;
string event_type = 2; // e.g., "updated", "deleted"
}
// --- Service Definition ---
service UserService {
// Queries → Unary RPCs
rpc GetUser(GetUserRequest) returns (User);
rpc GetUsers(GetUsersRequest) returns (stream User); // Server-streaming for list
rpc SearchUsers(SearchUsersRequest) returns (stream User);
// Mutations → Unary RPCs
rpc CreateUser(CreateUserRequest) returns (User);
rpc UpdateUser(UpdateUserRequest) returns (User);
rpc DeleteUser(DeleteUserRequest) returns (DeleteUserResponse);
// Subscription → Server-streaming RPC
rpc WatchUserUpdates(GetUserRequest) returns (stream UserUpdateEvent);
}
Notice several important design decisions:
- The
GetUsersandSearchUsersRPCs use server-streaming (stream User) rather than returning a repeated field in a wrapper message. This allows the client to process results incrementally as they arrive, reducing memory pressure on large result sets. - The GraphQL
subscriptionbecomesWatchUserUpdates, a server-streaming RPC that pushes events to the client whenever the specified user is modified. - Every enum value includes a
UNSPECIFIEDzero value, which is a protobuf best practice for forward compatibility. - Optional fields in
UpdateUserRequestuse theoptionalkeyword (proto3 feature) to distinguish between "not provided" and "set to empty/zero".
Step 3: Generate Code and Set Up the gRPC Server
With your protobuf definitions ready, generate the server and client stubs. This example uses TypeScript with the @grpc/grpc-js library, but the process is similar across all supported languages.
First, install the necessary tooling:
npm install @grpc/grpc-js @grpc/proto-loader google-protobuf
npm install -D grpc-tools grpc_tools_node_protoc_ts
Add a script to generate code from your proto files:
// package.json (scripts section)
{
"scripts": {
"generate:proto": "grpc_tools_node_protoc --js_out=import_style=commonjs,binary:./generated --grpc_out=grpc_js:./generated --ts_out=grpc_js:./generated -I ./proto ./proto/**/*.proto"
}
}
Now implement the gRPC server. This is where you port your existing GraphQL resolver logic:
// server.ts
import * as grpc from '@grpc/grpc-js';
import { UserServiceService } from './generated/user_service_grpc_pb';
import {
User,
GetUserRequest,
GetUsersRequest,
SearchUsersRequest,
CreateUserRequest,
UpdateUserRequest,
DeleteUserRequest,
DeleteUserResponse,
UserUpdateEvent,
UserRole
} from './generated/user_service_pb';
import { Timestamp } from 'google-protobuf/google/protobuf/timestamp_pb';
import { Struct } from 'google-protobuf/google/protobuf/struct_pb';
// This is your existing business logic, extracted from GraphQL resolvers
const userDatabase = new Map(); // In-memory for demo; use real DB
function buildUserResponse(userData: User.AsObject): User {
const user = new User();
user.setId(userData.id);
user.setName(userData.name);
user.setEmail(userData.email);
user.setRole(userData.role);
const ts = new Timestamp();
ts.fromDate(new Date(userData.createdAt));
user.setCreatedAt(ts);
if (userData.metadata) {
user.setMetadata(Struct.fromJavaScript(userData.metadata));
}
return user;
}
const server = new grpc.Server();
server.addService(UserServiceService, {
// Corresponds to: type Query { user(id: ID!): User }
getUser: (call, callback) => {
const request = call.request as GetUserRequest;
const userData = userDatabase.get(request.getId());
if (!userData) {
return callback({
code: grpc.status.NOT_FOUND,
message: `User ${request.getId()} not found`
});
}
callback(null, buildUserResponse(userData));
},
// Corresponds to: type Query { users(ids: [ID!]!): [User!]! }
getUsers: (call) => {
const request = call.request as GetUsersRequest;
const ids = request.getIdsList();
for (const id of ids) {
const userData = userDatabase.get(id);
if (userData) {
call.write(buildUserResponse(userData));
}
}
call.end();
},
// Corresponds to: type Query { searchUsers(query: String!, limit: Int): [User!]! }
searchUsers: (call) => {
const request = call.request as SearchUsersRequest;
const query = request.getQuery().toLowerCase();
const limit = request.getLimit() || 10;
let count = 0;
for (const [, userData] of userDatabase) {
if (count >= limit) break;
if (userData.name.toLowerCase().includes(query) ||
userData.email.toLowerCase().includes(query)) {
call.write(buildUserResponse(userData));
count++;
}
}
call.end();
},
// Corresponds to: type Mutation { createUser(input: CreateUserInput!): User! }
createUser: (call, callback) => {
const request = call.request as CreateUserRequest;
const id = `user_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
const now = new Date().toISOString();
const userData: User.AsObject = {
id,
name: request.getName(),
email: request.getEmail(),
role: request.getRole(),
createdAt: now,
metadata: request.hasMetadata()
? request.getMetadata()?.toJavaScript() as Record
: undefined
};
userDatabase.set(id, userData);
callback(null, buildUserResponse(userData));
},
// Corresponds to: type Mutation { updateUser(id: ID!, input: UpdateUserInput!): User! }
updateUser: (call, callback) => {
const request = call.request as UpdateUserRequest;
const id = request.getId();
const existing = userDatabase.get(id);
if (!existing) {
return callback({
code: grpc.status.NOT_FOUND,
message: `User ${id} not found`
});
}
// Apply partial updates (matching GraphQL optional input fields)
const updated: User.AsObject = { ...existing };
if (request.hasName()) updated.name = request.getName()!;
if (request.hasEmail()) updated.email = request.getEmail()!;
if (request.hasRole()) updated.role = request.getRole()!;
if (request.hasMetadata()) {
updated.metadata = request.getMetadata()!.toJavaScript() as Record;
}
userDatabase.set(id, updated);
callback(null, buildUserResponse(updated));
},
// Corresponds to: type Mutation { deleteUser(id: ID!): Boolean! }
deleteUser: (call, callback) => {
const request = call.request as DeleteUserRequest;
const id = request.getId();
const existed = userDatabase.delete(id);
const response = new DeleteUserResponse();
response.setSuccess(existed);
callback(null, response);
},
// Corresponds to: type Subscription { userUpdated(id: ID!): User! }
watchUserUpdates: (call) => {
const request = call.request as GetUserRequest;
const userId = request.getId();
// In a real implementation, you'd subscribe to a message bus or event emitter
// Here we simulate with a polling interval as a placeholder
const interval = setInterval(() => {
const userData = userDatabase.get(userId);
if (userData) {
const event = new UserUpdateEvent();
event.setUser(buildUserResponse(userData));
event.setEventType('updated');
call.write(event);
}
}, 5000);
// Clean up on client disconnect
call.on('cancelled', () => {
clearInterval(interval);
});
call.on('end', () => {
clearInterval(interval);
});
}
});
const PORT = 50051;
server.bindAsync(
`0.0.0.0:${PORT}`,
grpc.ServerCredentials.createInsecure(),
(error, boundPort) => {
if (error) {
console.error('Server bind failed:', error);
return;
}
server.start();
console.log(`gRPC server running on port ${boundPort}`);
}
);
Step 4: Port Your Business Logic to a Shared Service Layer
Before migrating clients, extract your resolver logic into a framework-agnostic service layer. This allows both your GraphQL resolvers and gRPC handlers to call the same code, ensuring consistency during the transition period.
// services/user_service.ts
// Framework-agnostic business logic shared by GraphQL resolvers AND gRPC handlers
export interface UserRecord {
id: string;
name: string;
email: string;
role: 'ADMIN' | 'EDITOR' | 'VIEWER';
createdAt: Date;
metadata?: Record;
}
export class UserService {
private db: Map = new Map();
async getUser(id: string): Promise {
return this.db.get(id) ?? null;
}
async getUsers(ids: string[]): Promise {
return ids
.map(id => this.db.get(id))
.filter((u): u is UserRecord => u !== undefined);
}
async searchUsers(query: string, limit: number = 10): Promise {
const lower = query.toLowerCase();
const results: UserRecord[] = [];
for (const [, user] of this.db) {
if (results.length >= limit) break;
if (user.name.toLowerCase().includes(lower) ||
user.email.toLowerCase().includes(lower)) {
results.push(user);
}
}
return results;
}
async createUser(input: Omit): Promise {
const record: UserRecord = {
id: crypto.randomUUID(),
...input,
createdAt: new Date(),
};
this.db.set(record.id, record);
return record;
}
async updateUser(id: string, patch: Partial>): Promise {
const existing = this.db.get(id);
if (!existing) throw new Error(`User ${id} not found`);
const updated = { ...existing, ...patch };
this.db.set(id, updated);
return updated;
}
async deleteUser(id: string): Promise {
return this.db.delete(id);
}
}
Now both your GraphQL resolvers and gRPC handlers can instantiate or inject UserService, keeping business rules identical across both APIs.
Step 5: Build a Dual-Running Bridge with a GraphQL-to-gRPC Proxy
The strangler fig pattern requires both systems to run concurrently. Instead of forcing every client to rewrite immediately, you can introduce a GraphQL gateway that delegates to gRPC under the hood. This gives you a migration path where:
- Existing GraphQL clients continue working unchanged
- New gRPC clients can be gradually introduced
- The gateway validates that your gRPC implementation matches the old GraphQL behavior
Here is a GraphQL gateway that wraps your gRPC service using Apollo Server:
// gateway.ts
import { ApolloServer } from '@apollo/server';
import { startStandaloneServer } from '@apollo/server/standalone';
import * as grpc from '@grpc/grpc-js';
import { UserServiceClient } from './generated/user_service_grpc_pb';
import {
GetUserRequest,
GetUsersRequest,
SearchUsersRequest,
CreateUserRequest,
UpdateUserRequest,
DeleteUserRequest,
UserRole
} from './generated/user_service_pb';
import { Struct } from 'google-protobuf/google/protobuf/struct_pb';
// gRPC client setup
const grpcClient = new UserServiceClient(
'localhost:50051',
grpc.credentials.createInsecure()
);
// Read the original GraphQL schema (SDL)
const typeDefs = `
type Query {
user(id: ID!): User
users(ids: [ID!]!): [User!]!
searchUsers(query: String!, limit: Int): [User!]!
}
type Mutation {
createUser(input: CreateUserInput!): User!
updateUser(id: ID!, input: UpdateUserInput!): User!
deleteUser(id: ID!): Boolean!
}
type User {
id: ID!
name: String!
email: String!
role: UserRole!
createdAt: String!
metadata: JSON
}
enum UserRole {
ADMIN
EDITOR
VIEWER
}
input CreateUserInput {
name: String!
email: String!
role: UserRole!
metadata: JSON
}
input UpdateUserInput {
name: String
email: String
role: UserRole
metadata: JSON
}
scalar JSON
`;
// Helper to convert gRPC User message to GraphQL shape
function grpcUserToGraphQL(grpcUser: any) {
return {
id: grpcUser.getId(),
name: grpcUser.getName(),
email: grpcUser.getEmail(),
role: grpcUser.getRole(),
createdAt: grpcUser.getCreatedAt()?.toDate().toISOString(),
metadata: grpcUser.getMetadata()?.toJavaScript()
};
}
// Resolvers that delegate to gRPC
const resolvers = {
Query: {
user: async (_: any, { id }: { id: string }) => {
const request = new GetUserRequest();
request.setId(id);
return new Promise((resolve, reject) => {
grpcClient.getUser(request, (error, response) => {
if (error) {
if (error.code === grpc.status.NOT_FOUND) return resolve(null);
return reject(new Error(error.message));
}
resolve(grpcUserToGraphQL(response));
});
});
},
users: async (_: any, { ids }: { ids: string[] }) => {
const request = new GetUsersRequest();
request.setIdsList(ids);
const users: any[] = [];
return new Promise((resolve, reject) => {
const stream = grpcClient.getUsers(request);
stream.on('data', (user: any) => users.push(grpcUserToGraphQL(user)));
stream.on('end', () => resolve(users));
stream.on('error', (err: any) => reject(err));
});
},
searchUsers: async (_: any, { query, limit }: { query: string; limit?: number }) => {
const request = new SearchUsersRequest();
request.setQuery(query);
if (limit !== undefined) request.setLimit(limit);
const users: any[] = [];
return new Promise((resolve, reject) => {
const stream = grpcClient.searchUsers(request);
stream.on('data', (user: any) => users.push(grpcUserToGraphQL(user)));
stream.on('end', () => resolve(users));
stream.on('error', (err: any) => reject(err));
});
}
},
Mutation: {
createUser: async (_: any, { input }: { input: any }) => {
const request = new CreateUserRequest();
request.setName(input.name);
request.setEmail(input.email);
request.setRole(input.role);
if (input.metadata) {
request.setMetadata(Struct.fromJavaScript(input.metadata));
}
return new Promise((resolve, reject) => {
grpcClient.createUser(request, (error: any, response: any) => {
if (error) return reject(new Error(error.message));
resolve(grpcUserToGraphQL(response));
});
});
},
updateUser: async (_: any, { id, input }: { id: string; input: any }) => {
const request = new UpdateUserRequest();
request.setId(id);
if (input.name !== undefined) request.setName(input.name);
if (input.email !== undefined) request.setEmail(input.email);
if (input.role !== undefined) request.setRole(input.role);
if (input.metadata !== undefined) {
request.setMetadata(Struct.fromJavaScript(input.metadata));
}
return new Promise((resolve, reject) => {
grpcClient.updateUser(request, (error: any, response: any) => {
if (error) return reject(new Error(error.message));
resolve(grpcUserToGraphQL(response));
});
});
},
deleteUser: async (_: any, { id }: { id: string }) => {
const request = new DeleteUserRequest();
request.setId(id);
return new Promise((resolve, reject) => {
grpcClient.deleteUser(request, (error: any, response: any) => {
if (error) return reject(new Error(error.message));
resolve(response.getSuccess());
});
});
}
}
};
const server = new ApolloServer({ typeDefs, resolvers });
startStandaloneServer(server, { listen: { port: 4000 } }).then(({ url }) => {
console.log(`GraphQL gateway (backed by gRPC) ready at ${url}`);
});
At this stage, your GraphQL endpoint functions identically to the original, but all data flows through the new gRPC service. This validates the gRPC implementation in production without breaking existing clients.
Step 6: Migrate Clients Incrementally
With the gRPC service running and validated, you can begin migrating client applications. The strategy depends on the client type:
For backend services (microservices, job workers): Replace the GraphQL client library with a gRPC client directly. Here is a before-and-after example:
// BEFORE: GraphQL client (using graphql-request)
import { request, gql } from 'graphql-request';
async function fetchUser(id: string) {
const query = gql`
query GetUser($id: ID!) {
user(id: $id) {
id
name
email
role
}
}
`;
const data = await request('http://localhost:4000/graphql', query, { id });
return data.user;
}
// AFTER: gRPC client
import * as grpc from '@grpc/grpc-js';
import { UserServiceClient } from './generated/user_service_grpc_pb';
import { GetUserRequest } from './generated/user_service_pb';
const client = new UserServiceClient('localhost:50051', grpc.credentials.createInsecure());
async function fetchUser(id: string): Promise {
const req = new GetUserRequest();
req.setId(id);
return new Promise((resolve, reject) => {
client.getUser(req, (error, user) => {
if (error) return reject(error);
resolve({
id: user.getId(),
name: user.getName(),
email: user.getEmail(),
role: user.getRole()
});
});
});
}
For frontend web applications: gRPC-web or a gateway pattern is required since browsers cannot speak raw gRPC. You have two options:
- Keep the GraphQL gateway indefinitely for browser clients, while backend services use gRPC directly. This is a common end-state for many organizations.
- Introduce a gRPC-web proxy (like Envoy or grpc-web-proxy) and use the
@grpc/grpc-webclient library in the browser, enabling direct gRPC calls from the frontend.
// Example: Frontend gRPC-web client (requires grpc-web proxy like Envoy)
import { UserServiceClient } from './generated/user_service_grpc_web_pb';
import { GetUserRequest } from './generated/user_service_pb';
const client = new UserServiceClient('https://api.example.com');
export async function fetchUser(id: string) {
const req = new GetUserRequest();
req.setId(id);
const response = await client.getUser(req, {});
return {
id: response.getId(),
name: response.getName(),
email: response.getEmail(),
role: response.getRole()
};
}
For mobile applications: gRPC has excellent native support on both iOS (Swift protobuf + gRPC) and Android (Java/Kotlin protobuf + gRPC). Mobile apps benefit enormously from gRPC's smaller payloads and connection multiplexing, especially on unreliable cellular networks.
Step 7: Decommission the GraphQL Endpoint
Once all clients have been migrated (or permanently routed through the gateway), you can decommission the original GraphQL server. Follow this checklist:
- Verify zero traffic to the old GraphQL endpoint for at least two full deployment cycles (monitor logs, metrics, and client IPs).
- Remove the gateway code if you no longer need it, or keep it as a thin compatibility layer for browser clients.
- Archive the GraphQL schema and resolver code in a tagged repository state for future reference.
- Update CI/CD pipelines to remove GraphQL-specific test suites and schema publishing steps.
- Notify all dependent teams that the GraphQL endpoint is deprecated and will be shut down on a specific date.
- Decommission infrastructure — remove the GraphQL server deployment, load balancer rules, and monitoring dashboards.
Best Practices for a Smooth Migration
Version Your Protobuf Packages from Day One
Always include a version component in your proto package name (e.g., user.service.v1) and in the generated code path. This allows you to run multiple versions of a service simultaneously during migrations and rollbacks. Never use an unversioned package like user — it creates a dead end for future evolution.
// Good
package user.service.v1;
// Bad — avoid this
package user;
Use Protobuf Backward-Compatibility Rules Religiously
Protobuf has strict compatibility rules that, when followed, guarantee that new servers can handle old clients and vice versa. The rules are:
- Never change the field number or name of an existing field
- Never remove a field (mark it deprecated instead)
- Only add new fields with new, unique field numbers
- Never change an enum value's numeric code (only add new values)
- Reserve deleted field numbers and names using the
reservedkeyword
message User {
string id = 1;
string name = 2;
string email = 3;
// REMOVED: string phone = 4; -- Never just delete!
reserved 4; // Reserve the field number
reserved "phone"; // Reserve the field name
UserRole role = 5;
google.protobuf.Timestamp created_at = 6;
}
Implement Comprehensive Error Handling
gRPC has a rich status code model. Map your GraphQL error patterns to appropriate gRPC status codes rather than returning generic errors:
NOT_FOUND— resource doesn't exist (replaces GraphQLnullreturns)INVALID_ARGUMENT— validation failures (replaces GraphQL field-level errors)PERMISSION_DENIED— authorization failuresUNAUTHENTICATED— missing or invalid credentialsALREADY_EXISTS— duplicate resource creation attemptsRESOURCE_EXHAUSTED— rate limiting or quota exceeded
// Example: Rich error handling in a gRPC handler
import { status } from '@grpc/grpc-js';
function createUserHandler(call: any, callback: any) {
const input = call.request;
if (!input.getName() || input.getName().length < 2) {
return callback({
code: status.INVALID_ARGUMENT,
message: 'Name must be at least 2 characters',
details: `Field 'name' validation failed`
});
}
// ... proceed with creation
}
Monitor and Observe Everything
During the migration, you must have visibility into both systems. Instrument your gRPC service with:
- OpenTelemetry tracing — propagate trace contexts across GraphQL gateway → gRPC service → downstream dependencies
- gRPC-specific metrics — request count, latency percentiles, error rates per method, and streaming message counts
- Deadline propagation — GraphQL resolvers often lack timeouts; gRPC deadlines must be threaded through the entire call chain