How to Properly Handle gRPC Error Handling

gRPC Request Message Format ​

A gRPC request message is essentially an HTTP/2 POST request. Its message structure is mainly divided into two parts: HTTP/2 Headers and a Length-Prefixed Message (usually Protobuf-encoded data).

Below is a structural diagram of a gRPC request message:

+---------------------------------------------------------------+
|                    HTTP/2 HEADERS Frame                       |
+---------------------------------------------------------------+
| :method: POST                                                 |
| :scheme: http (or https)                                      |
| :path: /<Package>.<Service>/<Method> (e.g., /hello/SayHello) |
| :authority: <Target Hostname>                                 |
| content-type: application/grpc                                |
| user-agent: grpc-go/1.x.x                                     |
| grpc-timeout: <Timeout> (Optional)                            |
| (Other Custom Metadata...)                                    |
+---------------------------------------------------------------+
|                      HTTP/2 DATA Frame                        |
|                   (Length-Prefixed Message)                   |
+---+-------------------+---------------------------------------+
|   | Compressed Flag   |  0 (Uncompressed) or 1 (Compressed)   |
|   +-------------------+---------------------------------------+
| D | Message Length    |  Occupies 4 bytes (Length of data)    |
| A +-------------------+---------------------------------------+
| T |                   |                                       |
| A | Message Data      |  Binary business payload serialized    |
|   |                   |  via Protobuf (Length is determined    |
|   |                   |  by the Message Length above)          |
+---+-------------------+---------------------------------------+

Structure Explanation:

1. Headers Frame: gRPC strictly requires the POST method, and content-type must be application/grpc (or its variants). The specific requested method is routed via the :path pseudo-header.
2. Data Frame: gRPC wraps the serialized payload (usually Protobuf) into a structure called a Length-Prefixed Message:
   • 1-byte Compressed Flag: Indicates whether the subsequent payload is compressed.
   • 4-byte Message Length: An unsigned integer representing the byte size of the specific business data.
   • Message Data: The actual binary stream of the business request.

gRPC Response Message Format ​

┌───────────────────────────────┐
│           HTTP/2              │
│                               │
│  ┌─────────────────────────┐  │
│  │       HEADERS Frame     │  │
│  │                         │  │
│  │ :status: 200            │  │
│  │ content-type:           │  │
│  │   application/grpc      │  │
│  │ grpc-encoding: gzip     │  │
│  │ grpc-accept-encoding    │  │
│  └─────────────────────────┘  │
│                               │
│  ┌─────────────────────────┐  │
│  │        DATA Frame       │  │
│  │                         │  │
│  │  gRPC Message           │  │
│  │                         │  │
│  │  ┌───────────────────┐  │  │
│  │  │Compressed-Flag(1) │  │  │
│  │  │Message-Length(4)  │  │  │
│  │  │Protobuf Message   │  │  │
│  │  └───────────────────┘  │  │
│  │                         │  │
│  └─────────────────────────┘  │
│                               │
│  ┌─────────────────────────┐  │
│  │      HEADERS Frame      │  │
│  │      (Trailers)         │  │
│  │                         │  │
│  │ grpc-status: 0          │  │
│  │ grpc-message: OK        │  │
│  │ grpc-status-details-bin │  │
│  └─────────────────────────┘  │
│                               │
└───────────────────────────────┘

The error information returned by gRPC is not placed in the traditional response message data body. Instead, it utilizes the HTTP/2 Headers and Trailers mechanism for encapsulation and transmission.

The specific transmission details are as follows:

• Core Status Code and Short Description: When an RPC stream terminates abnormally, the gRPC protocol mandates that the server returns a grpc-status field in the HTTP/2 Trailers to indicate the error status code. Optionally, a human-readable error description can be attached via the grpc-message field.
• Rich Error Model (Structured Metadata): For complex business scenarios requiring specific validation failure fields, gRPC supports appending any number of Protobuf-serialized messages as rich error details. These structured metadata are safely Base64-encoded and embedded into an HTTP/2 Trailer field named grpc-status-details-bin for cross-network transmission.

Physical Capacity Limit: It is highly noteworthy that, because error information is transmitted via headers and trailers, the gRPC specification explicitly states that the total capacity of Headers and Trailers is typically constrained by the 8KB (8,192 bytes) limit configured in the HTTP/2 protocol. Attaching an extremely large payload to the error details will lead to connection truncation failures. Therefore, the transmitted error information must be concise.

How Should gRPC-Go Return Errors? ​

Handling and returning errors in grpc-go requires moving beyond simple exception-catching logic found in monolithic applications. It must be treated as a core part of inter-service contracts, system resilience, and security defense. According to the gRPC protocol specifications and microservice architecture best practices, returning errors should follow these core principles:

1. Strictly Adhere to the Official Standardized Status Code System ​

• Mutually Exclusive Results Principle: A gRPC call must return a clear, mutually exclusive result: either a strongly-typed success payload message or a strictly-encoded status error message.
• Use Standard Status Codes: You must strictly use the 17 standard constant status codes (underlying type uint32) defined in the google.golang.org/grpc/codes package.

• Precise Matching and Rejecting Catch-alls: Developers should select status codes precisely based on the specific business scenario (e.g., use PERMISSION_DENIED for insufficient permissions, FAILED_PRECONDITION for unmet system prerequisites). It is strictly forbidden to use UNKNOWN or INTERNAL as a "universal catch-all" option for all errors, as this destroys the client’s automated recovery and retry strategies.
• Consistency of Status and Error: A non-nil error must never contain an OK status code, and an OK status code must correspond to a nil error.

2. Strictly Prohibit Passing Native Go Underlying Errors Directly; Standardized Encapsulation is Required ​

• Prevent Semantic Downgrade: The server must never directly return a native Go error created by errors.New or fmt.Errorf. If underlying errors (like the database’s gorm.ErrRecordNotFound) are passed through directly, the gRPC framework, unable to identify its protocol mapping logic, will forcefully downgrade and wrap it into a vague Unknown (Code 2) error, losing all business semantics.

// grpc-go/server.go/processUnaryRPC
if gs, ok := err.(grpcstatus); ok {
    grpcStatus := gs.GRPCStatus()
    if grpcStatus == nil {
        // Error has status nil, which maps to codes.OK. There
        // is no sensible behavior for this, so we turn it into
        // an error with codes.Unknown and discard the existing
        // status.
        return New(codes.Unknown, err.Error()), false
    }
    return grpcStatus, true
}

• Construct Errors Using the status Package: Standardized error construction should utilize the status.New, status.Error, or status.Errorf methods provided by the google.golang.org/grpc/status package.

st := status.New(codes.NotFound, "some description")
err := st.Err()

// vs.

err := status.Error(codes.NotFound, "some description")

• Implement Compatible Interfaces: Custom business error types can implement the GRPCStatus() *status.Status method, allowing the error object to be correctly recognized and seamlessly converted by the gRPC framework.

3. Adopt the Rich Error Model to Pass Structured Details ​

• Beyond Simple Strings: When integer status codes and short string descriptions are insufficient for complex business needs (such as form field validation failures), the rich error model defined based on google.rpc.Status should be used.
• Inject Standard Metadata Details: Use the status.Status.WithDetails() method to attach any number of Protobuf-serialized messages to the error. It is highly recommended to use the standard details defined in error_details.proto, such as errdetails.BadRequest (to explain exactly which field failed validation), errdetails.RetryInfo (to guide the client’s retry delay), errdetails.QuotaFailure (rate limiting breakdown), etc.

func (s *myServiceServer) CreateUser(ctx context.Context, req *pb.CreateUserRequest) (*pb.UserResponse, error) {
    // Assume some complex business-level validation is executed
    if req.GetEmail() == "" || len(req.GetPassword()) < 8 {
        // 1. Initialize the base status object (Codes: InvalidArgument)
        st := status.New(codes.InvalidArgument, "One or more fields are invalid")

        // 2. Construct the official BadRequest struct
        br := &errdetails.BadRequest{}

        if req.GetEmail() == "" {
            br.FieldViolations = append(br.FieldViolations, &errdetails.BadRequest_FieldViolation{
                Field:       "email",
                Description: "The email address cannot be empty",
            })
        }
        if len(req.GetPassword()) < 8 {
            br.FieldViolations = append(br.FieldViolations, &errdetails.BadRequest_FieldViolation{
                Field:       "password",
                Description: "The password must be at least 8 characters long",
            })
        }

        // 3. Pack and attach the detailed payload to the status object
        stWithDetails, err := st.WithDetails(br)
        if err != nil {
            // If serializing Any data fails (usually won't happen unless a non-proto struct is passed), return a fallback internal error
            return nil, status.Error(codes.Internal, "Failed to attach error details")
        }

        // 4. Convert the packed struct back to a Go error interface and return
        return nil, stWithDetails.Err()
    }

    //... Normal processing logic
    return &pb.UserResponse{}, nil
}

• Strongly-Typed Client Parsing: This pattern spares the client from fragile "string matching" error parsing. After extracting the status via status.FromError(err), the client can call st.Details() combined with type assertions to safely extract structured metadata.

resp, err := client.CreateUser(context.Background(), req)
if err != nil {
    // 1. Safely extract the native error into a *status.Status object
    st, ok := status.FromError(err)
    if !ok {
        // If ok is false, it means this is not a standardized error from the gRPC protocol layer
        // It might be a Context timeout or network-level DNS failure before the request was sent
        log.Fatalf("Non-gRPC error occurred: %v", err)
        return
    }

    // 2. Perform core branching logic based on the status code
    switch st.Code() {
    case codes.InvalidArgument:
        fmt.Println("Server rejected the request due to invalid arguments.")

        // 3. Safely extract and iterate over the attached rich error details
        for _, detail := range st.Details() {
            switch t := detail.(type) {
            // Use type assertion to match specific errdetails structs
            case *errdetails.BadRequest:
                fmt.Println("Validation Errors:")
                for _, violation := range t.GetFieldViolations() {
                    fmt.Printf(" - Field %q: %s\n", violation.GetField(), violation.GetDescription())
                }
            case *errdetails.RetryInfo:
                fmt.Printf("Rate limited. Please retry after: %v\n", t.GetRetryDelay().AsDuration())
            default:
                // Ignore unknown or unconcerned detail types
                fmt.Printf("Received unhandled detail type: %T\n", t)
            }
        }

    case codes.Unavailable:
        fmt.Println("Service is temporarily unavailable, initiating exponential backoff retry...")
    default:
        fmt.Printf("Unhandled gRPC error - Code: %s, Message: %s\n", st.Code(), st.Message())
    }
    return
}

fmt.Printf("Successfully created user: %v\n", resp)

4. Rely on Interceptors to Build Global Mapping and Anti-Leakage Boundaries ​

• Unified Exception Transformation: Unified error interception, dictionary lookup, and transformation logic should be executed at the server’s Interceptor layer (Unary/Stream Interceptor). Underlying native errors (like transforming a database record not found into codes.NotFound) are mapped to standard status codes before being dispatched to the network layer, avoiding massive repetitive code in every controller.
• Panic Recovery: A Recovery interceptor must be integrated to catch unhandled Panics. This gracefully transforms a system crash into a codes.Internal status code, preventing the underlying HTTP/2 stream from being abnormally severed, which could trigger cascading failures (avalanches).

Consequences of Not Returning Errors as Prescribed ​

If native Go errors created via errors.New or fmt.Errorf are directly returned in a gRPC-Go request, the gRPC framework’s underlying logic (like status.FromError) fails to find a valid status object implementing the GRPCStatus() interface within the error chain. Consequently, it is forced to uniformly downgrade and wrap this original error into a vague Unknown (Code 2) error.

This practice is widely considered an anti-pattern in distributed architectures and will trigger severe issues across several dimensions:

1. Semantic Erasure and Failure of Client Recovery Mechanisms ​

gRPC relies on 17 explicit normative status codes to guide clients in executing specific logic. When all abnormal situations are downgraded to Unknown (Code 2), the client loses the basis for determining the fault type.

• Destroyed Retry Resilience: For instance, a transient network dip should return UNAVAILABLE and trigger the client’s automated backoff retry. However, if downgraded to Unknown, most mature client interceptors will treat it as a non-retryable fatal error, causing the request to fail outright.
• Blocked Business Logic: Clients cannot distinguish between "invalid request parameters (INVALID_ARGUMENT)", "resource not found (NOT_FOUND)", or "insufficient permissions (PERMISSION_DENIED)", meaning they cannot provide accurate frontend UI guidance for users.

2. Sensitive Information Leakage and Security Risks ​

In standard Go development, fmt.Errorf("…: %w", err) is frequently used to stitch together underlying error call stacks and details. Returning this directly to external clients causes severe Information Disclosure:

• It may expose the absolute physical file paths of the server or internal OS usernames.
• If the underlying error is a database error, the returned string might contain SQL statement fragments, table structure names, or even database connection info. This provides enormous convenience for malicious attackers to probe system vulnerabilities.

3. Degradation of Observability and Failure of Monitoring Alerts ​

Modern microservice monitoring metrics (like Prometheus) heavily rely on the grpc_code label automatically extracted by interceptors to generate dashboards.

• Alert Blindness: With all exceptions bearing the Unknown label, it becomes impossible to distinguish whether a current error spike is due to clients sending massive invalid parameters (which should be INVALID_ARGUMENT, requiring no server alert) or an actual database crash (which should be INTERNAL, requiring immediate, high-priority paging).
• Monitoring System Avalanche (High Cardinality Problem): If monitoring tools or log aggregators mistakenly treat these lengthy error strings containing dynamic data (like timestamps, specific user IDs) as labels, it triggers a cardinality explosion that can overwhelm and crush the monitoring system.

4. Loss of the Capability for a Structured "Rich Error Model" ​

Directly returning native errors completely bypasses gRPC’s rich error model feature. You forfeit the ability to use HTTP/2 Trailers to pass richer, structured metadata.

• Cross-language clients (like Java, Python, Node.js, etc.) will be unable to extract strongly-typed error breakdowns (e.g., using errdetails.BadRequest to specify exactly which form field is incorrect, or RetryInfo to accurately inform how many seconds to wait before retrying). Clients are left to write fragile regular expressions to parse your error string.

5. Performance Degradation and HTTP/2 Underlying Transmission Risks ​

• HPACK Compression Failure: HTTP/2 uses the HPACK algorithm to compress headers, which achieves extremely high compression ratios for static, repetitive common status codes (like OK). Conversely, dynamically generated long-text native error strings are different every time, rendering the compression algorithm useless and adding significant wasted bandwidth overhead.
• Exceeding Connection Limits: If the error stack string is excessively long, it may approach or exceed the MAX_HEADER_LIST_SIZE limit specified by HTTP/2, causing the transport layer to directly block it, thereby ironically masking the true application error.

Unified Error Handling ​

In gRPC-Go, various non-standard errors thrown by underlying APIs and other services (e.g., database query errors, network request failures) should be unifiedly handled and safely converted into the status standard prescribed by gRPC. Instead of mapping statuses via errors.Is inside every single RPC method, you should map "error types" rather than specific errors. The best architectural practice is to rely on server-side interceptors to build a global error mapping and defense line, coupled with the implementation of custom error type interfaces.

I. Three-Tier Error Model ​

Underlying Error (mysql/oss/http)
        ↓
Business Error (domain error)
        ↓
gRPC status code

II. Error Conversion Flow ​

For example, in a single RPC:

UploadAvatar RPC
      ↓
Call OSS
      ↓
Call DB
      ↓
Call Image Processing Service

Underlying returns:

oss.ErrNoSuchBucket
sql.ErrNoRows
context.DeadlineExceeded
http timeout

Do not map these directly.

Instead, convert them first into Business Errors:

ErrUploadFailed
ErrUserNotFound
ErrDependencyTimeout

Then, uniformly convert them into a gRPC code.

III. Example Architecture ​

1. Define Business Errors ​

var (
    ErrUserNotFound      = errors.New("user not found")
    ErrUploadFailed      = errors.New("upload failed")
    ErrDependencyTimeout = errors.New("dependency timeout")
)

2. Service Layer Converts Underlying Errors ​

For example:

func (s *Service) UploadAvatar(ctx context.Context, uid int64, file []byte) error {
    err := s.oss.Put(file)
    if err != nil {
        if errors.Is(err, context.DeadlineExceeded) {
            return ErrDependencyTimeout
        }
        return ErrUploadFailed
    }

    return nil
}

Note: Do not return gRPC errors here.

3. RPC Middleware Unifies gRPC Code Mapping ​

func ErrorInterceptor() grpc.UnaryServerInterceptor {
    return func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
        resp, err := handler(ctx, req)
        if err != nil {
            // Unify error mapping logic
            switch {
            case errors.Is(err, ErrUserNotFound):
                return nil, status.Error(codes.NotFound, err.Error())
            case errors.Is(err, ErrDependencyTimeout):
                return nil, status.Error(codes.DeadlineExceeded, "upstream service timeout")
            case errors.Is(err, ErrUploadFailed):
                return nil, status.Error(codes.Internal, "file upload failed")
            default:
                // Catch-all processing: log unknown errors and return a safe Internal error to avoid leakage
                log.Printf("unknown error: %v", err)
                return nil, status.Error(codes.Internal, "internal server error")
            }
        }
        return resp, nil
    }
}