Build an Internal Service Layer for Serverless with a Private API Gateway

When a serverless estate grows past one team, services start calling each other. The usual answer is a cross-team grant of lambda:InvokeFunction on someone else’s function ARN. That grant turns a function name into a public contract. The callee can no longer rename, split, or replace the function without breaking a consumer that lives in a repository they cannot see. The naive fix, wrapping the callee in an SDK client library, does not help: the coupling is in the IAM policy, not in the code.

Before any of that, answer a harder question: does the caller need the answer in-band at all? AWS’s own Lambda guide is blunt about the alternative. Under a heading called Anti-patterns in Lambda-based event-driven applications, it says that “Lambda functions directly calling other Lambda functions is generally an anti-pattern due to cost and complexity concerns”, and prescribes an SQS queue or Step Functions instead (event-driven architectures). The same page notes that “Most Lambda invocations in production systems are asynchronous”. Events are the default shape for serverless east-west traffic, and they are the right one. If nobody is waiting on the reply, stop reading and put a bus in the middle.

This post is about the calls that survive that filter. Some genuinely need a response: a checkout that must know whether the inventory reservation succeeded, a request that cannot be eventually consistent. AWS’s prescriptive guidance grants the point rather than scolding you, listing predictable flow control, strong consistency, and simple error handling as real benefits of synchronous calls (synchronous integration). Those calls exist in every estate. The question is not whether to make them, it is what to make them over.

You do not deploy a mesh for that traffic; you compose one out of primitives you already own. The default is a private REST API in API Gateway. Callers reach it through a shared execute-api interface VPC endpoint. A resource policy plus AWS_IAM authorization gates every call. That combination buys the three things a mesh is worth at this scale: stable addressing, per-route authorization, and per-call observability. No sidecars, no control plane. It is not an alternative to events. It is the replacement for the direct invoke that AWS just told you not to write.

What a cross-team direct invoke actually costs

Direct RequestResponse invoke looks free because nothing sits between the two functions. Four costs are hiding in that gap.

The function name becomes the contract. Team B’s execution role holds lambda:InvokeFunction on arn:aws:lambda:...:function:teamA-orders. Every rename, every split into two handlers, every migration to a different compute model breaks a policy that Team A does not own and cannot grep for.

You pay twice for the same second. A synchronous invoke from A to B leaves A idle, billed at its own memory tier, for the whole duration of B. Chain three services deep and the same milliseconds land on three bills.

There is no throttling contract. The callee cannot cap one caller at 200 requests per second and have the platform enforce it. Its only lever is reserved concurrency, which is blunt: it throttles every caller at once, including the callee’s own async triggers.

Telemetry stops at the function boundary. There are per-function metrics, but no per-route latency or error rate for the orders API as a whole. East-west traffic also mixes with north-south traffic in the same numbers.

One more detail is worth pinning, because stale figures circulate. The Lambda invocation payload quota is 6 MB each way for synchronous calls and 1 MB for asynchronous ones. It is not the 256 KB that older posts still quote (Lambda quotas). Direct invoke also changes retry semantics based on a flag the caller picked, not a contract the callee published.

The shape of the composed layer

The substitution at the heart of this pattern is small: the caller’s execution role holds execute-api:Invoke on a route ARN instead of lambda:InvokeFunction on a function ARN. The callee’s function ARN never leaves the callee’s stack.

Two gates stack in that path and both must allow the call. The API resource policy decides which VPC or VPC endpoint may reach the API at all. The method authorization decides which IAM principal may call which route. AWS also offers a third, optional gate: a VPC endpoint policy. AWS describes it as the most secure data perimeter option.

Private endpoints are REST-only

The API Gateway developer guide is direct about this in Considerations for private APIs: “Only REST APIs are supported” (Private REST APIs).

That single line kills the most common cost objection. HTTP APIs are cheaper per request, and community threads reach for them constantly, but there is no private endpoint type for an HTTP API. What HTTP APIs have is VPC Link V2, and it runs in the opposite direction. It lets a public API Gateway reach into your VPC to hit an ALB, an NLB, or a Cloud Map service (VPC links V2). That is a private integration, not a private endpoint. If the API itself must be unreachable from the internet, REST is the only flavour that does it, and you pay the REST rate.

Three other considerations from the same page belong in your design notes: private APIs accept TLS 1.2 only, HTTP/2 requests are enforced to HTTP/1.1, and you cannot restrict a private API to IPv4 only, since only dualstack is supported. One more comes from the endpoint types page: private API endpoints pass all header names through as-is (API endpoint types). Edge-optimized endpoints capitalize them, so a service that quietly depended on that capitalization can break when it moves behind a private API.

The resource policy is the on switch

A freshly created private API is not reachable. AWS says so in step 3 of Create a private API: “Your current private API is inaccessible to all VPCs” (Create a private API). Until a resource policy names your VPC or endpoint, every call fails. The failure looks enough like a DNS problem that people go debugging Route 53 for an hour.

The canonical shape is allow-then-deny, keyed on aws:SourceVpce (resource policy examples):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "execute-api:Invoke",
      "Resource": ["execute-api:/*"]
    },
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": "execute-api:Invoke",
      "Resource": ["execute-api:/*"],
      "Condition": {
        "StringNotEquals": { "aws:SourceVpce": "vpce-1a2b3c4d" }
      }
    }
  ]
}

The execute-api:/* shorthand expands to a full ARN when API Gateway saves the policy. The route-level form is execute-api:/{stage}/{METHOD}/{path}, which is how you scope a policy statement to one route rather than the whole API.

Per-route grants with AWS_IAM

With AWS_IAM authorization on a method, the caller signs the request with SigV4 (service name execute-api), and API Gateway evaluates the caller’s identity policy. The grant looks like this:

execute-api:Invoke  on  arn:aws:execute-api:{region}:{account}:{apiId}/{stage}/POST/orders

That is the whole authorization story, and it needs no new policy language. The callee can rewrite, rename, or replace the backing Lambda and not one consumer policy changes, because no consumer policy ever mentioned the function.

There is a trap here worth stating plainly, because the error message points somewhere else. If the resource policy names an AWS principal such as a role ARN, and any method is still on NONE authorization, API Gateway returns User: anonymous is not authorized to perform: execute-api:Invoke. The troubleshooting section of the Create a private API page states the fix: every method must use AWS_IAM authorization. Set it once as a default in CDK rather than per method, so a new route cannot be added without it.

The CDK stacks

Two stacks, because the ownership boundary is the point. The platform stack owns the VPC and one shared interface endpoint. AWS’s own best practice is to use a single VPC endpoint for multiple private APIs. That is what amortises the fixed cost across an entire estate.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';

export class PlatformNetworkStack extends cdk.Stack {
  public readonly vpc: ec2.Vpc;
  public readonly apiEndpoint: ec2.InterfaceVpcEndpoint;

  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Isolated subnets only: east-west traffic never needs a NAT gateway.
    // Calls to other AWS APIs (DynamoDB, SQS) need their own endpoints.
    this.vpc = new ec2.Vpc(this, 'Vpc', {
      maxAzs: 2,
      natGateways: 0,
      subnetConfiguration: [
        { name: 'isolated', subnetType: ec2.SubnetType.PRIVATE_ISOLATED, cidrMask: 24 },
      ],
    });

    // The endpoint ENI must accept 443 from the caller Lambdas.
    // Omit securityGroups and CDK defaults to open: true, which opens
    // 443 to the whole VPC CIDR. This narrows it to the caller SG.
    const endpointSg = new ec2.SecurityGroup(this, 'ApiEndpointSg', {
      vpc: this.vpc,
      description: 'execute-api interface endpoint',
      allowAllOutbound: false,
    });
    endpointSg.addIngressRule(
      ec2.Peer.ipv4(this.vpc.vpcCidrBlock),
      ec2.Port.tcp(443),
      'HTTPS from in-VPC callers',
    );

    // One endpoint, many private APIs. This is the fixed cost of the layer.
    this.apiEndpoint = new ec2.InterfaceVpcEndpoint(this, 'ExecuteApiEndpoint', {
      vpc: this.vpc,
      service: ec2.InterfaceVpcEndpointAwsService.APIGATEWAY,
      privateDnsEnabled: true,
      securityGroups: [endpointSg],
      subnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
    });
  }
}

The service stack owns the API and the handler. This is where the single most useful CDK detail lives: endpointConfiguration.vpcEndpoints does not write the resource policy. The CDK reference describes that prop as “A list of VPC Endpoints against which to create Route53 ALIASes” (EndpointConfiguration). The policy comes from either the policy prop or from grantInvokeFromVpcEndpointsOnly(), documented as adding “a resource policy that only allows API execution from a VPC Endpoint” (RestApi). Two props, two jobs. Setting only the first is why an API that looks correctly wired returns 403 on every call.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as apigateway from 'aws-cdk-lib/aws-apigateway';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as nodejs from 'aws-cdk-lib/aws-lambda-nodejs';
import * as logs from 'aws-cdk-lib/aws-logs';

interface OrdersServiceStackProps extends cdk.StackProps {
  readonly apiEndpoint: ec2.IInterfaceVpcEndpoint;
}

export class OrdersServiceStack extends cdk.Stack {
  public readonly api: apigateway.RestApi;

  constructor(scope: Construct, id: string, props: OrdersServiceStackProps) {
    super(scope, id, props);

    const handler = new nodejs.NodejsFunction(this, 'OrdersHandler', {
      entry: 'src/orders/handler.ts',
      runtime: lambda.Runtime.NODEJS_22_X,
      timeout: cdk.Duration.seconds(10),
    });

    this.api = new apigateway.RestApi(this, 'OrdersApi', {
      restApiName: 'orders',
      endpointConfiguration: {
        types: [apigateway.EndpointType.PRIVATE],
        // Route53 ALIAS records only. This does NOT create the resource policy.
        vpcEndpoints: [props.apiEndpoint],
      },
      // Applied to every method, so a new route cannot ship without auth.
      defaultMethodOptions: {
        authorizationType: apigateway.AuthorizationType.IAM,
      },
      deployOptions: {
        stageName: 'v1',
        tracingEnabled: true,
        metricsEnabled: true,
        // The throttling contract direct invoke cannot express.
        throttlingRateLimit: 200,
        throttlingBurstLimit: 400,
        accessLogDestination: new apigateway.LogGroupLogDestination(
          new logs.LogGroup(this, 'OrdersAccessLogs', {
            retention: logs.RetentionDays.ONE_MONTH,
          }),
        ),
        accessLogFormat: apigateway.AccessLogFormat.jsonWithStandardFields(),
      },
    });

    // This is what makes the API reachable. Without it, every VPC is denied.
    this.api.grantInvokeFromVpcEndpointsOnly([props.apiEndpoint]);

    this.api.root
      .addResource('orders')
      .addMethod('POST', new apigateway.LambdaIntegration(handler));
  }
}

The caller’s stack grants on the route, and arnForExecuteApi builds the ARN for you:

import * as iam from 'aws-cdk-lib/aws-iam';

const checkout = new nodejs.NodejsFunction(this, 'CheckoutHandler', {
  entry: 'src/checkout/handler.ts',
  runtime: lambda.Runtime.NODEJS_22_X,
  vpc: props.vpc,
  vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
  environment: {
    ORDERS_URL: props.ordersApi.urlForPath('/orders'),
  },
});

// A grant on a route, not on a function ARN.
checkout.addToRolePolicy(new iam.PolicyStatement({
  actions: ['execute-api:Invoke'],
  resources: [props.ordersApi.arnForExecuteApi('POST', '/orders', 'v1')],
}));

One deployment detail to plan around: associating or dissociating a VPC endpoint with a private API requires redeploying the API, and DNS propagation takes minutes. Order the platform stack before the service stacks.

Signing the call from a Node 22 Lambda

The caller signs with SigV4 against the service name execute-api. Not lambda, not apigateway. Wrong service name is a 403 with an unhelpful message.

import { SignatureV4 } from '@smithy/signature-v4';
import { HttpRequest } from '@smithy/protocol-http';
import { Sha256 } from '@aws-crypto/sha256-js';
import { defaultProvider } from '@aws-sdk/credential-provider-node';

const ordersUrl = new URL(process.env.ORDERS_URL!);

// Built once per execution environment, reused across invocations.
const signer = new SignatureV4({
  service: 'execute-api',
  region: process.env.AWS_REGION!,
  credentials: defaultProvider(),
  sha256: Sha256,
});

export const handler = async (event: { cartId: string }) => {
  const body = JSON.stringify({ cartId: event.cartId });

  const request = new HttpRequest({
    protocol: ordersUrl.protocol,
    hostname: ordersUrl.hostname,
    path: ordersUrl.pathname,
    method: 'POST',
    headers: {
      host: ordersUrl.hostname, // required in the signed headers
      'content-type': 'application/json',
    },
    body,
  });

  const signed = await signer.sign(request);

  // Global fetch; no HTTP client dependency needed on Node 22.
  const response = await fetch(ordersUrl, {
    method: signed.method,
    headers: signed.headers,
    body: signed.body,
  });

  if (!response.ok) {
    throw new Error(`orders returned ${response.status}: ${await response.text()}`);
  }

  return response.json();
};

The body you sign and the body you send must be byte-identical. Serialising twice, or letting a client library re-encode the payload, produces a payload hash mismatch and another 403.

SigV4 proves the service, not the user

That signature answers one question: which service is calling. It says nothing about which user the call is for. Those are two different questions, and the layer built so far only answers the first. A checkout Lambda signing with its execution role proves it is checkout-service. It does not prove that the cart it is asking about belongs to the person who opened the session.

Start with what the callee gets for free. Under AWS_IAM with a Lambda proxy integration, the caller’s IAM principal arrives in the event with no mapping template at all. AWS states it directly: “When the authorization type is AWS_IAM, the authorized user information includes $context.identity.* properties” (mapping template reference). The useful fields are $context.identity.userArn, defined as “The Amazon Resource Name (ARN) of the effective user identified after authentication”, plus .caller and .accountId. In a proxy integration they land under event.requestContext.identity. So the callee can add a rule of its own on top of the IAM grant: only the checkout-service role may post to /orders.

That is service identity. The user is a separate problem, and the gateway does not solve it for you.

The plain x-user-id header is a trap. It looks harmless next to a signed request, and it is an unauthenticated assertion. Any service that already holds execute-api:Invoke on the route can set that header to any value, and the callee acts on a claim it never verified. AWS has a name for this shape: the confused deputy, “a security issue where an entity that doesn’t have permission to perform an action can coerce a more-privileged entity to perform the action” (confused deputy). The service is authenticated. The subject is not. One careless caller is enough for any service to act as any user.

A method has one authorizationType, and you already spent it. The obvious fix is to have the gateway verify the user token as well. API Gateway will not do it. A REST API method carries exactly one authorizationType; the valid values are NONE, AWS_IAM, CUSTOM, and COGNITO_USER_POOLS, and an authorizer can only attach when the type is CUSTOM or COGNITO_USER_POOLS (Method). So AWS_IAM and a Cognito or Lambda authorizer are mutually exclusive on the same method. Once the gateway slot goes to service identity, there is no gateway-side verifier left for the user token. Verifying the subject inside the callee is therefore not a preference. It is the only place left.

That constraint forces the design: sign the subject, and verify the signature where the work happens. Two shapes work.

Forward the original token. Pass the user’s OIDC or Cognito access token in a custom header and verify it in the callee against the issuer’s JWKS endpoint. For Node, AWS points at the awslabs library: “In a Node.js app, AWS recommends the aws-jwt-verify library to validate the parameters in the token that your user passes to your app” (aws-jwt-verify). It is cheap: one dependency, no new infrastructure, JWKS cached in the execution environment. The cost is that the raw user token now travels across every hop, and its audience and scope were minted for the edge, not for orders.

Exchange the token at the edge. Trade the incoming token once, at the entry point, for a short-lived internal token that carries both the user and the calling workload down the chain. The IETF is formalising this shape as Transaction Tokens, “designed to maintain and propagate user identity, workload identity and authorization context throughout the Call Chain within a trusted domain” (Transaction Tokens). Be honest about its status: it is an IETF draft in working-group last call, not an RFC, and there is no AWS-managed implementation to switch on. Read it as the direction of travel and as a specification of what a hand-rolled internal token should carry, not as something you turn on this quarter.

The callee then runs both checks in the same handler:

import type { APIGatewayProxyEvent, APIGatewayProxyResult } from 'aws-lambda';
import { CognitoJwtVerifier } from 'aws-jwt-verify';

// Built once per execution environment; the JWKS is cached after the first fetch.
const verifier = CognitoJwtVerifier.create({
  userPoolId: process.env.USER_POOL_ID!,
  tokenUse: 'access',
  clientId: process.env.APP_CLIENT_ID!,
});

// Pinned at deploy time. Do not parse the ARN: its shape depends on how the
// caller obtained its credentials.
const allowedCallers = new Set(
  (process.env.ALLOWED_CALLER_ARNS ?? '').split(',').filter(Boolean),
);

export const handler = async (
  event: APIGatewayProxyEvent,
): Promise<APIGatewayProxyResult> => {
  // Check 1: which service is calling. API Gateway verified the SigV4 signature.
  const callerArn = event.requestContext.identity.userArn;
  if (!callerArn || !allowedCallers.has(callerArn)) {
    return { statusCode: 403, body: JSON.stringify({ message: 'caller not allowed' }) };
  }

  // Check 2: which user this call is for. Nothing above verified this.
  // Lower case name: private endpoints pass header names through as-is.
  const assertion = event.headers['x-user-assertion'];
  if (!assertion) {
    return { statusCode: 401, body: JSON.stringify({ message: 'missing user assertion' }) };
  }

  try {
    const claims = await verifier.verify(assertion);
    return { statusCode: 200, body: JSON.stringify({ subject: claims.sub }) };
  } catch {
    return { statusCode: 401, body: JSON.stringify({ message: 'invalid user assertion' }) };
  }
};

Two checks, not one. Which service is calling comes from IAM and arrives already verified. Which user this is for comes from a signed assertion the callee verifies itself. A layer that runs only the first check is a layer where any service can act as any user.

One field to leave out of the handler: $context.identity.vpceId exists, and AWS documents it as present only for private APIs, but the documented enforcement point for the endpoint is the resource policy with aws:SourceVpce, which the service stack above already sets. Keep the network condition in the policy. Keep the handler to the two identity checks.

The encryption layer between services

The transport is encrypted already. It is worth being exact about how much that covers, because the phrase people reach for next is usually wrong.

Three facts are solid. API Gateway exposes HTTPS endpoints only: “API Gateway doesn’t support unencrypted (HTTP) endpoints” (data protection). Private APIs accept TLS 1.2 only. And the path stays inside AWS: “traffic to your private API uses secure connections and is isolated from the public internet. Traffic doesn’t leave the Amazon network.”

It is not end-to-end encryption. TLS terminates at API Gateway. The integration is a second connection with its own handshake. In the all-Lambda shape this post recommends, that second leg is also encrypted, since “Lambda API endpoints only support secure connections over HTTPS” (Lambda data protection). But AWS nowhere asserts one unbroken channel from caller code to callee code, so do not claim it in a design review. Two TLS legs with a service boundary between them is the accurate description. The sharp edge lives on the second leg. For the HTTP integration types, API Gateway states that “each integration can specify a protocol (http/https), port and path” (Integration). Plain http is a legal choice there. Passing through API Gateway does not by itself mean the second leg was encrypted.

Mutual TLS is not available on this layer. “Mutual TLS isn’t supported for private APIs” (mutual TLS). mTLS in API Gateway requires a custom domain name, and private APIs are excluded from it. There is no property to set. If a compliance requirement asks for mTLS on east-west calls, the workaround is to terminate it upstream of the gateway. AWS has written that up (consuming private APIs using mutual TLS), and it is a blog pattern, not documented service behaviour. Label it that way when someone puts it in an architecture document.

The custom header is where the user token can leak. This follows straight from the identity section. Under AWS_IAM, SigV4 already owns the Authorization header, so a forwarded user assertion has to ride in a custom header such as x-user-assertion or x-id-token. Now read the redaction promise carefully: “API Gateway redacts authorization headers, API key values, and similar sensitive request parameters from the logged data” (set up logging). Arbitrary custom headers are not on that list. Turn on data tracing to debug a route, and the JWT sitting in that custom header can land in CloudWatch Logs in cleartext. This is not something AWS does by default: data tracing is opt-in, and AWS says it “can result in logging sensitive data” and that “We recommend that you don’t use Data tracing for production APIs”. The point is narrower and sharper than a default-on leak. The one header you were forced to use is the one header the redaction list does not cover. Keep data tracing off on any stage that carries a user assertion, and keep the assertion out of your own structured logs while you are at it.

Encrypt the field when the channel is not the boundary you need. A callee that must never see a PII value, or a control that has to survive a misconfigured log, calls for payload-level encryption instead. The AWS Encryption SDK is “a client-side encryption library designed to make it easy for everyone to encrypt and decrypt data using industry standards and best practices” (AWS Encryption SDK), built on envelope encryption with KMS wrapping keys. The caller encrypts the field, the gateway and the logs see ciphertext, and only a principal holding kms:Decrypt on the wrapping key can read it. This is defense in depth, not the default for an internal layer. Reach for it per field, not per service, and only where cleartext on the second leg or in a log line is genuinely unacceptable.

Callers must be VPC-attached

The caller resolves and reaches the interface endpoint from inside the VPC, so it has to be attached to one. The reflex objection is that VPC Lambdas cold-start slowly. That has not been true since the Hyperplane rework. The current documentation states that the ENI is created when the function is created or its VPC settings are updated, not at invoke time. Each Hyperplane ENI supports up to 65,000 connections, shared across functions that use the same subnet and security group combination (Lambda VPC networking). The 2019 announcement recorded a cold start dropping from 14.8 seconds to 933 milliseconds in an X-Ray trace (improved VPC networking). That is historical context for why the myth died, not a benchmark to plan against.

Three operational consequences follow. A newly created VPC function stays in Pending until its ENI is ready, which can take minutes, so budget for it in a cold pipeline. Reusing one subnet plus security group combination across your functions lets Lambda share ENIs instead of minting new ones.

The third one bites a sparse internal service layer specifically. If a VPC function sits idle for 14 days, Lambda reclaims the unused Hyperplane ENI and moves the function to Inactive. The next invocation fails and the function re-enters Pending. A rarely-called internal endpoint is exactly that traffic shape, so treat the first call after a quiet fortnight as a failure the caller has to retry.

What the layer costs

Prices below are us-east-1 list rates, read on 2026-07-14. Re-run them against your own Region before you commit.

Component	Rate	Source
Private REST API requests	$3.50 per million (first 333M/month)	API Gateway pricing
Interface VPC endpoint	$0.01 per endpoint per AZ-hour	PrivateLink pricing
PrivateLink data processing	$0.01 per GB (first PB/month)	PrivateLink pricing

The fixed part is small and shared. One endpoint across two AZs runs about $14.60 per month (2 x $0.01 x 730 hours), and that one endpoint serves every private API in the VPC. Partial endpoint-hours bill as full hours, so the floor is real but flat.

The variable part is the honest tax. REST is the most expensive API Gateway tier, and it is the only tier that can be private. There is no escape to the cheaper HTTP API rate. At 50 million internal calls per month across the whole estate you are paying roughly $175 per month for plumbing that used to appear free. State that number to your team before someone finds it in a bill. Note the shape of that example: 50 million spread over a dozen services is a different bill from 50 million through one service, and the next section is about why.

It is worth paying when the calls are a contract boundary between teams. The $175 buys per-route authorization, a throttle limit the callee sets, and per-route latency and error metrics. It is not worth paying inside a single service’s own boundary. Two Lambdas owned by the same team, deployed by the same pipeline, with no independent consumer, should keep calling each other directly. Better still, they should stop being two Lambdas.

When to override the default

Before choosing any transport, ask whether the call needs a response at all. If the caller does not need the callee’s answer in-band, EventBridge or SQS is cheaper, more available, and removes the coupling instead of formalising it. Only a genuine request/response call belongs on this layer.

VPC Lattice is the credible competitor, and the crossover is arithmetic rather than taste. Lattice charges $0.025 per service per hour, roughly $18.25 per service per month, plus $0.025 per GB processed. Requests are free for the first 300,000 per hour per service, then $0.10 per additional million (VPC Lattice pricing, read 2026-07-14).

Read those two structures against each other. Lattice’s fixed fee scales linearly with service count. AWS’s own worked example puts a 100-service estate at about $1,825 per month in service-hours, before a single request arrives. API Gateway’s fixed cost does not scale that way: one shared endpoint, tens of dollars, however many APIs sit behind it. But Lattice’s request pricing is about 35 times cheaper per million. Its free tier of 300,000 requests per hour works out to about 219 million requests per month per service at no request charge, though only if the traffic is evenly spread. The allowance is hourly, so a spiky service can pay for a busy hour while sitting far below that monthly figure.

Per service, and ignoring data processing on both sides, the crossover is this. Lattice’s $18.25 monthly fee is repaid once API Gateway’s $3.50 per million would have cost more, which happens at roughly 5.2 million requests per service per month. Below that, the private API is cheaper. Above it, Lattice pulls away quickly. Two conditions take the override regardless of volume: targets that are not all Lambda (Lattice target groups cover ECS, EKS, EC2, ALB, and IP targets), and the network-owner versus service-owner role split that Lattice’s service network is built around. Note what is not on that list: gRPC. Switching to Lattice does not buy you gRPC while your compute is still Lambda functions, for reasons the next section spells out.

ALB with Lambda targets is cheaper per request than a REST API, and it is not a contract layer. It has no IAM authorization, no per-route usage plan, and an hourly plus LCU bill of its own. It balances load; it does not publish a contract.

Direct invoke remains correct inside a single service’s own boundary, where the caller and the callee ship together. It stops being correct at a team boundary. That is the whole rule.

Why gRPC is not on the table

Anyone arriving from the service-mesh world asks this within a minute. gRPC is the default east-west transport there: a typed contract, generated clients, streaming, one connection carrying many calls. It is a fair question, and the answer is not preference. It is structural.

gRPC needs HTTP/2, and a private REST API refuses to speak it. The gRPC wire spec is titled gRPC over HTTP2 and states that it “serves as a detailed description for an implementation of gRPC carried over HTTP2 framing” (gRPC over HTTP/2). The dependency is not cosmetic. A call terminates in HTTP/2 trailers, and the spec is blunt about it: “Status must be sent in Trailers even if the status code is OK.” HTTP/1.1 has no equivalent. The spec even ships a tripwire for intermediaries that get this wrong, defining TE: trailers as “Used to detect incompatible proxies”.

Now recall the consideration from earlier: on a private API, “If you make a request using HTTP/2 protocol, the request is enforced to use HTTP/1.1 protocol.” That single line settles it. The transport this post recommends downgrades away the exact protocol gRPC is carried over. AWS never publishes a sentence saying API Gateway does not support gRPC, so do not go looking for one. The downgrade is the proof.

Lattice does not rescue you, and this is the part people get wrong. The intuitive next move is to reach for VPC Lattice, which genuinely does support gRPC. Read the constraint before you plan around it. For the gRPC protocol version, Lattice says “The only supported target types are INSTANCE and IP”, and then, plainly: “You can’t use Lambda functions as targets” (Lattice target groups). ALB carries the identical exclusion for its own gRPC target groups. The HTTP/2 protocol version has the same target-type restriction on both.

That is the pattern worth internalising: every AWS surface that speaks gRPC excludes Lambda function targets. Lattice is not a gateway swap that unlocks gRPC. If you need gRPC east-west, you are changing your compute, not your gateway.

The reason underneath is Lambda’s execution model. A Lambda function is not a server. There is no process listening on a port, and nothing to hold a connection open: “Lambda freezes the execution environment when the runtime and each extension have completed and there are no pending events” (execution environment). Everything gRPC is prized for, a persistent connection, multiplexed streams, bidirectional streaming, presumes a long-lived server holding that connection. A frozen, request-and-response environment cannot offer it. Lambda’s response streaming is a different mechanism and gives you none of it: no gRPC framing, no trailers, no client streaming.

Be precise about the claim, because a sharp reader will test it. Lambda functions cannot terminate gRPC. Lambda MicroVMs can, and their networking documentation lists gRPC among the supported inbound protocols. That is not a counterexample. A MicroVM gives you a long-lived server process on a port, which is exactly the shape gRPC needs, and its existence confirms the rule rather than breaking it.

Keep the contract, drop the transport. Strip gRPC down and most teams want the typed contract and the generated clients, not bidirectional streaming between two Lambdas. You can keep all of that. Write .proto schemas, generate the clients, and ship protobuf or JSON over an ordinary HTTP/1.1 POST through the private API. The Connect protocol is built for exactly this shape and is explicit about it: “Bidirectional streaming requires HTTP/2, but the other RPC types also support HTTP/1.1. The protocol doesn’t use HTTP trailers at all, so it works with any networking infrastructure” (Connect protocol). What you give up is streaming and multiplexing. Lambda’s execution model was never going to give you those anyway, so the loss is smaller than it sounds.

The decision rule is short. gRPC east-west is a reason to move that service onto a long-lived container behind an ALB or Lattice with IP targets. It is not a reason to switch gateways, and it is not something a private API Gateway can be talked into.

One caveat so the picture stays honest. Lambda MicroVMs, launched in June 2026, do hand each MicroVM an HTTPS URL that speaks HTTP/2, gRPC, and WebSockets. AWS positions them as isolated sandboxes for untrusted or AI-generated code, not as an east-west transport, so do not read them as AWS steering internal traffic toward gRPC. Read them as confirmation of the rule: the protocol arrives exactly when a long-lived server process does.

The wire format is not the contract

Ask a team why they want gRPC and the honest answer is rarely HTTP/2 multiplexing. It is schema discipline: one source of truth for the payload, generated clients in every language, and a CI gate that fails the build when someone changes a field’s type. That is a contract concern. gRPC bundles it with a transport, and the transport is the part Lambda cannot take. The good news is that the bundle comes apart.

Protobuf is an encoding, not a transport. You can send binary protobuf over an ordinary HTTP/1.1 POST. This is not a hack, it is specified. The Connect protocol “does not depend on framing details specific to a particular HTTP version” and its own reference shows a plain POST /service/Method HTTP/1.1 carrying Content-Type: application/proto with an uncompressed binary protobuf body (Connect protocol). AWS agrees in its own house: the Smithy RPC v2 CBOR protocol declares http: ["h2", "http/1.1"] and moves payloads as application/cbor (Smithy RPC v2 CBOR). Binary payloads do not require HTTP/2. Only gRPC’s framing does.

But API Gateway taxes the bytes, and this is the trap. A REST API is text-first by default: “By default, the RestApi supports only UTF-8-encoded text payloads.” Post a protobuf body to a route that has not enrolled the content type in binaryMediaTypes, and the gateway treats your bytes as UTF-8 text and mangles them. Enroll the type properly and you meet the second rule: “When converting a binary payload to a text string, API Gateway always applies a base64 encoding on the binary data” (binary media types). The Lambda proxy event carries a JSON body string plus an isBase64Encoded flag, so a binary body arrives base64-encoded and your handler decodes it before protobuf ever sees it.

Base64 is four characters for every three bytes. That is a fixed one-third inflation, by definition, plus a decode you pay in the callee. Protobuf usually still lands smaller than the equivalent JSON, because it starts far below it, but the platform is quietly refunding a large part of what the binary format won. On this transport, “we switched to protobuf to shrink the payload” is a weaker argument than it sounds.

So take the contract and leave the bytes. Write the schema, generate the clients, and gate breaking changes in CI. buf breaking compares a schema against its previous version and fails on a change that would break existing clients, reporting things like a field changing type from int32 to string (Buf). None of that needs gRPC, HTTP/2, or even a binary body. It is the part you actually wanted. Ship JSON over the private API, keep the schema as the source of truth, and reach for binary only where a measured payload justifies paying the base64 tax.

JSON stays the reasonable default here, and not out of laziness. It is the platform’s native currency: the proxy event is already JSON, and CloudWatch Logs, X-Ray, and a console test invoke all read it without a decoder ring. For the contract layer, OpenAPI is the standard for HTTP and JSON APIs, now at version 3.2.0 (OpenAPI Specification).

The industry’s direction is worth naming, because AWS is the proof of it. The direction is schema-first contracts with a swappable wire format. Smithy is “a language for defining services and SDKs”, an IDL AWS says has been “widely used within Amazon and AWS for over a decade” (Smithy). The contract is the IDL; the protocol is a trait you can change. And AWS is changing it: service models such as CloudWatch and GameLift now declare smithy.protocols#rpcv2Cbor alongside their legacy protocol, and CBOR itself is a full Internet Standard, STD 94 (RFC 8949). This is a migration in flight, per service and per language, not a finished switch. The lesson generalises cleanly: AWS can move its own wire format under its own SDKs precisely because the contract was never the wire format. Build your east-west layer the same way and the format stays a decision you can revisit.

Where this is all heading

It would be easy to read this post as a serverless workaround: no mesh available, so improvise. That reading is backwards. The mesh world is walking toward the same conclusion from the other side, and AWS has already arrived.

AWS is retiring its own service mesh. The App Mesh user guide carries the notice on every page: “On September 30, 2026, AWS will discontinue support for AWS App Mesh. After September 30, 2026, you will no longer be able to access the AWS App Mesh console or AWS App Mesh resources” (App Mesh). New customers were shut out back in September 2024. AWS built a sidecar mesh, ran it for years, and is switching it off. That is not a small signal about where managed east-west connectivity is going.

Be precise about what AWS replaced it with, because the obvious summary is wrong. AWS did not abolish the proxy. For ECS it points customers at Service Connect, where, in AWS’s words, “the Envoy proxy, known as the Service Connect Proxy, is fully managed by AWS”, against App Mesh’s self-managed sidecar. Envoy is still in the task. It is simply no longer your problem. The retired idea is not the sidecar, it is you operating the sidecar.

The open-source mesh is moving the same way. Istio’s ambient mode reached GA in November 2024, and its pitch is a mesh without per-pod proxies: L4 handled by a per-node ztunnel, L7 proxying opt-in and separately scaled (ambient mode). Istio’s own motivation reads like a bill of indictment against the sidecar: proxies must be injected by rewriting the pod spec, upgrades mean restarting application pods, and “the CPU and memory resources must be provisioned for worst case usage of each individual pod” (introducing ambient mesh). Cilium reaches the same place through eBPF with a per-node Envoy rather than a per-pod one.

Do not overstate it, though, because it is not unanimous. Istio still supports sidecars and says they “are not going away”. Linkerd, the other CNCF graduated mesh, argues the opposite outright and is fixing sidecar lifecycle problems rather than removing the proxy. The honest claim is not that the industry abandoned sidecars. It is that the centre of gravity is moving off the per-workload proxy and into the platform.

Which is the whole point. Both worlds have concluded the same thing: identity, authorization, encryption, and telemetry for east-west calls belong to the platform, not bolted to the workload. Istio ambient is Kubernetes working to acquire a property serverless had by construction. Lambda never had a sidecar to remove. The gateway, IAM, and the managed network are the platform layer, and this post is just wiring them together.

The convergence is real, but the implementations are diverging, and that is the trade you are signing. Kubernetes is standardising on portable, vendor-neutral specifications that you operate yourself: SPIFFE and SPIRE for workload identity, a CNCF graduated project since 2022, and OpenTelemetry, graduated in May 2026, for tracing across service boundaries. Serverless standardises on cloud primitives that you do not operate and cannot move: IAM SigV4, a private API Gateway, EventBridge. SigV4 and SPIFFE answer the same question, which workload is calling, with the same shape of answer, cryptographic and short-lived. The difference is who owns the trust root. You are buying an operator-free platform and paying in portability. That is a defensible trade, and it should be a conscious one.

One gap is worth admitting rather than papering over. The industry is settling on a two-part answer to identity on an internal call, and only half of it has landed. SPIFFE covers which workload is calling. For the other half, on whose behalf, the IETF’s Transaction Tokens draft is explicit that the two are not interchangeable: “A workload MUST NOT use a transaction token to authenticate itself to another workload, service or the TTS. Transaction tokens represents information relevant to authorization decisions and are not workload identity credentials” (Transaction Tokens). That is exactly the two-check rule this post arrived at from the AWS side. The difference is that the standard is still a working-group draft, so for now you build the second check yourself.

Trade-offs you are signing up for

The account-level throttle bucket is the one most posts miss. API Gateway allows 10,000 requests per second per account per Region, shared across REST, HTTP, and WebSocket APIs. A documented list of smaller Regions carries a lower 2,500 RPS default (API Gateway quotas). Move east-west traffic onto API Gateway and it now competes with your public APIs for the same bucket. Track headroom before you migrate the tenth service.

The integration timeout defaults to 29 seconds. Raising it for Regional and private REST APIs is possible, but AWS notes that doing so “might require a reduction in your account-level throttle quota limit” (What’s New, June 2024). Direct invoke has no such ceiling under the 15-minute function timeout. Long-running east-west calls do not belong behind this layer; they belong in Step Functions or an async pattern.

Private DNS carries a trade-off too. With private DNS enabled for execute-api in a VPC, that VPC can no longer reach the default endpoint of public API Gateway APIs. A private hosted zone workaround is the way around it. If the same functions call a public third-party API hosted on API Gateway, know this before you flip the switch. The alternative addressing form is the endpoint-associated alias, https://{rest-api-id}-{vpce-id}.execute-api.{region}.amazonaws.com/{stage} (invoke a private API).

Finally, VPC attachment is a one-time architectural commitment: subnets, security groups, an ENI budget. Cheap once, not free.

Common pitfalls

1. Reaching for an HTTP API because it is cheaper. There is no private endpoint type for HTTP APIs. Use a REST API with EndpointType.PRIVATE and budget for $3.50 per million.
2. Setting vpcEndpoints in endpointConfiguration and assuming the resource policy came with it. It only creates Route 53 aliases. Call grantInvokeFromVpcEndpointsOnly() or set policy.
3. Naming an IAM role as Principal in the resource policy while a method still uses NONE authorization. The result is User: anonymous is not authorized to perform: execute-api:Invoke. Put AWS_IAM in defaultMethodOptions.
4. Leaving the interface endpoint on the default security group. Its ENI needs an explicit rule accepting 443 from the callers.
5. Signing with the wrong service name. It is execute-api.
6. Building one VPC endpoint per API. AWS’s best practice is one endpoint, many private APIs; doing it per API multiplies the fixed cost by N.
7. Quoting the 256 KB async payload limit. The current quota is 1 MB async and 6 MB each way sync.
8. Assuming VPC-attached Lambdas cold-start slowly. Cite the current ENI docs, not 2018 blog posts.
9. Trusting an x-user-id header because the request was signed. SigV4 proves the calling service and nothing about the subject. Forward a signed token and verify it in the callee.
10. Turning on data tracing for a stage whose calls carry a user assertion. Redaction covers authorization headers and API key values, not the custom header AWS_IAM forced you into.
11. Planning a migration to VPC Lattice in order to get gRPC while the compute stays on Lambda functions. Lattice’s gRPC target types are INSTANCE and IP only, and it says so: “You can’t use Lambda functions as targets.”

What I plan to measure

The POC is not deployed, so these are targets rather than results. Gateway overhead per call is the first number: Latency minus IntegrationLatency at p50 and p99, per route. That difference is exactly what the per-call tax buys or costs. Then the monthly bill: requests times $3.50 per million, plus endpoint AZ-hours, plus PrivateLink GB processed, set against the Lattice service-hour equivalent at the estate’s real service count. Then per-route 4XXError and 5XXError rates, which direct invoke cannot produce at all. And one adoption metric that matters more than any of them: the count of cross-team lambda:InvokeFunction grants, where the target is zero.

Where this default holds

The default holds for a multi-team serverless estate with all-Lambda targets, request/response calls, and moderate volume. That default is a private REST API behind one shared execute-api interface endpoint, with AWS_IAM per-route grants. It gives you a mesh’s authorization and observability story with no sidecar, no control plane, and a fixed cost measured in tens of dollars.

Apply the filters in order, because the first one removes the most work. If nobody is waiting on the reply, use an event bus and none of this applies. If the targets are not all Lambda, or a service crosses roughly 5 million calls per month, Lattice starts to win. If you need gRPC, you are changing compute, not gateways. And whatever you ship the bytes as, keep the schema as the contract, because that is the part you will still be living with when the wire format changes.

The first step is cheaper than the migration: count the cross-team lambda:InvokeFunction grants in your account today. That number is the size of the coupling you are carrying.

References

• Private REST APIs in API Gateway - The load-bearing page: “Only REST APIs are supported”, plus the best practice of one endpoint for many APIs and the TLS 1.2 and HTTP/1.1 downgrade considerations.
• Create a private API - “Your current private API is inaccessible to all VPCs”, the resource policy step, and the anonymous-principal troubleshooting entry that forces AWS_IAM on every method.
• gRPC over HTTP/2 - The wire spec. gRPC is carried over HTTP/2 framing, status is returned in trailers, and TE: trailers exists to detect incompatible proxies.
• VPC Lattice target groups - For the gRPC protocol version, “The only supported target types are INSTANCE and IP” and “You can’t use Lambda functions as targets”. The reason Lattice is not a gRPC escape hatch for Lambda.
• CDK: class RestApi - grantInvokeFromVpcEndpointsOnly() and arnForExecuteApi(), the two methods that carry this pattern.
• CDK: interface EndpointConfiguration - vpcEndpoints is “A list of VPC Endpoints against which to create Route53 ALIASes”; it does not create the resource policy.
• API Gateway REST API reference: Method - The single authorizationType per method and the rule that an authorizer only attaches to CUSTOM or COGNITO_USER_POOLS. The constraint that forces user verification into the callee.
• Event-driven architectures in Lambda - AWS’s own anti-pattern section: “Lambda functions directly calling other Lambda functions is generally an anti-pattern”, and the statement that most production invocations are asynchronous. The reason events are the default and this layer is the exception.
• The confused deputy problem - AWS’s definition of the failure an unsigned x-user-id header creates.
• IETF: OAuth 2.0 Transaction Tokens - The emerging standard for “on whose behalf”, and explicit that a transaction token is not a workload identity credential. Still a working-group draft.
• Configuring mutual TLS authentication for a REST API - “Mutual TLS isn’t supported for private APIs”, plus the custom domain name requirement behind that exclusion.
• Set up CloudWatch logging for a REST API - The scope of header redaction and the warning against data tracing on production APIs.
• Binary media types in API Gateway - A REST API is UTF-8 text by default, and “API Gateway always applies a base64 encoding on the binary data”. The one-third tax on any protobuf or CBOR payload through this transport.
• AWS App Mesh - “On September 30, 2026, AWS will discontinue support for AWS App Mesh.” AWS is switching off its own sidecar mesh.
• Smithy - AWS’s IDL, “widely used within Amazon and AWS for over a decade”. The contract is the model; the wire protocol is a swappable trait, which is why AWS can migrate its own SDKs to CBOR.