Build an Internal Service Layer with a Private API Gateway

When a serverless estate grows past one team, services start calling each other. The usual answer is a cross-team grant of lambda:InvokeFunction on someone else’s function ARN. That grant turns a function name into a public contract. The callee can no longer rename, split, or replace the function without breaking a consumer that lives in a repository they cannot see. The naive fix, wrapping the callee in an SDK client library, does not help: the coupling is in the IAM policy, not in the code.

Before any of that, answer a harder question: does the caller need the answer in-band at all? AWS’s own Lambda guide is blunt about the alternative. Under a heading called Anti-patterns in Lambda-based event-driven applications, it says that “Lambda functions directly calling other Lambda functions is generally an anti-pattern due to cost and complexity concerns”, and prescribes an SQS queue or Step Functions instead (event-driven architectures). The same page notes that “Most Lambda invocations in production systems are asynchronous”. Events are the default shape for serverless east-west traffic, and they are the right one. If nobody is waiting on the reply, stop reading and put a bus in the middle.

This series is about the calls that survive that filter. Some genuinely need a response: a checkout that must know whether the inventory reservation succeeded, a request that cannot be eventually consistent. AWS’s prescriptive guidance grants the point rather than scolding you, listing predictable flow control, strong consistency, and simple error handling as real benefits of synchronous calls (synchronous integration). Those calls exist in every estate. The question is not whether to make them, it is what to make them over.

You do not deploy a mesh for that traffic; you compose one out of primitives you already own. The default is a private REST API in API Gateway. Callers reach it through a shared execute-api interface VPC endpoint. A resource policy plus AWS_IAM authorization gates every call. No sidecars, no control plane. It is not an alternative to events. It is the replacement for the direct invoke that AWS just told you not to write.

Two things make it the default, and neither is identity. The fixed cost is shared: one interface endpoint serves every private API in the VPC for tens of dollars a month, while the obvious competitor bills per service. And API Gateway is an API management layer, so the callee can throttle a route, validate a request against a schema, and publish a contract. Those are the arguments, and part 104 makes them properly against VPC Lattice, which does the identity half at least as well as this does. Be clear about that up front: if you are choosing this layer because you think it is the only way to get per-route IAM authorization on an internal call, you are choosing it for the wrong reason.

What a cross-team direct invoke actually costs

Direct RequestResponse invoke looks free because nothing sits between the two functions. Four costs are hiding in that gap.

The function name becomes the contract. Team B’s execution role holds lambda:InvokeFunction on arn:aws:lambda:...:function:teamA-orders. Every rename, every split into two handlers, every migration to a different compute model breaks a policy that Team A does not own and cannot grep for.

You pay twice for the same second. A synchronous invoke from A to B leaves A idle, billed at its own memory tier, for the whole duration of B. Chain three services deep and the same milliseconds land on three bills.

There is no throttling contract. The callee cannot cap one caller at 200 requests per second and have the platform enforce it. Its only lever is reserved concurrency, which is blunt: it throttles every caller at once, including the callee’s own async triggers.

Telemetry stops at the function boundary. There are per-function metrics, but no per-route latency or error rate for the orders API as a whole. East-west traffic also mixes with north-south traffic in the same numbers.

One more detail is worth pinning, because stale figures circulate. The Lambda invocation payload quota is 6 MB each way for synchronous calls and 1 MB for asynchronous ones. It is not the 256 KB that older posts still quote (Lambda quotas). Direct invoke also changes retry semantics based on a flag the caller picked, not a contract the callee published.

The shape of the composed layer

The substitution at the heart of this pattern is small: the caller’s execution role holds execute-api:Invoke on a route ARN instead of lambda:InvokeFunction on a function ARN. The callee’s function ARN never leaves the callee’s stack.

Two gates stack in that path and both must allow the call. The API resource policy decides which VPC or VPC endpoint may reach the API at all. The method authorization decides which IAM principal may call which route. AWS also offers a third, optional gate: a VPC endpoint policy. AWS describes it as the most secure data perimeter option.

Private endpoints are REST-only

The API Gateway developer guide is direct about this in Considerations for private APIs: “Only REST APIs are supported” (Private REST APIs).

That single line kills the most common cost objection. HTTP APIs are cheaper per request, and community threads reach for them constantly, but there is no private endpoint type for an HTTP API. What HTTP APIs have is VPC Link V2, and it runs in the opposite direction. It lets a public API Gateway reach into your VPC to hit an ALB, an NLB, or a Cloud Map service (VPC links V2). That is a private integration, not a private endpoint. If the API itself must be unreachable from the internet, REST is the only flavour that does it, and you pay the REST rate.

Three other considerations from the same page belong in your design notes: private APIs accept TLS 1.2 only, HTTP/2 requests are enforced to HTTP/1.1, and you cannot restrict a private API to IPv4 only, since only dualstack is supported. One more comes from the endpoint types page: private API endpoints pass all header names through as-is (API endpoint types). Edge-optimized endpoints capitalize them, so a service that quietly depended on that capitalization can break when it moves behind a private API.

The resource policy is the on switch

A freshly created private API is not reachable. AWS says so in step 3 of Create a private API: “Your current private API is inaccessible to all VPCs” (Create a private API). Until a resource policy names your VPC or endpoint, every call fails. The failure looks enough like a DNS problem that people go debugging Route 53 for an hour.

The canonical shape is allow-then-deny, keyed on aws:SourceVpce (resource policy examples):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "execute-api:Invoke",
      "Resource": ["execute-api:/*"]
    },
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": "execute-api:Invoke",
      "Resource": ["execute-api:/*"],
      "Condition": {
        "StringNotEquals": { "aws:SourceVpce": "vpce-1a2b3c4d" }
      }
    }
  ]
}

The execute-api:/* shorthand expands to a full ARN when API Gateway saves the policy. The route-level form is execute-api:/{stage}/{METHOD}/{path}, which is how you scope a policy statement to one route rather than the whole API.

Per-route grants with AWS_IAM

With AWS_IAM authorization on a method, the caller signs the request with SigV4 (service name execute-api), and API Gateway evaluates the caller’s identity policy. The grant looks like this:

execute-api:Invoke  on  arn:aws:execute-api:{region}:{account}:{apiId}/{stage}/POST/orders

That is the whole authorization story, and it needs no new policy language. The callee can rewrite, rename, or replace the backing Lambda and not one consumer policy changes, because no consumer policy ever mentioned the function.

There is a trap here worth stating plainly, because the error message points somewhere else. If the resource policy names an AWS principal such as a role ARN, and any method is still on NONE authorization, API Gateway returns User: anonymous is not authorized to perform: execute-api:Invoke. The troubleshooting section of the Create a private API page states the fix: every method must use AWS_IAM authorization. Set it once as a default in CDK rather than per method, so a new route cannot be added without it.

The CDK stacks

Two stacks, because the ownership boundary is the point. The platform stack owns the VPC and one shared interface endpoint. AWS’s own best practice is to use a single VPC endpoint for multiple private APIs. That is what amortises the fixed cost across an entire estate.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';

export class PlatformNetworkStack extends cdk.Stack {
  public readonly vpc: ec2.Vpc;
  public readonly apiEndpoint: ec2.InterfaceVpcEndpoint;

  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Isolated subnets only: east-west traffic never needs a NAT gateway.
    // Calls to other AWS APIs (DynamoDB, SQS) need their own endpoints.
    this.vpc = new ec2.Vpc(this, 'Vpc', {
      maxAzs: 2,
      natGateways: 0,
      subnetConfiguration: [
        { name: 'isolated', subnetType: ec2.SubnetType.PRIVATE_ISOLATED, cidrMask: 24 },
      ],
    });

    // The endpoint ENI must accept 443 from the caller Lambdas.
    // Omit securityGroups and CDK defaults to open: true, which opens
    // 443 to the whole VPC CIDR. This narrows it to the caller SG.
    const endpointSg = new ec2.SecurityGroup(this, 'ApiEndpointSg', {
      vpc: this.vpc,
      description: 'execute-api interface endpoint',
      allowAllOutbound: false,
    });
    endpointSg.addIngressRule(
      ec2.Peer.ipv4(this.vpc.vpcCidrBlock),
      ec2.Port.tcp(443),
      'HTTPS from in-VPC callers',
    );

    // One endpoint, many private APIs. This is the fixed cost of the layer.
    this.apiEndpoint = new ec2.InterfaceVpcEndpoint(this, 'ExecuteApiEndpoint', {
      vpc: this.vpc,
      service: ec2.InterfaceVpcEndpointAwsService.APIGATEWAY,
      privateDnsEnabled: true,
      securityGroups: [endpointSg],
      subnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
    });
  }
}

The service stack owns the API and the handler. This is where the single most useful CDK detail lives: endpointConfiguration.vpcEndpoints does not write the resource policy. The CDK reference describes that prop as “A list of VPC Endpoints against which to create Route53 ALIASes” (EndpointConfiguration). The policy comes from either the policy prop or from grantInvokeFromVpcEndpointsOnly(), documented as adding “a resource policy that only allows API execution from a VPC Endpoint” (RestApi). Two props, two jobs. Setting only the first is why an API that looks correctly wired returns 403 on every call.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as apigateway from 'aws-cdk-lib/aws-apigateway';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as nodejs from 'aws-cdk-lib/aws-lambda-nodejs';
import * as logs from 'aws-cdk-lib/aws-logs';

interface OrdersServiceStackProps extends cdk.StackProps {
  readonly apiEndpoint: ec2.IInterfaceVpcEndpoint;
}

export class OrdersServiceStack extends cdk.Stack {
  public readonly api: apigateway.RestApi;

  constructor(scope: Construct, id: string, props: OrdersServiceStackProps) {
    super(scope, id, props);

    const handler = new nodejs.NodejsFunction(this, 'OrdersHandler', {
      entry: 'src/orders/handler.ts',
      runtime: lambda.Runtime.NODEJS_22_X,
      timeout: cdk.Duration.seconds(10),
    });

    this.api = new apigateway.RestApi(this, 'OrdersApi', {
      restApiName: 'orders',
      endpointConfiguration: {
        types: [apigateway.EndpointType.PRIVATE],
        // Route53 ALIAS records only. This does NOT create the resource policy.
        vpcEndpoints: [props.apiEndpoint],
      },
      // Applied to every method, so a new route cannot ship without auth.
      defaultMethodOptions: {
        authorizationType: apigateway.AuthorizationType.IAM,
      },
      deployOptions: {
        stageName: 'v1',
        tracingEnabled: true,
        metricsEnabled: true,
        // The throttling contract direct invoke cannot express.
        throttlingRateLimit: 200,
        throttlingBurstLimit: 400,
        accessLogDestination: new apigateway.LogGroupLogDestination(
          new logs.LogGroup(this, 'OrdersAccessLogs', {
            retention: logs.RetentionDays.ONE_MONTH,
          }),
        ),
        accessLogFormat: apigateway.AccessLogFormat.jsonWithStandardFields(),
      },
    });

    // This is what makes the API reachable. Without it, every VPC is denied.
    this.api.grantInvokeFromVpcEndpointsOnly([props.apiEndpoint]);

    this.api.root
      .addResource('orders')
      .addMethod('POST', new apigateway.LambdaIntegration(handler));
  }
}

The caller’s stack grants on the route, and arnForExecuteApi builds the ARN for you:

import * as iam from 'aws-cdk-lib/aws-iam';

const checkout = new nodejs.NodejsFunction(this, 'CheckoutHandler', {
  entry: 'src/checkout/handler.ts',
  runtime: lambda.Runtime.NODEJS_22_X,
  vpc: props.vpc,
  vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
  environment: {
    ORDERS_URL: props.ordersApi.urlForPath('/orders'),
  },
});

// A grant on a route, not on a function ARN.
checkout.addToRolePolicy(new iam.PolicyStatement({
  actions: ['execute-api:Invoke'],
  resources: [props.ordersApi.arnForExecuteApi('POST', '/orders', 'v1')],
}));

One deployment detail to plan around: associating or dissociating a VPC endpoint with a private API requires redeploying the API, and DNS propagation takes minutes. Order the platform stack before the service stacks.

Signing the call from a Node 22 Lambda

The caller signs with SigV4 against the service name execute-api. Not lambda, not apigateway. Wrong service name is a 403 with an unhelpful message.

import { SignatureV4 } from '@smithy/signature-v4';
import { HttpRequest } from '@smithy/protocol-http';
import { Sha256 } from '@aws-crypto/sha256-js';
import { defaultProvider } from '@aws-sdk/credential-provider-node';

const ordersUrl = new URL(process.env.ORDERS_URL!);

// Built once per execution environment, reused across invocations.
const signer = new SignatureV4({
  service: 'execute-api',
  region: process.env.AWS_REGION!,
  credentials: defaultProvider(),
  sha256: Sha256,
});

export const handler = async (event: { cartId: string }) => {
  const body = JSON.stringify({ cartId: event.cartId });

  const request = new HttpRequest({
    protocol: ordersUrl.protocol,
    hostname: ordersUrl.hostname,
    path: ordersUrl.pathname,
    method: 'POST',
    headers: {
      host: ordersUrl.hostname, // required in the signed headers
      'content-type': 'application/json',
    },
    body,
  });

  const signed = await signer.sign(request);

  // Global fetch; no HTTP client dependency needed on Node 22.
  const response = await fetch(ordersUrl, {
    method: signed.method,
    headers: signed.headers,
    body: signed.body,
  });

  if (!response.ok) {
    throw new Error(`orders returned ${response.status}: ${await response.text()}`);
  }

  return response.json();
};

The body you sign and the body you send must be byte-identical. Serialising twice, or letting a client library re-encode the payload, produces a payload hash mismatch and another 403.

Callers must be VPC-attached

The caller resolves and reaches the interface endpoint from inside the VPC, so it has to be attached to one. Do not file this under the cost of choosing API Gateway: VPC Lattice requires the caller to sit in a VPC associated with the service network, so an all-Lambda caller is VPC-attached either way. This is a property of doing private east-west on AWS, not a tax this design charges. The reflex objection is that VPC Lambdas cold-start slowly. That has not been true since the Hyperplane rework. The current documentation states that the ENI is created when the function is created or its VPC settings are updated, not at invoke time. Each Hyperplane ENI supports up to 65,000 connections, shared across functions that use the same subnet and security group combination (Lambda VPC networking). The 2019 announcement recorded a cold start dropping from 14.8 seconds to 933 milliseconds in an X-Ray trace (improved VPC networking). That is historical context for why the myth died, not a benchmark to plan against.

Three operational consequences follow. A newly created VPC function stays in Pending until its ENI is ready, which can take minutes, so budget for it in a cold pipeline. Reusing one subnet plus security group combination across your functions lets Lambda share ENIs instead of minting new ones.

The third one bites a sparse internal service layer specifically. If a VPC function sits idle for 14 days, Lambda reclaims the unused Hyperplane ENI and moves the function to Inactive. The next invocation fails and the function re-enters Pending. A rarely-called internal endpoint is exactly that traffic shape, so treat the first call after a quiet fortnight as a failure the caller has to retry.

Common pitfalls

1. Reaching for an HTTP API because it is cheaper. There is no private endpoint type for HTTP APIs. Use a REST API with EndpointType.PRIVATE and budget for $3.50 per million.
2. Setting vpcEndpoints in endpointConfiguration and assuming the resource policy came with it. It only creates Route 53 aliases. Call grantInvokeFromVpcEndpointsOnly() or set policy.
3. Naming an IAM role as Principal in the resource policy while a method still uses NONE authorization. The result is User: anonymous is not authorized to perform: execute-api:Invoke. Put AWS_IAM in defaultMethodOptions.
4. Leaving the interface endpoint on the default security group. Its ENI needs an explicit rule accepting 443 from the callers.
5. Signing with the wrong service name. It is execute-api.
6. Building one VPC endpoint per API. AWS’s best practice is one endpoint, many private APIs; doing it per API multiplies the fixed cost by N.
7. Quoting the 256 KB async payload limit. The current quota is 1 MB async and 6 MB each way sync.
8. Assuming VPC-attached Lambdas cold-start slowly. Cite the current ENI docs, not 2018 blog posts.

The layer is running, and it proves which service is calling. It proves nothing about which user the call is for. Part 102 closes that gap and states what the transport actually encrypts.

References

• Event-driven architectures in Lambda - AWS’s own anti-pattern section: “Lambda functions directly calling other Lambda functions is generally an anti-pattern”. The reason events are the default and this layer is the exception.
• Private REST APIs in API Gateway - The load-bearing page: “Only REST APIs are supported”, one endpoint for many APIs, and the TLS 1.2 and HTTP/1.1 considerations.
• Create a private API - “Your current private API is inaccessible to all VPCs”, and the anonymous-principal troubleshooting entry that forces AWS_IAM on every method.
• API Gateway resource policy examples - The canonical allow-then-deny aws:SourceVpce policy and the route-level shorthand.
• Set up VPC links V2 in API Gateway - Shows that the HTTP API private story is a private integration, gateway into VPC, not a private endpoint.
• CDK: class RestApi - grantInvokeFromVpcEndpointsOnly() and arnForExecuteApi(), the two methods that carry this pattern.
• CDK: interface EndpointConfiguration - vpcEndpoints is “A list of VPC Endpoints against which to create Route53 ALIASes”; it does not create the resource policy.
• Lambda VPC networking - Hyperplane ENI created at function create or update, 65,000 connections per ENI, and the 14-day idle reclaim that bites a sparse internal service.
• AWS Lambda quotas - Invocation payload of 6 MB each way synchronous and 1 MB asynchronous. Not the 256 KB that older posts still quote.
• Synchronous integration - AWS’s even-handed case for synchronous calls: predictable flow control, strong consistency, simple error handling, against tight coupling and cascading failures.