Query Batching

​

Overview

Query batching allows you to send multiple GraphQL operations in a single HTTP request. The router supports processing batched GraphQL requests where multiple operations are sent in a single JSON array. Each operation in the batch is processed concurrently, with configurable limits on the number of operations and concurrency.

​

Configuration

Query batching can be configured in your router configuration file:

batching:
  enabled: true # default: false
  max_concurrency: 10 # default: 10, minimum: 1
  max_entries_per_batch: 100 # default: 100, minimum: 1
  omit_extensions: false # default: false

• enabled: As we do not recommend using batching, it is disabled by default.
• max_concurrency: Specifies how many operations will be processed concurrently. If you want to disable concurrency and process everything serially, you can set the max_concurrency value to 1. The minimum allowed value is 1.
• max_entries_per_batch: Defines the maximum number of operations that can be present in a batch. While we do not support unlimited entries, you can set an extremely high number. The minimum allowed value is 1.
• omit_extensions: Certain errorCodes specific to batch operation failures can be omitted.

​

Usage

To use query batching, send a POST request to your GraphQL endpoint with a JSON array of operations:

[
 {
    "query": "query GetUser { user(id: \"1\") { id name } }"
 },
 {
    "query": "query GetPosts { posts { id title } }"
 },
 {
    "mutation": "mutation ModifyPost(id: \"7\", data: { \"title\": \"new title\" }) { post { id title } }"
 }
]

Note that while the operations may be processed concurrently, the responses will always be returned in the same order as the operations in the request array.

[
  {
    "data": {
      "user": {
        "id": "1",
        "name": "John Doe"
      }
    }
  },
  {
    "data": {
      "posts": [
        {
          "id": "1",
          "title": "Hello World"
        }
      ]
    }
  },
  {
    "data": {
      "post": {
        "id": "1",
        "title": "new title"
      }
    }
  }
]

Note that batches do not support subscriptions as part of their operations. If a subscription is included, the batched request will fail.

​

Tracing

When batching is enabled, we have added several parameters to make it easier to trace batch requests. For batched requests, the root span (or the start cosmo-router span) will always have the following attributes:

• wg.operation.batching.is_batched: Identifies whether the request is batched
• wg.operation.batching.operations_count: Indicates how many operations are expected in the batch request

Additionally, for every direct child span, the following attribute is available:

• wg.operation.batching.operation_index: Indicates which index of the batch operation array this operation belongs to

For more information, please refer to the tracing page.

​

Rate Limiting

For batching, the same subgraph rate limiting rules apply if enabled. This means that if a rate limit of 15 per 5 seconds is defined, a batch of 20 operations would have 5 of them fail with a rate-limiting error. For more information, please refer to the rate limiting page.

​

Feature Flags

Feature flags work the same as normal requests. The flag will be identified by the cookie or feature flag header. However, note that the feature flag will apply to all operations in the batch. For more information, please refer to the feature flag page.

​

Why We Do Not Recommend Batching

Batching as a feature has been built to support clients who are already using query batching in their infrastructure and to make it easy for those users to start using the Cosmo router. If you haven’t used batching yet and are planning to, here are a few reasons to reconsider:

1. Load Balancing: A single batch is processed by a single router instance. This means if you have 50 routers running, a batch of 200 will still be processed on one instance. Depending on your max_concurrency setting, this could result in slower responses compared to sending 200 separate load-balanced requests from the client. We recommend using HTTP/2 multiplexing instead, as described in the next section.

2. Security: If you have set an extremely high max_entries_per_batch value, it is much easier for an attacker to DDoS the router plus the services and infrastructure behind the router by sending multiple batch requests containing many operations. This requires much less effort than sending every single batch operation as an individual request.

3. Caching: It’s easier to cache individual operations than it is to cache batches. A single error will also make the entire batch uncacheable.

​

Alternatives to Batching

Instead of using query batching, consider using HTTP/2 Multiplexing. This allows sending multiple requests over a single connection while still enabling load balancing across multiple router instances. This approach maintains the benefits of efficient connection usage while distributing the load effectively.