Engineer a GraphQL federation observability suite tracking query plans, subgraph errors,…
- User
Engineer a GraphQL federation observability suite tracking query plans, subgraph errors, and tracing.
Assistant
The user wants an "Engineer a GraphQL federation observability suite tracking query plans, subgraph errors, and tracing."
This is a request to design and provide a solution for monitoring a federated GraphQL architecture (like Apollo Federation or similar). The key components requested are:
- Query Plans: Visualizing or tracking how the gateway breaks down a query into sub-queries.
- Subgraph Errors: Identifying, logging, and analyzing errors occurring in specific downstream services.
- Tracing: Distributed tracing to measure latency across the gateway and subgraphs.
This is a high-level architectural design task, but usually, users want code or concrete implementation details. I should provide a mix of architecture explanation, technology choices (OpenTelemetry, Apollo, etc.), and concrete code examples (perhaps in TypeScript/Node.js, which is standard for Apollo) to demonstrate how to instrument this.
Scope:
- Architecture: Gateway + Subgraphs.
- Tools: OpenTelemetry (standard), Apollo Server (common), Jaeger/Zipkin (visualizers), Prometheus (metrics).
- Implementation:
- How to export trace data?
- How to enrich trace data with Query Plans?
- How to handle errors specifically?
Plan:
- High-Level Architecture: Explain the components.
- Instrumentation (The "How-To"):
- Use OpenTelemetry (OTel) because it's vendor-neutral and robust.
- Set up the Gateway (Apollo) with OTel.
- Set up Subgraphs with OTel.
- Specific Features requested:
- Query Plans: Apollo has specific plugins for this. I need to show how to hook into the
executionDidStartor similar lifecycle events to log the plan. - Subgraph Errors: Standard error handling + specific error extensions.
- Tracing: OTel spans.
- Query Plans: Apollo has specific plugins for this. I need to show how to hook into the
- Code Example: A self-contained example using Apollo Server + OTel would be very helpful.
- Visualization/Analysis: Mention tools like Apollo Studio, Jaeger, or Grafana.
Refining the "Query Plan" aspect: Apollo Studio does this out of the box, but if we are building a custom suite (implied by "Engineer a..."), we might want to extract this data ourselves. Apollo Server's
ApolloServerPlugininterface is the key here. I can write a custom plugin that captures the query plan and attaches it to the current OTel span as an attribute or event.Refining the "Subgraph Errors" aspect: We need to ensure that if a subgraph returns an error, it's not just logged as "400 Bad Request" or "500 Internal Server Error" by the gateway, but the specific subgraph name and path are recorded. Again, custom plugins or Apollo's built-in tracing are relevant.
Refining the "Tracing" aspect: Context propagation is crucial. The gateway receives a request, creates a trace, and must propagate the context to subgraphs via HTTP headers.
Structure of the response:
- Introduction: The "Stack" (Apollo, OpenTelemetry, Jaeger/Zipkin).
- Core Concept: The Custom Observability Plugin.
- Implementation Details (Code):
- OTel setup (TracerProvider, Exporter).
- The
CustomObservabilityPlugin. - Gateway setup.
- Deep Dive:
- Tracking Query Plans (JSON parsing of the plan).
- Tracking Errors (intercepting
errorsarray). - Distributed Tracing (Propagators).
- Dashboard/Metrics: Suggesting what to look at in the UI.
Let's write the code in TypeScript.
Self-Correction during drafting: I initially thought about suggesting Apollo Studio. However, the prompt asks to "Engineer a suite," implying a do-it-yourself or highly customized solution, possibly for on-prem or proprietary usage. I will focus on OpenTelemetry (OTel) as the backbone, allowing the user to send data to any backend (Jaeger, Honeycomb, Datadog, etc.), rather than locking them into Apollo Studio.
Key Technical Challenge: Getting the Query Plan in Apollo Server. The
queryPlanis available in theexecutionDidStarthook in theexperimental__queryPlanproperty (though this changes versions, usually it's part of the request context in federation). Actually, the standard way now is usually via theApolloServerPlugin->executionDidStart->requestContext.queryPlan. Or simpler, just rely on the span attributes provided by Apollo's official OTel plugin, but the prompt asks for specific tracking of query plans, which usually implies seeing the structure. I will write a custom plugin to attach the query plan JSON to the span attributes (warning: large attributes can be an issue, so I might summarize it).Revised Plan for Code:
- Setup OTel: Standard NodeSDK setup.
- Plugin:
requestDidStart: Start a span for the whole operation.executionDidStart: This is where the query plan is available. I will attach it to the span.didEncounterErrors: Log specific subgraph errors as span events.
- Subgraphs: Briefly show how to enable OTel in a subgraph to show context propagation.
Let's execute this.
Dependencies:
@apollo/server,@apollo/gateway,@opentelemetry/sdk-node,@opentelemetry/instrumentation-graphql(maybe? No, better to instrument the server manually to get specific Federation details, or use the generic graphql instrumentation and enhance it).I will focus on a Custom Plugin approach. This gives the user the "Suite" control they asked for.
Features to implement in the Plugin:
- Query Plan Extraction: Extract the plan and serialize key parts (service names, operations) into the Trace.
- Error Enrichment: Detect
ApolloErrororGraphQLError, check extensions forserviceName, and log structured error events. - Latency Tracking: Built-in with OTel.
Let's write the solution. This solution engineers a custom GraphQL Federation Observability Suite using Apollo Server, OpenTelemetry (OTel), and a custom plugin architecture.
While managed solutions (like Apollo Studio) exist, building this suite yourself gives you full ownership of the data, allows you to stay within your own VPC, and enables custom logic on query plans.
Architecture Overview
- The Probe (Custom Plugin): An Apollo Server plugin that intercepts the request lifecycle. It extracts the "Query Plan" (the logic of how the gateway splits the query) and maps errors to specific subgraphs.
- The Wire (OpenTelemetry): We use the OpenTelemetry API to create spans. The Gateway acts as the root span, and we generate child spans for subgraph fetches.
- The Collector (Jaeger/Prometheus): The telemetry data is exported to a backend (we will use Jaeger for tracing visualization in this example).
Prerequisites
- Node.js
- Docker (for running Jaeger)
1. The Observability Suite Implementation
Create a file named
observability-suite.js. This is the core engineering artifact. It creates a standardized tracing and logging interface.// observability-suite.js const { trace, SpanStatusCode, SpanKind } = require('@opentelemetry/api'); const { BasicTracerProvider, SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base'); const { JaegerExporter } = require('@opentelemetry/exporter-jaeger'); // 1. Configure the OpenTelemetry Exporter (Sending data to Jaeger) const provider = new BasicTracerProvider(); const exporter = new JaegerExporter({ serviceName: 'federation-gateway', }); provider.addSpanProcessor(new SimpleSpanProcessor(exporter)); provider.register(); const tracer = trace.getTracer('federation-gateway-tracer'); /** * 2. The Custom Observability Plugin * This is the engine that tracks Query Plans, Errors, and Tracing. */ const FederationObservabilityPlugin = { // Hook: Request starts async requestDidStart(context) { const span = tracer.startSpan('graphql_gateway_operation', { kind: SpanKind.SERVER, attributes: { 'graphql.operation.name': context.operationName || 'Anonymous', 'graphql.query': context.request.query, }, }); return { // Hook: Response sent (Success or Error) async willSendResponse(requestContext) { const { response, errors } = requestContext; // Track Subgraph Errors if (errors) { span.setStatus({ code: SpanStatusCode.ERROR, message: errors.map(e => e.message).join(', ') }); span.setAttribute('graphql.errors.count', errors.length); // Enrich telemetry with specific subgraph error codes if present errors.forEach(err => { if (err.extensions && err.extensions.code) { span.addEvent('subgraph_error', { 'error.code': err.extensions.code, 'error.path': err.path ? err.path.join('.') : 'root', }); } }); } else { span.setStatus({ code: SpanStatusCode.OK }); } span.end(); }, // Hook: Execution begins (Where the Query Plan exists) async executionDidStart(requestContext) { return { // Hook: The specific execution of the Query Plan async executionDidEnd(err) { if (err) { span.recordException(err); } }, }; }, }; }, // 3. TRACKING QUERY PLANS // This is specific to Apollo Gateway. We hook into the 'didResolveQueryPlan' // to visualize exactly how the query was split. async didResolveQueryPlan(requestContext, queryPlan) { // Add the Query Plan structure to the Trace as an Event // In production, you might send this to a dedicated logger instead of the trace. const span = trace.getActiveSpan(); if (span && queryPlan) { span.addEvent('query_plan_resolved', { 'query.plan.node_count': queryPlan.node?.kind ? 1 : 0, // Simplified metric 'query.plan.raw': JSON.stringify(queryPlan), // Attach full plan }); // Create child spans for predicted subgraph fetches (Static Analysis) // Note: Actual network timing is handled by the subgraph service itself, // but we can log the plan structure here. if (queryPlan.node) { traceSubgraphNodes(queryPlan.node, span); } } }, }; // Helper: Recursively trace the query plan nodes function traceSubgraphNodes(node, parentSpan) { if (!node) return; if (node.kind === 'Fetch') { const childSpan = tracer.startSpan(`plan.fetch.${node.serviceName}`, { parent: parentSpan, attributes: { 'subgraph.service': node.serviceName, 'subgraph.operation': node.operation, }, }); // We end this immediately because this hook happens *before* network execution. // This serves as a structural log of "We plan to call this service". childSpan.end(); } // Recursion for parallel or sequence nodes if (node.nodes) { node.nodes.forEach(n => traceSubgraphNodes(n, parentSpan)); } } module.exports = { FederationObservabilityPlugin };2. The Federation Gateway Setup
Here we initialize the Apollo Gateway and apply our Observability Suite.
// gateway.js const { ApolloGateway } = require('@apollo/gateway'); const { ApolloServer } = require('@apollo/server'); const { startStandaloneServer } = require('@apollo/server/standalone'); const { FederationObservabilityPlugin } = require('./observability-suite'); // Define Subgraphs (For demo purposes, assuming these are running) // In a real setup, these URLs point to your running microservices. const subgraphs = [ { name: 'products', url: 'http://localhost:4001' }, { name: 'users', url: 'http://localhost:4002' }, ]; const gateway = new ApolloGateway({ supergraphSdl: ` schema @core(feature: "https://specs.apollo.dev/core/v0.1"), @core(feature: "https://specs.apollo.dev/join/v0.1") { query: Query } directive @join__field(graph: join__Graph, requires: join__FieldSet, provides: join__FieldSet) on FIELD_DEFINITION directive @join__type(graph: join__Graph!, key: join__FieldSet) repeatable on OBJECT | INTERFACE directive @join__owner(graph: join__Graph!) on OBJECT | INTERFACE directive @join__graph(name: String!, url: String!) on ENUM_VALUE scalar join__FieldSet enum join__Graph { PRODUCTS @join__graph(name: "products" url: "http://localhost:4001") USERS @join__graph(name: "users" url: "http://localhost:4002") } type Query { me: User @join__field(graph: USERS) } type User @join__owner(graph: USERS) @key(fields: "id") { id: ID! name: String! @join__field(graph: USERS) reviews: [Review] @join__field(graph: PRODUCTS) } type Review { body: String author: User @join__field(graph: PRODUCTS, provides: "name") } `, }); const server = new ApolloServer({ gateway, // Inject our Custom Observability Suite plugins: [FederationObservabilityPlugin], debug: true, }); startStandaloneServer(server, { listen: { port: 4000 }, }).then(({ url }) => { console.log(`🚀 Gateway ready at ${url}`); console.log(`📊 Observability suite active. Check Jaeger.`); });3. Dashboard & Visualization (Docker Compose)
To visualize the traces, spin up a standard OpenTelemetry backend. Create a
docker-compose.yml:version: '3.8' services: jaeger: image: jaegertracing/all-in-one:latest ports: - "16686:16686" # UI - "6831:6831/udp" # AgentRun
docker-compose upto start the dashboard.Key Engineering Features Explained
1. Query Plan Tracking
In
didResolveQueryPlan, we capture the abstract syntax tree of the federation plan before execution.- Why this matters: If a query is slow, you need to know if it's because the Gateway created a complex "cascade" of requests (waterfall problem) or if a single subgraph is slow.
- Implementation: We recursively walk the
nodestructure. If the plan containsnodes(Parallel) or a sequence, we log that structure.
2. Subgraph Error Attribution
In
willSendResponse, we inspecterrors.- Why this matters: A generic 500 error is useless. Federation errors usually contain
extensions.serviceorextensions.code. - Implementation: We extract
err.extensions.codeanderr.pathand attach them to the span. In your dashboard (Jaeger), you can filter traces wheresubgraph_errorexists.
3. Distributed Tracing
We use
@opentelemetry/api.- Propagation: When the Gateway makes HTTP calls to subgraphs, standard OTel propagation (via HTTP headers like
traceparent) automatically links the Gateway's span to the Subgraph's span (assuming subgraphs are also instrumented). - Attributes: We tag spans with
graphql.operation.nameto identify expensive operations in the aggregate view.
How to test it
- Start the Jaeger container.
- Run the
gateway.js. - Execute a query:
query GetMe { me { name reviews { body } } } - Open Jaeger at
http://localhost:16686. - Select service
federation-gateway-tracerand click "Find Traces". - Analyze: You will see the main operation. Clicking into it reveals the
query_plan_resolvedevent containing the JSON plan, and if any subgraph failed, theERRORstatus tag.