Building Modern, Observable .NET Applications: A Deep Dive into OpenTelemetry, Scalar, and OpenAPI Integration

Building Modern, Observable .NET Applications: A Deep Dive into OpenTelemetry, Scalar, and OpenAPI Integration

In today's distributed application landscape, observability isn't just a nice-to-have—it's an absolute necessity. When you're running multiple microservices across different environments, understanding what's happening inside your applications becomes critical for maintaining performance, debugging issues, and ensuring reliability. In this article, I'll walk you through how we've integrated OpenTelemetry, Scalar, and OpenAPI technologies into our .NET 9 solution to create a comprehensive observability and documentation strategy that scales with our microservices architecture.

The Foundation: .NET Aspire and Service Defaults

Our journey begins with .NET Aspire, Microsoft's opinionated stack for building observable, production-ready cloud applications. At the heart of our implementation lies the ServiceDefaults project—a centralized configuration hub that standardizes observability across all our microservices.

The ServiceDefaults project serves as our single source of truth for common configurations. Every service in our solution references this project, ensuring consistent telemetry collection, health checks, and service discovery patterns. This approach eliminates configuration drift and makes it incredibly easy to add new services that automatically inherit our observability standards.

public static TBuilder AddServiceDefaults<TBuilder>(this TBuilder builder) 
    where TBuilder : IHostApplicationBuilder
{
    builder.ConfigureOpenTelemetry();
    builder.AddDefaultHealthChecks();
    builder.Services.AddServiceDiscovery();

    builder.Services.ConfigureHttpClientDefaults(http =>
    {
        http.AddStandardResilienceHandler();
        http.AddServiceDiscovery();
    });

    return builder;
}

This extension method is called in every service's Program.cs, immediately providing OpenTelemetry instrumentation, health checks, and service discovery capabilities. It's a perfect example of how .NET Aspire promotes best practices by making the right thing the easy thing.

OpenTelemetry: The Observability Backbone

OpenTelemetry forms the cornerstone of our observability strategy. Rather than relying on proprietary monitoring solutions that lock us into specific vendors, we've embraced the open standard that provides vendor-neutral instrumentation for metrics, logs, and traces.

Comprehensive Instrumentation Strategy

Our OpenTelemetry configuration captures three critical signals:

Tracing helps us understand request flows across our distributed system. We instrument ASP.NET Core applications to automatically capture incoming HTTP requests, and we track outgoing HTTP client calls to understand inter-service dependencies. Each trace creates a complete picture of how a user request travels through our system, making it invaluable for debugging performance bottlenecks and understanding system behavior.

Metrics provide the quantitative data we need for monitoring system health. We collect ASP.NET Core metrics for request rates and response times, HTTP client metrics for outbound call performance, and .NET runtime metrics for garbage collection and memory usage patterns. This gives us both application-level and infrastructure-level insights.

Logging integrates seamlessly with .NET's built-in logging framework. OpenTelemetry automatically correlates log entries with traces, so when we're debugging an issue, we can see exactly which log entries belong to a specific request trace.

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        metrics.AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddRuntimeInstrumentation();
    })
    .WithTracing(tracing =>
    {
        tracing.AddSource(builder.Environment.ApplicationName)
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation();
    });

Flexible Export Strategy

One of OpenTelemetry's greatest strengths is its vendor-neutral approach to data export. Our configuration supports multiple export targets depending on the environment and requirements:

For development and testing environments, we can export to any OTLP-compatible endpoint by setting the OTEL_EXPORTER_OTLP_ENDPOINT environment variable. This might be a local Jaeger instance, a Grafana stack, or any other OpenTelemetry-compatible tool.

For production workloads, we integrate with Azure Monitor, leveraging Microsoft's OpenTelemetry distro to send telemetry data directly to Application Insights. This provides us with enterprise-grade monitoring capabilities while maintaining the flexibility to switch monitoring backends if needed.

private static TBuilder AddOpenTelemetryExporters<TBuilder>(this TBuilder builder) 
    where TBuilder : IHostApplicationBuilder
{
    var useOtlpExporter = !string.IsNullOrWhiteSpace(
        builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]);

    if (useOtlpExporter)
    {
        builder.Services.AddOpenTelemetry().UseOtlpExporter();
    }

    if(builder.Environment.IsDevelopment() || builder.Environment.IsProduction())
    {
        if (!string.IsNullOrEmpty(
            builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"]))
        {
            builder.Services.AddOpenTelemetry().UseAzureMonitor();
        }
    }

    return builder;
}

This dual-export capability means our development teams can use lightweight, local observability tools during development while production systems benefit from Azure's robust monitoring infrastructure.

Custom Instrumentation Through Action Filters

Beyond automatic instrumentation, we've implemented custom telemetry collection through our ActionControllerLogging filter. This filter captures detailed audit information for every API call, including user context, session information, request bodies, and routing data.

public class ActionControllerLogging : IAsyncActionFilter
{
    public async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
    {
        HttpAuditEntryMessageModel msg = new HttpAuditEntryMessageModel();

        msg.UserName = context.HttpContext.Request.Headers
            .Where(w => w.Key.ToLower() == "username").FirstOrDefault().Value;
        msg.Controller = ((ControllerBase)context.Controller)
            .ControllerContext.ActionDescriptor.ControllerName;
        msg.ActionMethod = ((ControllerBase)context.Controller)
            .ControllerContext.ActionDescriptor.ActionName;

        // Additional audit data collection...

        await next();
    }
}

This filter demonstrates how OpenTelemetry's extensible nature allows us to capture domain-specific telemetry that goes beyond standard web framework metrics. The audit data flows through Azure Service Bus to dedicated audit storage, creating a complete audit trail for compliance and security analysis.

Scalar: Modern API Documentation

While OpenTelemetry handles observability, Scalar addresses another critical need: comprehensive, developer-friendly API documentation. Traditional Swagger UI, while functional, often feels outdated and lacks the polish that modern development teams expect. Scalar provides a beautiful, responsive interface that makes API exploration intuitive and enjoyable.

Seamless Integration with OpenAPI

Our Scalar integration builds on .NET 9's native OpenAPI support. The Microsoft.AspNetCore.OpenApi package automatically generates OpenAPI specifications from our controller definitions, and Scalar consumes these specifications to create interactive documentation.

public static WebApplication ConfigureOpenApiAndScaler(this WebApplication app)
{
    app.MapOpenApi();
    app.MapScalarApiReference(options =>
    {
        options.WithPreferredScheme("Bearer")
            .WithHttpBearerAuthentication(bearer =>
            {
                bearer.Token = "your-bearer-token";
            });
        options.Title = $"{app.Environment.EnvironmentName} - {app.Environment.ApplicationName}";
    });
    return app;
}

This configuration provides several key benefits. The documentation automatically stays in sync with our code—when we add new endpoints or modify existing ones, the documentation updates automatically. Scalar's interface allows developers to test APIs directly from the documentation, eliminating the need to switch between documentation and testing tools.

Security-Aware Documentation

Our Scalar configuration includes built-in authentication support. The Bearer token authentication is pre-configured, so developers can easily test secured endpoints. In development environments, this dramatically reduces the friction involved in API testing and exploration.

The environment-aware titles ({app.Environment.EnvironmentName} - {app.Environment.ApplicationName}) help developers immediately understand which environment and service they're working with, reducing confusion when working across multiple environments.

Strategic Documentation Deployment

We only expose Scalar documentation in non-production environments, maintaining security best practices while maximizing developer productivity:

if (!app.Environment.IsProduction())
{
    app.ConfigureOpenApiAndScaler();
}

This approach ensures that our production endpoints remain secure while development and staging environments provide rich documentation experiences for both internal teams and external API consumers.

OpenAPI: The Contract-First Approach

OpenAPI serves as the foundation for both our documentation and our inter-service communication strategy. By leveraging .NET 9's built-in OpenAPI support, we ensure that our API contracts are automatically generated, accurate, and versioned.

API Versioning Strategy

Our solution implements comprehensive API versioning through the Asp.Versioning libraries. This allows us to evolve our APIs while maintaining backward compatibility for existing consumers.

builder.Services.AddApiVersioning(options =>
{
    options.DefaultApiVersion = new ApiVersion(primaryVersion, 0);
    options.ReportApiVersions = true;
    options.AssumeDefaultVersionWhenUnspecified = true;
    options.ApiVersionReader = ApiVersionReader.Combine(
        new UrlSegmentApiVersionReader(),
        new HeaderApiVersionReader("x-api-version"),
        new QueryStringApiVersionReader("api-version")
    );
});

This configuration supports multiple versioning strategies—URL segments, headers, and query strings—giving API consumers flexibility in how they specify version requirements. The ReportApiVersions setting ensures that API responses include version information, helping consumers understand what versions are available.

Code Generation and Type Safety

While not explicitly shown in our current codebase, our OpenAPI specifications enable powerful code generation scenarios. Teams can generate strongly-typed HTTP clients using tools like Refit (which we include in our shared project) or NSwag, ensuring that inter-service communication is type-safe and automatically stays in sync with API changes.

The presence of Refit in our dependencies suggests we're using this approach for HTTP client generation:

<PackageReference Include="Refit" Version="8.0.0" />
<PackageReference Include="Refit.HttpClientFactory" Version="8.0.0" />

This enables us to define HTTP APIs as C# interfaces and let Refit handle the HTTP implementation details, creating a more maintainable and less error-prone approach to service communication.

Architectural Integration: How It All Works Together

The true power of our observability and documentation strategy emerges from how these technologies work together to support our microservices architecture.

Service-to-Service Communication

When a request flows through our system, OpenTelemetry automatic instrumentation creates a distributed trace that spans multiple services. If our User Service calls the Profile Service, which then calls the Image Service, we get a complete trace showing the entire request flow, including timing information for each service call.

The OpenAPI specifications for each service define clear contracts, while Refit-generated clients provide type-safe communication. If the Profile Service needs to call the User Service, it uses a strongly-typed interface generated from the User Service's OpenAPI specification.

Development Workflow

During development, our setup provides an exceptional developer experience. Developers can:

  1. Explore APIs using Scalar's beautiful interface, testing endpoints directly from the documentation
  2. Monitor request flows using local OpenTelemetry exporters to understand how their changes affect the broader system
  3. Debug issues using correlated logs and traces that show exactly what happened during a specific request
  4. Ensure compatibility by testing against versioned APIs that clearly communicate breaking changes

Production Operations

In production, our observability stack provides operations teams with the insights they need to maintain system reliability:

  • Azure Monitor integration provides enterprise-grade alerting, dashboards, and analysis capabilities
  • Distributed tracing enables rapid root cause analysis when issues occur across service boundaries
  • Comprehensive metrics support proactive monitoring and capacity planning
  • Structured logging with trace correlation makes troubleshooting specific customer issues straightforward

Security and Compliance

Our ActionControllerLogging filter creates detailed audit trails for compliance requirements, while the audit data flows through Azure Service Bus to ensure reliable delivery and processing. The combination of OpenTelemetry traces and custom audit logs provides both technical observability and business-level audit capabilities.

Performance and Overhead Considerations

Implementing comprehensive observability always raises questions about performance impact. OpenTelemetry is designed with performance in mind, using efficient sampling strategies and asynchronous export mechanisms to minimize impact on application performance.

Our instrumentation focuses on high-value, low-overhead data collection. ASP.NET Core instrumentation adds minimal latency to request processing, while HTTP client instrumentation helps us identify slow downstream dependencies. Runtime instrumentation provides valuable insights into garbage collection and memory usage with negligible overhead.

The key is selective instrumentation—we capture the telemetry that provides actionable insights while avoiding over-instrumentation that could impact performance or create overwhelming amounts of data.

Future Enhancements and Roadmap

Looking ahead, our observability and documentation strategy continues to evolve. Some areas we're exploring include:

Enhanced Custom Metrics: Adding business-specific metrics that help us understand not just technical performance but business outcomes.

Improved Documentation: Exploring ways to include more context in our API documentation, such as usage examples and integration guides.

Advanced Tracing: Investigating custom span creation for critical business processes to provide even more detailed insights into application behavior.

Multi-Environment Correlation: Developing strategies to correlate telemetry across development, staging, and production environments to better understand how code changes affect system behavior.

Conclusion

The integration of OpenTelemetry, Scalar, and OpenAPI in our .NET 9 solution creates a powerful foundation for building and operating modern microservices. OpenTelemetry provides vendor-neutral observability that scales with our application, Scalar delivers beautiful, developer-friendly API documentation, and OpenAPI ensures our service contracts are clear, versioned, and automatically maintained.

This combination supports both development velocity and operational excellence. Developers get the tools they need to understand and test APIs, while operations teams get the observability data required to maintain reliable, performant systems.

Most importantly, this approach is future-proof. By embracing open standards like OpenTelemetry and OpenAPI, we avoid vendor lock-in while still benefiting from excellent tooling and integrations. As our architecture evolves and new requirements emerge, our observability and documentation strategy can adapt without requiring fundamental changes.

The investment in proper observability and documentation pays dividends throughout the application lifecycle—from initial development through production operations to long-term maintenance and evolution. In today's fast-paced development environment, these capabilities aren't just helpful; they're essential for building systems that teams can confidently develop, deploy, and operate at scale.

Comments ()