Schemio Framework

Introduction

Schemio is a powerful .NET library designed to aggregate data from heterogeneous data stores using a schema-driven approach. It enables developers to hydrate complex object graphs by fetching data from multiple sources (SQL databases, Web APIs, NoSQL stores) using XPath and JSONPath schema mappings.

Key Benefits

The Need for Data Aggregation

Modern Application Challenges

In today's microservices and distributed system architectures, applications often need to:

1. Combine Data from Multiple Sources

• User profiles from identity services
• Order history from e-commerce APIs
• Product catalogs from different databases
• Analytics data from various platforms

2. Handle Different Data Formats

• SQL database records
• JSON responses from REST APIs
• XML from legacy systems
• NoSQL document stores

3. Manage Complex Dependencies

• Parent-child relationships across systems
• Conditional data loading based on context
• Performance optimization through selective loading

Traditional Approaches and Their Limitations

Manual Data Assembly

// Traditional approach - brittle and hard to maintain
var customer = GetCustomerFromDatabase(customerId);
var orders = GetOrdersFromAPI(customerId);
var communication = GetCommunicationFromService(customerId);

// Manual assembly - error-prone
customer.Orders = orders;
customer.Communication = communication;

Schemio's Solution

Schemio provides a declarative, schema-driven approach:

// Schemio approach - declarative and maintainable
public class CustomerConfiguration : EntityConfiguration<Customer>
{
    public override IEnumerable<Mapping<Customer, IQueryResult>> GetSchema()
    {
        return CreateSchema.For<Customer>()
            .Map<CustomerQuery, CustomerTransform>(For.Paths("customer"),
                customer => customer.Dependents
                    .Map<CommunicationQuery, CommunicationTransform>(For.Paths("customer/communication"))
                    .Map<OrdersQuery, OrdersTransform>(For.Paths("customer/orders")))
            .End();
    }
}

Core Concepts

Entities

Entities represent the final aggregated data structure implementing IEntity. They define the complete object graph that will be hydrated with data from multiple sources:

public class Customer : IEntity
{
    // Level 1: Root properties
    public int CustomerId { get; set; }
    public string CustomerCode { get; set; }
    public string CustomerName { get; set; }
    
    // Level 2: Nested objects
    public Communication Communication { get; set; }
    public Address Address { get; set; }
    public Order[] Orders { get; set; }
}

public class Order
{
    public int OrderId { get; set; }
    public string OrderNumber { get; set; }
    public DateTime OrderDate { get; set; }
    
    // Level 3: Deep nesting
    public OrderItem[] Items { get; set; }
    public Payment Payment { get; set; }
}

public class OrderItem
{
    public int ItemId { get; set; }
    public string ProductName { get; set; }
    public int Quantity { get; set; }
    public decimal Price { get; set; }
}

Queries and Parent-Child Relationships

Queries form a hierarchical structure where child queries depend on data from their parent queries. This creates a powerful data flow where each query can use the results of its parent to customize its own execution.

Parent Query (Root Level)

Parent queries execute first and don't depend on other query results:

public class CustomerQuery : SQLQuery<CustomerResult>
{
    protected override Func<IDbConnection, Task<CustomerResult>> GetQuery(
        IDataContext context, IQueryResult parentQueryResult)
    {
        // parentQueryResult is null for root queries
        var request = (CustomerRequest)context.Request;
        
        return connection => connection.QueryFirstOrDefaultAsync<CustomerResult>(
            "SELECT CustomerId as Id, CustomerName as Name, CustomerCode as Code FROM Customers WHERE CustomerId = @Id",
            new { Id = request.CustomerId });
    }
}

Child Query (Dependent Level)

Child queries receive their parent's result and use it to determine what data to fetch:

public class OrdersQuery : SQLQuery<CollectionResult<OrderResult>>
{
    protected override Func<IDbConnection, Task<CollectionResult<OrderResult>>> GetQuery(
        IDataContext context, IQueryResult parentQueryResult)
    {
        // parentQueryResult contains the CustomerResult from the parent query
        var customer = (CustomerResult)parentQueryResult;
        
        return async connection =>
        {
            var orders = await connection.QueryAsync<OrderResult>(
                "SELECT OrderId, OrderNumber, OrderDate FROM Orders WHERE CustomerId = @CustomerId",
                new { CustomerId = customer.Id });
                
            return new CollectionResult<OrderResult>(orders);
        };
    }
}

Query Execution Flow

1. Parent Query (CustomerQuery) executes first
   ↓ (CustomerResult passed to children)
   
2. Child Queries execute in parallel:
   - OrdersQuery (uses CustomerId from CustomerResult)
   - CommunicationQuery (uses CustomerId from CustomerResult)
   - AddressQuery (uses CustomerId from CustomerResult)
   ↓ (OrderResult passed to grandchildren)
   
3. Grandchild Queries execute:
   - OrderItemsQuery (uses OrderId from OrderResult)
   - PaymentQuery (uses OrderId from OrderResult)

Package Overview

Core Packages

1. Schemio.Core

Purpose: Foundation package providing core interfaces and implementations.

Key Components:

• IEntity, IQuery, ITransformer interfaces
• DataProvider<T> - Main orchestration class
• QueryBuilder<T> - Builds query execution plan
• EntityBuilder<T> - Assembles final entity
• Path matchers for XPath and JSONPath

2. Schemio.SQL

Purpose: SQL database support using Dapper for query execution.

Key Components:

• SQLQuery<TResult> - Base class for SQL queries
• QueryEngine - Dapper-based query execution
• SQLConfiguration - Connection and query settings

Supported Databases:

• SQL Server
• SQLite
• MySQL
• PostgreSQL
• Oracle (with appropriate providers)

3. Schemio.EntityFramework

Purpose: Entity Framework Core integration for advanced ORM scenarios.

Key Components:

• SQLQuery<TResult> - EF Core query implementation
• QueryEngine<T> - DbContext factory integration
• Full LINQ query support

4. Schemio.API

Purpose: HTTP/REST API data source support using HttpClient.

Key Components:

• WebQuery<TResult> - Base class for API queries
• QueryEngine - HttpClient-based execution
• WebHeaderResult - Support for response headers
• Request/response header management

Package Compatibility Matrix

Getting Started

1. Basic Setup

First, install the required packages:

Install-Package Schemio.Core
Install-Package Schemio.SQL  # For SQL database support
Install-Package Schemio.API  # For REST API support
Install-Package Schemio.EntityFramework # For Entity Framework support

2. Define Your Entity

public class Product : IEntity
{
    public int ProductId { get; set; }
    public string Name { get; set; }
    public decimal Price { get; set; }
    public Category Category { get; set; }
    public Review[] Reviews { get; set; }
}

public class Category
{
    public int CategoryId { get; set; }
    public string Name { get; set; }
}

public class Review
{
    public int ReviewId { get; set; }
    public string Comment { get; set; }
    public int Rating { get; set; }
}

3. Create Query Results

public class ProductResult : IQueryResult
{
    public int Id { get; set; }
    public string Name { get; set; }
    public decimal Price { get; set; }
    public int CategoryId { get; set; }
}

public class CategoryResult : IQueryResult
{
    public int Id { get; set; }
    public string Name { get; set; }
}

public class ReviewResult : IQueryResult
{
    public int Id { get; set; }
    public string Comment { get; set; }
    public int Rating { get; set; }
    public int ProductId { get; set; }
}

4. Implement Queries

SQL Query Example:

public class ProductQuery : SQLQuery<ProductResult>
{
    protected override Func<IDbConnection, Task<ProductResult>> GetQuery(
        IDataContext context, IQueryResult parentQueryResult)
    {
        var request = (ProductRequest)context.Request;
        
        return connection => connection.QueryFirstOrDefaultAsync<ProductResult>(
            "SELECT ProductId as Id, Name, Price, CategoryId FROM Products WHERE ProductId = @Id",
            new { Id = request.ProductId });
    }
}

API Query Example:

public class ReviewsApiQuery : WebQuery<CollectionResult<ReviewResult>>
{
    public ReviewsApiQuery() : base("https://api.reviews.com/") { }

    protected override Func<Uri> GetQuery(IDataContext context, IQueryResult parentQueryResult)
    {
        var product = (ProductResult)parentQueryResult;
        return () => new Uri($"products/{product.Id}/reviews", UriKind.Relative);
    }
}

5. Create Transformers

public class ProductTransformer : BaseTransformer<ProductResult, Product>
{
    public override void Transform(ProductResult queryResult, Product entity)
    {
        entity.ProductId = queryResult.Id;
        entity.Name = queryResult.Name;
        entity.Price = queryResult.Price;
    }
}

public class CategoryTransformer : BaseTransformer<CategoryResult, Product>
{
    public override void Transform(CategoryResult queryResult, Product entity)
    {
        if (entity.Category == null)
            entity.Category = new Category();
            
        entity.Category.CategoryId = queryResult.Id;
        entity.Category.Name = queryResult.Name;
    }
}

6. Configure Entity Schema

public class ProductConfiguration : EntityConfiguration<Product>
{
    public override IEnumerable<Mapping<Product, IQueryResult>> GetSchema()
    {
        return CreateSchema.For<Product>()
            .Map<ProductQuery, ProductTransformer>(For.Paths("product"),
                product => product.Dependents
                    .Map<CategoryQuery, CategoryTransformer>(For.Paths("product/category"))
                    .Map<ReviewsApiQuery, ReviewsTransformer>(For.Paths("product/reviews")))
            .End();
    }
}

7. Register Dependencies

// Using fluent interface
services.UseSchemio()
    .WithEngine(c => new QueryEngine(sqlConfiguration))  // SQL support
    .WithEngine<Schemio.API.QueryEngine>()              // API support
    .WithPathMatcher(c => new XPathMatcher())
    .WithEntityConfiguration<Product>(c => new ProductConfiguration());

// Enable logging
services.AddLogging();

// For API queries
services.AddHttpClient();

// For SQL queries  
DbProviderFactories.RegisterFactory("System.Data.SqlClient", SqlClientFactory.Instance);

8. Use the Data Provider

public class ProductService
{
    private readonly IDataProvider<Product> dataProvider;
    
    public ProductService(IDataProvider<Product> dataProvider)
    {
        this.dataProvider = dataProvider;
    }
    
    public Product GetProduct(int productId)
    {
        var request = new ProductRequest { ProductId = productId };
        return dataProvider.GetData(request);
    }
    
    public Product GetProductWithReviews(int productId)
    {
        var request = new ProductRequest 
        { 
            ProductId = productId,
            SchemaPaths = new[] { "product", "product/reviews" }
        };
        return dataProvider.GetData(request);
    }
}

Architecture Deep Dive

Execution Flow

Client Request
    ↓
DataProvider
    ↓
QueryBuilder
    ↓
Generate Query Plan
    ↓
QueryExecutor
    ↓
Execute Level 1 Queries
    ↓
Resolve Dependencies
    ↓
Execute Level 2 Queries
    ↓
Continue Until All Levels Complete
    ↓
EntityBuilder
    ↓
Apply Transformers
    ↓
Return Aggregated Entity

Component Interactions

1. DataProvider Orchestration

The DataProvider<T> serves as the main orchestrator:

public TEntity GetData(IEntityRequest request)
{
    var context = new DataContext(request);
    
    // Build execution plan
    var queries = queryBuilder.Build(context);
    
    // Execute queries
    var results = queryExecutor.Execute(context, queries);
    
    // Build final entity
    var entity = entityBuilder.Build(context, results);
    
    return entity;
}

2. Query Building Process

The QueryBuilder<T> creates an optimized execution plan:

1. Filter by Schema Paths: Only include queries matching requested paths
2. Resolve Dependencies: Build parent-child query relationships
3. Optimize Execution: Determine optimal query execution order

3. Query Execution Strategy

The QueryExecutor manages parallel execution with dependency resolution:

1. Level-by-Level Execution: Execute queries level by level to handle dependencies
2. Parallel Processing: Run independent queries at the same level in parallel
3. Result Propagation: Pass parent results to child queries
4. Caching: Cache results marked with [CacheResult] attribute

4. Entity Building Process

The EntityBuilder<T> assembles the final entity:

1. Transformer Resolution: Match query results to appropriate transformers
2. Sequential Application: Apply transformers in dependency order
3. Type Safety: Ensure type compatibility between results and transformers

Memory Management

• Query Results: Results are collected and passed to transformers
• Caching: Optional caching for expensive operations
• Disposal: Proper disposal of database connections and HTTP clients

Configuration Guide

Entity Configuration Patterns

1. Simple Linear Configuration

return CreateSchema.For<Customer>()
    .Map<CustomerQuery, CustomerTransformer>(For.Paths("customer"))
    .Map<AddressQuery, AddressTransformer>(For.Paths("customer/address"))
    .End();

2. Branching Configuration

return CreateSchema.For<Customer>()
    .Map<CustomerQuery, CustomerTransformer>(For.Paths("customer"),
        customer => customer.Dependents
            .Map<ContactQuery, ContactTransformer>(For.Paths("customer/contact"))
            .Map<PreferencesQuery, PreferencesTransformer>(For.Paths("customer/preferences")))
    .End();

3. Deep Nesting Configuration

return CreateSchema.For<Customer>()
    .Map<CustomerQuery, CustomerTransformer>(For.Paths("customer"),
        customer => customer.Dependents
            .Map<OrdersQuery, OrdersTransformer>(For.Paths("customer/orders"),
                orders => orders.Dependents
                    .Map<OrderItemsQuery, OrderItemsTransformer>(For.Paths("customer/orders/order/items"))))
    .End();

Path Matching Strategies

XPath Patterns

• customer - Exact match
• customer/orders - Nested path
• customer/orders/order/items - Deep nesting
• //orders - Descendant matching (with ancestor support)

JSONPath Patterns

• $.customer - Root level
• $.customer.orders - Nested property
• $.customer.orders[*].items - Array elements

Schema Path Filtering

Control which parts of the object graph to load:

// Load only customer basic info
var request = new CustomerRequest
{
    CustomerId = 123,
    SchemaPaths = new[] { "customer" }
};

// Load customer with orders but no items
var request = new CustomerRequest
{
    CustomerId = 123,
    SchemaPaths = new[] { "customer", "customer/orders" }
};

// Load everything
var request = new CustomerRequest
{
    CustomerId = 123
    // SchemaPaths = null loads all configured paths
};

Query Implementation Guide

SQL Queries with Dapper

Basic Query

public class CustomerQuery : SQLQuery<CustomerResult>
{
    protected override Func<IDbConnection, Task<CustomerResult>> GetQuery(
        IDataContext context, IQueryResult parentQueryResult)
    {
        var request = (CustomerRequest)context.Request;
        
        return connection => connection.QueryFirstOrDefaultAsync<CustomerResult>(
            @"SELECT CustomerId as Id, 
                     CustomerName as Name, 
                     CustomerCode as Code 
              FROM Customers 
              WHERE CustomerId = @CustomerId",
            new { CustomerId = request.CustomerId });
    }
}

Collection Query

public class OrdersQuery : SQLQuery<CollectionResult<OrderResult>>
{
    protected override Func<IDbConnection, Task<CollectionResult<OrderResult>>> GetQuery(
        IDataContext context, IQueryResult parentQueryResult)
    {
        var customer = (CustomerResult)parentQueryResult;
        
        return async connection =>
        {
            var orders = await connection.QueryAsync<OrderResult>(
                @"SELECT OrderId, OrderNumber, OrderDate, TotalAmount
                  FROM Orders 
                  WHERE CustomerId = @CustomerId",
                new { CustomerId = customer.Id });
                
            return new CollectionResult<OrderResult>(orders);
        };
    }
}

Complex Query with Parameters

public class ProductSearchQuery : SQLQuery<CollectionResult<ProductResult>>
{
    protected override Func<IDbConnection, Task<CollectionResult<ProductResult>>> GetQuery(
        IDataContext context, IQueryResult parentQueryResult)
    {
        var request = (ProductSearchRequest)context.Request;
        
        return async connection =>
        {
            var sql = @"
                SELECT p.ProductId as Id, p.Name, p.Price, p.CategoryId
                FROM Products p
                WHERE (@CategoryId IS NULL OR p.CategoryId = @CategoryId)
                  AND (@MinPrice IS NULL OR p.Price >= @MinPrice)
                  AND (@MaxPrice IS NULL OR p.Price <= @MaxPrice)
                  AND (@SearchTerm IS NULL OR p.Name LIKE @SearchPattern)
                ORDER BY p.Name";
            
            var products = await connection.QueryAsync<ProductResult>(sql, new
            {
                CategoryId = request.CategoryId,
                MinPrice = request.MinPrice,
                MaxPrice = request.MaxPrice,
                SearchTerm = request.SearchTerm,
                SearchPattern = $"%{request.SearchTerm}%"
            });
            
            return new CollectionResult<ProductResult>(products);
        };
    }
}

Entity Framework Queries

Basic EF Query

public class CustomerQuery : SQLQuery<CustomerResult>
{
    protected override Func<DbContext, Task<CustomerResult>> GetQuery(
        IDataContext context, IQueryResult parentQueryResult)
    {
        var request = (CustomerRequest)context.Request;
        
        return async dbContext =>
        {
            var result = await dbContext.Set<CustomerEntity>()
                .Where(c => c.CustomerId == request.CustomerId)
                .Select(c => new CustomerResult
                {
                    Id = c.CustomerId,
                    Name = c.Name,
                    Code = c.Code,
                    Email = c.Email
                })
                .FirstOrDefaultAsync();
                
            return result;
        };
    }
}

Web API Queries

Basic API Query

public class UserProfileQuery : WebQuery<UserProfileResult>
{
    public UserProfileQuery() : base("https://api.userservice.com/") { }

    protected override Func<Uri> GetQuery(IDataContext context, IQueryResult parentQueryResult)
    {
        var request = (UserRequest)context.Request;
        return () => new Uri($"users/{request.UserId}", UriKind.Relative);
    }
}

API Query with Headers

public class AuthenticatedApiQuery : WebQuery<UserDataResult>
{
    public AuthenticatedApiQuery() : base("https://api.secure.com/") { }

    protected override Func<Uri> GetQuery(IDataContext context, IQueryResult parentQueryResult)
    {
        var request = (SecureRequest)context.Request;
        return () => new Uri($"secure/data/{request.Id}", UriKind.Relative);
    }

    protected override IDictionary<string, string> GetRequestHeaders()
    {
        return new Dictionary<string, string>
        {
            { "Authorization", "Bearer " + GetAccessToken() },
            { "X-Client-Version", "1.0" },
            { "Accept", "application/json" }
        };
    }

    protected override IEnumerable<string> GetResponseHeaders()
    {
        return new[] { "X-Rate-Limit-Remaining", "X-Request-Id" };
    }

    private string GetAccessToken()
    {
        // Implement token retrieval logic
        return "your-access-token";
    }
}

Query Result Types

Simple Result

public class CustomerResult : IQueryResult
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Code { get; set; }
    public string Email { get; set; }
}

Collection Result

public class CollectionResult<T> : List<T>, IQueryResult
{
    public CollectionResult(IEnumerable<T> items) : base(items) { }
    public CollectionResult() { }
}

Cached Result

[CacheResult]
public class CategoryResult : IQueryResult
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Description { get; set; }
}

Transformer Implementation Guide

Basic Transformers

Simple Property Mapping

public class CustomerTransformer : BaseTransformer<CustomerResult, Customer>
{
    public override void Transform(CustomerResult queryResult, Customer entity)
    {
        entity.CustomerId = queryResult.Id;
        entity.CustomerName = queryResult.Name;
        entity.CustomerCode = queryResult.Code;
        entity.Email = queryResult.Email;
    }
}

Collection Transformation

public class OrdersTransformer : BaseTransformer<CollectionResult<OrderResult>, Customer>
{
    public override void Transform(CollectionResult<OrderResult> queryResult, Customer entity)
    {
        if (queryResult == null || !queryResult.Any())
        {
            entity.Orders = new Order[0];
            return;
        }

        entity.Orders = queryResult.Select(orderResult => new Order
        {
            OrderId = orderResult.OrderId,
            OrderNumber = orderResult.OrderNumber,
            OrderDate = orderResult.OrderDate,
            TotalAmount = orderResult.TotalAmount
        }).ToArray();
    }
}

Advanced Transformers

Context-Aware Transformation

public class PersonalizedProductTransformer : BaseTransformer<ProductResult, Product>
{
    public override void Transform(ProductResult queryResult, Product entity)
    {
        entity.ProductId = queryResult.Id;
        entity.Name = queryResult.Name;
        entity.Price = queryResult.Price;

        // Access request context
        var request = Context.Request as ProductRequest;
        if (request?.UserId.HasValue == true)
        {
            // Apply user-specific logic
            entity.IsInWishlist = CheckWishlist(request.UserId.Value, entity.ProductId);
            entity.UserRating = GetUserRating(request.UserId.Value, entity.ProductId);
        }
    }

    private bool CheckWishlist(int userId, int productId)
    {
        // Check if product is in user's wishlist
        return false;
    }

    private int? GetUserRating(int userId, int productId)
    {
        // Get user's rating for this product
        return null;
    }
}

Transformation Best Practices

1. Null Safety

public override void Transform(CustomerResult queryResult, Customer entity)
{
    if (queryResult == null) return;

    entity.CustomerId = queryResult.Id;
    entity.CustomerName = queryResult.Name ?? string.Empty;
    entity.Email = queryResult.Email?.ToLowerInvariant();
    
    // Initialize collections to prevent null reference exceptions
    if (entity.Orders == null)
        entity.Orders = new List<Order>();
}

2. Data Validation

public override void Transform(ProductResult queryResult, Product entity)
{
    entity.ProductId = queryResult.Id;
    entity.Name = ValidateAndCleanName(queryResult.Name);
    entity.Price = Math.Max(0, queryResult.Price); // Ensure non-negative price
    entity.Description = SanitizeHtml(queryResult.Description);
}

private string ValidateAndCleanName(string name)
{
    if (string.IsNullOrWhiteSpace(name))
        return "Unknown Product";
        
    return name.Trim().Length > 100 
        ? name.Trim().Substring(0, 100) + "..."
        : name.Trim();
}

Advanced Features

Caching Support

Schemio provides built-in caching for expensive query results:

Enable Caching

[CacheResult]
public class CategoryResult : IQueryResult
{
    public int Id { get; set; }
    public string Name { get; set; }
}

Access Cached Results

The cached results are available to queries and transformers via context object.

public class ProductTransformer : BaseTransformer<ProductResult, Product>
{
    public override void Transform(ProductResult queryResult, Product entity)
    {
        entity.ProductId = queryResult.Id;
        entity.Name = queryResult.Name;

        // Access cached category data
        if (Context.Cache.TryGetValue("CategoryResult", out var cachedCategory))
        {
            var category = (CategoryResult)cachedCategory;
            entity.CategoryName = category.Name;
        }
    }
}

Selective Data Loading

Control which parts of the object graph to load based on request parameters:

public class CustomerRequest : IEntityRequest
{
    public int CustomerId { get; set; }
    public string[] SchemaPaths { get; set; }
    public bool IncludeOrders { get; set; }
    public bool IncludeOrderItems { get; set; }
}

// Usage
var fullCustomer = dataProvider.GetData(new CustomerRequest
{
    CustomerId = 123,
    SchemaPaths = new[] { "customer", "customer/orders", "customer/orders/order/items" }
});

var basicCustomer = dataProvider.GetData(new CustomerRequest
{
    CustomerId = 123,
    SchemaPaths = new[] { "customer" }
});

Error Handling and Resilience

Query-Level Error Handling

public class ResilientApiQuery : WebQuery<UserResult>
{
    public ResilientApiQuery() : base("https://api.external.com/") { }

    protected override Func<Uri> GetQuery(IDataContext context, IQueryResult parentQueryResult)
    {
        var request = (UserRequest)context.Request;
        return () => new Uri($"users/{request.UserId}", UriKind.Relative);
    }

    // Override to handle HTTP errors
    protected override async Task<IQueryResult> HandleError(Exception ex)
    {
        if (ex is HttpRequestException httpEx)
        {
            // Return default result or try alternative endpoint
            return new UserResult { Id = -1, Name = "Unknown User" };
        }
        
        throw ex; // Re-throw if not handled
    }
}

Parallel Query Execution

Schemio automatically executes independent queries in parallel:

// These queries will execute in parallel since they're at the same level
return CreateSchema.For<Customer>()
    .Map<CustomerQuery, CustomerTransformer>(For.Paths("customer"),
        customer => customer.Dependents
            .Map<ContactQuery, ContactTransformer>(For.Paths("customer/contact"))        // Parallel
            .Map<PreferencesQuery, PreferencesTransformer>(For.Paths("customer/preferences")) // Parallel
            .Map<AddressQuery, AddressTransformer>(For.Paths("customer/address")))       // Parallel
    .End();

Extension Points

Creating Custom Query Engines

Implement IQueryEngine to support new data sources:

public class RedisQueryEngine : IQueryEngine
{
    private readonly IConnectionMultiplexer redis;

    public RedisQueryEngine(IConnectionMultiplexer redis)
    {
        this.redis = redis;
    }

    public bool CanExecute(IQuery query) 
        => query != null && query is IRedisQuery;

    public async Task<IQueryResult> Execute(IQuery query)
    {
        var serviceBusQuery = (IServiceBusQuery)query;
        var sender = serviceBusClient.CreateSender(serviceBusQuery.QueueName);
        
        var message = new ServiceBusMessage(serviceBusQuery.GetMessageBody());
        await sender.SendMessageAsync(message);
        
        // Wait for response (implement request-response pattern)
        var response = await WaitForResponse(serviceBusQuery.CorrelationId);
        return serviceBusQuery.ProcessResponse(response);
    }
}

2. AWS Lambda Integration

public class LambdaQueryEngine : IQueryEngine
{
    private readonly IAmazonLambda lambdaClient;

    public LambdaQueryEngine(IAmazonLambda lambdaClient)
    {
        this.lambdaClient = lambdaClient;
    }

    public bool CanExecute(IQuery query) => query is ILambdaQuery;

    public async Task<IQueryResult> Execute(IQuery query)
    {
        var lambdaQuery = (ILambdaQuery)query;
        
        var request = new InvokeRequest
        {
            FunctionName = lambdaQuery.FunctionName,
            Payload = JsonSerializer.Serialize(lambdaQuery.GetPayload())
        };

        var response = await lambdaClient.InvokeAsync(request);
        var responsePayload = Encoding.UTF8.GetString(response.Payload.ToArray());
        
        return lambdaQuery.ProcessLambdaResponse(responsePayload);
    }
}

Performance Monitoring Extensions

1. Application Insights Integration

public class ApplicationInsightsDataProvider<T> : IDataProvider<T> where T : IEntity
{
    private readonly IDataProvider<T> innerProvider;
    private readonly TelemetryClient telemetryClient;

    public T GetData(IEntityRequest request)
    {
        using var operation = telemetryClient.StartOperation<DependencyTelemetry>("Schemio.GetData");
        operation.Telemetry.Type = "Schemio";
        operation.Telemetry.Target = typeof(T).Name;

        try
        {
            var result = innerProvider.GetData(request);
            operation.Telemetry.Success = true;
            
            telemetryClient.TrackMetric($"Schemio.{typeof(T).Name}.Success", 1);
            
            return result;
        }
        catch (Exception ex)
        {
            operation.Telemetry.Success = false;
            telemetryClient.TrackException(ex);
            telemetryClient.TrackMetric($"Schemio.{typeof(T).Name}.Error", 1);
            throw;
        }
    }
}

2. Prometheus Metrics

public class PrometheusDataProvider<T> : IDataProvider<T> where T : IEntity
{
    private readonly IDataProvider<T> innerProvider;
    private static readonly Counter RequestCounter = Metrics
        .CreateCounter("schemio_requests_total", "Total requests", new[] { "entity_type", "status" });
    private static readonly Histogram RequestDuration = Metrics
        .CreateHistogram("schemio_request_duration_seconds", "Request duration", new[] { "entity_type" });

    public T GetData(IEntityRequest request)
    {
        var entityType = typeof(T).Name;
        
        using (RequestDuration.WithLabels(entityType).NewTimer())
        {
            try
            {
                var result = innerProvider.GetData(request);
                RequestCounter.WithLabels(entityType, "success").Inc();
                return result;
            }
            catch
            {
                RequestCounter.WithLabels(entityType, "error").Inc();
                throw;
            }
        }
    }
}

Testing Extensions

1. Test Data Providers

public class TestDataProvider<T> : IDataProvider<T> where T : IEntity, new()
{
    private readonly Dictionary<string, T> testData;

    public TestDataProvider(Dictionary<string, T> testData)
    {
        this.testData = testData;
    }

    public T GetData(IEntityRequest request)
    {
        var key = GenerateKey(request);
        return testData.TryGetValue(key, out var data) ? data : new T();
    }

    private string GenerateKey(IEntityRequest request)
    {
        // Generate key based on request properties
        return request.GetType().Name + "_" + GetRequestId(request);
    }
}

Usage in Tests

[Test]
public void Should_Return_Customer_Data()
{
    var testData = new Dictionary<string, Customer>
    {
        ["CustomerRequest_123"] = new Customer
        {
            CustomerId = 123,
            Name = "Test Customer",
            Email = "test@example.com"
        }
    };

    var testProvider = new TestDataProvider<Customer>(testData);
    var request = new CustomerRequest { CustomerId = 123 };
    
    var result = testProvider.GetData(request);
    
    Assert.AreEqual(123, result.CustomerId);
    Assert.AreEqual("Test Customer", result.Name);
}

2. Mock Query Engines

public class MockQueryEngine : IQueryEngine
{
    private readonly Dictionary<Type, Func<IQuery, IQueryResult>> mockImplementations;

    public MockQueryEngine()
    {
        mockImplementations = new Dictionary<Type, Func<IQuery, IQueryResult>>();
    }

    public MockQueryEngine Setup<TQuery>(Func<TQuery, IQueryResult> implementation) 
        where TQuery : IQuery
    {
        mockImplementations[typeof(TQuery)] = query => implementation((TQuery)query);
        return this;
    }

    public bool CanExecute(IQuery query) 
        => mockImplementations.ContainsKey(query.GetType());

    public Task<IQueryResult> Execute(IQuery query)
    {
        var implementation = mockImplementations[query.GetType()];
        var result = implementation(query);
        return Task.FromResult(result);
    }
}

Conclusion

Schemio provides a powerful, extensible framework for aggregating data from heterogeneous data sources. Its schema-driven approach, combined with flexible query and transformation capabilities, makes it an ideal choice for modern applications that need to combine data from multiple systems.

Key Takeaways

When to Use Schemio

• Microservices Architectures: When you need to aggregate data from multiple services
• Legacy System Integration: When modernizing applications that need to pull from various data sources
• API Gateway Patterns: When building composite APIs that combine multiple backend services
• Data Migration Projects: When moving from monolithic to distributed architectures
• Reporting Systems: When building reports that require data from multiple sources

Getting Support