How do I define custom/dynamic columns/fields/properties in OData (v8 in .NET 6)?

Question

There is a sample here for creating 100% dynamic OData models in Microsoft.AspNetCore.OData 8.x. However, in our case we have an existing model that we are happy with, but we want to add custom fields to it.

In other words, we want an OData model with entities that have some fixed columns/properties and some dynamically-generated columns/properties that come from the database, like this:

public class ODataEntity
{
    [Key]
    public int Id { get; set; }
    public string Name { get; set; } = "";

    // From the perspective of clients like Power BI, this should produce 
    // a series of additional columns (the columns are the same on all 
    // instances, but the schema can change at any time)
    public Dictionary<string, object> CustomFields { get; set; }
}

To my tremendous surprise, the key-value pairs in CustomFields become properties in the JSON output (i.e. there is no CustomFields column; its contents are inserted into the parent object). However, the custom fields are not recognized by Power BI:

I assume that this is because there is no metadata for the custom fields in https://.../odata/$metadata. So my question is:

1. How can I modify the following code so that the custom columns are included in the IEdmModel?

   static IEdmModel GetEdmModel(params CustomFieldDef[] customFields)
   {
       var builder = new ODataConventionModelBuilder() {
           Namespace = "Namespace",
           ContainerName = "Container", // no idea what this is for
       };
       builder.EntitySet<ODataEntity>("objects");
   
       return builder.GetEdmModel();
   }
   
   public class CustomFieldDef
   {
       public string FieldName;
       public Type Type;
   }
   

2. How can I modify the following startup code so that the IEdmModel is regenerated every time https://.../odata/$metadata is accessed?

   IMvcBuilder mvc = builder.Services.AddControllers();
   
   mvc.AddOData(opt => opt.AddRouteComponents("odata", GetEdmModel())
      .Select().Filter().OrderBy().Count().Expand().SkipToken());
   

Answer 1

Regarding the first question, there are two basic approaches that one could take:

1. Dynamic Everything

Use the 100% dynamic approach that is used in the ODataDynamicModel sample. This approach is difficult, especially if you already have a working model, because (i) the way the sample code works is difficult to understand, and (ii) you have to completely rewrite your code, or use reflection to help generate both the schema ($metadata) and the output data.

2. Modify the EdmModel

This is the approach I'm taking in this answer.

Step 1: change the model

The IEdmModel returned by ODataConventionModelBuilder.GetEdmModel is mutable (and actually has type EdmModel), so you can add custom fields to it.

static IEdmModel GetEdmModel(params CustomFieldDef[] customFields)
{
    var builder = new ODataConventionModelBuilder() {
        Namespace = "Namespace",
        ContainerName = "Container", // no idea what this is for
    };

    builder.EntitySet<ODataEntity>("objects");

    var model = (EdmModel) builder.GetEdmModel();

    IODataTypeMapper mapper = model.GetTypeMapper();

    foreach (var edmType in model.SchemaElements.OfType<EdmEntityType>()) {
        if (edmType.Name == nameof(ODataEntity)) {
            foreach (var field in customFields) {
                var typeRef = mapper.GetEdmTypeReference(model, field.Type);
                edmType.AddStructuralProperty(field.FieldName, typeRef);
            }
        }
    }

    return model;
}

Step 2: Obtain the custom fields

Normally you call mvc.AddOData in ConfigureServices in Startup.cs, passing it a lambda that will create the IEdmModel. But wait, are your custom field definitions stored in your database? If so, how can you access the database from inside ConfigureServices? Don't worry: the lambda passed to AddOData is called after Configure, so it is possible to access the database or any other required service with some slightly hacky code. This code also installs a class called ODataResourceSerializerForCustomFields which is the subject of step 3:

    public void ConfigureServices(IServiceCollection services)
    {
        ...
        IMvcBuilder mvc = services.AddControllers(...);

        // OData Configuration
        mvc.AddOData(opt => {
            var scopeProvider = _serviceProvider!.CreateScope().ServiceProvider;
            var cfdm = scopeProvider.GetRequiredService<CustomFieldDefManager>();
            var edmModel = GetEdmModel(cfdm.GetAll());

            opt.AddRouteComponents("odata", edmModel, services => {
                services.AddScoped<ODataResourceSerializer, 
                    ODataResourceSerializerForCustomFields>();
            }).Select().Filter().OrderBy().Count().Expand().SkipToken();
        });
        ...
    }

    IServiceProvider? _serviceProvider;

    public void Configure(IApplicationBuilder app, ...)
    {
        _serviceProvider = app.ApplicationServices;
        ...
    }

Of course, this code is not dynamic: it generates the EdmModel only once on startup. I will figure out later how to make this dynamic.

Step 3: Stop it from crashing

Microsoft.AspNetCore.OData.dll isn't designed to support custom fields in the EdmModel. It's a young product, you understand, only version 8.0. As soon as you add a custom field, you'll get an InvalidOperationException like "The EDM instance of type '[ODataEntity Nullable=True]' is missing the property 'ExampleCustomField'., because the library assumes that all properties in the EdmModel are real CLR properties.

I found a way around this problem by overriding a few methods of ODataResourceSerializer. But first, define an IHasCustomFields interface and make sure that any OData entity with custom fields implements this interface:

public interface IHasCustomFields
{
    public Dictionary<string, object?> CustomFields { get; set; }
}

Now let's add ODataResourceSerializerForCustomFields, which uses special behavior when this interface is present.

/// <summary>
/// This class modifies the behavior of ODataResourceSerializer to 
/// stop it from crashing when the EdmModel contains custom fields.
/// Note: these modifications are designed for simple custom fields 
/// (e.g. string, bool, DateTime).
/// </summary>
public class ODataResourceSerializerForCustomFields : ODataResourceSerializer
{
    public ODataResourceSerializerForCustomFields(IODataSerializerProvider serializerProvider)
        : base(serializerProvider) { }

    IHasCustomFields? _hasCustomFields;
    HashSet<string>? _realProps;

    public override Task WriteObjectInlineAsync(
        object graph, IEdmTypeReference expectedType,
        ODataWriter writer, ODataSerializerContext writeContext)
    {
        _hasCustomFields = null;
        if (graph is IHasCustomFields hasCustomFields) {
            _hasCustomFields = hasCustomFields;
            var BF = BindingFlags.Public | BindingFlags.Instance;
            _realProps = graph.GetType().GetProperties(BF).Select(p => p.Name).ToHashSet();
        }
        return base.WriteObjectInlineAsync(graph, expectedType, writer, writeContext);
    }

    public override ODataResource CreateResource(
        SelectExpandNode selectExpandNode, ResourceContext resourceContext)
    {
        return base.CreateResource(selectExpandNode, resourceContext);
    }

    public override ODataProperty CreateStructuralProperty(
        IEdmStructuralProperty structuralProperty, ResourceContext resourceContext)
    {
        // Bypass tne base class if the current property doesn't physically exist
        if (_hasCustomFields != null && !_realProps!.Contains(structuralProperty.Name)) {
            _hasCustomFields.CustomFields.TryGetValue(structuralProperty.Name, out object? value);

            return new ODataProperty {
                Name = structuralProperty.Name,
                Value = ToODataValue(value)
            };
        }

        return base.CreateStructuralProperty(structuralProperty, resourceContext);
    }

    public static ODataValue ToODataValue(object? value)
    {
        if (value == null)
            return new ODataNullValue();
        if (value is DateTime date)
            value = (DateTimeOffset)date;

        return new ODataPrimitiveValue(value);
    }

    // The original implementation of this method can't be prevented from
    // crashing, so replace it with a modified version based on the original
    // source code. This version is simplified to avoid calling `internal`
    // methods that are inaccessible, but as a result I'm not sure that it
    // behaves quite the same way. If properties that aren't in the EdmModel
    // aren't needed in the output, the method body is optional and deletable.
    public override void AppendDynamicProperties(ODataResource resource, 
        SelectExpandNode selectExpandNode, ResourceContext resourceContext)
    {
        if (_hasCustomFields == null) {
            base.AppendDynamicProperties(resource, selectExpandNode, resourceContext);
            return;
        }

        if (!resourceContext.StructuredType.IsOpen || // non-open type
            (!selectExpandNode.SelectAllDynamicProperties && selectExpandNode.SelectedDynamicProperties == null)) {
            return;
        }

        IEdmStructuredType structuredType = resourceContext.StructuredType;
        IEdmStructuredObject structuredObject = resourceContext.EdmObject;
        object value;
        if (structuredObject is IDelta delta) {
            value = ((EdmStructuredObject)structuredObject).TryGetDynamicProperties();
        } else {
            PropertyInfo dynamicPropertyInfo = resourceContext.EdmModel.GetDynamicPropertyDictionary(structuredType);
            if (dynamicPropertyInfo == null || structuredObject == null ||
                !structuredObject.TryGetPropertyValue(dynamicPropertyInfo.Name, out value) || value == null) {
                return;
            }
        }

        IDictionary<string, object> dynamicPropertyDictionary = (IDictionary<string, object>)value;

        // Build a HashSet to store the declared property names.
        // It is used to make sure the dynamic property name is different from all declared property names.
        HashSet<string> declaredPropertyNameSet = new HashSet<string>(resource.Properties.Select(p => p.Name));
        List<ODataProperty> dynamicProperties = new List<ODataProperty>();

        // To test SelectedDynamicProperties == null is enough to filter the dynamic properties.
        // Because if SelectAllDynamicProperties == true, SelectedDynamicProperties should be null always.
        // So `selectExpandNode.SelectedDynamicProperties == null` covers `SelectAllDynamicProperties == true` scenario.
        // If `selectExpandNode.SelectedDynamicProperties != null`, then we should test whether the property is selected or not using "Contains(...)".
        IEnumerable<KeyValuePair<string, object>> dynamicPropertiesToSelect =
            dynamicPropertyDictionary.Where(x => selectExpandNode.SelectedDynamicProperties == null || selectExpandNode.SelectedDynamicProperties.Contains(x.Key));
        foreach (KeyValuePair<string, object> dynamicProperty in dynamicPropertiesToSelect) {
            if (string.IsNullOrEmpty(dynamicProperty.Key))
                continue;
            if (declaredPropertyNameSet.Contains(dynamicProperty.Key))
                continue;

            dynamicProperties.Add(new ODataProperty {
                Name = dynamicProperty.Key,
                Value = ToODataValue(dynamicProperty.Value)
            });
        }

        if (dynamicProperties.Count != 0)
            resource.Properties = resource.Properties.Concat(dynamicProperties);
    }
}