How to consume Azure Databricks data table from external Client developed in C#

Question

Our company has migrated all its data into a Azure Databricks. Now we have to connect our API and client app developed in .Net C# to this Azure Databricks to retrieve data.

I found that it is possible to use JDBC/ODBC Databricks Simba driver to connect directly to the cluster Spark, but I'm pretty sure that it is not the good practice to use directly Databricks, more over on Azure we can not install new ODBC driver without using Self hosted VM.

I'm unable to find the good and best pratices Architecture for my need.

I'm looking for the best Architecture ratio between simplicity, performance and cost :-)

Do I have to create an Azure DB/CosmoDB and populate it in realtime with a specific pipeline from Databricks and retrieve data from those DB, do I have to use Data Factory, Event Hud, etc...

Any ideas ?

Many thanks Regards