Siili Solutions builds near real-time data dashboard using Azure Databricks

Siili Solutions Plc has detailed a methodology for constructing near real-time data dashboards by leveraging Microsoft's Azure Databricks service.

The approach involves ingesting real-time traffic data from the Finnish Transport Agency's open data API. This data is captured by a custom .NET Core container application, processed via Azure Databricks using Event Hub, and finally visualized with Microsoft Power BI.

Siili Solutions highlights Azure Databricks as a flexible, distributed computing platform suitable for complex data science problems, ETL tasks, and real-time data handling. The chosen architecture, featuring a custom container, also allows for future scalability, such as migration to Kubernetes.

Data is sent to Azure Event Hub for intermediate processing and temporary storage (up to seven days). Azure Databricks then refines the data, which is stored in Databricks Delta tables before being presented in Power BI.

Upcoming steps include configuring Azure Databricks to receive events from Event Hub, implementing the Digitraffic API interface, and designing the Power BI visualizations.