Data Ingestion Architecture for Ignition

Getting operational data from industrial systems into business-ready dashboards requires an end-to-end data pipeline from ingestion to transformation to visualization.

Ignition’s Azure Injector Module enables real-time event publishing from the Ignition Gateway directly into Azure Event Hub. From there, Azure’s ecosystem—Stream Analytics, Storage, Databricks—can process and prepare the data for use in Power BI.

This approach offers a low-latency, Azure-native path from the plant floor to the boardroom, without adding intermediate brokers or additional infrastructure layers. An overview of the dataflow is shown below.

Using the Azure Injector Module with Azure Event Hub

The Azure Injector Module sends data from Ignition tags, alarms, or queries directly to Azure Event Hub using AMQP. Event Hub then acts as the ingestion endpoint for your entire analytics pipeline.

Advantages:

• Low latency — data appears in Event Hub within seconds
• No extra broker layer — reduced complexity and maintenance
• Azure-native — tight integration with Stream Analytics, Databricks, and Azure Storage
• Flexible message formats — supports JSON or Avro with customizable field mappings

Considerations:

• Requires Event Hub throughput units and Stream Analytics jobs (Azure cost model applies)
• For bulk historical data exports, a separate batch export strategy may be needed

Once data is in Event Hub, Stream Analytics can:

• Store messages as Parquet in Blob Storage
• Stream data into Delta tables in Databricks for transformation
• Trigger alerts or feed real-time Power BI dashboards

Transforming Bronze Data into Delta Live Tables

Event Hub messages typically arrive in a narrow, raw event log format. In Databricks, Delta Live Tables (DLTs) can be used to:

• Filter out irrelevant records
• Add context such as equipment names, units, or location metadata
• Reshape data from narrow to wide format for BI consumption
• Output clean, structured datasets for analytics

The transformed datasets—often called silver or gold data—are then ready for direct use in BI tools.

Power BI Integration

Once the curated Delta tables are available, Power BI can connect directly to the Databricks SQL Warehouse. This enables:

• Live queries on up-to-date datasets
• Interactive dashboards without manual exports
• Centralized data governance within the Azure environment

Narrow vs Wide Format for BI

Most IoT and OT ingestion pipelines deliver narrow format data, where each row represents a single measurement with a timestamp, sensor ID, metric type, and value.

BI tools like Power BI perform better with wide format data, where each row contains all metrics from a single sensor at a single timestamp.

The ETL process in Databricks handles this pivot, making the data more intuitive and improving query performance for dashboards.

Conclusion: Choosing the Right Flow

The Azure Injector Module provides a streamlined and effective path to get Ignition data into Azure for analysis and visualization.

Ideal when:

• Low-latency dashboards and alerting are required
• You want a direct connection to Azure without Kafka or other intermediaries
• You already use Azure services and want to minimize architectural complexity

The result is a lean but powerful architecture: