Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks

Databricks announced that Real-Time Mode (RTM) in Spark Declarative Pipelines (SDP) is now Generally Available, bringing millisecond-range end-to-end latency to SDP while retaining its orchestration, incremental processing, failure recovery and data quality features. RTM streams data continuously and runs all query stages simultaneously, and Databricks reports it was up to 92% faster than Apache Flink on customer feature-computation workloads. Users enable it by setting the pipeline to continuous mode with spark.databricks.streaming.realTimeMode.enabled set to true and defining an update flow with pipelines.trigger set to "RealTime", and can write to custom sinks such as Lakebase or Redis via a ForEach sink.