Spark Declarative Pipelines

Apache Spark Declarative Pipelines (part of Databricks Lakeflow) simplifies batch and streaming ETL: engineers declare the data transformations they need while the platform automatically handles ingestion from any Apache Spark-supported source, dependency management, scaling and recovery, and data quality rules via Expectations. It offers unified batch and streaming processing, end-to-end incremental processing, an integrated pipeline development IDE, and is built on Unity Catalog and open table formats.

Official product page

16published use cases

12industries

7countries on record

Predictive Analytics

top AI capability

Evidence mix: High 16 · Medium 0 · Low 0 — bands are computed from each record's evidence signals.

Industry

Country

16 use cases

HealthcarePredictive AnalyticsRecommendation & Personalization

Tata 1mg expands healthcare accessibility across India with Databricks

Indian healthcare company Tata 1mg migrated from a legacy data warehouse to the Databricks Data + AI Platform with Unity Catalog and Delta Lake, reducing ETL costs by 50% and enabling over 1,000 non-data business users to access insights for location analytics, delivery route optimization and predictive maintenance.

Tata 1mg· IndiaDelta Lake · Delta Sharing · Spark Declarative Pipelines +1

MarketingMachine LearningPredictive Analytics

MissionWired drives donor campaigns through predictive data on Databricks

MissionWired used Databricks (Spark Declarative Pipelines, Unity Catalog, Databricks SQL) to process 275 billion donor data points across its Cooperative Email Database aggregating 150+ nonprofit organizations, helping raise $350 million for clients since 2020 with member organizations achieving full ROI within 12 months.

MissionWired· United StatesDatabricks SQL · Delta Lake · Spark Declarative Pipelines +2

Technology & SoftwareMachine Learning

Making urban mobility effortless with smarter parking solutions

Estapar, Brazil's largest parking network, unified over 600 data sources on the Databricks Data + AI Platform, using AI/BI Genie for self-service analytics and Spark Declarative Pipelines for automation, cutting license plate analysis from three days to real time and delivering a 20x productivity boost.

Estapar· BrazilDelta Lake · Lakeflow Jobs · Spark Declarative Pipelines +3

E-commercePredictive AnalyticsFraud & Anomaly DetectionRecommendation & Personalization

ElasticRun helps small businesses improve supply chain with Databricks

Indian B2B e-commerce platform ElasticRun rebuilt its data infrastructure on Databricks with Delta Lake, Spark Declarative Pipelines and MLflow to manage over 10,000 machine learning models for supply chain and demand forecasting, cutting data pipeline slowdowns by 90% and IT costs by 33%.

ElasticRun· IndiaDelta Lake · Spark Declarative Pipelines · Lakeflow Jobs +2

AgriculturePredictive Analytics

UPL optimizes supply chain operations with demand forecasting on Databricks

Global agriculture company UPL unified internal and third-party data sources (SAP, Salesforce, IoT, weather) on the Databricks Data + AI Platform with Unity Catalog to build a demand forecasting model that reduces product spoilage and supports AI-driven crop management and pest control across 20 countries.

UPLDatabricks SQL · Delta Lake · Delta Sharing +2

Technology & SoftwareAI Model Development & MLOps

Kongsberg Digital achieves 60% cost reduction and real-time data refresh with Databricks

Norwegian industrial SaaS provider Kongsberg Digital modernized its data warehouse on Databricks, increasing data refresh frequency from four times daily to hourly (a 300% increase) and cutting data processing costs by 60%.

Kongsberg Digital· NorwayDelta Lake · Spark Declarative Pipelines · Lakeflow Jobs +2

CybersecurityGenerative AIFraud & Anomaly Detection

Palo Alto Networks advances cybersecurity with the Databricks Data + AI Platform

Palo Alto Networks adopted the Databricks Data + AI Platform, including Unity Catalog, Delta Lake and Spark Declarative Pipelines, to unify fragmented data across its Prisma Cloud modules, achieving 3x faster iterations on AI/ML features, a 20% reduction in COGS, 3x decrease in engineering development time, and 40% less time on data preparation.

Palo Alto NetworksUnity Catalog · Delta Lake · Spark Declarative Pipelines +2

EnergyPredictive AnalyticsMachine Learning

Using data to power-fuel the transition to a carbon-neutral world

Helen, Helsinki's energy utility, built a centralized data and AI platform on Databricks to power forecasting and optimization models for its district heating system serving about 90% of Helsinki's population, processing real-time streaming sensor and IoT data to optimize distributed energy resources and EV charging placement as part of a plan to cut carbon emissions over 80% by decommissioning coal plants.

Helen· FinlandDatabricks Data + AI Platform · Delta Lake · Spark Declarative Pipelines +3

Technology & SoftwareRetrieval-Augmented GenerationGenerative AI

Transforming supply chain data into real-time intelligence with GenAI

Accuris built a GenAI-powered engineering assistant on the Databricks Data + AI Platform, using RAG, AI Search, MLflow and Model Serving to index over 40 million records across 1 billion+ parts, achieving a 10x increase in development velocity from concept to customer-facing feature.

AccurisAgent Bricks · Delta Lake · Lakeflow Jobs +2

Travel & HospitalityRecommendation & PersonalizationPredictive Analytics

Fueling personalized travel with real-time insights

Adani Digital Labs replaced legacy systems with the Databricks Data + AI Platform to power real-time, AI-driven personalization on the Adani OneApp, launching 15+ production use cases including geospatial analysis and ML-driven product recommendations, cutting operational costs by 29% and development time by 20%.

Adani Digital Labs· IndiaDatabricks SQL · Delta Lake · Spark Declarative Pipelines +1

Financial ServicesConversational AI

Connecting the global investment ecosystem

Addepar adopted the Databricks Data + AI Platform to centralize disparate datasets, streamline workflows and leverage advanced AI capabilities at scale for a platform managing more than $7 trillion in assets, reducing pipeline development times by 3-5x, accelerating client onboarding, and cutting infrastructure costs by 32%, saving $2 million annually.

Addepar, a global technology and data company connecting the investment ecosystemAgent Bricks · Databricks SQL · Delta Lake +3

HealthcareAgentic AIConversational AI

Novo Nordisk Transforms Clinical Trials With Generative AI

Novo Nordisk, a Denmark-headquartered global healthcare and pharmaceutical company, used the Databricks Data + AI Platform to unify previously siloed clinical trial data across its FounData and DataCore platforms. In 9 months it moved from concept to MVP, and now over 20% of its development organization actively uses clinical data via FounData. The company built a 'Co-Scientist' AI agent system on Databricks Agent Bricks, Unity Catalog and AI/BI Genie to search clinical data and surface curated insights, with an expected $157M+ in net new value and $14M in efficiency gains from optimized clinical trials over five years.

Novo Nordisk· DenmarkUnity Catalog · Agent Bricks · AI/BI Genie +2

RetailGenerative AIPredictive Analytics

Cultivating the Perfect Lawn for 2.3 Million Customers

TruGreen migrated to the Databricks Data + AI Platform, using Databricks SQL on Azure for its data warehouse, Unity Catalog for governance, Lakeflow Spark Declarative Pipelines to stream data from Dynamics 365, and Qlik Replicate for legacy ERP integration. It built TruSight, an AI-powered reporting solution delivering daily insights to 200+ branch and regional managers before crews start work at 7 AM, evaluated against ChatGPT and multiple Microsoft Copilot variants before selecting Databricks for accuracy, cost, and control over context and LLM choice. TruGreen deployed Claude Sonnet 4.5 via Databricks Model Serving (External Models), using MLflow's LLM judges to evaluate AI output quality, and has churn and lifetime-value machine learning models in production. Databricks SQL cut data warehousing costs by roughly 50%, and automated data lineage in Unity Catalog reduced ERP impact analysis from weeks to minutes (a 99% improvement).

TruGreenAgent Bricks · Databricks SQL · Spark Declarative Pipelines +4

Media & EntertainmentConversational AI

Amagi Ensures Millions Never Miss Their Shows

Amagi, a media technology company supporting 8,000+ streaming channels and 300+ distributors, consolidated fragmented data systems (Dataproc, Snowflake, and in-house platforms) across AWS and GCP into the Amagi Data Platform (ADP), a unified lakehouse built on Databricks with Unity Catalog governance. The platform serves real-time data for ad decisioning, content performance analytics, billing and customer reporting to internal teams (via notebooks, Genie, SQL warehouses) and customer-facing products. Amagi reports 45% cost savings from consolidating fragmented data systems, 15x faster time to market for new data products, and 35% fewer production incidents due to improved data reliability.

AmagiDatabricks SQL · Delta Lake · Spark Declarative Pipelines +2

RetailPredictive Analytics

Operationalizing Analytics Data at Scale (DEICHMANN SE)

DEICHMANN SE, Europe's largest footwear retailer with roughly 4,700 stores in more than 30 countries and about 8.7 billion euros in 2024 revenue, adopted Databricks Lakebase, a managed PostgreSQL database integrated with its Databricks Lakehouse, to activate governed customer and product data for e-commerce marketing without building separate integration pipelines. Lakebase reduced a data-product publishing pipeline to 3 clicks and about 30 seconds, and is now in production across Europe feeding SAP Emarsys for e-commerce marketing.

DEICHMANN SE· GermanyDatabricks Lakebase · Databricks SQL · Delta Lake +3

EducationMachine LearningFraud & Anomaly DetectionAI Model Development & MLOps

GoGuardian: Safer schools, empowered teachers, thriving students

GoGuardian, which powers safe, focused learning for half of U.S. K-12 students, migrated its ML infrastructure to Databricks to manage billions of daily inferences for web filtering, classroom management and harm prevention while maintaining a PII-free, COPPA/FERPA-compliant data environment. Using Delta Lake, Lakeflow, Unity Catalog, MLflow and Databricks Model Serving, GoGuardian achieved up to 50% reduction in machine learning operational costs, 90% operational cost savings with its Delphi website classification model, and a 62% reduction in inappropriate device use among students. AI-driven prioritization also cut the volume of records requiring human review for high-risk content by over 95%, from 1 million to 35,000-45,000.

GoGuardian· United StatesDelta Lake · Databricks Lakeflow · Unity Catalog +6