Overview

We are looking for a Senior Data Engineer to build and operate a clinical data platform on Databricks, with a strong focus on robust data pipelines, data models, and data quality. Project Overview: You will work on a platform that processes clinical and real-world data (EHRs, labs, registries, trial data) and powers analytics, reporting, and data products for a healthcare / clinical research client.

Responsibilities:
  • Design, build, and maintain end-to-end Databricks data pipelines (ingestion, transformation, publishing) for production use
  • Work with data models (staging, curated, canonical, or dimensional) and help evolve them together with architects and analysts
  • Embed data quality and data governance rules into all pipelines (checks, validation, monitoring, alerting)
  • Optimize Databricks jobs for performance and cost (cluster configuration, partitioning, caching, file layout)
  • Collaborate with data architects, analysts, and domain experts to clarify requirements and refine technical solutions
Required Qualifications:
  • 5+ years of experience in data engineering, DWH, or big data, including production data pipelines
  • Strong hands-on experience with Databricks: Spark (PySpark/Scala), Delta Lake, Databricks Jobs / Workflows
  • Proven experience designing and operating end-to-end pipelines on Databricks for batch or near-real-time data
  • Experience with data pipelines and CI/CD for data
  • Practical experience with data modeling (layered models, canonical or dimensional models) for analytics and reporting
  • Experience embedding data quality and data governance rules into pipelines (schema checks, business rules, SLOs, monitoring)
  • Good communication skills, upper-intermediate or higher English proficiency, and the ability to work closely with stakeholders in distributed teams and communicate directly with clients
Nice To Have:
  • Experience designing and delivering PoC solutions on Databricks to quickly validate ideas using real data
  • Experience with ontologies or a semantic layer (business concepts, metrics, mappings) on top of analytical data
Note:

✨ Our intelligent job search engine discovered this job and republished it for your convenience.
Please be aware that the job information may be incorrect or incomplete. The job announcement remains the property of its original publisher. To view the original job and its full details, please visit the job's URL on the owner’s page.

Please clearly mention that you have heard of this job opportunity on https://ijob.am.