Overview

We are looking for a Data Engineer who will design, develop, and optimize data pipelines and warehouse layers on Google BigQuery. You will work with real time streaming, batch processing, and MLflow based workflows while ensuring data quality, reliability, and performance across all components of the platform. Project Overview: We are building a modern data warehousing platform that supports both real time and batch analytical workflows. The project focuses on designing scalable data pipelines, implementing a medallion architecture, and enabling seamless collaboration across data engineering, analytics, and data science teams.

Responsibilities:
  • Plan, develop, and maintain ETL and ELT pipelines across Bronze, Silver, and Gold layers
  • Implement the medallion architecture with a focus on data lineage and quality
  • Build and optimize real time data streaming pipelines using Pub/Sub and Apache Beam on Dataflow
  • Create and orchestrate batch workflows using Cloud Composer and Apache Airflow
  • Write performant and cost efficient BigQuery SQL using partitioning, clustering, and query optimization
  • Collaborate with analysts and data scientists to understand data needs and deliver reliable solutions
  • Prepare and maintain technical documentation, data dictionaries, and monitoring dashboards
Required Qualifications:
  • 4 years of Data Engineering experience
  • 3 or more years of experience designing and building cloud based ETL and ELT pipelines
  • Hands on experience with data modeling including dimensional modeling and schema design
  • Experience working with real time streaming architectures and event driven data processing
  • Proficiency in SQL and at least one programming language such as Python, Java, or Scala
  • Experience with Google Cloud Pub/Sub for message driven ingestion
  • Practical experience building Apache Beam pipelines on Google Cloud Dataflow
  • Experience with workflow orchestration in Cloud Composer or Apache Airflow
  • Hands on experience with optimizing BigQuery SQL, partitioning, clustering, resource usage, and cost management
Nice To Have:
  • Experience using BigQuery ML for model creation and deployment
  • Foundational understanding of machine learning workflows and feature engineering
  • Experience with Terraform for infrastructure provisioning
  • Familiarity with data quality frameworks such as Great Expectations or dbt tests
  • Google Cloud Professional Data Engineer certification
Note:

✨ Our intelligent job search engine discovered this job and republished it for your convenience.
Please be aware that the job information may be incorrect or incomplete. The job announcement remains the property of its original publisher. To view the original job and its full details, please visit the job's URL on the owner’s page.

Please clearly mention that you have heard of this job opportunity on https://ijob.am.