data
Posted 3 days agoSenior Data Engineer
at CAE
On-site
Responsibilities
- Support and improve the enterprise data platform, including data warehouses and data lakes.
- Design, build, and maintain data pipelines using SQL and big data tools such as Spark, Kafka, or Hadoop.
- Develop reliable processes to ingest, transform, and deliver data from a variety of sources.
- Integrate data from internal, external, and IoT sources to support business needs.
- Build and maintain backend data solutions that support analysts, data scientists, and other business users.
- Identify opportunities to automate manual work and improve data delivery processes.
- Monitor data pipelines and provide support when issues arise.
- Collaborate with cross-functional teams on data and AI initiatives.
Requirements
- experience in modern data engineering. The ideal candidate has
- experience with Python, Spark, Git, Microsoft Fabric, and medallion architecture across Bronze, Silver, and Gold layers.
- Qualifications Bachelor’s degree in Computer Science , Engineering, or a related field. A master’s degree is an asset. At least 6 years of
- experience in data engineering with strong hands-on coding in Python, Spark, SQL, and related scripting languages.
- Experience using Visual Studio Code and Git in modern development workflows. Required hands-on
- experience with Microsoft Fabric, including Lakehouse s , OneLake , pipelines, notebooks, and medallion architecture across Bronze, Silver, and Gold layers.
- Strong understanding of computer science fundamentals, including data structures and algorithms.
- Experience with Terraform.
- Experience working with cross-functional teams in a fast-paced environment.
- Experience with relational and NoSQL databases, including SQL Server, PostgreSQL, and Cosmos DB. Preferred
- Experience with workflow and orchestration tools such as Azure Data Factory and Airflow.
- Experience with DevOps and Github and infrastructure deployment tools such as ARM.
- Experience with Microsoft cloud and data platforms such as Fabric, Azure, Databricks, and Purview. Knowledge of medallion architecture across Bronze, Silver, and Gold layers.