Azure Data Engineer

Our client is a large enterprise in the energy sector with a substantial Azure data platform at the center of its reporting and operations. It is hiring an experienced data engineer to design, build and support the pipelines that move and shape that data, from ingestion through to the reporting layer.

This is a hands-on individual contributor role for an engineer who is deep in the Azure data stack: Synapse, Data Factory and Databricks, backed by strong SQL Server and Python/PySpark skills. You will build complex orchestration workflows, process large datasets in Spark, write the stored procedures and dynamic SQL behind them, and promote your work through a Git-based CI/CD process from development to production.

Production support is part of the role, so you will also troubleshoot data issues and work closely with business, application and data teams. It is a long-term engagement, expected to run 12 months or more.

Job responsibilities

• Design, build and maintain data solutions on Azure Synapse Analytics, Azure Data Factory and Databricks, using Azure Storage, SQL Pools and Azure Key Vault

• Develop ADF and Synapse linked services, datasets, pipelines, activities and integration runtimes

• Build complex orchestration workflows using ForEach, If, Switch, Until, Lookup, Stored Procedure, Script, Validation, Copy Data and Data Flow activities

• Write and maintain SQL Server stored procedures, functions, dynamic SQL and complex data-processing logic

• Process large datasets with Spark Pools, PySpark and Databricks notebooks, including incremental/delta loads and data-merging solutions

• Develop Python solutions with PySpark, Pandas, NumPy and PyMySQL, retrieving and processing data through REST APIs and stored procedures

• Debug Python and PySpark code, resolving library, dependency and runtime issues

• Build and support integrations using Azure Functions and Logic Apps

• Implement and support CI/CD pipelines, promoting code from development through QA to production with Git-based deployment

• Develop and support Power BI reporting and data visualization

• Troubleshoot production data issues alongside business, application and data teams

Candidate requirements

• Strong hands-on experience with Azure Synapse Analytics, Azure Data Factory and Azure Databricks; this is core to the role, not a nice-to-have

• Deep SQL Server skills, including stored procedures, dynamic SQL and complex data-processing logic

• Fluency in Python and PySpark, with Spark experience processing large datasets

• Experience building ETL/data pipelines with delta and incremental loads

• Working knowledge of Azure Storage, Azure Key Vault and SQL Pools

• Experience with Git and CI/CD pipelines across development, QA and production environments

• Comfortable working with APIs and troubleshooting production data issues

• Power BI experience and exposure to Azure Functions and Logic Apps are desirable

• Candidates must be able to work onsite in Jackson, Michigan three days a week; this is a hybrid role