Senior Azure Data Engineering Expert

Project Overview

We are seeking an experienced Senior Data Engineer with strong Azure cloud, data engineering, and AI/ML platform development expertise to independently design, develop, and deliver a scalable enterprise-grade data platform.

The project involves building and improving data processing solutions, developing reliable data pipelines, integrating cloud-based services, and preparing data infrastructure to support AI, machine learning, and advanced analytics applications.

This is an independent contract engagement. The selected developer will take end-to-end responsibility for technical execution, from architecture and implementation through testing, documentation, and final delivery.

We are looking for someone who can work autonomously, make sound technical decisions, troubleshoot complex problems, and deliver production-ready solutions.

Key Responsibilities

Data Platform Architecture and Development

Design and implement scalable Azure-based data engineering architectures.

Develop enterprise-grade data platforms supporting structured and unstructured datasets.

Build batch and real-time data processing solutions.

Design efficient data storage, processing, and integration architectures.

Identify opportunities to improve platform performance, reliability, scalability, and cost efficiency.

Data Engineering and ETL Development

Build, maintain, and optimize end-to-end ETL/ELT pipelines.

Develop data ingestion and transformation workflows.

Implement data integration between internal systems, external APIs, and cloud services.

Develop advanced SQL queries, stored procedures, database functions, and data models.

Implement data validation, error handling, and data quality controls.

Azure Cloud and Distributed Processing

Develop solutions using Azure Data Factory, Azure Functions, Event Hubs, and related cloud-native services.

Implement event-driven architectures using Azure Event Grid and Service Bus.

Work with Azure Databricks and distributed processing technologies.

Optimize data workflows for high-volume and real-time processing.

Support cloud migration, integration, and infrastructure improvements.

AI and Machine Learning Data Infrastructure

Develop data processing frameworks supporting AI/ML applications.

Prepare and transform datasets for machine learning and advanced analytics.

Implement reliable data workflows throughout the AI/ML data lifecycle.

Develop data algorithms and processing logic for analytical applications.

Ensure data pipelines are scalable, consistent, and suitable for downstream AI/ML workloads.

Software Development and Integration

Develop backend services and RESTful APIs.

Integrate databases, cloud applications, and third-party systems.

Implement secure and maintainable application components.

Follow software engineering best practices, including version control, code reviews, and automated testing.

Deployment, Testing, and Documentation

Configure and maintain CI/CD pipelines using Azure DevOps and GitHub.

Perform unit, integration, and end-to-end testing.

Identify and resolve technical issues independently.

Produce technical documentation, architecture diagrams, and implementation guides.

Provide a complete handover of the developed solution.

Required Technical Stack

Cloud and Infrastructure

Microsoft Azure

Azure Data Factory

Azure Databricks

Azure Functions

Azure Event Hubs

Azure Event Grid

Azure Service Bus

Azure Cosmos DB

Azure SQL Data Warehouse / Azure SQL

Programming and Development

Python

SQL / T-SQL

C# / .NET

Java

Node.js

Unix Shell Scripting

Data Engineering and Processing

ETL / ELT

Apache Spark / PySpark

Spark SQL

Hive SQL

Data Modeling

RDBMS

Structured and Unstructured Data Processing

Batch and Stream Processing

Event-Driven Architecture

DevOps and Integration

Azure DevOps

Git / GitHub

CI/CD Pipelines

RESTful APIs

JSON / JSON-LD / XML

Version Control and Automated Deployment