Role Title: Principal Data Engineering Lead
Minimum years of experience: 10 – 15 years
Detailed Job Description:
Microsoft Azure, Experience developing and deploying data intensive application at enterprise scale. Evolve architecture to support scaling to other OUs and bottlers. Topics include data privacy, clean team, data security, ML scaling and performance
• Technical Oversight: Provide technical leadership and hands-on oversight in developing data processing applications on Spark using Scala programming, focusing on Microsoft Azure Synapse Spark Runtime.
• Data Pipeline Optimization: Design and optimize data pipelines processing through various zones in Medallion architecture using Azure Synapse pipelines.
• Data Ingestion and Quality: Manage data ingestion, ensure data quality checks with tools like Deequ, and handle data validation and error management.
• Configuration Management: Develop and manage configuration settings using JSON files read by classes such as ApplicationConfig and TableConfig for various zones.
• Cross-Functional Collaboration: Collaborate with data scientists, analysts, and cross-functional teams to ensure seamless integration and alignment of data engineering practices with marketing strategies.
• Logging and Auditing: Oversee logging, auditing, and error handling processes to track and ensure data processing integrity. Knowledge of Azure Log Analytics and KQL queries a plus.
• Testing and Validation: Implement unit testing with tools like Scala Test and maintain data quality checks for reliable data processing outcomes.