Data Engineering Lead

Purpose

• As a Data Engineering Lead, you oversee the design, development, and maintenance of data pipelines and architectures. Ensure alignment with business objectives, manage data engineering teams, and optimize data processing workflows. Collaborate with stakeholders to enhance data integration, quality, and performance. Support organisational data strategy and objectives.

Responsibilities

• Lead and participate in the design and development of data engineering solutions, ensuring robust data pipelines and ETL processes.
• Collaborate with data scientists, analysts, and IT teams to understand data needs and deliver scalable and efficient data solutions.
• Ensure adherence to best practices in data engineering, including data modeling, data integration, and data quality management.
• Manage the development and maintenance of ETL processes to ensure the accurate and timely movement of data across systems.
• Implement and maintain data governance policies, ensuring data quality, security, and compliance with industry standards and regulations.
• Provide technical guidance and mentorship to less experienced data engineers, fostering a culture of continuous improvement and innovation.
• Troubleshoot common issues in data engineering and implement solutions to optimize performance and reliability.
• Develop and maintain comprehensive documentation for data engineering processes, including data models, data flow diagrams, and metadata definitions.
• Work with project managers and stakeholders to accurately forecast man-hours, resource needs, and completion dates for deliverables.
• Stay updated with the latest trends and technologies in data engineering and apply this knowledge to improve the organization's data capabilities.
• Facilitate communication and collaboration among cross-functional teams involved in data projects.
• Report on the progress, performance, and impact of data engineering initiatives to senior leadership, providing recommendations for continuous improvement.

Education

• Minimum Bachelor's degree in Computer Science, Information Technology, Data Science, or a related field is required. Certifications such as TOGAF (The Open Group Architecture Framework), ITIL (Information Technology Infrastructure Library), or relevant data engineering and management certifications are preferred. Proficiency in English is essential for effective communication with team members, stakeholders, and external vendors. A Master's degree in Computer Science, Data Science, Business Administration, or a related field is preferred but not mandatory.

Experience

• Minimum of 7 years of experience in data engineering, data architecture, or a related field, with at least 3 years in a leadership or people management role.
• Proven experience in designing and implementing data engineering strategies and frameworks that align with business objectives and IT strategy.
• Strong technical background with hands-on experience in managing complex data infrastructures, including data modeling, ETL processes, cloud environments, and big data technologies.
• Demonstrated ability to develop and enforce data governance policies, standards, guidelines, and best practices to ensure data quality, security, and compliance.
• Experience with data engineering tools and frameworks such as Apache Hadoop, Spark, or similar platforms.
• Strong understanding of Agile and DevOps methodologies and their application in data engineering.
• Proven track record of leading cross-functional teams and collaborating with stakeholders at all levels of the organization.
• Excellent communication and interpersonal skills, with the ability to articulate complex technical concepts to non-technical stakeholders.
• Problem solving and solution oriented skills are essential.

Functional Competencies