Senior Data Engineer

Purpose

• As a Senior Data Engineer, you design, develop, and optimize data pipelines and architectures. Ensure alignment with business objectives, manage complex data processing workflows, and enhance data integration and quality. Collaborate with cross-functional teams to deliver robust data solutions.

Responsibilities

• Lead and participate in the design, development, and implementation of data engineering solutions with minimal supervision, ensuring adherence to best practices and standards in data engineering.
• Collaborate with data scientists, analysts, and stakeholders to understand data requirements and ensure the timely delivery of high-quality data solutions.
• Design and manage data pipelines, ensuring efficient data flow, processing, and storage from various sources to designated destinations.
• Develop, test, and deploy data integration processes, including ETL (Extract, Transform, Load) workflows, to ensure data consistency, quality, and reliability.
• Implement and maintain scalable data architectures, including data lakes, data warehouses, and cloud-based data solutions, to support the organization's data needs.
• Optimize data storage and retrieval performance by designing efficient database schemas, indexing strategies, and partitioning techniques.
• Monitor and troubleshoot data pipeline performance, addressing any issues related to data quality, latency, or failures promptly and effectively.
• Collaborate with cross-functional teams, including IT, product management, and business units, to ensure alignment on data strategies and goals.
• Mentor and guide junior data engineers, providing technical support, training, and development opportunities to enhance the team's skills and capabilities.
• Maintain and optimize technical documentation, including data flow diagrams, data dictionaries, and architecture blueprints, ensuring clear communication and knowledge sharing.
• Stay current with emerging data engineering technologies, tools, and best practices, applying this knowledge to improve existing processes and solutions.
• Ensure compliance with data governance policies, data security standards, and regulatory requirements throughout the data engineering lifecycle.
• Work closely with project managers and SCRUM masters to accurately forecast project timelines, resource needs, and deliverable milestones.

Education

• Bachelor's degree in computer science, information technology, data science, or a related field is required. Advanced certifications in data engineering or related fields, such as Google Cloud Professional Data Engineer, AWS Certified Big Data – Specialty, or Microsoft Certified: Azure Data Engineer Associate, are preferred. A Master's degree in data science, computer science, or a related field is an added advantage but not mandatory. Proficiency in English is essential for effective communication with team members, stakeholders, and external vendors.

Experience

• Minimum of 7 years of experience in data engineering, data architecture, or a related field, with at least 3 years in a leadership or senior technical role.
• Proven experience in designing, implementing, and managing complex data pipelines, ETL processes, and data integration solutions.
• Strong technical background with hands-on experience in using big data technologies such as Hadoop, Spark, and Kafka, as well as cloud data platforms such as AWS, Azure, or Google Cloud.
• Demonstrated ability to develop and implement data governance policies, data quality standards, and data security measures to ensure compliance with industry standards and regulatory requirements.
• Experience with data modeling, data warehousing, and database design using tools such as SQL, NoSQL, and columnar databases.
• Proficiency in programming languages such as Python, Java, or Scala for data processing and analysis.
• Strong knowledge of data visualization tools and techniques, with experience in using tools such as Tableau, Power BI, or Looker.
• Excellent communication and interpersonal skills, with the ability to liaise effectively with stakeholders at all levels of the organization.
• Strong leadership and team-building skills, with the ability to mentor and guide technical staff to achieve performance goals and objectives.
• Familiarity with Agile and DevOps methodologies, with a proven ability to implement continuous integration, continuous delivery (CI/CD) pipelines.
• Relevant certifications such as Google Cloud Professional Data Engineer, AWS Certified Big Data – Specialty, or Microsoft Certified: Azure Data Engineer Associate are preferred.

Functional Competencies