This role involves designing, developing, and maintaining data infrastructure and data pipelines to ensure that data is efficiently collected, processed, stored, and made accessible for analysis and decision-making within an organization. The role of a Data Engineering Lead is instrumental in supporting data-driven initiatives. This role is instrumental in shaping an organization’s data landscape. The ability to blend technical expertise with a strong understanding of business needs is essential in this role.

Job Description​

• Design, develop, and maintain data pipelines for extracting, transforming, and loading (ETL) data from various sources into data storage systems.
• Create efficient and scalable ETL processes to ensure the smooth flow of data and its availability for analysis.
• Integrate data from multiple sources, such as databases, APIs, third-party services, and logs, into a unified data ecosystem.
• Implement data quality checks and validation processes to ensure data accuracy and consistency.
• Develop and maintain data models, schemas, and database structures to optimize data storage and retrieval.
• Design and maintain data warehouses, data lakes, or other storage solutions for data analytics.
• Utilize big data technologies such as Hadoop, Spark, Kafka, and NoSQL databases for processing and analysing large volumes of data.
• Stay up to date with emerging data technologies and evaluate their applicability to the organization’s data infrastructure.
• Create and maintain data documentation, data dictionaries, and data lineage to support data governance and audit requirements.
• Implement data governance best practices and adhere to data security and compliance standards.
• Optimize database performance, including query tuning, indexing, and data partitioning, to enhance system performance and reduce latency.
• Implement data compression, archiving, and data retention policies to manage storage costs.
• Implement data security measures, encryption, and access controls to protect sensitive data and ensure compliance with data protection regulations.
• Collaborate with data scientists, analysts, and other stakeholders to understand their data needs and provide the necessary infrastructure for data-driven decision-making.
• Work with cross-functional teams to define data requirements for new projects and initiatives.
• Set up monitoring and alerting systems to proactively identify and resolve data pipeline issues, data quality problems, or performance bottlenecks.
• Conduct root cause analysis and troubleshooting of data-related problems.
• Ensure that data infrastructure is scalable to accommodate growing data volumes and evolving business requirements.
• Continuously optimize data pipelines and infrastructure for improved performance and cost-efficiency.
• Implement automation for deployment, data pipeline scheduling, and routine maintenance tasks to increase operational efficiency.
• Keep the team and relevant stakeholders informed about data engineering best practices and new technologies.
• Provide training and knowledge sharing sessions to help others understand and utilize data systems effectively.

Qualifications

Bachelor’s Degree – Computer and Information Science, Experience in a similar environment ideally at executive management level