Responsibilities:
- Design, develop, and maintain scalable data pipelines and infrastructure to support data collection, storage, processing, and analysis.
- Work closely with data scientists, analysts, and other stakeholders to understand their data requirements and translate them into technical solutions.
- Implement best practices for data management, including data governance, data quality monitoring, and data security.
- Optimize performance of data systems through monitoring, tuning, and troubleshooting.
- Collaborate with software engineers to integrate data infrastructure with existing systems and applications.
- Stay current with emerging technologies and trends in data engineering and recommend new tools and techniques to enhance productivity and efficiency.
Requirements:
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
- Proven experience as a Data Engineer or similar role, with a strong background in data architecture, database design, and software development.
- Proficiency in programming languages such as Python, Java, or Scala.
- Experience with big data technologies such as Hadoop, Spark, Kafka, etc.
- Hands-on experience with cloud platforms such as AWS, Azure, or GCP.
- Strong SQL skills and experience with relational and non-relational databases.
- Excellent problem-solving skills and attention to detail.
- Strong communication and collaboration skills, with the ability to work effectively in a cross-functional team environment.