Python with Pyspark at N Consulting Ltd

Responsibilities:

1. Design, develop, and maintain scalable data processing solutions using Python and PySpark.
2. Collaborate with data engineers, data scientists, and other cross-functional teams to understand data requirements and implement effective solutions.
3. Develop and optimize Spark jobs for efficient data processing and analysis.
4. Implement data pipelines for ETL (Extract, Transform, Load) processes using PySpark.
5. Work with big data technologies and frameworks, handling large-scale data sets efficiently.
6. Troubleshoot and optimize existing PySpark applications for performance and reliability.
7. Collaborate with database administrators and data architects to ensure data consistency and integrity.
8. Stay updated with the latest developments in PySpark and related technologies to recommend improvements and best practices.

Requirements:

1. Bachelor's degree in Computer Science, Engineering, or a related field.
2. Proven experience as a Python developer with a focus on PySpark.
3. Strong understanding of distributed computing concepts and frameworks.
4. Hands-on experience with big data technologies such as Apache Spark.
5. Proficient in SQL and experience with relational databases.
6. Familiarity with data modeling and design principles.
7. Solid understanding of data processing and ETL concepts.
8. Experience with version control systems, such as Git.
9. Strong problem-solving skills and attention to detail.
10. Excellent communication and collaboration abilities.
11. Ability to work independently and in a team environment.

Preferred:

1. Experience with cloud platforms, such as AWS or Azure.
2. Knowledge of data warehousing concepts.
3. Familiarity with machine learning frameworks and libraries.