Role: Data Engineer / Data Architect – Databricks & Unity Catalog
Location: Poland, Remote
Job Title: Data Engineer / Data Architect – Databricks & Unity Catalog
Job Summary:
We are looking for a skilled StData Engineer /with expertise in Databricks and Unity Catalog to design, implement, and manage scalable data solutions.
Key Responsibilities:
- Design and implement scalable data pipelines and ETL workflows using Databricks.
- Implement Unity Catalog for data governance, access control, and metadata management across multiple workspaces.
- Develop Delta Lake architectures for optimized data storage and retrieval.
- Establish best practices for data security, compliance, and lineage tracking in Unity Catalog.
- Optimize data lakehouse architecture for performance and cost efficiency.
- Collaborate with data scientists, engineers, and business teams to support analytical workloads.
- Monitor and troubleshoot Databricks clusters, performance tuning, and cost management.
- Implement data quality frameworks and observability solutions to maintain high data integrity.
- Work with Azure/AWS/GCP cloud environments to deploy and manage data solutions.
Required Skills & Qualifications:
- 8-19 years of experience in data engineering, data architecture, or cloud data solutions.
- Strong hands-on experience with Databricks and Unity Catalog.
- Expertise in PySpark, Scala, or SQL for data processing.
- Deep understanding of Delta Lake, Lakehouse architecture, and data partitioning strategies.
- Experience with RBAC, ABAC, and access control mechanisms within Unity Catalog.
- Knowledge of data governance, compliance standards (GDPR, HIPAA, etc.), and audit logging.
- Familiarity with cloud platforms (Azure, AWS, or GCP) and their respective data services.
- Strong understanding of CI/CD pipelines, DevOps, and Infrastructure as Code (IaC).
- Experience integrating BI tools (Tableau, Power BI, Looker) and ML frameworks is a plus.
- Excellent problem-solving, communication, and collaboration skills.