Job Description

We are seeking a skilled Databricks Engineer with over 5 years of experience in designing, developing, and maintaining scalable data pipelines and big data solutions. The ideal candidate will have extensive experience with Databricks, Apache Spark, and data engineering best practices to optimize and manage large datasets in cloud environments. As part of our team, you will be instrumental in building efficient data architectures, integrating disparate data sources, and driving innovation in data processing and analytics. Key Responsibilities Design, build, and maintain scalable data pipelines and architectures using Databricks, Spark, and cloud technologies (Azure, AWS, GCP). Collaborate with data scientists, analysts, and product teams to create data solutions that support machine learning and business intelligence initiatives. Write efficient, reusable, and scalable code for ETL processes in Databricks and other big data environments. Develop and optimize Apache Spark jobs, improving performance, scalability, and reliability. Work on data modeling, transformation, and cleaning using Databricks notebooks and clusters. Implement data governance, security protocols, and data quality standards. Troubleshoot and resolve issues related to data pipelines and data quality, ensuring seamless operations. Manage and monitor Databricks clusters for optimal performance and resource utilization. Stay up-to-date with the latest Databricks and cloud technologies, and contribute to the adoption of new tools and practices. Mentor junior engineers and participate in code reviews to maintain high-quality coding standards.

More Info

Education