Data Analyst

Job Description

Key Responsibilities:

- Design and develop scalable batch and real-time data pipelines using Spark

- Build and maintain distributed streaming systems using Kafka

- Develop ETL/ELT workflows for large-scale structured and unstructured datasets

- Optimize data processing jobs for performance

- Implement data ingestion frameworks for high-volume data sources

- Ensure data quality, integrity, and reliability across pipelines

- Collaborate with cross-functional teams to define data architecture and solutions

- Monitor, troubleshoot, and resolve data pipeline issues

- Implement data security and governance standards

Required Skills & Qualifications

- Bachelors/Master’s degree in Computer Science, Engineering, or related field

- 4+ years of experience in data engineering or big data development

- Strong hands-on experience with Apache Spark (Core, SQL, Structured Streaming)

- Experience with Apache Kafka, Kafka Streams, and connectors

- Proficiency in Python

- Experience with distributed data processing and parallel computing

- Strong knowledge of data structures and algorithms

- Experience with ETL tools and workflow orchestration

- Familiarity with data lake and data warehouse architectures

- Experience with SQL and NoSQL databases

Preferred Qualifications

- Knowledge of modern data formats (Parquet, Avro, ORC)

- Experience with CI/CD pipelines and DevOps practices

- Familiarity with data governance and metadata management tools

- Real-time analytics and event-driven architecture experience

- Experience working in Agile/Scrum environments

Key Competencies

- Strong problem-solving and analytical skills

- Excellent communication and teamwork abilities

- Ability to work with large, complex datasets

- Strong attention to detail

- Self-motivated with ability to work independently

Nice to Have

- Experience with machine learning pipelines

- Knowledge of data visualization tools

- Certification in big data technologies

More Info

Education