Senior Data Engineer QA

Job Description

Job Description - Senior QA Data Engineer

Key Responsibilities:

Testing Strategy: Plan, design, and execute comprehensive testing strategies for new features, enhancements, and system changes.
Hands-on Testing: Create, maintain, and execute manual and automated test cases ensuring thorough coverage.
Automation Leadership: Champion automation best practices by designing scalable, maintainable solutions supporting continuous delivery.
Functional & Non-functional Testing: Validate system quality, performance, and reliability across the SDLC.
Defect Management: Identify, analyse, and troubleshoot defects with root cause analysis, collaborating with cross-functional teams for resolution.
Reporting: Produce clear, detailed defect reports to support efficient remediation.
Stakeholder Communication: Share progress, risks, and quality insights with technical and non-technical stakeholders.

Knowledge & Experience Required:

Data Quality & Validation (Azure Data Platforms)

ADF & Databricks Validation: Ensure accuracy, reliability, and consistency across complex workflows.
ETL/ELT Testing: Validate ingestion, transformations, schema, and reconciliation between source and target systems.
Data Profiling: Identify anomalies, duplicates, and integrity issues.
Lineage & Monitoring: Validate lineage, auditing, and monitoring using observability tools.

Test Automation for Data Platforms

Automation Frameworks: Design frameworks using Python (PyTest, PySpark, Pandas) or C#.
Reusable Scripts: Build scripts for pipelines, transformations, and integrations.
SQL Integration Tests: Develop integration tests for queries and workflows.

SQL & Database Testing

Advanced SQL: Validate business rules, transformations, and workflows.
Source-to-Target Validation: Ensure accuracy post-transformations.
Database Structures: Validate indexes, constraints, stored procedures, and performance.

Business Intelligence & Reporting Testing

CI/CD & Data Pipeline Testing

Pipeline Integration: Embed validation tests into Azure DevOps/GitHub Actions.
Deployment Validation: Support deployment validation for ETL pipelines and BI models.

Data Warehouse & Traditional ETL Tools

Warehouse Testing: Validate architectures, data modelling, and enterprise platforms.
ETL Tools: Experience with SAP BO Data Services, Informatica, SSIS.

Quality Engineering & Collaboration

Enterprise QA: Strong focus on automation and quality engineering practices.
Agile Practices: Active contribution to sprint ceremonies and Three Amigos sessions.
Mentorship: Mentor QA engineers, conduct code reviews, and promote best practices.
Collaboration: Partner with BAs and Developers to refine requirements and acceptance criteria.

Essential Skills:

Cloud & Data Platforms: Azure Data Factory, Databricks, Synapse, SQL.
Databases & Storage: SQL Server, Delta Lake.
Programming & Scripting: Python (PySpark, Pandas), SQL, C#, REST APIs, DAX.
Automation Tools: PyTest, Great Expectations, DataDiffPy, Postman.
CI/CD & DevOps: Azure DevOps, GitHub Actions.
Monitoring: Azure Monitor, Log Analytics, Databricks Monitoring.
BI & Power Platform: Power BI, Power Apps, Dataverse.
Database Testing: Advanced SQL for validation and performance analysis.
Collaboration & Soft Skills: Strong teamwork, communication, and Agile collaboration.