Software Developer

Uchai Infra Projects

Posted: 1 year ago
Openings: 10
Applicants: 0

Job Description

Were seeking a skilled Software Developer - Data Acquisition and Pipeline with 3+ years of experience to build and maintain a robust system for extracting and processing publicly available information from various websites. Youll design, implement, and scale data scraping solutions, ensuring data accuracy and automating the data pipeline. This requires strong skills in data scraping techniques, data management, cloud technologies, and monitoring. Responsibilities Project Refinement: Validate and scale existing data acquisition projects. Data Extraction: Develop scripts to scrape specific data points from target websites and implement mechanisms for capturing updates. Automation: Schedule automated scraping tasks using cron or similar tools. Scalable Infrastructure: Design and deploy a scalable pipeline using Docker and Kubernetes in a cloud environment. Website Navigation: Implement solutions to efficiently navigate website structures and bypass any access restrictions. Data Management: Configure MongoDB Atlas and Azure Data Factory for data storage and transformation. Monitoring: Set up monitoring dashboards using Grafana and Azure Monitor. System Reliability: Implement logging, retry mechanisms, and alerts. Full-Scale Deployment: Expand the solution to 20-25 target websites and launch full-scale data acquisition operations. Qualifications Essential Skills: 3+ years of experience in data scraping and pipeline development. Expertise in data scraping frameworks and libraries. Experience with task scheduling, Docker, Kubernetes, and cloud platforms (Azure, AWS, GCP). Familiarity with Python, MongoDB Atlas, Azure Data Factory, and monitoring tools. Strong problem-solving and debugging skills. Preferred Qualifications: Experience with similar data scraping projects. Knowledge of OCR tools. Experience with scalable data pipelines and monitoring dashboards. Familiarity with agile project management. Additional Requirements: Construction domain expertise ERP systems knowledge is a plus. Willingness to relocate.