Über
Overview We are seeking a Data Engineer to lead the modernization of legacy ETL systems by migrating Ab Initio workflows to scalable, modular PySpark pipelines on Databricks. The role involves transforming complex data ecosystems into cloud-native architectures while ensuring data integrity, performance, and reliability. Key Responsibilities ETL Modernization & Development • nalyze and migrate legacy ETL workflows from Ab Initio to PySpark-based pipelines • Design and develop scalable data pipelines on Databricks • Refactor monolithic processes into modular, reusable components • Leverage existing enterprise datasets to avoid redundancy Data Integration & Processing • Build and maintain ETL/ELT pipelines integrating data from Snowflake and other sources • Process and publish enriched datasets for downstream applications • Support batch and near real-time data processing Data Lineage & Optimization • Create end-to-end data lineage and data flow diagrams • Identify redundancies and drive process consolidation and optimization • Ensure adherence to data governance and quality standards Testing & Validation • Develop unit, integration, and reconciliation frameworks • Perform dual-run comparisons with legacy systems • Validate outputs in UAT and pre-production environments Deployment & Operations • Support cutover and migration strategy from legacy systems • Decommission legacy workflows and optimize scheduling (e.g., Control-M) • Develop runbooks, monitoring, and operational documentation Collaboration • Work with data architects, analysts, and downstream application teams • Coordinate user acceptance testing (UAT/FAT) and stakeholder sign-offs
About ApTask: ApTask is a leading global provider of workforce solutions and talent acquisition services, dedicated to shaping the future of work. As an African American-owned and Veteran-owned company, ApTask offers a comprehensive suite of services, including staffing and recruitment solutions, managed services, IT consulting, and project management. With a focus on excellence, collaboration, and innovation, ApTask provides unparalleled opportunities for professional growth and development. As a member of the ApTask team, you will have the chance to connect businesses with top-tier professionals, optimize workforce performance, and drive success across diverse industries. Join us at ApTask and be part of our mission to empower organizations to thrive while fostering a diverse and inclusive work environment.
Applicants may be required to attend interviews in person or by video conference. In addition, candidates may be required to present their current state or government issued ID during each interview.
Candidate Data Collection Disclaimer: At ApTask, we prioritize safeguarding your privacy. As part of our recruitment process, certain Personally Identifiable Information (PII) may be requested by our clients for verification and application purposes. Rest assured, we strictly adhere to confidentiality standards and comply with all relevant data protection laws. Please note that we only collect the necessary information as specified by each client and do not request sensitive details during the initial stages of recruitment.
If you have any concerns or queries about your personal information, please feel free to contact our compliance team at businessexcellence@aptask.com.
Applicant Consent: By submitting your application, you agree to ApTask's (www.aptask.com) Terms of Use and Privacy Policy, and provide your consent to receive SMS and voice call communications regarding employment opportunities that match your resume and qualifications. You understand that your personal information will be used solely for recruitment purposes and that you can withdraw your consent at any time by contacting us at 732-355-8000 or help@aptask.com. Message frequency may vary. Msg & data rates may apply.
Sprachkenntnisse
- English
Hinweis für Nutzer
Dieses Stellenangebot stammt von einer Partnerplattform von TieTalent. Klick auf „Jetzt Bewerben”, um deine Bewerbung direkt auf deren Website einzureichen.