Application Architect – data pipeline

We are looking for an experienced developer to lead the design and development of high-performance and scalable enterprise solutions using PySpark, Java, Databricks etc. The ideal candidate will have over 10 years of experience in software development role, designing enterprise-grade, fault-tolerant, even-driven applications. You will work closely with cross-functional teams to design, build and deploy complex systems, providing technical direction and expertise to ensure the delivery of robust, efficient, and scalable solutions.

Key Responsibilities:

• Architect, design, develop and deploy high-performance and scalable data pipe lines -based applications that meet business complex requirements.
• Familiarity with streaming architectures and patterns such as event-driven pipelines, near real-time scoring, and anomaly monitoring.
• Experience working with high-volume, sensitive data while adhering to security, compliance, and privacy guidelines.
• Proficiency in Python for data processing, automation, API integration, anomaly-detection scripts, and model-ready dataset preparation
• Lead a team of engineers and work with cross-functional team to timely deliver high volume data pipeline and streaming solutions.
• Provide technical leadership across all aspects of the software development lifecycle, from initial design through production deployment.
• Design and implement data pipelines using PySpark, DataBricks, Java and related tech stack.
• Ensure high availability and scalability of systems using Kubernetes, containerization, and cloud infrastructure.
• Implement and manage schedulers, event-driven architecture, and asynchronous processes.
• Collaborate with DevOps and infrastructure teams to automate deployment, scaling, and monitoring of applications.
• Drive the adoption of best practices in coding, design, testing, and deployment to improve team productivity.
• Strong SQL skills, including query optimization, performance tuning, and working with both relational and non-relational stores.

Required Skills:

• 10+ years of total experience in software development, with at least 5 years in a design lead role.
• Deep experience with PySpark for distributed data processing, data quality validation, data enrichment, and feature engineering
• Excellent problem-solving, analytical, and interpersonal skills.
• Expertise in Java, J2EE, Kafka, and Spring Boot.
• Extensive hands-on experience on spring-boot, Kafka, and API development activities.
• Experience in designing scalable, distributed systems and microservices architecture.
• Familiarity with schedulers, event-driven architecture, and messaging systems (e.g., Kafka, RabbitMQ).
• Proficiency in working with cloud platforms such as AWS and Azure.
• Hands-on experience with caching strategies (ECH), performance tuning, and security best practices.
• Experience with version control systems (Git), CI/CD pipelines, and Agile methodologies.
• Experience of working with relational and NoSQL databases.

Job Type: Full-time

Pay: $113, $156,274.91 per year

Benefits:

• Paid time off

Work Location: In person