Lead Databricks Engineer with Pyspark & Python Exp---USA

• Work on Finance data related to Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo
• Design, develop, and deploy Databricks jobs to process and analyze large volumes of data.
• Collaborate with data engineers and data scientists to understand data requirements and implement appropriate data processing pipelines.
• Optimize Databricks jobs for performance and scalability to handle big data workloads.
• Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks.
• Implement best practices for data management, security, and governance within the Databricks environment. Experience designing and developing Enterprise Data Warehouse solutions.
• Demonstrated proficiency with Data Analytics, Data Insights
• Proficient writing SQL queries and programming including stored procedures and reverse engineering existing process
• Azure Synapse/Bigquery/Redshift is good to have.
• Perform code reviews to ensure fit to requirements, optimal execution patterns and adherence to established standards.

• 5+ years' - Strong experience in Finance / Banking industry - Capital markets, investment banking - Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo
• 10+ years - Enterprise Data Management
• 10+ years - SQL Server based development of large datasets
• 5+ years with Data Warehouse Architecture, hands-on experience with Databricks platform. Extensive experience in PySpark coding.
• Azure Synapse/BigqExperience with Cloud based data architectures, messaging, and analytics
• Superior communication skills
• Cloud certification(s)
• Any experience with Regulatory Reporting is a Plus

• Minimally a BA degree within an engineering and/or computer science discipline
• Master's degree strongly preferred

Nice-to-have skills

• PySpark
• Python
• SQL
• Redshift

• New York, New York

Work experience

• Data Engineer
• Data Infrastructure
• Data Analyst

Languages

• English