Lead Databricks Data Engineer

Job Description
Perficient is a global AI-first consultancy. Our team of strategists, designers, technologists, and engineers partners with the world's most innovative enterprises and admired brands to deliver real business results through the transformative power of AI. As part of our AI-First strategy, we empower every employee to build AI fluency and actively engage with AI tools to drive innovation and efficiency. We break boundaries, obsess over outcomes, and shape the future for our clients. Join a company where bold ideas and brilliant minds converge to redefine what's possible - while building a career filled with growth, balance, and purpose.

Perficient currently has a career opportunity for a
Lead Databricks Data Engineer
with AWS
to join our team located in the United States. This role is primarily office-based at a local Perficient location, with travel and on-site client engagements expected as needed.

Job Overview
As a Lead Databricks Data Engineer, you are expected to lead the development team and have strong development and programming skills in Python, SQL, Pyspark, Databricks, Notebook Repositories, CI/CD, and ETL platforms associated with Data Lakehouse solutions. You are expected to be experienced and fluent in agile development and agile tools.

You will work with architects and infrastructure teams to develop, test, deploy and troubleshoot your code as well as provide input into solutions and design of the system. You will collaborate with some of the best talent in the industry to create and implement innovative high-quality solutions focused on our clients' business needs.

About Us
The Perficient Data & Analytics Solutions organization supports a wide variety of clients, technologies, and applications. Our Team members drive innovation in the arena of data and analytics solutions. We work on innovative digital transformation deployment for various clients across the globe. We had explosive growth in the last few years to meet the demands of a massive transition to digital with an AI First approach. Our practice groups include Artificial Intelligence, Analytics, Content Management, Data Governance, Data Management, Data Strategy and Search.

Perficient is always looking for the best and brightest talent and we need you We're a quickly growing, global digital consulting leader, and we're transforming the world's largest enterprises and biggest brands. You'll work with the latest technologies, expand your skills, and become a part of our global community of talented, diverse, and knowledgeable colleagues.

Responsibilities

• Design and develop scalable data solutions.
• Build high-performance ETL pipelines using Spark/PySpark.
• Implement CI/CD pipelines and orchestrate Databricks workflows.
• Monitor production jobs and troubleshoot issues.
• Participate in Agile ceremonies.

Qualifications

• Passionate developer with 8+ years of data engineering experience, with at least 3 years in a lead or senior data engineer role.
• Minimum 5 years of hands-on Teradata and Databricks experience using ( Delta Lake, Notebooks, Pipelines, cluster management).
• Experience in A WS services (S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, CloudWatch)
• DevOps experience with Databricks CI/CD ( Git, Jenkins, Artifactory)
• Experience with U nix/Linux shell scripting and administration
• Experience with S QL (Teradata SQL, Hive SQL, Spark SQL)
• Experience with J ob scheduling tools (CA7 Enterprise Scheduler)
• Demonstrated ability to leverage AI tools to enhance productivity, streamline workflows, and support data-informed task execution.
• Familiarity with AI-enhanced platforms is a plus.
• A solid understanding of AI capabilities and limitations including ethical considerations is expected.
• Flexible and adaptable attitude, disciplined to manage multiple responsibilities and adjust to varied environments.
• Ability to produce high quality products within deadlines
• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field.

Preferred Skills & Education:

• Certification in Databricks, Spark, AWS, Azure or other cloud platform
• Master's degree in Computer Science or related field
• Client-facing or consulting experience is highly preferred.

Perficient, Inc. proudly provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, gender, sexual orientation, national origin, age, disability, genetic information, marital status, amnesty, or status as a protected veteran in accordance with applicable federal, state and local laws. Perficient, Inc. complies with applicable state and local laws governing nondiscrimination in employment in every location in which the company has facilities. This policy applies to all terms and conditions of employment, including, but not limited to, hiring, placement, promotion, termination, layoff, recall,