Senior AI Data Engineer
Karsun Solutions
- Herndon, Virginia, United States
- Herndon, Virginia, United States
About
What You'll Be Doing
Design, build, and maintain scalable data pipelines and workflows using Databricks to directly support AI/ML and analytics workloads. Leverage core capabilities like Delta Lake, Delta Live Tables, and Databricks Workflows to create high-performance data platforms.
Set up, establish, and operationalize MLOps practices directly within the Databricks environment, including version control, CI/CD for data pipelines, automated testing, and model deployment strategies.
Utilize and integrate open‑source tools such as Python, PySpark, and Apache Airflow for distributed data processing and workflow orchestration.
Implement GenAI‑enhanced workflows using LLMs to automate metadata generation, create data dictionaries, validate data quality, and track data lineage.
Leverage medallion architecture (Bronze, Silver, Gold layers) following data lakehouse best practices. Integrate Unity Catalog for enterprise data governance and access control. Implement and operationalize best practices.
Collaborate with AI/ML teams to curate, prepare, and serve high‑quality datasets for model training and inference.
Required Qualifications
BA or BS degree in Computer Science, Computer Engineering, Data Science, or a related field (Master's degree is a plus).
5+ years of strong proficiency in open‑source languages and frameworks, specifically Python and PySpark, for distributed data processing. Strong knowledge of open‑source data orchestration tools like Apache Airflow.
5+ years of proven experience building large‑scale data platforms, with at least 2 years incorporating Generative AI into data engineering workflows.
3+ years of hands‑on experience with the Databricks platform, specifically leveraging data engineering and AI features (Delta Lake, DLT, Workflows, Unity Catalog).
Proven experience setting up, maintaining, and operationalizing MLOps frameworks within Databricks.
3+ years of experience with AWS data services (e.g., S3, Glue, Lambda) and a deep understanding of data lakehouse architecture.
Certifications in Databricks Data Engineer Associate/Professional or AWS Data Analytics.
Preferred Qualifications
Experience with open‑source streaming data processing tools like Apache Kafka or Structured Streaming.
Familiarity with open‑source data quality and analytics engineering tools such as dbt, Great Expectations, or Sweetviz.
Experience with open‑source containerization (Docker) and orchestration (Kubernetes) for data applications.
Understanding of vector databases and embedding pipelines for AI/ML applications.
Commitment to Non‑Discrimination All qualified applicants will receive consideration for employment without regard to disability, status as a protected veteran or any other status protected by applicable federal, state, local, or international law.
Salary Range The proposed salary range for this role is $165,000 to $180,000 USD. The salary range provided is a good faith estimate representative of all experience levels. Karsun considers several factors when extending an offer, including but not limited to the role, function and associated responsibilities, a candidate's work experience, location, education/training, and key skills.
Location To be considered for this role, you must reside in one of the following states: CA, CO, DC, FL, GA, IL, MD, NJ, NY, NC, OH, OK, PA, SC, TX, VA, WV.
Security Clearance This position requires the eligibility to obtain a security clearance.
Applicants must be authorized to work in the U.S. We may consider candidates currently in H‑1B status who are eligible for transfer.
#J-18808-Ljbffr
Languages
- English
Notice for Users
This job comes from a TieTalent partner platform. Click "Apply Now" to submit your application directly on their site.