Software Engineer

Software Engineer, Job Description

Summary

As organizations accelerate their adoption of AI/ML capabilities, the demand for robust data management and governance solutions is becoming increasingly critical. Atlas Data Science is seeking a Software Engineer to help build a central metadata hub that streamlines discovery, access, use, and sharing of diverse data types across an expanding ecosystem of suppliers and consumers. By orchestrating the full data lifecycle – from ingestion and enrichment to discovery and dissemination – we aim to deliver a new-to-market data management and governance product that is foundational to enable AI/ML at scale.

Essential Duties & Responsibilities

We are seeking a talented and motivated Software Engineer to join our dynamic and growing team. You will work alongside experienced engineers, leveraging a modern technology stack that includes Oracle, NestJS, Python, Java, Kubernetes, Apache Spark, Terraform, and GitLab. You will contribute to the development, testing, and productization of solution components integrating relational databases, microservices, cloud services, and containerized applications.

Key Responsibilities

• Design, develop, and maintain data ingestion, transformation, and delivery pipelines.
• Integrate structured data from relational databases (Oracle, PostgreSQL, MySQL, etc.) into pipelines.
• Optimize and troubleshoot pipelines for performance, reliability, and scalability.
• Contribute to backend service development using NestJS ) and Python.
• Support cloud-based workflows leveraging AWS Services such as Glue, S3, Lambda, and ECS/EKS.
• Implement infrastructure as code using Terraform to provision and manage cloud resources.
• Build, deploy, and manage containerized application using Docker and Kubernetes.
• Collaborate with DevOps to integrate pipelines into CI/CD workflows using GitLab.
• Write unit, integration, and system tests to ensure code quality and functionality.
• Support AI/ML initiatives by building and optimizing data pipelines for LLM training and data processing, enabling seamless integration across diverse datasets.
• Stay current with emerging technologies, development best practices, modern data lake/lakehouse frameworks, and evolving approaches in data brokerage services.

Requirements

• Familiarity with relational databases (Oracle, PostgreSQL, MySQL, etc.).
• Experience or interest in modern data lake/lakehouse technologies (Apache Iceberg, Spark, etc.).
• Exposure to AWS services, including Glue, S3, Lambda and container platforms (EKS/ECS).
• Experience with Terraform for infrastructure as code on cloud platforms.
• Exposure to workflow orchestration tools such as Airflow, Step Functions, or Argo.
• Understanding of Docker and Kubernetes concepts for containerization and orchestration of applications and services.
• Proficiency with GitLab (or a similar tool) for source control and CI/CD workflows.
• Experience or interest in LLMs for AI/ML data processing.
• Strong problem-solving skills and willingness to learn complex data systems.
• Ability to work collaboratively in a team-oriented Agile environment.
• Clear written and verbal communication skills.

Education and/or Experience

• Bachelor's Degree in Computer Science or related field.
• 5 - 10 years of experience.

Requirements

• Candidate must be a US Citizen.

Location

• Remote. Strong preference for DMV area.

Anticipated Salary Range

• Competitive base salary range of $125k - $160k. Opportunity for performance-based bonus. Total compensation package will be commensurate with skillset and experience.