Sr Staff/Sr Data Engineer

Job DescriptionJob Description

We are seeking an experienced Data Engineer to join our Mission Data Processing program. In this role, you will design, build, and maintain scalable ETL pipelines for processing terabyte-scale streaming data and architect databases optimized for machine learning on on-premises hardware using open-source software. The ideal candidate will have expertise in data design patterns such as the Medallion Architecture and data lakehouse technologies to ensure efficient and reliable data processing. You should be skilled at handling high-throughput, low-latency data ingestion, managing data bursts, and implementing features like time-based partitioning, versioning, auditing, and rollback for historical data replay and event reproducibility. Additionally, you will bring DevOps expertise for pipeline automation, Infrastructure as Code (IaC) skills with tools like Terraform and Ansible, and a strong understanding of DevSecOps practices for maintaining secure and compliant data workflows.

Key Responsibilities :

Design and optimize ETL pipelines capable of handling high-throughput, low-latency data ingestion, especially during large data bursts

Implement robust asynchronous processing systems using ZeroMQ to handle large, serialized Protobuf messages

Create systems that efficiently process sudden, large volumes of data while maintaining performance

Design strategies for managing backpressure to prevent system overload during high data volumes

Develop fault-tolerant systems to safeguard data integrity and maintain reliability

Set up monitoring and alerting mechanisms for proactive response to sudden data load changes

Build and sustain high-performance databases on on-premises infrastructure, leveraging MinIO or similar object storage solutions for seamless integration with ML workflows

Apply and manage data design patterns such as the Medallion Architecture to organize data into Bronze, Silver, and Gold layers

Deploy Delta Lake solutions to combine the flexibility of data lakes with data warehouse performance

Implement containerization and orchestration solutions using Docker and Kubernetes, and build CI/CD pipelines for automated ETL workflows

Implement infrastructure provisioning and deployment automation using Terraform and/or Ansible

Uphold data governance and security protocols to ensure data integrity and compliance with DoD standards, including vulnerability scans and secure configurations

Other duties as assigned

Requirements

Required Qualifications :

Minimum 4 years of experience building and maintaining data pipelines/ETL solutions at scale

Proficiency in Python, C++, SQL, and RDBMS (PostgreSQL or similar)

Experience with object storage (e.g., MinIO), Protocol Buffers, and ZeroMQ

Familiarity with Data Version Control (DVC), Delta Lake, and the Medallion Architecture

Skilled in Docker, Kubernetes, CI/CD pipelines, and infrastructure automation (Terraform/Ansible)

Experience with high-throughput, low-latency systems, fault tolerance, and backpressure handling

Knowledge of data governance, versioning, auditing, rollback, and DevSecOps practices.

Active DoD Secret Clearance

Detail Oriented

Good verbal and written communication skills

Qualifications :

Knowledge of Java, Rust, Scala, and NoSQL databases (e.g., Apache Cassandra).

Familiar with Apache Iceberg, Yugabyte, Apache Hudi, Ceph, OpenStack Swift, Redis, and high-performance alternatives (DragonflyDB, KeyDB, Apache Ignite)

Experienced with data processing tools (e.g., Apache Airflow, Prefect, Dagster, Apache NiFi, Apache Spark, Flink, Beam, Dask) and data quality tools (e.g., Great Expectations, Soda Core)

Familiar with performance optimization and observability tools (e.g., Prometheus, Grafana, Loki)

Experience with data management, compliance, and security platforms (e.g., AWS Secrets Manager)

Education:

Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field.

Relevant certifications are a plus.

Benefits

SciTec offers a highly competitive salary and benefits package, including:

Employee Stock Ownership Plan (ESOP)

3% Fully Vested Company 401K Contribution (no employee contribution required)

100% company paid HSA Medical insurance, with a choice of 2 buy-up options

80% company paid Dental insurance

100% company paid Vision insurance

100% company paid Life insurance

100% company paid Long-term insurance

Short-term insurance

Annual Profit-Sharing Plan

Discretionary Performance Bonus

Paid Parental Leave

Generous Paid Time Off, including Holiday, Vacation, and Sick Pay

Flexible Work Hours

The pay range for this position is $113,000 - $168,000 / year. SciTec considers several factors when extending an offer of employment, including but not limited to the role and associated responsibilities, a candidate's work experience, education/training, and key skills. This is not a guarantee of compensation.

#ND123

SciTec is committed to hiring and retaining a diverse workforce and is proud to be an Equal Opportunity/Affirmative Action employer.