Data Engineer (Forensic & Streaming)Oslitanditech • Washington, Utah, United States
This job offer is no longer available
Data Engineer (Forensic & Streaming)
Oslitanditech
- Washington, Utah, United States
- Washington, Utah, United States
About
This is a pivotal data-centric role responsible for managing the division's massive data requirements. The engineer will design and maintain scalable pipelines for ingesting, cleansing, and normalizing terabyte-scale forensic datasets. A key responsibility is building and administering high-throughput, real-time data streaming architectures (Kafka/Pulsar) and time-series databases (TimescaleDB) to ensure the AI/ML models are continuously fueled with high-integrity, mission-critical sensor data. About Oslitandi Tech LLC
Our company works with clients to achieve their tactical and strategic goals by unifying sustainable technology solutions which reduce costs, decrease cycle times, and seamlessly manage processes throughout the enterprise.Oslitandi Tech specialty lies in integrating sustainable IT services & solutions, Management & IT Consulting, & Network/Cyber Security. Description
Primary Responsibilities (Operational Duties)
ETL Pipeline Design:
Design, build, and maintain fault-tolerant
ETL pipelines
to ingest, cleanse, transform, and normalize massive, multi-terabyte datasets of historical mission data (logs, transcripts, sensor records) using technologies like
Apache Spark
or equivalent distributed processing frameworks. Real-Time Streaming Architecture:
Implement and administer high-throughput, low-latency
real-time data streaming architectures
utilizing
Apache Kafka
or
Apache Pulsar
to handle live feeds from numerous sensor sources. Time-Series Database Management:
Administer and optimize specialized databases, such as
TimescaleDB
or
InfluxDB , for high-speed storage and retrieval of time-stamped sensor data. Data Governance & Lineage:
Implement comprehensive data governance policies, metadata management, and
data lineage tracking
tools to ensure the integrity, quality, and audibility of data consumed by the AI/ML Squad. Query Optimization:
Work closely with the Mission Software Squad to optimize complex
SQL
and distributed query performance for near real-time retrieval of historical mission data. API Integration:
Perform integration activities to configure, connect, and pull data from 3rd party software APIs, and collaborate with separate engineering teams to configure data sources for pipeline integration. Basic Qualifications (Experience & Technical Stack)
A minimum of
4+ years
of progressive experience in Data Engineering, with specific experience handling high-volume (terabyte-scale), high-velocity datasets. At least
3+ years
of experience designing and managing
Apache Kafka
or similar distributed messaging systems in production. Expert-level proficiency in advanced
SQL
and relational/time-series database optimization (e.g.,
TimescaleDB ). Strong scripting and development skills ( Python ) for pipeline orchestration and data manipulation. Experience with distributed data processing frameworks such as
Apache Spark . Proficiency in developing log ingestion, data normalization strategies, and implementing data models for complex datasets. The candidate shall have a
Bachelor's or Master's
degree in Computer Science, Data Engineering, or a related technical field. Must be eligible for a U.S. Government
Secret Clearance .
#J-18808-Ljbffr
Languages
- English
Notice for Users
This job was posted by one of our partners. You can view the original job source here.