Data Engineer

Job Title: Data Engineer US

Company Overview: Atreides helps organizations transform large and complex multi-modal datasets into information-rich, geo-spatial data subscriptions that can be used across a wide spectrum of use cases. Currently, Atreides focuses on providing defense intelligence professionals with high-fidelity data solutions that enable these customers to derive insights in minutes. The company is a fast-moving, and high-performance startup.

We are a fast-growing early-stage company in the process of closing our seed funding led by a well-recognized venture firm. We value a diverse team and believe inclusion performs better. We believe that trust + autonomy = better results and more job satisfaction. Since we are building something new and helping unlock the power of massive-scale data to make the world safer, stronger, and more prosperous, a mission-driven mindset and entrepreneurial spirit are a must.

Team Overview: We are a passionate team of technologists, data scientists, and analysts with backgrounds in operational intelligence, law enforcement, large multinationals, and cybersecurity operations. We obsess about designing products that will change the way global companies, governments and nonprofits protect themselves from external threats and global adversaries.

Position Overview: We're looking for a skilled Data Engineer to specialize in designing, implementing, and optimizing data pipelines and storage solutions to manage our extensive datasets, ranging from hundreds of terabytes to petabytes. You will play a key role in ensuring data integrity, accessibility, and security while working closely with our infrastructure engineers to automate and streamline data processes.

Team Principles: At Atreides, we believe that teams work best when they: • • Remain curious and passionate in all aspects of our work • Promote clear, direct, and transparent communication • Embrace the 'measure twice, cut once' philosophy • Value and encourage diverse ideas and technologies • Lead with empathy in all interactions

Responsibilities: • Design, develop, and maintain scalable data pipelines using Spark, Scala, Java and Python. Nice to have: Golang • Implement and manage Iceberg tables and ensure efficient data storage and retrieval • Develop and enforce data retention policies and ensure compliance with data governance standards • Work with Postgres, GraphQL, and other data manipulation tools. • Collaborate with infrastructure engineers to integrate data processes with AWS infrastructure and automation tools • Ensure data security and implement necessary measures to protect sensitive information • Monitor and troubleshoot data pipelines to ensure high availability and performance • Document data engineering processes and maintain comprehensive documentation

Desired Qualifications: • 3+ years of experience in data engineering or related roles • Proficiency in Spark, Scala, Java, and Python. Nice to have: Golang • Experience with Postgres, GraphQL, and other data manipulation tools • Strong understanding of data optimization, storage solutions, and data retention policies • Familiarity with AWS or Azure infrastructure and site reliability engineering principles. AWS/Azure Data Eng/DevOps Associate or Professional AWS/Azure certificates are a plus. • Knowledge of big data tools and environments, having previously worked with at least 100TB of data at production scale • Familiarity with geospatial data formats such as Parquet/GeoParquet, GeoJSON, and Shapefiles is a plus. Visualization of geospatial data to facilitate understanding with Analysts and End Users a plus. • Experience with data orchestration tools such as Airflow, Prefect, Dagster, or Temporal • Excellent problem-solving skills and the ability to think quickly in a high-performance environment • Effective communication skills to convey technical concepts to both technical and non-technical stakeholders

Compensation and Benefits: • Competitive salary • Comprehensive health, dental, and vision insurance plans • Flexible hybrid work environment • Additional benefits like flexible hours, work travel opportunities, competitive vacation time and parental leave

While meeting all of these criteria would be ideal, we understand that some candidates may meet most, but not all. If you're passionate, curious and ready to "work smart and get things done," we'd love to hear from you.