Lead Data Engineer

Altimate

United States

United States

Find similar jobs

About

Join Altimate AI, a pioneering company founded in 2022 in San Francisco, as we transform enterprise data operations through innovative AI solutions. Our mission is to empower enterprise data teams overwhelmed by the demands of their roles by providing cutting-edge technologies that automate and streamline data tasks. With our flagship product, DataPilot, and our groundbreaking DataMates technology, we enable data professionals to enhance their productivity with virtual AI teammates that integrate seamlessly into everyday tools like VSCode, Git, and Slack. We utilize a unique framework combining various language models and a custom-built knowledge graph for contextual AI agents that integrate smoothly into existing workflows. Our solutions include ambient AI for continuous monitoring and optimization, responding to the ever-growing needs of data operations, business intelligence, and analytics amid surging data volumes. Our products are already trusted by thousands of users globally, with backing from top investors, positioning us at the forefront of the AI-driven data engineering landscape. What We Are Looking For We are seeking a Lead Data Engineer who is not only passionate about innovation and problem-solving but is also eager to think beyond traditional data engineering approaches. This role is suited for individuals with extensive experience in constructing large-scale data pipelines, optimizing SQL, and managing cloud infrastructure. You will play a vital role in enhancing our AI capabilities and developing high-performance data systems that integrate advanced AI technology into the daily workflows of thousands of data teams. Qualifications: 8+ years of experience in data engineering with a focus on scalable data pipeline development. Strong proficiency in Python and SQL. Extensive experience with SQL query profiling, optimization, and performance tuning, preferably with Snowflake. Deep understanding of SQL Abstract Syntax Tree (AST) and familiarity with SQL parsers (e.g., sqlglot) for generating detailed column-level lineage and dynamic ETLs. Experience with data pipeline construction using Airflow or dbt. [Optional] Solid understanding of cloud platforms, particularly AWS. [Optional] Knowledge of Kubernetes (K8s) for containerized deployments. Projects You'll Work On: Building Petabyte Scale Data Pipelines:
Develop high-performance infrastructure that scales to support 100K+ jobs and manages PB-scale data daily. Cloud-Native Data Infrastructure:
Design and implement robust, scalable data architecture on AWS, employing Kubernetes and Airflow for effective resource management and deployment. Intelligent SQL Ecosystem:
Create a comprehensive SQL intelligence system that includes query optimization, dynamic pipeline generation, and data lineage tracking. Utilize your expertise in SQL profiling, AST analysis, and parsing to enhance query performance and implement detailed column-level lineage. Impact: Innovation at the Forefront:
Redefine data engineering by merging conventional approaches with leading-edge AI technologies. High Visibility:
Influence the productivity and efficiency of global data teams, contributing directly to the daily operations of users in hundreds of countries. Open Source Contribution:
Engage with the developer community by contributing to our open-source projects and gaining recognition within the tech sphere. Career Growth:
This position serves as a launchpad in the rapidly evolving field of AI-driven data engineering, offering exposure to state-of-the-art technologies and applications in generative AI. What We Offer: Competitive salary and equity. Extensive professional development opportunities and resources for ongoing learning. A dynamic and intellectually stimulating environment with a team of talented engineers. Opportunities to influence the company's direction and make a lasting impact. If this resonates with you, we can't wait to meet you! Apply now and let's create something incredible together.

United States

Languages

English

Notice for Users

This job was posted by one of our partners. You can view the original job source here.

Find similar jobs