Senior Big Data Engineer

LiveRamp

New York, New York, United States

New York, New York, United States

Über

Job Overview LiveRamp is a data collaboration platform that helps companies build a connected customer view while protecting privacy and compliance requirements. Responsibilities Design, build, and optimize scalable data processing pipelines using Spark, Airflow, and related big‑data technologies for both batch and real‑time use cases. Lead and collaborate with technical, application, and security stakeholders to deliver reliable, secure Big Data infrastructure using tools such as Spark, Dataproc, Redpanda, and Temporal. Own and participate in on‑call responsibilities for Big Data platforms, triaging incidents, resolving tickets, and ensuring systems consistently meet defined SLAs for availability, performance, and data quality. Design, develop, and automate large‑scale infrastructure on Kubernetes using Terraform and other infrastructure‑as‑code patterns to support high‑performance data processing, analytics, and AI/ML workloads. Define and implement monitoring, alerting, and runbooks to provide end‑to‑end observability and drive continuous improvement in reliability and operational excellence with tools such as Grafana. Onboard, train, and mentor vendor teams and external partners so they can effectively support, operate, and extend the solutions owned by the Big Data Infrastructure (BDI) team. Drive the technical roadmap for emerging data and infrastructure technologies by evaluating options, building proofs of concept (POCs), and authoring solution selection and design documents. Champion engineering best practices (code reviews, testing strategies, CI/CD, security and compliance standards) across the Big Data ecosystem to ensure maintainable, resilient, and cost‑efficient systems. Team Context Build and operate the core BDI platform that powers large‑scale data processing, analytics, and AI/ML workloads across LiveRamp using Dataproc, Redpanda, Airflow and Temporal. Provide secure, self‑service, and cost‑efficient data environments that enable product and data teams to experiment, ship, and scale data collaboration applications quickly. Partner closely with application, security, SRE, and platform teams to ensure data systems meet LiveRamp’s standards for privacy, compliance, reliability, and performance. Evolve common patterns, tooling, and reference architectures that reduce friction for teams adopting new big data, streaming, and AI/ML capabilities. Continuously improve platform reliability and developer experience by automating operations, simplifying workflows, and incorporating feedback from internal customers. About You 7+ years of experience in software engineering or data engineering focused on large‑scale data processing and distributed systems. Hands‑on experience building and operating pipelines with Apache Spark (batch and/or streaming) and at least one orchestration framework such as Airflow or Temporal. Practical experience running Big Data workloads on cloud platforms (e.g., Dataproc or similar managed compute services). Proficiency with Kubernetes and infrastructure‑as‑code tools such as Terraform for provisioning, configuring, and managing production services. Experience with modern streaming and messaging technologies (e.g., Redpanda, Kafka, or similar) and real‑time data processing patterns. Solid understanding of systems design, scalability, reliability, and observability for data‑intensive workloads. Experience participating in or leading on‑call rotations and incident management processes for production systems. Strong collaboration and communication skills, including working with cross‑functional partners (security, application owners, vendors) and documenting designs and decisions clearly. Ability to evaluate new technologies, build POCs, and make data‑informed recommendations that influence team and platform roadmaps. Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related technical field, or equivalent practical experience. Preferred Skills Experience working on Big Data Infrastructure, data platform, or core infrastructure teams in a cloud‑native environment. 3+ years of software development experience in Python, Go, Java, or Scala. 3+ years of experience with Linux operating systems. 3+ years of experience setting up, managing, and optimizing Big Data ecosystems such as Kafka, Airflow, Redpanda, and Temporal. 3+ years of experience using cloud‑based platforms such as AWS, GCP, Azure, or similar. 3+ years of experience with a version control system (GIT, Subversion, or similar). Experience in using AI tools, NLP, and agent creation. Exposure to security and compliance considerations for data platforms (access control, encryption, data governance, auditing). Prior experience mentoring engineers, leading technical initiatives, or coordinating work with vendors and external partners. Compensation & Benefits Annual base compensation range: $130,000 to $196,500. The actual offer reflects experience, skills, geography, and internal equity. Benefits include medical, dental, vision, life and disability insurance, an employee assistance program, voluntary benefits, and a 401(k) matching plan of 1:1 up to 6% of salary. Flexible paid time off, paid holidays, options for working from home, and paid parental leave are also provided. Equal Opportunity LiveRamp is an affirmative action and equal opportunity employer (AA/EOE/W/M/Vet/Disabled) and does not discriminate in recruiting, hiring, training, promotion or other employment of associates or the awarding of subcontracts because of a person’s race, color, sex, age, religion, national origin, protected veteran, disability, sexual orientation, gender identity, genetics or other protected status. California residents: Please see the California Personnel Privacy Policy for more information about how we collect, use, and disclose the personal information you provide during the job application process. Qualified applicants with arrest and conviction records will be considered for the position in accordance with the San Francisco Fair Chance Ordinance. We use automated decision systems (ADS) as part of our recruitment and hiring process. If you require an accommodation or believe that the use of an ADS may create a barrier to your application or participation in the hiring process due to a disability or other protected characteristic, please let us know. We are committed to providing reasonable accommodations and ensuring an equitable hiring experience for all candidates.
#J-18808-Ljbffr

New York, New York, United States

Sprachkenntnisse

English

Hinweis für Nutzer

Dieses Stellenangebot wurde von einem unserer Partner veröffentlicht. Sie können das Originalangebot einsehen hier.

Ähnliche Jobs finden