This job offer is no longer available
Big Data Engineer
Tata Consultancy Services
- Irving, Texas, United States
- Irving, Texas, United States
About
Responsibilities
Stakeholder Unification: Serve as a single point of contact and unified stakeholder for all Hadoop‑related concerns, bridging the gap between platform administrators and application teams.
Platform Upgrade Management: Review and assess upcoming Hadoop platform upgrades, including new features, libraries, and patches. Conduct impact analysis on existing applications and services, identify potential risks and opportunities, and coordinate and communicate upgrade schedules and requirements with relevant teams.
Technical Feature and Library Evaluation: Identify and evaluate new technical features and libraries within the Hadoop ecosystem that can benefit application teams or improve platform efficiency, and advocate for adoption of new technologies and methodologies.
Cluster Health and Optimization: Monitor overall cluster health, performance metrics, and resource utilization; propose and implement optimization strategies to improve cluster efficiency, scalability, and cost‑effectiveness; collaborate with the admin team to troubleshoot and resolve complex platform‑level issues.
Resource Management and Housekeeping: Oversee and manage the allocation of cluster resources (CPU, memory, storage) across applications and tenants; establish and enforce policies for resource quota management, data lifecycle, and storage optimization; implement housekeeping strategies to maintain a clean and efficient cluster environment.
Best Practices and Overall Excellence: Define, document, and promote best practices for Hadoop application development, deployment, and operations; ensure operational stability and resilience of the Hadoop platform, implementing measures to prevent outages and minimize downtime; contribute to disaster recovery and business continuity plans for the Hadoop ecosystem.
Solution Proposal and Innovation: Research and propose suitable technical solutions to address emerging business needs, performance bottlenecks, or architectural challenges; stay abreast of industry trends and advancements in big data technologies, continuously seeking opportunities for innovation.
Qualifications
Education: Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
Experience: 5+ years in a big data environment with a focus on Hadoop; proven technical leadership or architect role working closely with operations and development teams; experience with distributed systems, data processing frameworks such as Spark and Hive, and data warehousing concepts; familiarity with cloud platforms (AWS, Azure, GCP) and containerization technologies (Docker, Kubernetes) is a plus.
Technical Skills: Deep understanding of Hadoop ecosystem components (HDFS, YARN, MapReduce, Hive, Spark, Kafka, etc.); strong understanding of Spark architecture and core concepts; proficiency in Linux scripting for automation and system management; basic to intermediate proficiency in Python and Scala for scripting and data manipulation; experience with monitoring tools such as Grafana and Prometheus, and logging frameworks; awareness of various data engineering solutions and consumption tools; strong understanding of security best practices in a big data environment.
Salary Range: $120,000 to $145,000 per year
#J-18808-Ljbffr
Languages
- English
Notice for Users
This job was posted by one of our partners. You can view the original job source here.