Hadoop Administrator

Job Description: Hadoop Administrator

Role Overview

We are looking for an experienced Hadoop Administrator to manage, maintain, and upgrade our on-premises Hadoop clusters. The role involves working on Apache Hadoop versions 2.7.1 and 3.4.x, ensuring cluster stability, performance, and scalability, along with ecosystem components like Hive and YARN.

Key Responsibilities

• Cluster Administration:
• Install, configure, and manage on-prem Hadoop clusters based on Apache Hadoop (versions 2.7.1 and 3.4.x)
• Administer and monitor Hadoop Distributed File System (HDFS) including storage management, replication, and data balancing
• Manage resource allocation and scheduling using Apache YARN

• Upgrade & Migration:
• Hadoop upgrades (2.7 → 3.4) in on-prem environments
• Perform Cluster migration, Data validation (HDFS checksums, block integrity) & Backward compatibility validation for jobs
• Experience with migration tools and distribution changes

• Ecosystem Management
• Administer and optimize Apache Hive (query tuning, metastore management)
• Work with ecosystem tools like Spark (basic admin exposure preferred) & Tez (execution engine for Hive)
• Ensure high availability for Hive Metastore and critical services

• Performance & Optimization
• Monitor cluster health, job performance, and resource utilization
• Tune YARN queues and capacity scheduler, Hive queries and execution plans, Disk, memory, and network utilization
• Implement performance improvements for large-scale data processing

• Security & Governance
• Configure authentication/authorization
• Ensure data security and compliance policies
• Manage access controls and audit logs

• Troubleshooting & Support
• Provide support for Hadoop cluster issues
• Troubleshoot Node failures, Job failures (Hive/YARN) & Data inconsistencies
• Perform log analysis and root cause identification

• Backup & DR
• Plan and manage backup and disaster recovery strategies
• Implement DistCp-based replication & DR cluster setup (active-passive / active-active)

Required Skills

• Strong hands-on experience with Apache Hadoop (2.7.x and 3.x)
• Expertise in Apache YARN, Apache Hive & HDFS administration
• Experience in on-prem cluster setup and hardware-level optimization
• Linux administration (RHEL/CentOS/Ubuntu)
• Shell scripting for automation