Manager Data
Resume:
MAHENDAR
Sr.Hadoop Administrator
ac1a2w@r.postjobfree.com
Professional Summary:
Over 8+ years of experience including 4+ years of experience with Hadoop Ecosystem in installation and administrated of all UNIX/LINUX servers and configuration of different Hadoop eco-system components in the existing cluster project.
Extensive experience in installing, configuring and administrating Hadoop cluster for major Hadoop distributions like CDH and HDP.
Experience in deploying and managing the multi-node development, testing and production Hadoop cluster with different Hadoop components (HIVE, PIG, SQOOP, OOZIE, FLUME, HBASE, ZOOKEEPER) using Cloudera Manager and Ambari.
Hadoop Cluster capacity planning, performance tuning, cluster Monitoring, Troubleshooting.
Hands on experience on configuring a Hadoop cluster in a professional environment and on Amazon Web Services (AWS) using an EC2instance and Rack space.
Experience in importing and exporting the data using Sqoop from HDFS to Relational Database systems/mainframe and vice-versa.
Excellent knowledge of in NOSQL databases like HBase, Cassandra and MongoDB.
Configuring Kerberos and integrating with Directory services.
Providing security for Hadoop Cluster with Kerberos, Active Directory/LDAP, and TLS/SSL utilizations and dynamic tuning to make cluster available and efficient.
Expertise in Managing, Monitoring and Administration of Hadoop for Multi Hundred Node Cluster with different distributions like Cloudera CDH and Horton works HDP.
Experience in managing the cluster resources by implementing fair scheduler and capacity scheduler.
Worked with Sqoop for importing and exporting data from different databases like RDBMS, Oracle, Mysql into HDFS
Worked with Flume for collecting the logs from log collector into HDFS.
Experience in developing and scheduling ETL workflows in Hadoop using Oozie.
Worked on Disaster Management with Hadoop cluster.
Experience in performing minor and major upgrades, commissioning and decommissioning of data nodes on Hadoop cluster.
Strong Capability to Design Big Data solutions for traditional enterprise businesses.
Installation, patching, upgrading, tuning, configuring and troubleshooting Linux based operating systems Red Hat and Centos and virtualization in a large set of servers.
Having Experience in Install and configuration of Web hosting administration.
Worked on Firewall implementation & Load balancer between various Windows servers.
Working on different kind of compression techniques like LZO, GZIP, Snappy.
Worked with Puppet for automated deployments.
Experience in Installation of VMware ESX server and creation of VMs and install different guest OS.
Extensive Experience in monitoring and troubleshooting issues with Linux memory, CPU, OS, storage and network
Good Experience in Managing and Scheduling Cron jobs such as enabling system logging, network logging of servers for maintenance, performance tuning and testing
Extensively worked on configuring & administering YUM, RPM's, NFS, DNS, and DHCP, Mail servers.
Experience in Managing Various Network related tasks such as TCP/IP, NFS, DNS, DHCP and SMTP.
Working with Datameer for Big data Analytics solutions.
Proficient in OS upgrades and Patch loading as and when required.
Expert in setting up SSH, SCP and VSFTP connectivity between UNIX hosts.
Experience in supporting users to debug their job failures and supporting systems with 24X7 availability and monitoring.
TECHNICAL SKILLS:
Hadoop Ecosystem
HDFS, Map reduce, YARN, Hive, Pig, Sqoop, Oozie, Flume and Zoo Keeper
Languages
C, Java, and Python
RDBMS Databases
Oracle Sql, PL/SQL,Mysql and Sql server
NoSql Databases
Hbase and Cassandra
Monitoring Tools
Ganglia and Nagios
Configuration Management Tools
Puppet and Chef
Authentication
Kerberos and sentry
Analytical Tool
Datameer
EDUCATION:
Bachelor of Engineering in Computer Science at Kakatiya University, warangal, India.
PROFESSIONAL EXPERIENCE:
Client: Equifax, Atlanta, GA
Title: Sr. Hadoop Administrator
Duration: Jun 2015 to Present
Responsibilities:
Currently working as Sr. Hadoop Administrator on Cloudera (CDH5) distribution and responsible for clusters from POC (Proof-of-Concept) to PROD clusters.
Responsible for Cluster maintenance, Monitoring, commissioning and decommissioning Data nodes, Troubleshooting, Manage and review data backups, Manage & review log files.
Day to day responsibilities includes solving developer issues, deployments moving code from one environment to other environment, providing access to new users and providing instant solutions to reduce the impact and documenting the same and preventing future issues.
Installed Oozie workflow engine to run multiple Hive and pig jobs.
Creating collections and configurations, Register a Lily HBase Indexer configuration with the Lily HBase Indexer Service.
Implemented test scripts to support test driven development and continuous integration.
Creating and truncating HBase tables in hue and taking backup of submitter ID(s).
Adding/installation of new components and removal of them through Cloudera Manager.
Collaborating with application teams to install operating system and Hadoop updates, patches, version upgrades.
Integrated Hadoop cluster with Active Directory and enabled Kerberos for Authentication.
Implemented Capacity schedulers on the Yarn Resource Manager to share the resources of the cluster for the Map Reduce jobs given by the users.
Monitored workload, job performance and capacity planning using Cloudera Manager.
Involved in Analyzing system failures, identifying root causes, and recommended course of actions.
Involved in Sentry services setup with security configuration with LDAP Users.
Interacting with Cloudera support and log the issues in Cloudera portal and fixing them as per their recommendations.
Configured Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scala.
Experience in Apache Spark cluster and streams processing using Spark Streaming.
Used Spark Streaming API with Kafka to build live dashboards; Worked on Transformations & actions in RDD, Spark Streaming, Pair RDD Operations, Check-pointing, and SBT.
Designed and implemented Disaster Recovery Plan for Hadoop Clusters.
Imported logs from web servers with Flume to ingest the data into HDFS.
Partitioned and queried the data in Hive for further analysis by the BI team.
Extending the functionality of Hive and Pig with custom UDF’s and UDAF's.
Involved in extracting the data from various sources into Hadoop HDFS for processing.
Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, Hbase database and Sqoop.
Installed Apache Kafka and monitored it with ganglia and nagios.
Monitoring Hadoop Clusters using Ganglia and Nagios with 24x7 on-call support.
Configuring, Managing permissions for the users in Hue.
Responsible for building scalable distributed data solutions using Hadoop.
Commissioned and Decommissioned nodes on CDH5 Hadoop cluster on Redhat 6.7.
Involved in loading data from LINUX file system to HDFS.
Worked on tuning the performance Pig queries.
Responsible to manage data coming from different sources.
Experience in managing and reviewing Hadoop log files.
Exported the analyzed data to the relational databases using Sqoop for visualization and to generate report for the BI team.
Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop.
Environment: HDFS, Hive, Hue, Pig, Flume, Yarn, Oozie, Sqoop, Scala, CDH5, Spark, Storm, Cloudera Manager, Datameer, Red Hat, MySQL and Oracle.
Client: Citizens Insurance, Tallahassee- FL
Title: Hadoop Administrator
Duration: Aug 2014 to June 2015
Responsibilities:
Currently working as admin in Hortonworks (HDP) distribution for 4 clusters ranges from POC to PROD.
Responsible for Cluster maintenance, Monitoring, commissioning and decommissioning Data nodes, Troubleshooting, Manage and review data backups, Manage & review log files.
Experienced on adding/installation of new components and removal of them through Ambari.
Monitoring systems and services through Ambari dashboard to make the clusters available for the business.
Architecture design and implementation of deployment, configuration management, backup, and disaster recovery systems and procedures.
Hand on experience on cluster up gradation and patch upgrade without any data loss and with proper backup plans.
Changing the configurations based on the requirements of the users for the better performance of the jobs.
Experienced in Ambary-alerts configuration for various components and managing the alerts.
Provided security and authentication with ranger where ranger admin provides administration and user sync adds the new users to the cluster.
Good troubleshooting skills on Hue, which provides GUI for developer's/business users for day to day activities.
Experience in Ranger, Knox configuration to provide the security for Hadoop services.
Developed Map Reduce programs to cleanse the data in HDFS obtained from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis.
Implemented Name Node HA in all environments to provide high availability of clusters.
Create queues and allocated the clusters resources to provide the priority for jobs.
Experienced in Setting up the project and volume setups for the new projects.
Involved in snapshots and mirroring to maintain the backup of cluster data and even remotely.
Implementing the SFTP for the projects to transfer data from External servers to servers.
Experienced in managing and reviewing log files.
Working experience on maintaining MySQL databases creation and setting up the users and maintain the backup of cluster metadata databases with cron jobs.
Setting up MySQL master and slave replications and helping business applications to maintain their data in MySQL Servers.
Managed and reviewed Log files as a part of administration for troubleshooting purposes. Communicate and escalate issues appropriately.
Involved in Analyzing system failures, identifying root causes, and recommended course of actions.
Monitored multiple clusters environments using AMBRI Alerts, Metrics and Nagios.
Environment: Hadoop Hdfs, Mapreduce, Hive, Scala, Pig, Flume, Nagios, Ranger, Knox, Oozie, Sqoop, Eclipse, Hortonworks, Ambari.
Client: Arista Networks - Santa Clara- CA
Title: Hadoop Administrator
Duration: Nov 2013 – July 2014
Responsibilities:
Experienced as admin in MapR distribution for clusters ranges from POC to PROD.
Implemented and Configured High Availability Hadoop Cluster (Quorum Based).
Involved in managing and reviewing Hadoop log files.
Implemented Fair scheduler on the job tracker to share the resources of the cluster for the Map reduces jobs given by the users.
Used Sqoop to import and export data from HDFS to RDBMS and vice-versa.
Hands on experience working on Hadoop ecosystem components like HDFS, Map Reduce, YARN, Zookeeper, Pig, Hive, Sqoop, Flume.
Worked on setting up high availability for major production cluster and designed automatic failover control using zookeeper and quorum journal nodes.
Effectively used Oozie workflow engine to run multiple Hive and Pig jobs.
Implemented rack aware topology on the Hadoop cluster.
Experience in using Flume to stream data into HDFS from various sources.
Responsible for troubleshooting issues in the execution of Map Reduce jobs by inspecting and reviewing log files.
Implemented Kerberos for authenticating all the services in Hadoop Cluster.
Experience in configuring Zookeeper to coordinate the servers in clusters to maintain the data consistency.
Created HBase tables to store various data formats of data coming from different portfolios.
Involved in Commissioning and Decommissioning of nodes depending upon the amount of data.
Automated the work flow using shell scripts.
Performance tuning of the hive queries, written by other developers.
Installed and maintain puppet-based configuration management system.
Environment: - HDFS, Map Reduce, Hive, Sqoop, PIG, Flume, SQL Server, UNIX, Red hat and Centos.
Client: Flagstar Bank – Detroit- MI
Title: Hadoop Administrator
Duration: Dec 2012 to Oct 2013
Responsibilities:
Experienced in Installation and configuration of Cloudera CDH
Responsible on-boarding new users to the Hadoop cluster (adding user a home directory and providing access to the datasets).
Played responsible role for deciding the hardware configurations for the cluster along with other teams in the company.
Resolved tickets submitted by users, P1 issues, troubleshoot the errors, documenting, resolving the errors.
Responsible for giving presentations about new ecosystems to be implemented in the cluster with the teams and managers.
Helped the users in production deployments throughout the process.
Managed and reviewed Hadoop Log files as a part of administration for troubleshooting purposes.
Communicate and escalate issues appropriately.
Applied patches to cluster.
Added new Data Nodes when needed and ran balancer.
Responsible for building scalable distributed data solutions using Hadoop.
Continuous monitoring and managing the Hadoop cluster through Ganglia and Nagios.
Installed Oozie workflow engine to run multiple Hive and Pig jobs, which run independently with time and data availability.
Done major and minor upgrades to the Hadoop cluster.
Upgraded the Cloudera Hadoop ecosystems in the cluster using Cloudera distribution packages.
Commissioned and decommissioned the Data Nodes in the cluster in case of the problems.
Debug and solve the major issues with Cloudera manager by interacting with the Cloudera team from Cloudera.
Environment: Flume, Oozie, Pig, Sqoop, Mongo, Hbase, Hive, Map-Reduce, YARN and Cloudera Manager
Client: Mcraan System Inc., Chennai, India.
Title: Linux Administrator
Duration: Jan 2011 to Nov 2012
Responsibilities:
Design, build, install and configure Red Hat Enterprise Linux servers (RHEL5, RHEL6) on bare metal servers of HP ProLiant DL380.
Installation and configuration of RHEL, CENTOS OS in servers in virtual VM machines (VMware) and proxmox virtual machine.
Configuring Fortigate firewalls and monitoring firewall logs (to enhance security, Network load balance, Application and user monitoring).
User, Group creation monitoring and maintain log for system status/system health using Linux commands and Nagios, Top, Genome system monitor.
Performance monitoring on Linux servers using iostat, netstat, vmstat, sar, top&prstat.
Experience in Managing and Scheduling Cron jobs such as enabling system logging, network logging of servers for maintenance, performance tuning and testing
Setup ACL and Sticky bit for critical files and directories.
Involve into user and group level issues like change Shell and directories.
Installed and Configured Red hat Linux Kick start and booting from SAN/NAS.
Experience in using protocols like NFS, SSH, VSFTP & DNS.
Performed package installations, maintenance, periodic updates and patch management.
Installation/Administration of TCP/IP, NFS, DNS, NTP, Auto mounts, Send mail and Print servers as per the client's requirement.
Experience in troubleshooting samba related issues.
Performed disk administration using LVM, Linux Volume Manager (LVM), and Veritas Volume Manager 4.x/5.x.
Installed VMWare ESX4.1 to perform virtualization of RHEL servers.
Configured Iptables on Linux servers.
Installing MysqlDB in Linux and Customize the MySQL DB parameters.
Schedule MySQL DB’s backup to remote host
Performed Package administration on Linux using rpm, yum and Satellite server.
Automation of various administrative tasks on multiple servers using Puppet.
Deployed Puppet, Puppet Dashboard, and Puppet DB for configuration management to existing infrastructure.
Proficient in installation, configuration and maintenance of applications like Apache, LDAP, PHP Resolved config issues and problems related to OS, NFS mounts, LDAP user ids DNS and issues.
Worked on VMware, VMware View, vSphere 4.0. Dealt with ESX, ESXi servers.
Performed installing, configuring and trouble-Shooting web servers like IBM HTTP Web Server, Apache Web Server, Web sphere Application Servers, and Samba Server on Linux (Redhat & Cantos).
Create and update technical documentation for team members.
Environment: Red-Hat Linux Enterprise servers (HP Proliant DL 585, BL ML Series, SAN (Netapp), VMware Virtual Client 3.5, VMware Infrastructure 3.5, Bash, Centos, LVM, Windows 2003 server, Net Backup, Veritas Volume Manager, Samba, Mysql and NFS
Client: Solaris IT, Hyderabad, India
Title: Linux Administrator
Duration: July‘2008 to Dec’2010
Responsibilities:
Administration of RHEL, which includes installation, testing, tuning, upgrading and loading patches, troubleshooting both physical and virtual server issues.
Creating, cloning Linux Virtual Machines.
Installing Red Hat Linux using kick start and applying security polices for hardening the server based on the company policies.
RPM and YUM package installations, patch and other server management.
Managing systems routine backup, scheduling jobs like disabling and enabling cronjobs, enabling system logging, network logging of servers for maintenance, performance tuning, testing.
Tech and non-tech refresh of Linux servers, which includes new hardware, OS, upgrade, application installation, testing.
Set up user and group login ID's, printing parameters, network configuration, password, resolving permissions issues, and user and group quota.
Installing MySQLDB in Linux and Customize the MySQL DB parameters.
Working with Service Now incident tool.
Creating physical volumes, volume groups and logical volumes.
Samba Server configuration with Samba Clients.
Knowledge of IP tables, SELINUX.
Modified existing Linux file systems to a Standard EXT3.
Configuration and administration of NFS FTP, SAMBA, NIS.
Maintenance of DNS, DHCP and APACHE services on Linux machines.
Installing and configuring Apache and supporting them on Linux production servers.
Environment: Red-Hat Linux Enterprise servers (HP Proliant DL 585, BL [ ] ML Series, SAN (Netapp), VERITAS Cluster Server 5.0, Windows 2003 server, Shell programming.
Contact this candidate