Hadoop Learning Tutorials

Hadoop Overview:-------------------------

Hadoop MapReduce is a software framework to easily write applications that deal with large quantities of data (multi-terabyte datasets) in parallel on major groups (of thousands of nodes) of material in reliable, tolerant manner.

Generally, a MapReduce job splits data into independent pieces set input which is processed by map in completely parallel tasks. The framework sorts cards, which are then entered for tasks to reduce output. As well general the input and output of the job are stored in a file system. The framework takes care of planning tasks, their monitoring and re - runs the task failed.

In general, the compute nodes and the storage nodes are the same, i.e. the framework MapReduce and the Hadoop Distributed File System run on the same set of nodes. This configuration allows the framework to effectively schedule tasks on the nodes where the data is already present, resulting in very high aggregate bandwidth in the whole of the cluster.

The MapReduce framework consists of one master, JobTracker and a TaskTracker slave by the node in the cluster. The captain is responsible for the planning of tasks component employment on the slaves, their follow-up and re-running the task failed. Slaves perform the tasks as directed by the master.

At a minimum, applications specify the locations of the inputs and outputs and provide map and reduce functions via the implementations of appropriate interfaces or abstract classes. These and other parameters of work, include the job configuration. The Hadoop work client then submits the work (jar/executable, etc.) and configuration of the JobTracker which then assumes the responsibility to distribute the software/configuration to slaves, planning tasks and their follow-up, providing the status and diagnostic information to the employment-client.

Although the Hadoop framework is completely implemented in Java, MapReduce applications not need to be written in Java.