com.twitter.elephantbird.util

Class HadoopCompat

java.lang.Object

com.twitter.elephantbird.util.HadoopCompat

public class HadoopCompat
extends Object

Utility methods to allow applications to deal with inconsistencies between MapReduce Context Objects API between Hadoop 1.x and 2.x.

Constructor Summary

Constructors

Constructor and Description
HadoopCompat()

Method Summary

Methods

Modifier and Type	Method and Description
static org.apache.hadoop.conf.Configuration	getConfiguration(org.apache.hadoop.mapreduce.JobContext context) Invoke getConfiguration() on JobContext.
static org.apache.hadoop.mapreduce.Counter	getCounter(org.apache.hadoop.mapreduce.TaskInputOutputContext context, String groupName, String counterName) Invoke getCounter() on TaskInputOutputContext.
static org.apache.hadoop.mapreduce.TaskAttemptID	getTaskAttemptID(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext) returns TaskAttemptContext.getTaskAttemptID().
static void	incrementCounter(org.apache.hadoop.mapreduce.Counter counter, long increment) Increment the counter.
static boolean	isVersion2x() True if runtime Hadoop version is 2.x, false otherwise.
static org.apache.hadoop.mapreduce.Counter	newGenericCounter(String name, String displayName, long value)
static org.apache.hadoop.mapreduce.JobContext	newJobContext(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.mapreduce.JobID jobId) Creates JobContext from a JobConf and jobId using the correct constructor for based on Hadoop version.
static org.apache.hadoop.mapreduce.MapContext	newMapContext(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptID, org.apache.hadoop.mapreduce.RecordReader recordReader, org.apache.hadoop.mapreduce.RecordWriter recordWriter, org.apache.hadoop.mapreduce.OutputCommitter outputCommitter, org.apache.hadoop.mapreduce.StatusReporter statusReporter, org.apache.hadoop.mapreduce.InputSplit inputSplit) Instantiates MapContext under Hadoop 1 and MapContextImpl under Hadoop 2.
static org.apache.hadoop.mapreduce.TaskAttemptContext	newTaskAttemptContext(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptId) Creates TaskAttempContext from a JobConf and jobId using the correct constructor for based on Hadoop version.
static void	setStatus(org.apache.hadoop.mapreduce.TaskAttemptContext context, String status) Invoke setStatus() on TaskAttemptContext.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopCompat

public HadoopCompat()

Method Detail

isVersion2x

public static boolean isVersion2x()

True if runtime Hadoop version is 2.x, false otherwise.

newJobContext

public static org.apache.hadoop.mapreduce.JobContext newJobContext(org.apache.hadoop.conf.Configuration conf,
                                                   org.apache.hadoop.mapreduce.JobID jobId)

Creates JobContext from a JobConf and jobId using the correct constructor for based on Hadoop version. jobId could be null.

newTaskAttemptContext

public static org.apache.hadoop.mapreduce.TaskAttemptContext newTaskAttemptContext(org.apache.hadoop.conf.Configuration conf,
                                                                   org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptId)

Creates TaskAttempContext from a JobConf and jobId using the correct constructor for based on Hadoop version.

newMapContext

public static org.apache.hadoop.mapreduce.MapContext newMapContext(org.apache.hadoop.conf.Configuration conf,
                                                   org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptID,
                                                   org.apache.hadoop.mapreduce.RecordReader recordReader,
                                                   org.apache.hadoop.mapreduce.RecordWriter recordWriter,
                                                   org.apache.hadoop.mapreduce.OutputCommitter outputCommitter,
                                                   org.apache.hadoop.mapreduce.StatusReporter statusReporter,
                                                   org.apache.hadoop.mapreduce.InputSplit inputSplit)

Instantiates MapContext under Hadoop 1 and MapContextImpl under Hadoop 2.

newGenericCounter

public static org.apache.hadoop.mapreduce.Counter newGenericCounter(String name,
                                                    String displayName,
                                                    long value)

Returns:

with Hadoop 2 : new GenericCounter(args),
with Hadoop 1 : new Counter(args)

getConfiguration

public static org.apache.hadoop.conf.Configuration getConfiguration(org.apache.hadoop.mapreduce.JobContext context)

Invoke getConfiguration() on JobContext. Works with both Hadoop 1 and 2.

setStatus

public static void setStatus(org.apache.hadoop.mapreduce.TaskAttemptContext context,
             String status)

Invoke setStatus() on TaskAttemptContext. Works with both Hadoop 1 and 2.

getTaskAttemptID

public static org.apache.hadoop.mapreduce.TaskAttemptID getTaskAttemptID(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)

returns TaskAttemptContext.getTaskAttemptID(). Works with both Hadoop 1 and 2.

getCounter

public static org.apache.hadoop.mapreduce.Counter getCounter(org.apache.hadoop.mapreduce.TaskInputOutputContext context,
                                             String groupName,
                                             String counterName)

Invoke getCounter() on TaskInputOutputContext. Works with both Hadoop 1 and 2.

incrementCounter

public static void incrementCounter(org.apache.hadoop.mapreduce.Counter counter,
                    long increment)

Increment the counter. Works with both Hadoop 1 and 2