com.twitter.elephantbird.util
Class HadoopCompat

java.lang.Object
  com.twitter.elephantbird.util.HadoopCompat

public class HadoopCompat
extends Object

Utility methods to allow applications to deal with inconsistencies between MapReduce Context Objects API between Hadoop 1.x and 2.x.

Constructor Summary

HadoopCompat()

Method Summary

static org.apache.hadoop.conf.Configuration	getConfiguration(org.apache.hadoop.mapreduce.JobContext context) Invoke getConfiguration() on JobContext.
static org.apache.hadoop.mapreduce.Counter	getCounter(org.apache.hadoop.mapreduce.TaskInputOutputContext context, Enum<?> key) Hadoop 1 & 2 compatible context.getCounter()
static org.apache.hadoop.mapreduce.Counter	getCounter(org.apache.hadoop.mapreduce.TaskInputOutputContext context, String groupName, String counterName) Invoke getCounter() on TaskInputOutputContext.
static long	getCounterValue(org.apache.hadoop.mapreduce.Counter counter) Hadoop 1 & 2 compatible counter.getValue()
static long	getDefaultBlockSize(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) This method invokes getDefaultBlocksize() without path on Hadoop 1 and with Path on Hadoop 2.
static short	getDefaultReplication(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) This method invokes getDefaultReplication() without path on Hadoop 1 and with Path on Hadoop 2.
static org.apache.hadoop.mapreduce.InputSplit	getInputSplit(org.apache.hadoop.mapreduce.MapContext mapContext) Hadoop 1 & 2 compatible MapContext.getInputSplit();
static org.apache.hadoop.mapreduce.JobID	getJobID(org.apache.hadoop.mapreduce.JobContext jobContext) Hadoop 1 & 2 compatible JobContext.getJobID()
static String	getJobName(org.apache.hadoop.mapreduce.JobContext jobContext) Hadoop 1 & 2 compatible JobContext.getJobName()
static org.apache.hadoop.mapreduce.TaskAttemptID	getTaskAttemptID(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext) returns TaskAttemptContext.getTaskAttemptID().
static void	incrementCounter(org.apache.hadoop.mapreduce.Counter counter, long increment) Increment the counter.
static boolean	isVersion2x() True if runtime Hadoop version is 2.x, false otherwise.
static org.apache.hadoop.mapreduce.Counter	newGenericCounter(String name, String displayName, long value)
static org.apache.hadoop.mapreduce.JobContext	newJobContext(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.mapreduce.JobID jobId) Creates JobContext from a JobConf and jobId using the correct constructor for based on Hadoop version.
static org.apache.hadoop.mapreduce.MapContext	newMapContext(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptID, org.apache.hadoop.mapreduce.RecordReader recordReader, org.apache.hadoop.mapreduce.RecordWriter recordWriter, org.apache.hadoop.mapreduce.OutputCommitter outputCommitter, org.apache.hadoop.mapreduce.StatusReporter statusReporter, org.apache.hadoop.mapreduce.InputSplit inputSplit) Instantiates MapContext under Hadoop 1 and MapContextImpl under Hadoop 2.
static org.apache.hadoop.mapreduce.TaskAttemptContext	newTaskAttemptContext(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptId) Creates TaskAttempContext from a JobConf and jobId using the correct constructor for based on Hadoop version.
static void	progress(org.apache.hadoop.mapreduce.TaskAttemptContext context) Invoke TaskAttemptContext.progress().
static void	setStatus(org.apache.hadoop.mapreduce.TaskAttemptContext context, String status) Invoke setStatus() on TaskAttemptContext.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopCompat

public HadoopCompat()

Method Detail

isVersion2x

public static boolean isVersion2x()

True if runtime Hadoop version is 2.x, false otherwise.

newJobContext

public static org.apache.hadoop.mapreduce.JobContext newJobContext(org.apache.hadoop.conf.Configuration conf,
                                                                   org.apache.hadoop.mapreduce.JobID jobId)

Creates JobContext from a JobConf and jobId using the correct constructor for based on Hadoop version. jobId could be null.

newTaskAttemptContext

public static org.apache.hadoop.mapreduce.TaskAttemptContext newTaskAttemptContext(org.apache.hadoop.conf.Configuration conf,
                                                                                   org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptId)

Creates TaskAttempContext from a JobConf and jobId using the correct constructor for based on Hadoop version.

newMapContext

public static org.apache.hadoop.mapreduce.MapContext newMapContext(org.apache.hadoop.conf.Configuration conf,
                                                                   org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptID,
                                                                   org.apache.hadoop.mapreduce.RecordReader recordReader,
                                                                   org.apache.hadoop.mapreduce.RecordWriter recordWriter,
                                                                   org.apache.hadoop.mapreduce.OutputCommitter outputCommitter,
                                                                   org.apache.hadoop.mapreduce.StatusReporter statusReporter,
                                                                   org.apache.hadoop.mapreduce.InputSplit inputSplit)

Instantiates MapContext under Hadoop 1 and MapContextImpl under Hadoop 2.

newGenericCounter

public static org.apache.hadoop.mapreduce.Counter newGenericCounter(String name,
                                                                    String displayName,
                                                                    long value)

Returns:

with Hadoop 2 : new GenericCounter(args),
with Hadoop 1 : new Counter(args)

getConfiguration

public static org.apache.hadoop.conf.Configuration getConfiguration(org.apache.hadoop.mapreduce.JobContext context)

Invoke getConfiguration() on JobContext. Works with both Hadoop 1 and 2.

setStatus

public static void setStatus(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                             String status)

Invoke setStatus() on TaskAttemptContext. Works with both Hadoop 1 and 2.

getTaskAttemptID

public static org.apache.hadoop.mapreduce.TaskAttemptID getTaskAttemptID(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)

returns TaskAttemptContext.getTaskAttemptID(). Works with both Hadoop 1 and 2.

getCounter

public static org.apache.hadoop.mapreduce.Counter getCounter(org.apache.hadoop.mapreduce.TaskInputOutputContext context,
                                                             String groupName,
                                                             String counterName)

Invoke getCounter() on TaskInputOutputContext. Works with both Hadoop 1 and 2.

progress

public static void progress(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Invoke TaskAttemptContext.progress(). Works with both Hadoop 1 and 2.

getCounter

public static org.apache.hadoop.mapreduce.Counter getCounter(org.apache.hadoop.mapreduce.TaskInputOutputContext context,
                                                             Enum<?> key)

Hadoop 1 & 2 compatible context.getCounter()

Returns:

TaskInputOutputContext.getCounter(Enum key)

incrementCounter

public static void incrementCounter(org.apache.hadoop.mapreduce.Counter counter,
                                    long increment)

Increment the counter. Works with both Hadoop 1 and 2

getCounterValue

public static long getCounterValue(org.apache.hadoop.mapreduce.Counter counter)

Hadoop 1 & 2 compatible counter.getValue()

Returns:

Counter.getValue()

getJobID

public static org.apache.hadoop.mapreduce.JobID getJobID(org.apache.hadoop.mapreduce.JobContext jobContext)

Hadoop 1 & 2 compatible JobContext.getJobID()

Returns:

JobContext.getJobID()

getJobName

public static String getJobName(org.apache.hadoop.mapreduce.JobContext jobContext)

Hadoop 1 & 2 compatible JobContext.getJobName()

Returns:

JobContext.getJobName()

getInputSplit

public static org.apache.hadoop.mapreduce.InputSplit getInputSplit(org.apache.hadoop.mapreduce.MapContext mapContext)

Hadoop 1 & 2 compatible MapContext.getInputSplit();

Returns:

MapContext.getInputSplit()

getDefaultBlockSize

public static long getDefaultBlockSize(org.apache.hadoop.fs.FileSystem fs,
                                       org.apache.hadoop.fs.Path path)

This method invokes getDefaultBlocksize() without path on Hadoop 1 and with Path on Hadoop 2. Older versions of Hadoop 1 do not have FileSystem#getDefaultBlocksize(Path). This api exists in Hadoop 2 and recent versions of Hadoop 1. Path argument is required for viewfs filesystem in Hadoop 2.

getDefaultReplication

public static short getDefaultReplication(org.apache.hadoop.fs.FileSystem fs,
                                          org.apache.hadoop.fs.Path path)

This method invokes getDefaultReplication() without path on Hadoop 1 and with Path on Hadoop 2. Older versions of Hadoop 1 do not have FileSystem.getDefaultReplication(Path). This api exists in Hadoop 2 and recent versions of Hadoop 1. Path argument is required for viewfs filesystem in Hadoop 2.