org.apache.hadoop.hive.shims
Class Hadoop20Shims

java.lang.Object
  org.apache.hadoop.hive.shims.Hadoop20Shims

All Implemented Interfaces:

HadoopShims

public class Hadoop20Shims
extends Object
implements HadoopShims

Implemention of shims against Hadoop 0.20.0.

Nested Class Summary

static class	Hadoop20Shims.CombineFileInputFormatShim<K,V>
static class	Hadoop20Shims.CombineFileRecordReader<K,V>
static class	Hadoop20Shims.InputSplitShim
class	Hadoop20Shims.MiniDFSShim MiniDFSShim.
class	Hadoop20Shims.MiniMrShim Shim for MiniMrCluster
static class	Hadoop20Shims.NullOutputCommitter

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.shims.HadoopShims

HadoopShims.ByteBufferPoolShim, HadoopShims.DirectCompressionType, HadoopShims.DirectDecompressorShim, HadoopShims.HCatHadoopShims, HadoopShims.JobTrackerState, HadoopShims.WebHCatJTShim, HadoopShims.ZeroCopyReaderShim

Field Summary

Fields inherited from interface org.apache.hadoop.hive.shims.HadoopShims

LOG

Constructor Summary

Hadoop20Shims()

Method Summary

String	addServiceToToken(String tokenStr, String tokenService)
void	authorizeProxyAccess(String proxyUser, org.apache.hadoop.security.UserGroupInformation realUserUgi, String ipAddress, org.apache.hadoop.conf.Configuration conf)
void	closeAllForUGI(org.apache.hadoop.security.UserGroupInformation ugi)
Path	createDelegationTokenFile(org.apache.hadoop.conf.Configuration conf)
int	createHadoopArchive(org.apache.hadoop.conf.Configuration conf, Path sourceDir, Path destDir, String archiveName)
FileSystem	createProxyFileSystem(FileSystem fs, URI uri)
org.apache.hadoop.security.UserGroupInformation	createProxyUser(String userName)
org.apache.hadoop.security.UserGroupInformation	createRemoteUser(String userName, List<String> groupNames)
<T> T	doAs(org.apache.hadoop.security.UserGroupInformation ugi, PrivilegedExceptionAction<T> pvea)
HadoopShims.CombineFileInputFormatShim	getCombineFileInputFormat()
org.apache.hadoop.conf.Configuration	getConfiguration(org.apache.hadoop.mapreduce.JobContext context)
long	getDefaultBlockSize(FileSystem fs, Path path)
short	getDefaultReplication(FileSystem fs, Path path)
HadoopShims.DirectDecompressorShim	getDirectDecompressor(HadoopShims.DirectCompressionType codec)
Map<String,String>	getHadoopConfNames()
URI	getHarUri(URI original, URI base, URI originalBase)
HadoopShims.HCatHadoopShims	getHCatShim()
String	getInputFormatClassName()
String	getJobLauncherHttpAddress(org.apache.hadoop.conf.Configuration conf)
String	getJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf)
HadoopShims.JobTrackerState	getJobTrackerState(org.apache.hadoop.mapred.ClusterStatus clusterStatus)
BlockLocation[]	getLocations(FileSystem fs, FileStatus status)
Comparator<org.apache.hadoop.io.LongWritable>	getLongComparator()
HadoopShims.MiniDFSShim	getMiniDfs(org.apache.hadoop.conf.Configuration conf, int numDataNodes, boolean format, String[] racks)
Hadoop20Shims.MiniMrShim	getMiniMrCluster(org.apache.hadoop.conf.Configuration conf, int numberOfTaskTrackers, String nameNode, int numDir) Returns a shim to wrap MiniMrCluster
FileSystem	getNonCachedFileSystem(URI uri, org.apache.hadoop.conf.Configuration conf)
String	getShortUserName(org.apache.hadoop.security.UserGroupInformation ugi)
String	getTaskAttemptLogUrl(org.apache.hadoop.mapred.JobConf conf, String taskTrackerHttpAddress, String taskAttemptId)
String	getTokenFileLocEnvName()
String	getTokenStrForm(String tokenSignature)
org.apache.hadoop.security.UserGroupInformation	getUGIForConf(org.apache.hadoop.conf.Configuration conf)
HadoopShims.WebHCatJTShim	getWebHCatShim(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.security.UserGroupInformation ugi)
HadoopShims.ZeroCopyReaderShim	getZeroCopyReader(FSDataInputStream in, HadoopShims.ByteBufferPoolShim pool)
void	hflush(FSDataOutputStream stream)
boolean	isLocalMode(org.apache.hadoop.conf.Configuration conf)
boolean	isLoginKeytabBased()
boolean	isSecureShimImpl()
boolean	isSecurityEnabled()
List<FileStatus>	listLocatedStatus(FileSystem fs, Path path, PathFilter filter)
void	loginUserFromKeytab(String principal, String keytabFile)
org.apache.hadoop.security.UserGroupInformation	loginUserFromKeytabAndReturnUGI(String principal, String keytabFile)
boolean	moveToAppropriateTrash(FileSystem fs, Path path, org.apache.hadoop.conf.Configuration conf)
org.apache.hadoop.mapreduce.JobContext	newJobContext(org.apache.hadoop.mapreduce.Job job)
org.apache.hadoop.mapreduce.TaskAttemptContext	newTaskAttemptContext(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.util.Progressable progressable)
org.apache.hadoop.mapreduce.TaskAttemptID	newTaskAttemptID(org.apache.hadoop.mapreduce.JobID jobId, boolean isMap, int taskId, int id)
void	prepareJobOutput(org.apache.hadoop.mapred.JobConf conf)
void	reLoginUserFromKeytab()
void	setJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf, String val)
void	setTokenStr(org.apache.hadoop.security.UserGroupInformation ugi, String tokenStr, String tokenService)
void	setTotalOrderPartitionFile(org.apache.hadoop.mapred.JobConf jobConf, Path partitionFile)
String	unquoteHtmlChars(String item)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Hadoop20Shims

public Hadoop20Shims()

Method Detail

getMiniMrCluster

public Hadoop20Shims.MiniMrShim getMiniMrCluster(org.apache.hadoop.conf.Configuration conf,
                                                 int numberOfTaskTrackers,
                                                 String nameNode,
                                                 int numDir)
                                          throws IOException

Returns a shim to wrap MiniMrCluster

Specified by:

getMiniMrCluster in interface HadoopShims

Throws:

IOException

getMiniDfs

public HadoopShims.MiniDFSShim getMiniDfs(org.apache.hadoop.conf.Configuration conf,
                                          int numDataNodes,
                                          boolean format,
                                          String[] racks)
                                   throws IOException

Specified by:

getMiniDfs in interface HadoopShims

Throws:

IOException

getCombineFileInputFormat

public HadoopShims.CombineFileInputFormatShim getCombineFileInputFormat()

Specified by:

getCombineFileInputFormat in interface HadoopShims

setTotalOrderPartitionFile

public void setTotalOrderPartitionFile(org.apache.hadoop.mapred.JobConf jobConf,
                                       Path partitionFile)

Specified by:

setTotalOrderPartitionFile in interface HadoopShims

getLongComparator

public Comparator<org.apache.hadoop.io.LongWritable> getLongComparator()

Specified by:

getLongComparator in interface HadoopShims

getInputFormatClassName

public String getInputFormatClassName()

Specified by:

getInputFormatClassName in interface HadoopShims

createHadoopArchive

public int createHadoopArchive(org.apache.hadoop.conf.Configuration conf,
                               Path sourceDir,
                               Path destDir,
                               String archiveName)
                        throws Exception

Specified by:

createHadoopArchive in interface HadoopShims

Throws:

Exception

getHarUri

public URI getHarUri(URI original,
                     URI base,
                     URI originalBase)
              throws URISyntaxException

Specified by:

getHarUri in interface HadoopShims

Throws:

URISyntaxException

prepareJobOutput

public void prepareJobOutput(org.apache.hadoop.mapred.JobConf conf)

Specified by:

prepareJobOutput in interface HadoopShims

getUGIForConf

public org.apache.hadoop.security.UserGroupInformation getUGIForConf(org.apache.hadoop.conf.Configuration conf)
                                                              throws LoginException

Specified by:

getUGIForConf in interface HadoopShims

Throws:

LoginException

isSecureShimImpl

public boolean isSecureShimImpl()

Specified by:

isSecureShimImpl in interface HadoopShims

getShortUserName

public String getShortUserName(org.apache.hadoop.security.UserGroupInformation ugi)

Specified by:

getShortUserName in interface HadoopShims

getTokenStrForm

public String getTokenStrForm(String tokenSignature)
                       throws IOException

Specified by:

getTokenStrForm in interface HadoopShims

Throws:

IOException

setTokenStr

public void setTokenStr(org.apache.hadoop.security.UserGroupInformation ugi,
                        String tokenStr,
                        String tokenService)
                 throws IOException

Specified by:

setTokenStr in interface HadoopShims

Throws:

IOException

addServiceToToken

public String addServiceToToken(String tokenStr,
                                String tokenService)
                         throws IOException

Specified by:

addServiceToToken in interface HadoopShims

Throws:

IOException

doAs

public <T> T doAs(org.apache.hadoop.security.UserGroupInformation ugi,
                  PrivilegedExceptionAction<T> pvea)
       throws IOException,
              InterruptedException

Specified by:

doAs in interface HadoopShims

Throws:

IOException

InterruptedException

createDelegationTokenFile

public Path createDelegationTokenFile(org.apache.hadoop.conf.Configuration conf)
                               throws IOException

Specified by:

createDelegationTokenFile in interface HadoopShims

Throws:

IOException

createRemoteUser

public org.apache.hadoop.security.UserGroupInformation createRemoteUser(String userName,
                                                                        List<String> groupNames)

Specified by:

createRemoteUser in interface HadoopShims

loginUserFromKeytab

public void loginUserFromKeytab(String principal,
                                String keytabFile)
                         throws IOException

Specified by:

loginUserFromKeytab in interface HadoopShims

Throws:

IOException

loginUserFromKeytabAndReturnUGI

public org.apache.hadoop.security.UserGroupInformation loginUserFromKeytabAndReturnUGI(String principal,
                                                                                       String keytabFile)
                                                                                throws IOException

Specified by:

loginUserFromKeytabAndReturnUGI in interface HadoopShims

Throws:

IOException

reLoginUserFromKeytab

public void reLoginUserFromKeytab()
                           throws IOException

Specified by:

reLoginUserFromKeytab in interface HadoopShims

Throws:

IOException

isLoginKeytabBased

public boolean isLoginKeytabBased()
                           throws IOException

Specified by:

isLoginKeytabBased in interface HadoopShims

Throws:

IOException

createProxyUser

public org.apache.hadoop.security.UserGroupInformation createProxyUser(String userName)
                                                                throws IOException

Specified by:

createProxyUser in interface HadoopShims

Throws:

IOException

listLocatedStatus

public List<FileStatus> listLocatedStatus(FileSystem fs,
                                          Path path,
                                          PathFilter filter)
                                   throws IOException

Specified by:

listLocatedStatus in interface HadoopShims

Throws:

IOException

getLocations

public BlockLocation[] getLocations(FileSystem fs,
                                    FileStatus status)
                             throws IOException

Specified by:

getLocations in interface HadoopShims

Throws:

IOException

hflush

public void hflush(FSDataOutputStream stream)
            throws IOException

Specified by:

hflush in interface HadoopShims

Throws:

IOException

authorizeProxyAccess

public void authorizeProxyAccess(String proxyUser,
                                 org.apache.hadoop.security.UserGroupInformation realUserUgi,
                                 String ipAddress,
                                 org.apache.hadoop.conf.Configuration conf)
                          throws IOException

Specified by:

authorizeProxyAccess in interface HadoopShims

Throws:

IOException

isSecurityEnabled

public boolean isSecurityEnabled()

Specified by:

isSecurityEnabled in interface HadoopShims

getTaskAttemptLogUrl

public String getTaskAttemptLogUrl(org.apache.hadoop.mapred.JobConf conf,
                                   String taskTrackerHttpAddress,
                                   String taskAttemptId)
                            throws MalformedURLException

Specified by:

getTaskAttemptLogUrl in interface HadoopShims

Throws:

MalformedURLException

getJobTrackerState

public HadoopShims.JobTrackerState getJobTrackerState(org.apache.hadoop.mapred.ClusterStatus clusterStatus)
                                               throws Exception

Specified by:

getJobTrackerState in interface HadoopShims

Throws:

Exception

unquoteHtmlChars

public String unquoteHtmlChars(String item)

Specified by:

unquoteHtmlChars in interface HadoopShims

newTaskAttemptContext

public org.apache.hadoop.mapreduce.TaskAttemptContext newTaskAttemptContext(org.apache.hadoop.conf.Configuration conf,
                                                                            org.apache.hadoop.util.Progressable progressable)

Specified by:

newTaskAttemptContext in interface HadoopShims

newTaskAttemptID

public org.apache.hadoop.mapreduce.TaskAttemptID newTaskAttemptID(org.apache.hadoop.mapreduce.JobID jobId,
                                                                  boolean isMap,
                                                                  int taskId,
                                                                  int id)

Specified by:

newTaskAttemptID in interface HadoopShims

newJobContext

public org.apache.hadoop.mapreduce.JobContext newJobContext(org.apache.hadoop.mapreduce.Job job)

Specified by:

newJobContext in interface HadoopShims

closeAllForUGI

public void closeAllForUGI(org.apache.hadoop.security.UserGroupInformation ugi)

Specified by:

closeAllForUGI in interface HadoopShims

isLocalMode

public boolean isLocalMode(org.apache.hadoop.conf.Configuration conf)

Specified by:

isLocalMode in interface HadoopShims

getJobLauncherRpcAddress

public String getJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf)

Specified by:

getJobLauncherRpcAddress in interface HadoopShims

setJobLauncherRpcAddress

public void setJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf,
                                     String val)

Specified by:

setJobLauncherRpcAddress in interface HadoopShims

getJobLauncherHttpAddress

public String getJobLauncherHttpAddress(org.apache.hadoop.conf.Configuration conf)

Specified by:

getJobLauncherHttpAddress in interface HadoopShims

moveToAppropriateTrash

public boolean moveToAppropriateTrash(FileSystem fs,
                                      Path path,
                                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException

Specified by:

moveToAppropriateTrash in interface HadoopShims

Throws:

IOException

getDefaultBlockSize

public long getDefaultBlockSize(FileSystem fs,
                                Path path)

Specified by:

getDefaultBlockSize in interface HadoopShims

getDefaultReplication

public short getDefaultReplication(FileSystem fs,
                                   Path path)

Specified by:

getDefaultReplication in interface HadoopShims

getTokenFileLocEnvName

public String getTokenFileLocEnvName()

Specified by:

getTokenFileLocEnvName in interface HadoopShims

getHCatShim

public HadoopShims.HCatHadoopShims getHCatShim()

Specified by:

getHCatShim in interface HadoopShims

getWebHCatShim

public HadoopShims.WebHCatJTShim getWebHCatShim(org.apache.hadoop.conf.Configuration conf,
                                                org.apache.hadoop.security.UserGroupInformation ugi)
                                         throws IOException

Specified by:

getWebHCatShim in interface HadoopShims

Throws:

IOException

createProxyFileSystem

public FileSystem createProxyFileSystem(FileSystem fs,
                                        URI uri)

Specified by:

createProxyFileSystem in interface HadoopShims

getHadoopConfNames

public Map<String,String> getHadoopConfNames()

Specified by:

getHadoopConfNames in interface HadoopShims

getZeroCopyReader

public HadoopShims.ZeroCopyReaderShim getZeroCopyReader(FSDataInputStream in,
                                                        HadoopShims.ByteBufferPoolShim pool)
                                                 throws IOException

Specified by:

getZeroCopyReader in interface HadoopShims

Throws:

IOException

getDirectDecompressor

public HadoopShims.DirectDecompressorShim getDirectDecompressor(HadoopShims.DirectCompressionType codec)

Specified by:

getDirectDecompressor in interface HadoopShims

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration(org.apache.hadoop.mapreduce.JobContext context)

Specified by:

getConfiguration in interface HadoopShims

getNonCachedFileSystem

public FileSystem getNonCachedFileSystem(URI uri,
                                         org.apache.hadoop.conf.Configuration conf)
                                  throws IOException

Specified by:

getNonCachedFileSystem in interface HadoopShims

Throws:

IOException