com.twitter.elephantbird.cascading2.scheme
Class LzoBinaryScheme<M,T extends BinaryWritable<M>>

java.lang.Object
  cascading.scheme.Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,Object[],T>
      com.twitter.elephantbird.cascading2.scheme.LzoBinaryScheme<M,T>

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

LzoByteArrayScheme, LzoProtobufScheme, LzoThriftScheme

public abstract class LzoBinaryScheme<M,T extends BinaryWritable<M>>
extends cascading.scheme.Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,Object[],T>

Scheme for lzo binary encoded files. Handles both block and base 64. Can be used for Protobuf and Thrift.

Author:

Argyris Zymnis

See Also:

Serialized Form

Constructor Summary

LzoBinaryScheme()

Method Summary

protected abstract T	prepareBinaryWritable()
void	sink(cascading.flow.FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess, cascading.scheme.SinkCall<T,org.apache.hadoop.mapred.OutputCollector> sinkCall)
void	sinkPrepare(cascading.flow.FlowProcess<org.apache.hadoop.mapred.JobConf> fp, cascading.scheme.SinkCall<T,org.apache.hadoop.mapred.OutputCollector> sinkCall)
boolean	source(cascading.flow.FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess, cascading.scheme.SourceCall<Object[],org.apache.hadoop.mapred.RecordReader> sourceCall)
void	sourceCleanup(cascading.flow.FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess, cascading.scheme.SourceCall<Object[],org.apache.hadoop.mapred.RecordReader> sourceCall)
void	sourcePrepare(cascading.flow.FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess, cascading.scheme.SourceCall<Object[],org.apache.hadoop.mapred.RecordReader> sourceCall) This sets up the state between succesive calls to source

Methods inherited from class cascading.scheme.Scheme

equals, getNumSinkParts, getSinkFields, getSourceFields, getTrace, hashCode, isSink, isSource, isSymmetrical, presentSinkFields, presentSinkFieldsInternal, presentSourceFields, presentSourceFieldsInternal, retrieveSinkFields, retrieveSourceFields, setNumSinkParts, setSinkFields, setSourceFields, sinkCleanup, sinkConfInit, sourceConfInit, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

LzoBinaryScheme

public LzoBinaryScheme()

Method Detail

sink

public void sink(cascading.flow.FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess,
                 cascading.scheme.SinkCall<T,org.apache.hadoop.mapred.OutputCollector> sinkCall)
          throws IOException

Specified by:

sink in class cascading.scheme.Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,Object[],T extends BinaryWritable<M>>

Throws:

IOException

sinkPrepare

public void sinkPrepare(cascading.flow.FlowProcess<org.apache.hadoop.mapred.JobConf> fp,
                        cascading.scheme.SinkCall<T,org.apache.hadoop.mapred.OutputCollector> sinkCall)

Overrides:

sinkPrepare in class cascading.scheme.Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,Object[],T extends BinaryWritable<M>>

prepareBinaryWritable

protected abstract T prepareBinaryWritable()

source

public boolean source(cascading.flow.FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess,
                      cascading.scheme.SourceCall<Object[],org.apache.hadoop.mapred.RecordReader> sourceCall)
               throws IOException

Specified by:

source in class cascading.scheme.Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,Object[],T extends BinaryWritable<M>>

Throws:

IOException

sourceCleanup

public void sourceCleanup(cascading.flow.FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess,
                          cascading.scheme.SourceCall<Object[],org.apache.hadoop.mapred.RecordReader> sourceCall)

Overrides:

sourceCleanup in class cascading.scheme.Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,Object[],T extends BinaryWritable<M>>

sourcePrepare

public void sourcePrepare(cascading.flow.FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess,
                          cascading.scheme.SourceCall<Object[],org.apache.hadoop.mapred.RecordReader> sourceCall)

This sets up the state between succesive calls to source

Overrides:

sourcePrepare in class cascading.scheme.Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,Object[],T extends BinaryWritable<M>>