org.apache.beam.sdk.util

Class GcsUtil

java.lang.Object

org.apache.beam.sdk.util.GcsUtil

public class GcsUtil
extends Object

Provides operations on GCS.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	GcsUtil.GcsUtilFactory This is a DefaultValueFactory able to create a GcsUtil using any transport flags specified on the PipelineOptions.
static class	GcsUtil.StorageObjectOrIOException A class that holds either a StorageObject or an IOException.

Method Summary

Modifier and Type	Method and Description
boolean	bucketAccessible(GcsPath path) Returns whether the GCS bucket exists and is accessible.
long	bucketOwner(GcsPath path) Returns the project number of the project which owns this bucket.
void	copy(Iterable<String> srcFilenames, Iterable<String> destFilenames)
WritableByteChannel	create(GcsPath path, String type) Creates an object in GCS.
void	createBucket(String projectId, com.google.api.services.storage.model.Bucket bucket) Creates a Bucket under the specified project in Cloud Storage or propagates an exception.
List<GcsPath>	expand(GcsPath gcsPattern) Expands a pattern into matched paths.
long	fileSize(GcsPath path) Returns the file size from GCS or throws FileNotFoundException if the resource does not exist.
static String	getGlobPrefix(String globExp) Returns the prefix portion of the glob that doesn't contain wildcards.
com.google.api.services.storage.model.StorageObject	getObject(GcsPath gcsPath) Returns the StorageObject for the given GcsPath.
List<GcsUtil.StorageObjectOrIOException>	getObjects(List<GcsPath> gcsPaths) Returns StorageObjectOrIOExceptions for the given GcsPaths.
static String	globToRegexp(String globExp) Expands glob expressions to regular expressions.
static boolean	isGcsPatternSupported(String gcsPattern) Returns true if the given GCS pattern is supported otherwise fails with an exception.
static boolean	isGlob(GcsPath spec) Returns true if the given spec contains glob.
com.google.api.services.storage.model.Objects	listObjects(String bucket, String prefix, String pageToken) Lists Objects given the bucket, prefix, pageToken.
SeekableByteChannel	open(GcsPath path) Opens an object in GCS.
void	remove(Collection<String> filenames)
protected void	setStorageClient(com.google.api.services.storage.Storage storageClient)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

isGcsPatternSupported

public static boolean isGcsPatternSupported(String gcsPattern)

Returns true if the given GCS pattern is supported otherwise fails with an exception.

getGlobPrefix

public static String getGlobPrefix(String globExp)

Returns the prefix portion of the glob that doesn't contain wildcards.

globToRegexp

public static String globToRegexp(String globExp)

Expands glob expressions to regular expressions.

Parameters:

globExp - the glob expression to expand

Returns:

a string with the regular expression this glob expands to

isGlob

public static boolean isGlob(GcsPath spec)

Returns true if the given spec contains glob.

setStorageClient

protected void setStorageClient(com.google.api.services.storage.Storage storageClient)

expand

public List<GcsPath> expand(GcsPath gcsPattern)
                     throws IOException

Expands a pattern into matched paths. The pattern path may contain globs, which are expanded in the result. For patterns that only match a single object, we ensure that the object exists.

Throws:

IOException

fileSize

public long fileSize(GcsPath path)
              throws IOException

Returns the file size from GCS or throws FileNotFoundException if the resource does not exist.

Throws:

IOException

getObject

public com.google.api.services.storage.model.StorageObject getObject(GcsPath gcsPath)
                                                              throws IOException

Returns the StorageObject for the given GcsPath.

Throws:

IOException

getObjects

public List<GcsUtil.StorageObjectOrIOException> getObjects(List<GcsPath> gcsPaths)
                                                    throws IOException

Returns StorageObjectOrIOExceptions for the given GcsPaths.

Throws:

IOException

listObjects

public com.google.api.services.storage.model.Objects listObjects(String bucket,
                                                                 String prefix,
                                                                 @Nullable
                                                                 String pageToken)
                                                          throws IOException

Lists Objects given the bucket, prefix, pageToken.

Throws:

IOException

open

public SeekableByteChannel open(GcsPath path)
                         throws IOException

Opens an object in GCS.

Returns a SeekableByteChannel that provides access to data in the bucket.

Parameters:

path - the GCS filename to read from

Returns:

a SeekableByteChannel that can read the object data

Throws:

IOException

create

public WritableByteChannel create(GcsPath path,
                                  String type)
                           throws IOException

Creates an object in GCS.

Returns a WritableByteChannel that can be used to write data to the object.

Parameters:

path - the GCS file to write to

type - the type of object, eg "text/plain".

Returns:

a Callable object that encloses the operation.

Throws:

IOException

bucketAccessible

public boolean bucketAccessible(GcsPath path)
                         throws IOException

Returns whether the GCS bucket exists and is accessible.

Throws:

IOException

bucketOwner

public long bucketOwner(GcsPath path)
                 throws IOException

Returns the project number of the project which owns this bucket. If the bucket exists, it must be accessible otherwise the permissions exception will be propagated. If the bucket does not exist, an exception will be thrown.

Throws:

IOException

createBucket

public void createBucket(String projectId,
                         com.google.api.services.storage.model.Bucket bucket)
                  throws IOException

Creates a Bucket under the specified project in Cloud Storage or propagates an exception.

Throws:

IOException

copy

public void copy(Iterable<String> srcFilenames,
                 Iterable<String> destFilenames)
          throws IOException

Throws:

IOException

remove

public void remove(Collection<String> filenames)
            throws IOException

Throws:

IOException