kedro.extras.datasets.spark.spark_dataset.KedroHdfsInsecureClient

class documentation

class KedroHdfsInsecureClient(InsecureClient): (source)

View In Hierarchy

Subclasses hdfs.InsecureClient and implements hdfs_exists and hdfs_glob methods required by SparkDataSet

Method hdfs_exists Determines whether given hdfs_path exists in HDFS.
Method hdfs_glob Perform a glob search in HDFS using the provided pattern.
def hdfs_exists(self, hdfs_path: str) -> bool: (source)

Determines whether given hdfs_path exists in HDFS.

Parameters
hdfs_path:strPath to check.
Returns
boolTrue if hdfs_path exists in HDFS, False otherwise.
def hdfs_glob(self, pattern: str) -> List[str]: (source)

Perform a glob search in HDFS using the provided pattern.

Parameters
pattern:strGlob pattern to search for.
Returns
List[str]List of HDFS paths that satisfy the glob pattern.