tf.raw_ops.ExperimentalAutoShardDataset

Creates a dataset that shards the input dataset.

Creates a dataset that shards the input dataset by num_workers, returning a sharded dataset for the index-th worker. This attempts to automatically shard a dataset by examining the Dataset graph and inserting a shard op before the inputs to a reader Dataset (e.g. CSVDataset, TFRecordDataset).

This dataset will throw a NotFound error if we cannot shard the dataset automatically.

Args
input_dataset A Tensor of type variant. A variant tensor representing the input dataset.
num_workers A Tensor of type int64. A scalar representing the number of workers to distribute this dataset across.
index A Tensor of type int64. A scalar representing the index of the current worker out of num_workers.
output_types A list of tf.DTypes that has length >= 1.
output_shapes A list of shapes (each a tf.TensorShape or list of ints) that has length >= 1.
auto_shard_policy An optional int. Defaults to 0.
name A name for the operation (optional).
Returns
A Tensor of type variant.

© 2020 The TensorFlow Authors. All rights reserved.
Licensed under the Creative Commons Attribution License 3.0.
Code samples licensed under the Apache 2.0 License.
https://www.tensorflow.org/versions/r2.4/api_docs/python/tf/raw_ops/ExperimentalAutoShardDataset