Training at scale

The following tips apply to large datasets and/or large models.

Single vs. distributed training

We do not support distributed training for scikit-learn and XGBoost (scale-tier STANDARD_1 and PREMIUM_1). Please only use the scale-tier values that correspond to a single worker instance.

Large datasets

When dealing with large datasets, it's possible that downloading the entire dataset into the training worker VM and loading it into pandas does not scale. In these cases, consider using TensorFlow's stream-read/file_io API (this API is preinstalled on the VM).

import pandas as pd

from pandas.compat import StringIO
from tensorflow.python.lib.io import file_io

# Access iris data from Cloud Storage
iris_data_filesteam = file_io.FileIO(os.path.join(data_dir, iris_data_filename),
                                     mode='r')
iris_data = pd.read_csv(StringIO(iris_data_filesteam.read())).values
iris_target_filesteam = file_io.FileIO(os.path.join(data_dir,
                                                    iris_target_filename),
                                       mode='r')
iris_target = pd.read_csv(StringIO(iris_target_filesteam.read())).values
iris_target = iris_target.reshape((iris_target.size,))


# Your training program goes here
...
..
.

# Close all filestreams
iris_data_filesteam.close()
iris_target_filesteam.close()

Large models

Training worker VMs with higher memory needs can be requested by setting scale-tier to CUSTOM and setting the masterType via an accompanying config file. For more details, refer to the scale tier documentation.

To do this:

  1. Create config.yaml locally with the following contents:

    trainingInput:
      masterType: large_model
    
  2. Submit your job:

    CONFIG=path/to/config.yaml
    
    gcloud ai-platform jobs submit training $JOB_NAME \
      --job-dir $JOB_DIR \
      --package-path $TRAINER_PACKAGE_PATH \
      --module-name $MAIN_TRAINER_MODULE \
      --region us-central1 \
      --runtime-version=$RUNTIME_VERSION \
      --python-version=$PYTHON_VERSION \
      --scale-tier CUSTOM \
      --config $CONFIG
    

Note that only BASIC and CUSTOM are supported for scale-tier. In the configuration file, only masterType is supported (parameterType and workerType are not supported).

Esta página foi útil? Conte sua opinião sobre:

Enviar comentários sobre…

Precisa de ajuda? Acesse nossa página de suporte.