In 2017, there seems no doubt that if you aren’t running your ML training on a GPU you just aren’t doing things right ?. At home, my only computer is my MacBook Pro which is great to develop on, but would take an extremely long time to train something such as an image classification task. Saying this, I’d love to have a GPU machine at home, but I also love the opportunity to use this hardware in the cloud without having to power it, upgrade it and generally take care of something you actually own. Thus I am lead to investigate using Google’s ML Engine to train my models.
Changes in your TensorFlow code
The first problems I came across were in how I would access files. I had been using a local disk and had been using
open or the python
os module for filesystem interaction. When training in ML engine, these locations will change to be storage buckets, with a path like
gs://my-bucketand as such will need a filesystem object that knows how to interact with these paths. Solution: use the
file_io module in TensorFlow and see below for a few samples to replace. This code can then be utilised when reading from a local
data_dir on disk or from google cloud storage.
from tensorflow.python.lib.io import file_io # Instead of with open(path, 'rb') as p: with file_io.FileIO(os.path.join(data_dir, 'pickle_object.p'), 'rb') as p: # use file pointer # Instead of glob.glob tf_record_filepaths = file_io.get_matching_files(os.path.join(data_dir, '*.tfrecords'))
To have the flexiility to run locally and in Google cloud, any parameters that need to be changed (i.e. data_dir) should be able to be set via command line arguments. This may seem obvious if developing using python from the terminal locally, but prior to this, I had done development in Jupyter notebooks where I would define constants linearly with the flow of the notebook. Defining these is pretty easy in python using the
argparse.ArgumentParser as we will show below, but ML engine will also pass the
--job-dir parameter to your program from the runtime. If you use
parser.parse_args() this will throw an error. You can either define this as a parameter yourself, or use
parser.parse_known_args() to only parse those you define.
from argparse import ArgumentParser if __name__ == '__main__': parser = ArgumentParser() parser.add_argument('--data-dir') parser.add_argument('--job-dir') args = parser.parse_known_args()
You can find the Cloud SDK Quickstarts for you platform. The only thing missing for the macOS guide was locations. For me, I extracted the files to ~/.gcloud and updated my ~/.bash_profile with the lines
source ~/.gcloud/path.bash.inc source ~/.gcloud/completion.bash.inc
Finally Running Something
My starter task was preprocessing images with labels to TFRecords: basically a task I new wouldn’t take too long and could validate everything was working. I’ll be using the CLI and you can following the ML Engine Quickstart to get the basics of that setup. This should create a project, enable billing, enable the ML Engine APIs and you should have a storage bucket.
Project Directory Structure
The trainer requires a python module based folder structure and a setup.py script for setup. We will be running a ‘preprocess’ module and the directory structure should be
Project __init__.py setup.py mlengine preproc-config.yaml trainer __init__.py preprocess.py
The __init__.py files can be empty, but firstly the setup.py file to define a few pip dependencies we want:
from setuptools import find_packages, setup REQUIRED_PACKAGES = [ 'tensorflow==1.4.0', 'h5py', 'numpy', 'image' ] setup( name='plant_classifer', version='0.1', install_requires=REQUIRED_PACKAGES, packages=find_packages(), include_package_data=True, description='Plant classifier application' )
Secondly, the mlengine/preproc-config.yaml file for a few Job resource configuration. This is very minimal to just define that we only need a BASIC machine, but can be used to set much more complex run properties in ML Engine.
trainingInput: scaleTier: BASIC
The preprocess.py file is our python program and can be viewed on GitHub with the rest of the source, but next to how we would start off this in ML Engine. Using the CLI and having our currend directory in the root of the project in terminal, run:
gcloud ml-engine jobs submit training plants_preproc \ --package-path trainer \ --module-name trainer.preprocess \ --job-dir gs://my-bucket \ --region us-central1 \ --config mlengine/preproc-config.yaml \ -- \ --data-dir gs://my-bucket/data
Here we are Submitting the job and defining that the trainer.preprocess module should be run. Arguments after the isolated – will be passed onto your program. This should queue the job and the run logs and job can be viewed on the cloud project.
The code used can be found at https://github.com/damienpontifex/plant-classification-ml-engine