Skip to content
TensorFlow serving can handle a variable batch size when doing predictions. I never understood how to configure this and also the shape of the results returned. Finally figuring this out, here’s the changes to our previous serving setup to accept a variable number of images to classify for our model.
Serving input function First thing is to update our serving input receiver function placeholder. In the past we had set the placeholder to have a shape of , for variable batch size, this is as easy as setting it to [None].
Azure provides low priority VMs that we can use to save some money when utilising GPUs for machine learning. All the major cloud providers offer spare compute for a discount: Google cloud has Preemptible VMs, AWS has spot instances and Azure has low priority VMs. On Azure these low priority VMs can only be provisioned in Azure Batch or virtual machine scale sets and we can use the latter for hosting a jupyter environment.
Here we’ll look at exporting our previously trained dog and cat classifier and call that with local or remote files to test it out. To do this, I’ll use TensorFlow Serving in a docker container and use a python client to call to the remote host.
_Update 12th June, 2018: I used the gRPC interface here, but TensorFlow serving now has a REST API that could be beneficial or of more interest_
This notebook is available as a codelab
TensorFlow Hub was announced at TensorFlow Dev Summit 2018 and promises to reduce the effort required to use existing machine learning models and weights in your own custom model. From the overview page
TensorFlow Hub is a library to foster the publication, discovery, and consumption of reusable parts of machine learning models. A module is a self-contained piece of a TensorFlow graph, along with its weights and assets, that can be reused across different tasks in a process known as transfer learning.
Azure Batch AI provides us the PaaS opportunity to use GPU resources in the cloud. The basis is to use virtual machines in a managed cluster (i.e. you don’t have to maintain them) and run jobs as you see fit. For my use case, the opportunity of low-priority VMs to reduce the cost of using GPU machines is also particularly promising. What I’ll run through is running our first job on Azure Batch AI.
PowerShell 6.0 has now become generally available and with it the ability to use PowerShell cross platform. With all of its goodness, I was most excited about Azure powershell cross platform capabilities that this brings about. Firstly, the azure-cli is also a great cross platform tool, but sometimes the powershell capabilities seem to outshine this CLI.
Installing PowerShell on macOS The installation instructions on the repo are pretty straight forward, but for summary:
The TensorFlow canned estimators got promoted to core in version 1.3 to make training and evaluation of machine learning models very easy. This API allows you to describe your input data (categorical, numeric, embedding etc) through the use of feature columns. The estimator API also allows you to write a custom model for your unique job, and the feature columns capabilities can be utilised here as well to simplify or enhance things.
Understanding the shape of your model is sometimes non-trivial when it comes to machine learning. Look at convolutional neural nets with the number of filters, padding, kernel sizes etc and it’s quickly evident why understanding what shapes your inputs and outputs are will keep you sane and reduce the time spent digging into strange errors. TensorFlow’s RNN API exposed me to similar frustrations and misunderstandings about what I was expected to give it and what I was getting in return.
TFrecord files are TensorFlow’s suggested data format, although they are very difficult to inspect given their binary nature. Inspecting the contents of existing record files and ensuring the data in your input pipeline is as you expect is a good technique to have.
Inspecting TFRecord values The first trick is reading in the tfrecord files and inspecting their values in python. As you’d expect, the TensorFlow API allows this (although a little hidden down).
I’m a big advocate of the cloud and it’s ability to provide just enough resources ad hoc. You can use whatever you want, and pay for it just when using it. In machine learning there are services such as Google’s ML Engine or Azure’s upcoming Batch AI but during development, data preprocessing etc sometimes you want immediate iterative processes. In these cases, you can’t go past a Jupyter notebook and in this case, running that on a VM.