Tech Blog

Tensorflow 2.0

Overview

Google released the TensorFlow 2.0 alpha version in March 2019. The official 2.0 release is expected in Q2 2019. TensorFlow 2.0 promises simplicity and ease of execution while maintaining TensorFlow’s flexibility and scalability. This post outlines the key differences between TensorFlow 1.X and 2.0, and explains how to upgrade to TensorFlow 2.0.

Key changes

Eager execution

TensorFlow 2.0 uses eager execution by default. The result is more intuitive, object oriented and pythonic.

Nothing like a simple example to conceptualize the difference:

import tensorflow as tf
sum = tf.add(1, 2)

If we print the sum result in TensorFlow 1.X, we get the operation definition in the graph and not the actual result. It is only when the operation is executed inside a session that we get the expected result:

# Using TensorFlow 1.X
print(sum)
# Tensor("Add:0", shape=(), dtype=int32)
 with tf.Session() as sess:
    print(sess.run(sum))
    # 3

If we repeat the operation in TensorFlow 2.0, the result is as follows

# Using TensorFlow 2.0
print(sum)
# tf.Tensor(3, shape=(), dtype=int32)

Eager execution is intuitive and corresponds to the expected behavior when using Python. It also has some practical benefits. For example, it simplifies debugging as the code does not wait for the session to be executed and, potentially, fail. This allows using print() to quickly inspect results for debugging.

Applications that require graphs can still use them with TensorFlow 2.0, but graphs are no longer the default behavior.

Keras

TensorFlow 2.0 consolidates high-level APIs under the Keras header. A great decision, in our opinion. Keras’ focus on ease to learn and use are ideal for a high-level API.

Keras’ capabilities have also been greatly expanded. All advanced features from TensorFlow can now be accessed through Keras API. Google has explained that the design objective was to minimize the effort to move from experimenting with Keras to production.

An example of the new Keras capabilities is tf.distribute that allows distributing model training through multiple GPUs. Google claims that tf.distribute and Keras achieve 90% scaling efficiency across multiple GPUs. In addition to being simple, Keras API is now scalable.

As part of its simplification process, TensorFlow 2.0 has removed duplicated functionality throughout its API. In the case of high-level API, layers (tf.layers) will now only be accessible through Keras (tf.keras.layers). Losses and metrics that used to be duplicated between Keras and TensorFlow are now grouped into single sets.

Remove global namespaces

TensorFlow 1.X relied on global namespaces and, indirectly, the assumption that there was a single model per graph. Again, this was not the expected Python behavior. After defining the same variable twice

# Using TensorFlow 1.X
x = tf.Variable(1, name='x1')
x = tf.Variable(2, name='x2')

the first definition with namespace “x1” was not garbage collected, even if we lost track of the Python variable pointing to it. The tf.Variable “x1” could be recovered, but only if the name that it had been created with was known. This behavior was the root of many mistakes among new TensorFlow users. For example, issues in TensorFlow 1.X can arise through name conflicts when using the same graph for multiple models, or a graph accumulating multiple unused definitions of the same variable.

In addition to being error prone, the TensorFlow 1.X design led to multiple mechanisms and functions to facilitate finding untracked variables (e.g. variable scopes, global collections and tf.get_global_step()). TensorFlow 2.0 removes all of these functionalities in favor of the default Python behavior. A tf.Variable that is not being tracked is now garbage collected.

Upgrade to TensorFlow 2.0

If you would like to give the new TensorFlow a go, the TensorFlow 2.0 alpha version can be installed through

pip install -U --pre tensorflow

As with TensorFlow 1.X, the best approach is to install TensorFlow in a virtual environment (e.g. virtualenv).

It is always tricky to perform major version upgrades. To facilitate the transition, the TensorFlow 2.0 default installation includes the command tf_upgrade_v2 that automatically updates a script from TensorFlow 1.X to TensorFlow 2.0. We tested the upgrade command in some of our code and it worked as expected. The command is fairly “conservative”. If a function does not have an exact equivalent in TensorFlow 2.0, it avoids any potential incompatibility by relying on the old function definition through the library tf.compat.v1.

After the automatic transformation, additional work is required to change the code style into TensorFlow 2.0. We recommend checking out the section Recommendations for idiomatic TensorFlow 2.0 for more details.

What next?

If you have any questions about TensorFlow or machine learning, please contact us.