基于cs20si的Tensorflow笔记，课程主页；

本节主要内容：Basic Operations, Constants, Variables, Control Dependencies, Feeding inputs, TensorBoard

## TensorBoard

The computations you’ll use TensorFlow for - like training a massive deep neural network - can be complex and confusing. To make it easier to understand,debug, and optimize TensorFlow programs, we’ve included a suite of visualization tools called TensorBoard

1 | import tensorflow as tf |

_note : Create the summary writer after graph definition and before running your session_

#### Run it

Go to terminal, run:

1 | $ python [yourprogram].py |

Then open your browser and go to: http://localhost:6006/

“Const” and “Const_1” correspond to a and b, and the node “Add” corresponds to x. The names we give them (a, b, and x) are for us to access them when we need. They mean nothing for the internal TensorFlow. To make TensorBoard display the names of your ops, you have to explicitly name them.

1 | a = tf.constant(2, name="a") |

The graph itself defines the ops and dependencies, but not displays the values. It only cares about the values when we run the session with some values to fetch in mind.

_note : If you’ve run your code several times, there will be multiple event files in ‘~/dev/cs20si/graphs/lecture01’, TF will show only the latest graph and display the warning of multiple event files. To get rid of the warning, delete all the event files you no longer need._

**Summary : Learn to use TensorBoard well and often.It will help a lot when you build complicated models.**

## Constant Types

Link to documentation : https://www.tensorflow.org/api_docs/python/constant_op/

```
tf.constant(value, dtype=None, shape=None,name='Const', verify_shape=False)
```

1 | # constant of 1d tensor (vector) |

#### Constants as Sequences

1 | tf.linspace(start, stop, num, name=None) |

1 | tf.range(start, limit=None, delta=1, dtype=None, name='range') |

_Note that unlike NumPy or Python sequences, TensorFlow sequences are not iterable._

1 | for _ in np.linspace(0, 10, 4): # OK |

#### Randomly Generated Constants

1 | tf.random_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None) |

#### tf.set_random_seed(seed)

To generate different sequences across sessions, set neither graph-level nor op-level seeds:

1 | a = tf.random_uniform([1]) |

To generate the same repeatable sequence for an op across sessions, set the seed for the op:

1 | a = tf.random_uniform([1], seed=1) |

To make the random sequences generated by all ops be repeatable across sessions, set a graph-level seed:

1 | tf.set_random_seed(1234) |

_More details in https://www.tensorflow.org/api_docs/python/tf/set_random_seed_

## Math Operations

1 | a = tf.constant([3, 6]) |

_More details in

https://www.tensorflow.org/api_guides/python/math_ops_

## Data Types

#### Python Native Types

TensorFlow takes in Python native types such as Python boolean values, numeric values (integers, floats), and strings. Single values will be converted to 0-d tensors (or scalars), lists of

values will be converted to 1-d tensors (vectors), lists of lists of values will be converted to 2-d tensors (matrices), and so on. Example below is adapted and modified from the book

“TensorFlow for Machine Intelligence”.

1 | t_0 = 19 # Treated as a 0-d tensor, or "scalar" |

#### TensorFlow Native Types

Like NumPy, TensorFlow also its own data types as you’ve seen tf.int32, tf.float32.Below is a list of current TensorFlow data types, taken from TensorFlow’s official documentation.

#### NumPy Data Types

TensorFlow was designed to integrate seamlessly with Numpy, the package that has become the lingua franca(免费) of data science.

TensorFlow’s data types are based on those of NumPy; in fact, np.int32 == tf.int32 returns True.You can pass NumPy types to TensorFlow ops.

1 | tf.ones([2, 2], np.float32) ==> [[1.0 1.0], [1.0 1.0]] |

#### Summary

- Do not use Python native types for tensors because TensorFlow has to infer(推断) Python type
- Beware when using NumPy arrays because NumPy and TensorFlow might become not so compatible in the future!
- Both TensorFlow and NumPy are n-d array libraries. NumPy supports ndarray, but doesn’t offer methods to create tensor functions and automatically compute derivatives, nor GPU support. So TensorFlow still wins!
- It’s possible to convert the data into the appropriate type when you pass it into TensorFlow, but certain data types still may be difficult to declare correctly, such as complex numbers. Because of this, it is common to create hand-defined Tensor objects as NumPy arrays. However, always use TensorFlow types when possible, because both TensorFlow and NumPy can evolve to a point that such compatibility no longer exists.

## Variables

The difference between a constant and a variable:

- A constant is constant. A variable can be assigned to, its value can be changed.
- A constant’s value is stored in the graph and its value is replicated(复制) wherever the graph is loaded. A variable is stored separately, and may live on a parameter server.

Point 2 basically means that constants are stored in the graph definition. When constants are memory expensive, it will be slow each time you have to load the graph.

To see the graph’s definition and what’s stored in the graph’s definition, simply print out the graph’s protobuf.Protobuf stands for protocol buffer :

“Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler.”

1 | import tensorflow as tf |

_Only use constants for primitive types.

Use variables or readers for more data that

requires more memory._

#### Declare Variables

To declare a variable, you create an instance of the class tf.Variable. Note that it’s tf.constant but tf.Variable and not tf.variable because tf.constant is an op, while tf.Variable is a class.

1 | # create variable a with scalar value |

#### Initialize Variables

**You have to initialize variables before using them.**

If you try to evaluate the variables before initializing them you’ll run into FailedPreconditionError: Attempting to use uninitialized value tensor.

1 | # The easiest way is initializing all variables at once |

To initialize only a subset of variables, you use tf.variables_initializer() with a list of variables you want to initialize :

1 | init_ab = tf.variables_initializer([a, b], name="init_ab") |

You can also initialize each variable separately using tf.Variable.initializer :

1 | W = tf.Variable(tf.zeros([784,10])) |

#### Evaluate Values of Variables

To get the value of a variable, we need to evaluate it using eval() :

1 | # W is a random 700 x 100 variable object |

#### Assign Values to Variables

We can assign a value to a variable using tf.Variable.assign() :

1 | W = tf.Variable(10) |

Why 10 and not 100? W.assign(100) doesn’t assign the value 100 to W, but instead create an assign op to do that. For this op to take effect, we have to run this op in session.

1 | W = tf.Variable(10) |

Note that we don’t have initialize W in this case, because assign() does it for us. In fact,initializer op is the assign op that assigns the variable’s initial value to the variable itself.

1 | a = tf.Variable(2, name="scalar") |

Unlike tf.Variable.assign(),tf.Variable.assign_add() and tf.Variable.assign_sub() don’t initialize your variables for you

because these ops depend on the initial values of the variable.

1 | my_var = tf.Variable(10) |

**Because TensorFlow sessions maintain values separately, each Session can have its own current value for a variable defined in a graph.**

1 | W = tf.Variable(10) |

Use a variable to initialize another variable :

1 | # not safe |

In this case, you should use initialized_value() to make sure that W is initialized before its value is used to initialize U.

1 | W = tf.Variable(2) |

_More details in https://www.tensorflow.org/api_docs/python/tf/Variable._

## InteractiveSession

The only difference is an InteractiveSession makes itself the default session so you can call run() or eval() without explicitly call the session. This is convenient in interactive shells and IPython notebooks, as it avoids having to pass an explicit Session object to run ops. However, it is complicated when you have multiple sessions to run.

1 | sess = tf.InteractiveSession() |

## Control Dependencies

Sometimes, we will have two independent ops but you’d like to specify which op should be run first, then you use tf.Graph.control_dependencies(control_inputs).

1 | # your graph g have 5 ops: a, b, c, d, e |

## Placeholders

**tf.placeholder(dtype, shape=None, name=None)** :

- Dtype is the required parameter that specifies of the data type of the value of the placeholder.
- Shape specifies the shape of the tensor that can be accepted as actual value for the placeholder. shape=None means that tensors of any shape will be accepted. Using shape=None is easy to construct graphs, but nightmarish for debugging. You should always define the shape of your placeholders as detailed as possible.
1

2

3

4

5

6

7

8# create a placeholder of type float 32-bit, shape is a vector of 3 elements

a = tf.placeholder(tf.float32, shape=[3])

# create a constant of type float 32-bit, shape is a vector of 3 elements

b = tf.constant([5, 5, 5], tf.float32)

# use the placeholder as you would a constant or a variable

c = a + b # Short for tf.add(a, b)

with tf.Session() as sess:

print sess.run(c) # Error because a doesn’t have any value

Feed the values to placeholders using a dictionary :

1 | # create a placeholder of type float 32-bit, shape is a vector of 3 elements |

We can feed as any data points to the placeholder as we want by iterating through the data set and feed in the value one at a time.

1 | with tf.Session() as sess: |

You can feed_dict any feedable tensor.Placeholder is just a way to indicate that something must be fed.

You can feed values to tensors that aren’t placeholders. Any tensors that are feedable can be fed. To check if a tensor is feedable or not, use:

```
tf.Graph.is_feedable(tensor)
```

#### Feeding Values to TF Ops

1 | # create operations, tensors, etc (using the default graph) |

feed_dict can be extremely useful to test your model. When you have a large graph and just want to test out certain parts, you can provide dummy values so TensorFlow won’t waste time doing unnecessary computations.

_More details in https://www.tensorflow.org/api_docs/python/tf/placeholder._

## The Trap of Lazy Loading

Lazy loading is a term that refers to a programming pattern when you defer(推迟) declaring/initializing an object until it is loaded.. In the context of TensorFlow, it means you defer creating an op until you need to compute it.

#### Normal Loading

1 | x = tf.Variable(10, name='x') |

#### Lazy Loading

1 | x = tf.Variable(10, name='x') |

Both give the same value of z.What’s the problem?

Let’s see the graphs for them on TensorBoard.Normal loading graph looks just like we expected.

Well, the node “Add” is missing, which is understandable since we added the note “Add” after we’ve written the graph to FileWriter. This makes it harder to read the graph but it’s not a bug.

Let’s look at the graph definition.The protobuf for the graph in normal loading has only 1 node “Add”,On the other hand, the protobuf for the graph in lazy loading has 10 copies of the node “Add”. It adds a new node “Add” every time you want to compute z!

1 | # normal loading |

You probably think: “This is stupid.Why would I want to compute the same value more than once?” and think that it’s a bug that nobody will ever commit. It happens more often than you think. For example, you might want to compute the same loss function or make some prediction after a certain number of training samples. Before you know it, you’ve computed it for thousands of times, and added thousands of unnecessary nodes to your graph. Your graph definition becomes bloated, slow to load and expensive to pass around.

There are two ways to avoid this bug :

- always separate the definition of ops and their

execution when you can. - But when it is not possible because you want to group related ops into classes, you can use Python property to ensure that your function is only loaded once when it’s first called.

_More details in https://danijar.com/structuring-your-tensorflow-models/_