Tensorflow Using Adam optimizer

Question

I am experimenting with some simple models in tensorflow  including one that looks very similar to the first MNIST for ML Beginners example  but with a somewhat larger dimensionality  I am able to use the gradient descent optimizer with no problems  getting good enough convergence  When I try to use the ADAM optimizer  I get errors like this   tensorflow python framework errors FailedPreconditionError  Attempting to use uninitialized value Variable 21 Adam        Node  Adam 2 update Variable 21 ApplyAdam   ApplyAdam T DT FLOAT  use locking false   device   job localhost replica 0 task 0 cpu 0   Variable 21  Variable 21 Adam  Variable 21 Adam 1  beta1 power 2  beta2 power 2  Adam 2 learning rate  Adam 2 beta1  Adam 2 beta2  Adam 2 epsilon  gradients 11 add 10 grad tuple control dependency 1      where the specific variable that complains about being uninitialized changes depending on the run  What does this error mean  And what does it suggest is wrong  It seems to occur regardless of the learning rate I use

User · Answer

You need to call tf global variables initializer   on you session  like  init   tf global variables initializer   sess run init    Full example is available in this great tutorial https   www tensorflow org get started mnist mechanics

User · Answer

The AdamOptimizer class creates additional variables  called  slots   to hold values for the  m  and  v  accumulators   See the source here if you re curious  it s actually quite readable  https   github com tensorflow tensorflow blob master tensorflow python training adam py L39   Other optimizers  such as Momentum and Adagrad use slots too   These variables must be initialized before you can train a model   The normal way to initialize variables is to call tf initialize all variables   which adds ops to initialize the variables present in the graph when it is called    Aside  unlike its name suggests  initialize all variables   does not initialize anything  it only add ops that will initialize the variables when run    What you must do is call initialize all variables   after you have added the optimizer      build your model      Add the optimizer train op   tf train AdamOptimizer 1e-4  minimize cross entropy    Add the ops to initialize variables   These will include    the optimizer slots added by AdamOptimizer    init op   tf initialize all variables      launch the graph in a session sess   tf Session     Actually intialize the variables sess run init op    now train your model for        sess run train op

User · Answer

I was having a similar problem   No problems training with GradientDescent optimizer  but error raised when using to Adam Optimizer  or any other optimizer with its own variables   Changing to an interactive session solved this problem for me   sess   tf Session     into  sess   tf InteractiveSession

User · Answer

run init after AdamOptimizer and without define init before or run init  sess run tf initialize all variables      or   sess run tf global variables initializer

User · Answer

FailedPreconditionError  Attempting to use uninitialized value is one of the most frequent errors related to tensorflow  From official documentation  FailedPreconditionError      This exception is most commonly raised when running an operation that   reads a tf Variable before it has been initialized    In your case the error even explains what variable was not initialized  Attempting to use uninitialized value Variable 1  One of the TF tutorials explains a lot about variables  their creation initialization saving loading  Basically to initialize the variable you have 3 options    initialize all global variables with tf global variables initializer   initialize variables you care about with tf variables initializer list of vars   Notice that you can use this function to mimic global variable initializer  tf variable initializers tf global variables    initialize only one variable with var name initializer   I almost always use the first approach  Remember you should put it inside a session run  So you will get something like this   with tf Session   as sess      sess run tf global variables initializer      If your are curious about more information about variables  read this documentation to know how to report uninitialized variables and check is variable initialized

[python] Tensorflow: Using Adam optimizer

Examples related to python

Examples related to tensorflow