ConvergenceWarning Liblinear failed to converge increase the number of iterations

Question

Running the code of linear binary pattern for Adrian  This program runs but gives the following warning    C  Python27 lib site-packages sklearn svm base py 922  ConvergenceWarning  Liblinear failed to converge  increase the number of iterations    the number of iterations    ConvergenceWarning   I am running python2 7 with opencv3 7  what should I do

User · Answer

Please incre max iter to 10000 as default value is 1000  Possibly  increasing no  of iterations will help algorithm to converge  For me it converged and solver was - lbfgs  log reg   LogisticRegression solver  lbfgs  class weight  balanced   max iter 10000

User · Answer

Normally when an optimization algorithm does not converge  it is usually because the problem is not well-conditioned  perhaps due to a poor scaling of the decision variables  There are a few things you can try   Normalize your training data so that the problem hopefully becomes more well conditioned  which in turn can speed up convergence  One possibility is to scale your data to 0 mean  unit standard deviation using Scikit-Learn s StandardScaler for an example  Note that you have to apply the StandardScaler fitted on the training data to the test data  Related to 1   make sure the other arguments such as regularization weight  C  is set appropriately  Set max iter to a larger value  The default is 1000  Set dual   True if number of features  gt  number of examples and vice versa  This solves the SVM optimization problem using the dual formulation   Thanks  Nino van Hooff for pointing this out  and  JamesKo for spotting my mistake  Use a different solver  for e g   the L-BFGS solver if you are using Logistic Regression  See  5ervant s answer   Note  One should not ignore this warning  This warning came about because  Solving the linear SVM is just solving a quadratic optimization problem  The solver is typically an iterative algorithm that keeps a running estimate of the solution  i e   the weight and bias for the SVM   It stops running when the solution corresponds to an objective value that is optimal for this convex optimization problem  or when it hits the maximum number of iterations set   If the algorithm does not converge  then the current estimate of the SVM s parameters are not guaranteed to be any good  hence the predictions can also be complete garbage    Edit In addition  consider the comment by  Nino van Hooff and  5ervant to use the dual formulation of the SVM  This is especially important if the number of features you have  D  is more than the number of training examples N  This is what the dual formulation of the SVM is particular designed for and helps with the conditioning of the optimization problem  Credit to  5ervant for noticing and pointing this out  Furthermore   5ervant also pointed out the possibility of changing the solver  in particular the use of the L-BFGS solver  Credit to him  i e   upvote his answer  not mine   I would like to provide a quick rough explanation for those who are interested  I am     why this matters in this case  Second-order methods  and in particular approximate second-order method like the L-BFGS solver  will help with ill-conditioned problems because it is approximating the Hessian at each iteration and using it to scale the gradient direction  This allows it to get better convergence rate but possibly at a higher compute cost per iteration  That is  it takes fewer iterations to finish but each iteration will be slower than a typical first-order method like gradient-descent or its variants  For e g   a typical first-order method might update the solution at each iteration like x k   1    x k  - alpha k    gradient f x k    where alpha k   the step size at iteration k  depends on the particular choice of algorithm or learning rate schedule  A second order method  for e g   Newton  will have an update equation x k   1    x k  - alpha k    Hessian x k    -1    gradient f x k    That is  it uses the information of the local curvature encoded in the Hessian to scale the gradient accordingly  If the problem is ill-conditioned  the gradient will be pointing in less than ideal directions and the inverse Hessian scaling will help correct this  In particular  L-BFGS mentioned in  5ervant s answer is a way to approximate the inverse of the Hessian as computing it can be an expensive operation  However  second-order methods might converge much faster  i e   requires fewer iterations  than first-order methods like the usual gradient-descent based solvers  which as you guys know by now sometimes fail to even converge  This can compensate for the time spent at each iteration  In summary  if you have a well-conditioned problem  or if you can make it well-conditioned through other means such as using regularization and or feature scaling and or making sure you have more examples than features  you probably don t have to use a second-order method  But these days with many models optimizing non-convex problems  e g   those in DL models   second order methods such as L-BFGS methods plays a different role there and there are evidence to suggest they can sometimes find better solutions compared to first-order methods  But that is another story

User · Answer

I reached the point that I set  up to max iter 1200000 on my LinearSVC classifier  but still the  ConvergenceWarning  was still present  I fix the issue by just setting dual False and leaving max iter to its default   With LogisticRegression solver  lbfgs   classifier  you should increase max iter  Mine have reached max iter 7600 before the  ConvergenceWarning  disappears when training with large dataset s features

User · Answer

Explicitly specifying the max iter resolves the warning as the default max iter is 100   For Logistic Regression        logreg   LogisticRegression max iter 1000

[python] ConvergenceWarning: Liblinear failed to converge, increase the number of iterations

Examples related to python

Examples related to opencv

Examples related to lbph-algorithm