UndefinedMetricWarning F-score is ill-defined and being set to 0 0 in labels with no predicted samples

Question

I m getting this weird error   classification py 1113  UndefinedMetricWarning  F-score is ill-defined and being set to 0 0 in labels with no predicted samples   precision    predicted   average  warn for     but then it also prints the f-score the first time I run    metrics f1 score y test  y pred  average  weighted     The second time I run  it provides the score without error  Why is that    gt  gt  gt  y pred   test predict X test   gt  gt  gt  y test array   1  10  35   9   7  29  26   3   8  23  39  11  20   2   5  23  28         30  32  18   5  34   4  25  12  24  13  21  38  19  33  33  16  20         18  27  39  20  37  17  31  29  36   7   6  24  37  22  30   0  22         11  35  30  31  14  32  21  34  38   5  11  10   6   1  14  12  36         25   8  30   3  12   7   4  10  15  12  34  25  26  29  14  37  23         12  19  19   3   2  31  30  11   2  24  19  27  22  13   6  18  20          6  34  33   2  37  17  30  24   2  36   9  36  19  33  35   0   4          1    gt  gt  gt  y pred array   1  10  35   7   7  29  26   3   8  23  39  11  20   4   5  23  28         30  32  18   5  39   4  25   0  24  13  21  38  19  33  33  16  20         18  27  39  20  37  17  31  29  36   7   6  24  37  22  30   0  22         11  35  30  31  14  32  21  34  38   5  11  10   6   1  14  30  36         25   8  30   3  12   7   4  10  15  12   4  22  26  29  14  37  23         12  19  19   3  25  31  30  11  25  24  19  27  22  13   6  18  20          6  39  33   9  37  17  30  24   9  36  39  36  19  33  35   0   4          1    gt  gt  gt  metrics f1 score y test  y pred  average  weighted   C  Users Michael Miniconda3 envs snowflakes lib site-packages sklearn metrics classification py 1113  UndefinedMetricWarning  F-score is ill-defined and being set to 0 0 in labels with no predicted samples     precision    predicted   average  warn for  0 87282051282051276  gt  gt  gt  metrics f1 score y test  y pred  average  weighted   0 87282051282051276  gt  gt  gt  metrics f1 score y test  y pred  average  weighted   0 87282051282051276   Also  why is there a trailing  precision    predicted   average  warn for  error message  There is no open parenthesis so why does it end with a closing parenthesis  I am running sklearn 0 18 1 using Python 3 6 0 in a conda environment on Windows 10   I also looked at here and I don t know if it s the same bug  This SO post doesn t have solution either

User · Answer

the same problem also happened to me when i training my classification model. the reason caused this problem is as what the warning message said "in labels with no predicated samples", it will caused the zero-division when compute f1-score. I found another solution when i read sklearn.metrics.f1_score doc, there is a note as follows:

When true positive + false positive == 0, precision is undefined; When true positive + false negative == 0, recall is undefined. In such cases, by default the metric will be set to 0, as will f-score, and UndefinedMetricWarning will be raised. This behavior can be modified with zero_division

the zero_division default value is "warn", you could set it to 0 or 1 to avoid UndefinedMetricWarning. it works for me ;) oh wait, there is another problem when i using zero_division, my sklearn report that no such keyword argument by using scikit-learn 0.21.3. Just update your sklearn to the latest version by running pip install scikit-learn -U

User · Answer

According to  Shovalt s answer  but in short  Alternatively you could use the following lines of code     from sklearn metrics import f1 score     metrics f1 score y test  y pred  labels np unique y pred    This should remove your warning and give you the result you wanted  because it no longer considers the difference between the sets  by using the unique mode

User · Answer

The accepted answer explains already well why the warning occurs  If you simply want to control the warnings  one could use precision recall fscore support  It offers a  semi-official  argument warn for that could be used to mute the warnings           f1       metrics precision recall fscore support y test  y pred                                                          average  weighted                                                            warn for tuple      As mentioned already in some comments  use this with care

User · Answer

As the error message states  the method used to get the F score is from the  Classification  part of sklearn - thus the talking about  labels    Do you have a regression problem  Sklearn provides a  F score  method for regression under the  feature selection  group  http   scikit-learn org stable modules generated sklearn feature selection f regression html  In case you do have a classification problem   Shovalt s answer seems correct to me

User · Answer

As I have noticed this error occurs under two circumstances     If you have used train test split   to split your data  you have to make sure that you reset the index of the data  specially when taken using a pandas series object   y train  y test indices should be resetted  The problem is when you try to use one of the scores from sklearn metrics such as  precision score  this will try to match the shuffled indices of the y test that you got from train test split       so use  either np array y test  for y true in scores or y test reset index drop True    Then again you can still have this error if your predicted  True Positives  is 0  which is used for precision  recall and f1 scores  You can visualize this using a confusion matrix  If the classification is multilabel and you set param  average  weighted  micro macro you will get an answer as long as the diagonal line in the matrix is not 0   Hope this helps

User · Answer

As mentioned in the comments  some labels in y test don t appear in y pred  Specifically in this case  label  2  is never predicted   gt  gt  gt  set y test  - set y pred   2   This means that there is no F-score to calculate for this label  and thus the F-score for this case is considered to be 0 0  Since you requested an average of the score  you must take into account that a score of 0 was included in the calculation  and this is why scikit-learn is showing you that warning  This brings me to you not seeing the error a second time  As I mentioned  this is a warning  which is treated differently from an error in python  The default behavior in most environments is to show a specific warning only once  This behavior can be changed  import warnings warnings filterwarnings  always       quot error quot    quot ignore quot    quot always quot    quot default quot    quot module quot  or  quot once quot   If you set this before importing the other modules  you will see the warning every time you run the code  There is no way to avoid seeing this warning the first time  aside for setting warnings filterwarnings  ignore    What you can do  is decide that you are not interested in the scores of labels that were not predicted  and then explicitly specify the labels you are interested in  which are labels that were predicted at least once    gt  gt  gt  metrics f1 score y test  y pred  average  weighted   labels np unique y pred   0 91076923076923078  The warning is not shown in this case

[python] UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples

Examples related to python

Examples related to scikit-learn