I'm getting this weird error:
classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.
'precision', 'predicted', average, warn_for)`
but then it also prints the f-score the first time I run:
metrics.f1_score(y_test, y_pred, average='weighted')
The second time I run, it provides the score without error. Why is that?
>>> y_pred = test.predict(X_test)
>>> y_test
array([ 1, 10, 35, 9, 7, 29, 26, 3, 8, 23, 39, 11, 20, 2, 5, 23, 28,
30, 32, 18, 5, 34, 4, 25, 12, 24, 13, 21, 38, 19, 33, 33, 16, 20,
18, 27, 39, 20, 37, 17, 31, 29, 36, 7, 6, 24, 37, 22, 30, 0, 22,
11, 35, 30, 31, 14, 32, 21, 34, 38, 5, 11, 10, 6, 1, 14, 12, 36,
25, 8, 30, 3, 12, 7, 4, 10, 15, 12, 34, 25, 26, 29, 14, 37, 23,
12, 19, 19, 3, 2, 31, 30, 11, 2, 24, 19, 27, 22, 13, 6, 18, 20,
6, 34, 33, 2, 37, 17, 30, 24, 2, 36, 9, 36, 19, 33, 35, 0, 4,
1])
>>> y_pred
array([ 1, 10, 35, 7, 7, 29, 26, 3, 8, 23, 39, 11, 20, 4, 5, 23, 28,
30, 32, 18, 5, 39, 4, 25, 0, 24, 13, 21, 38, 19, 33, 33, 16, 20,
18, 27, 39, 20, 37, 17, 31, 29, 36, 7, 6, 24, 37, 22, 30, 0, 22,
11, 35, 30, 31, 14, 32, 21, 34, 38, 5, 11, 10, 6, 1, 14, 30, 36,
25, 8, 30, 3, 12, 7, 4, 10, 15, 12, 4, 22, 26, 29, 14, 37, 23,
12, 19, 19, 3, 25, 31, 30, 11, 25, 24, 19, 27, 22, 13, 6, 18, 20,
6, 39, 33, 9, 37, 17, 30, 24, 9, 36, 39, 36, 19, 33, 35, 0, 4,
1])
>>> metrics.f1_score(y_test, y_pred, average='weighted')
C:\Users\Michael\Miniconda3\envs\snowflakes\lib\site-packages\sklearn\metrics\classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.
'precision', 'predicted', average, warn_for)
0.87282051282051276
>>> metrics.f1_score(y_test, y_pred, average='weighted')
0.87282051282051276
>>> metrics.f1_score(y_test, y_pred, average='weighted')
0.87282051282051276
Also, why is there a trailing 'precision', 'predicted', average, warn_for)
error message? There is no open parenthesis so why does it end with a closing parenthesis? I am running sklearn 0.18.1 using Python 3.6.0 in a conda environment on Windows 10.
I also looked at here and I don't know if it's the same bug. This SO post doesn't have solution either.
This question is related to
python
scikit-learn
As the error message states, the method used to get the F score is from the "Classification" part of sklearn - thus the talking about "labels".
Do you have a regression problem? Sklearn provides a "F score" method for regression under the "feature selection" group: http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_regression.html
In case you do have a classification problem, @Shovalt's answer seems correct to me.
According to @Shovalt's answer, but in short:
Alternatively you could use the following lines of code
from sklearn.metrics import f1_score
metrics.f1_score(y_test, y_pred, labels=np.unique(y_pred))
This should remove your warning and give you the result you wanted, because it no longer considers the difference between the sets, by using the unique mode.
As I have noticed this error occurs under two circumstances,
so use, either np.array(y_test) for y_true in scores or y_test.reset_index(drop=True)
Hope this helps.
The accepted answer explains already well why the warning occurs. If you simply want to control the warnings, one could use precision_recall_fscore_support
. It offers a (semi-official) argument warn_for
that could be used to mute the warnings.
(_, _, f1, _) = metrics.precision_recall_fscore_support(y_test, y_pred,
average='weighted',
warn_for=tuple())
As mentioned already in some comments, use this with care.
the same problem also happened to me when i training my classification model. the reason caused this problem is as what the warning message said "in labels with no predicated samples", it will caused the zero-division when compute f1-score. I found another solution when i read sklearn.metrics.f1_score doc, there is a note as follows:
When true positive + false positive == 0, precision is undefined; When true positive + false negative == 0, recall is undefined. In such cases, by default the metric will be set to 0, as will f-score, and UndefinedMetricWarning will be raised. This behavior can be modified with zero_division
the zero_division
default value is "warn"
, you could set it to 0
or 1
to avoid UndefinedMetricWarning
.
it works for me ;) oh wait, there is another problem when i using zero_division
, my sklearn report that no such keyword argument by using scikit-learn 0.21.3. Just update your sklearn to the latest version by running pip install scikit-learn -U
Source: Stackoverflow.com