[python] T-test in Pandas

If I want to calculate the mean of two categories in Pandas, I can do it like this:

data = {'Category': ['cat2','cat1','cat2','cat1','cat2','cat1','cat2','cat1','cat1','cat1','cat2'],
        'values': [1,2,3,1,2,3,1,2,3,5,1]}
my_data = DataFrame(data)
my_data.groupby('Category').mean()

Category:     values:   
cat1     2.666667
cat2     1.600000

I have a lot of data formatted this way, and now I need to do a T-test to see if the mean of cat1 and cat2 are statistically different. How can I do that?

This question is related to python pandas scipy statistics hypothesis-test

The answer is


it depends what sort of t-test you want to do (one sided or two sided dependent or independent) but it should be as simple as:

from scipy.stats import ttest_ind

cat1 = my_data[my_data['Category']=='cat1']
cat2 = my_data[my_data['Category']=='cat2']

ttest_ind(cat1['values'], cat2['values'])
>>> (1.4927289925706944, 0.16970867501294376)

it returns a tuple with the t-statistic & the p-value

see here for other t-tests http://docs.scipy.org/doc/scipy/reference/stats.html


I simplify the code a little bit.

from scipy.stats import ttest_ind
ttest_ind(*my_data.groupby('Category')['value'].apply(lambda x:list(x)))

EDIT: I had not realized this was about the data format. You could use

import pandas as pd
import scipy
two_data = pd.DataFrame(data, index=data['Category'])

Then accessing the categories is as simple as

scipy.stats.ttest_ind(two_data.loc['cat'], two_data.loc['cat2'], equal_var=False)

The loc operator accesses rows by label.


As @G Garcia said

one sided or two sided dependent or independent

If you have two independent samples but you do not know that they have equal variance, you can use Welch's t-test. It is as simple as

scipy.stats.ttest_ind(cat1['values'], cat2['values'], equal_var=False)

For reasons to prefer Welch's test, see https://stats.stackexchange.com/questions/305/when-conducting-a-t-test-why-would-one-prefer-to-assume-or-test-for-equal-vari.

For two dependent samples, you can use

scipy.stats.ttest_rel(cat1['values'], cat2['values'])

Examples related to python

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to pandas

xlrd.biffh.XLRDError: Excel xlsx file; not supported Pandas Merging 101 How to increase image size of pandas.DataFrame.plot in jupyter notebook? Trying to merge 2 dataframes but get ValueError Python Pandas User Warning: Sorting because non-concatenation axis is not aligned How to show all of columns name on pandas dataframe? Pandas/Python: Set value of one column based on value in another column Python Pandas - Find difference between two data frames Pandas get the most frequent values of a column Python convert object to float

Examples related to scipy

Reading images in python Numpy Resize/Rescale Image How to get the indices list of all NaN value in numpy array? ImportError: cannot import name NUMPY_MKL numpy.where() detailed, step-by-step explanation / examples Scikit-learn train_test_split with indices Matplotlib: Specify format of floats for tick labels Installing NumPy and SciPy on 64-bit Windows (with Pip) Can't install Scipy through pip Plotting a fast Fourier transform in Python

Examples related to statistics

Function to calculate R2 (R-squared) in R pandas: find percentile stats of a given column What exactly does numpy.exp() do? Find p-value (significance) in scikit-learn LinearRegression How to plot ROC curve in Python Pandas - Compute z-score for all columns Calculating percentile of dataset column How to normalize an array in NumPy to a unit vector? How to find row number of a value in R code np.mean() vs np.average() in Python NumPy?

Examples related to hypothesis-test

T-test in Pandas