[python] How do I properly set the Datetimeindex for a Pandas datetime object in a dataframe?

I have a pandas dataframe:

    lat         lng         alt days              date        time
0   40.003834   116.321462  211 39745.175405      2008-10-24  04:12:35
1   40.003783   116.321431  201 39745.175463  2008-10-24      04:12:40
2   40.003690   116.321429  203 39745.175521      2008-10-24      04:12:45
3   40.003589   116.321427  194 39745.175579      2008-10-24      04:12:50
4   40.003522   116.321412  190 39745.175637      2008-10-24      04:12:55
5   40.003509   116.321484  188 39745.175694      2008-10-24      04:13:00

For which I am trying to convert the df['date'] and df['time'] columns into a datetime. I can do:

df['Datetime'] = pd.to_datetime(df['date']+df['time'])
df = df.set_index(['Datetime'])
del df['date']
del df['time']

And I get:

                    lat         lng         alt days
Datetime                            
2008-10-2404:12:35  40.003834   116.321462  211 39745.175405    
2008-10-2404:12:40  40.003783   116.321431  201 39745.175463
2008-10-2404:12:45  40.003690   116.321429  203 39745.175521    
2008-10-2404:12:50  40.003589   116.321427  194 39745.175579    
2008-10-2404:12:55  40.003522   116.321412  190 39745.175637

But then if I try:

df.between_time(time(1),time(22,59,59))['lng'].std()

I get an error - 'TypeError: Index must be DatetimeIndex'

So, I've also tried setting the DatetimeIndex:

df['Datetime'] = pd.to_datetime(df['date']+df['time'])
#df = df.set_index(['Datetime'])
df = df.set_index(pd.DatetimeIndex(df['Datetime']))
del df['date']
del df['time']

And this throws an error also - 'DateParseError: unknown string format'

How do I create the datetime column and DatetimeIndex correctly so that df.between_time() works right?

This question is related to python datetime pandas

The answer is


To simplify Kirubaharan's answer a bit:

df['Datetime'] = pd.to_datetime(df['date'] + ' ' + df['time'])
df = df.set_index('Datetime')

And to get rid of unwanted columns (as OP did but did not specify per se in the question):

df = df.drop(['date','time'], axis=1)

You are not creating datetime index properly,

format = '%Y-%m-%d %H:%M:%S'
df['Datetime'] = pd.to_datetime(df['date'] + ' ' + df['time'], format=format)
df = df.set_index(pd.DatetimeIndex(df['Datetime']))

Examples related to python

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to datetime

Comparing two joda DateTime instances How to format DateTime in Flutter , How to get current time in flutter? How do I convert 2018-04-10T04:00:00.000Z string to DateTime? How to get current local date and time in Kotlin Converting unix time into date-time via excel Convert python datetime to timestamp in milliseconds SQL Server date format yyyymmdd Laravel Carbon subtract days from current date Check if date is a valid one Why is ZoneOffset.UTC != ZoneId.of("UTC")?

Examples related to pandas

xlrd.biffh.XLRDError: Excel xlsx file; not supported Pandas Merging 101 How to increase image size of pandas.DataFrame.plot in jupyter notebook? Trying to merge 2 dataframes but get ValueError Python Pandas User Warning: Sorting because non-concatenation axis is not aligned How to show all of columns name on pandas dataframe? Pandas/Python: Set value of one column based on value in another column Python Pandas - Find difference between two data frames Pandas get the most frequent values of a column Python convert object to float