Converting between datetime Timestamp and datetime64

Question

How do I convert a numpy datetime64 object to a datetime datetime  or Timestamp    In the following code  I create a datetime  timestamp and datetime64 objects   import datetime import numpy as np import pandas as pd dt   datetime datetime 2012  5  1    A strange way to extract a Timestamp object  there s surely a better way  ts   pd DatetimeIndex  dt   0  dt64   np datetime64 dt   In  7   dt Out 7   datetime datetime 2012  5  1  0  0   In  8   ts Out 8    lt Timestamp  2012-05-01 00 00 00 gt   In  9   dt64 Out 9   numpy datetime64  2012-05-01T01 00 00 000000 0100     Note  it s easy to get the datetime from the Timestamp   In  10   ts to datetime   Out 10   datetime datetime 2012  5  1  0  0    But how do we extract the datetime or Timestamp from a numpy datetime64  dt64       Update  a somewhat nasty example in my dataset  perhaps the motivating example  seems to be   dt64   numpy datetime64  2002-06-28T01 00 00 000000000 0100     which should be datetime datetime 2002  6  28  1  0   and not a long      1025222400000000000L

User · Answer

import numpy as np import pandas as pd   def np64toDate np64       return pd to datetime str np64   replace tzinfo None  to datetime     use this function to get pythons native datetime object

User · Answer

To convert numpy datetime64 to datetime object that represents time in UTC on numpy-1 8    gt  gt  gt  from datetime import datetime  gt  gt  gt  import numpy as np  gt  gt  gt  dt   datetime utcnow    gt  gt  gt  dt datetime datetime 2012  12  4  19  51  25  362455   gt  gt  gt  dt64   np datetime64 dt   gt  gt  gt  ts    dt64 - np datetime64  1970-01-01T00 00 00Z      np timedelta64 1   s    gt  gt  gt  ts 1354650685 3624549  gt  gt  gt  datetime utcfromtimestamp ts  datetime datetime 2012  12  4  19  51  25  362455   gt  gt  gt  np   version    1 8 0 dev-7b75899    The above example assumes that a naive datetime object is interpreted by np datetime64 as time in UTC     To convert datetime to np datetime64 and back  numpy-1 6     gt  gt  gt  np datetime64 datetime utcnow    astype datetime  datetime datetime 2012  12  4  13  34  52  827542    It works both on a single np datetime64 object and a numpy array of np datetime64   Think of np datetime64 the same way you would about np int8  np int16  etc and apply the same methods to convert beetween Python objects such as int  datetime and corresponding numpy objects   Your  nasty example  works correctly    gt  gt  gt  from datetime import datetime  gt  gt  gt  import numpy   gt  gt  gt  numpy datetime64  2002-06-28T01 00 00 000000000 0100   astype datetime  datetime datetime 2002  6  28  0  0   gt  gt  gt  numpy   version    1 6 2    current version available via pip install numpy   I can reproduce the long value on numpy-1 8 0 installed as   pip install git https   github com numpy numpy git egg numpy-dev   The same example    gt  gt  gt  from datetime import datetime  gt  gt  gt  import numpy  gt  gt  gt  numpy datetime64  2002-06-28T01 00 00 000000000 0100   astype datetime  1025222400000000000L  gt  gt  gt  numpy   version    1 8 0 dev-7b75899    It returns long because for numpy datetime64 type  astype datetime  is equivalent to  astype object  that returns Python integer  long  on numpy-1 8    To get datetime object you could    gt  gt  gt  dt64 dtype dtype   lt M8 ns     gt  gt  gt  ns   1e-9   number of seconds in a nanosecond  gt  gt  gt  datetime utcfromtimestamp dt64 astype int    ns  datetime datetime 2002  6  28  0  0    To get datetime64 that uses seconds directly    gt  gt  gt  dt64   numpy datetime64  2002-06-28T01 00 00 000000000 0100    s    gt  gt  gt  dt64 dtype dtype   lt M8 s     gt  gt  gt  datetime utcfromtimestamp dt64 astype int   datetime datetime 2002  6  28  0  0    The numpy docs say that the datetime API is experimental and may change in future numpy versions

User · Answer

I think there could be a more consolidated effort in an answer to better explain the relationship between Python s datetime module  numpy s datetime64 timedelta64 and pandas  Timestamp Timedelta objects   The datetime standard library of Python  The datetime standard library has four main objects   time - only time  measured in hours  minutes  seconds and microseconds date - only year  month and day datetime - All components of time and date timedelta - An amount of time with maximum unit of days   Create these four objects   gt  gt  gt  import datetime  gt  gt  gt  datetime time hour 4  minute 3  second 10  microsecond 7199  datetime time 4  3  10  7199    gt  gt  gt  datetime date year 2017  month 10  day 24  datetime date 2017  10  24    gt  gt  gt  datetime datetime year 2017  month 10  day 24  hour 4  minute 3  second 10  microsecond 7199  datetime datetime 2017  10  24  4  3  10  7199    gt  gt  gt  datetime timedelta days 3  minutes   55  datetime timedelta 3  3300    gt  gt  gt    add timedelta to datetime  gt  gt  gt  datetime timedelta days 3  minutes   55          datetime datetime year 2017  month 10  day 24  hour 4  minute 3  second 10  microsecond 7199  datetime datetime 2017  10  27  4  58  10  7199    NumPy s datetime64 and timedelta64 objects  NumPy has no separate date and time objects  just a single datetime64 object to represent a single moment in time  The datetime module s datetime object has microsecond precision  one-millionth of a second   NumPy s datetime64 object allows you to set its precision from hours all the way to attoseconds  10   -18   It s constructor is more flexible and can take a variety of inputs   Construct NumPy s datetime64 and timedelta64 objects  Pass an integer with a string for the units  See all units here  It gets converted to that many units after the UNIX epoch  Jan 1  1970   gt  gt  gt  np datetime64 5   ns    numpy datetime64  1970-01-01T00 00 00 000000005     gt  gt  gt  np datetime64 1508887504   s   numpy datetime64  2017-10-24T23 25 04     You can also use strings as long as they are in ISO 8601 format    gt  gt  gt  np datetime64  2017-10-24   numpy datetime64  2017-10-24     Timedeltas have a single unit   gt  gt  gt  np timedelta64 5   D     5 days  gt  gt  gt  np timedelta64 10   h   10 hours   Can also create them by subtracting two datetime64 objects   gt  gt  gt  np datetime64  2017-10-24T05 30 45 67   - np datetime64  2017-10-22T12 35 40 123   numpy timedelta64 147305547  ms     Pandas Timestamp and Timedelta build much more functionality on top of NumPy  A pandas Timestamp is a moment in time very similar to a datetime but with much more functionality  You can construct them with either pd Timestamp or pd to datetime    gt  gt  gt  pd Timestamp 1239 1238934   defautls to nanoseconds Timestamp  1970-01-01 00 00 00 000001239     gt  gt  gt  pd Timestamp 1239 1238934  unit  D     change units Timestamp  1973-05-24 02 58 24 355200     gt  gt  gt  pd Timestamp  2017-10-24 05     partial strings work Timestamp  2017-10-24 05 00 00     pd to datetime works very similarly  with a few more options  and can convert a list of strings into Timestamps    gt  gt  gt  pd to datetime  2017-10-24 05   Timestamp  2017-10-24 05 00 00     gt  gt  gt  pd to datetime   2017-1-1    2017-1-2    DatetimeIndex   2017-01-01    2017-01-02    dtype  datetime64 ns    freq None    Converting Python datetime to datetime64 and Timestamp   gt  gt  gt  dt   datetime datetime year 2017  month 10  day 24  hour 4                      minute 3  second 10  microsecond 7199   gt  gt  gt  np datetime64 dt  numpy datetime64  2017-10-24T04 03 10 007199     gt  gt  gt  pd Timestamp dt    or pd to datetime dt  Timestamp  2017-10-24 04 03 10 007199     Converting numpy datetime64 to datetime and Timestamp   gt  gt  gt  dt64   np datetime64  2017-10-24 05 34 20 123456    gt  gt  gt  unix epoch   np datetime64 0   s    gt  gt  gt  one second   np timedelta64 1   s    gt  gt  gt  seconds since epoch    dt64 - unix epoch    one second  gt  gt  gt  seconds since epoch 1508823260 123456   gt  gt  gt  datetime datetime utcfromtimestamp seconds since epoch   gt  gt  gt  datetime datetime 2017  10  24  5  34  20  123456    Convert to Timestamp   gt  gt  gt  pd Timestamp dt64  Timestamp  2017-10-24 05 34 20 123456     Convert from Timestamp to datetime and datetime64  This is quite easy as pandas timestamps are very powerful   gt  gt  gt  ts   pd Timestamp  2017-10-24 04 24 33 654321     gt  gt  gt  ts to pydatetime       Python s datetime datetime datetime 2017  10  24  4  24  33  654321    gt  gt  gt  ts to datetime64   numpy datetime64  2017-10-24T04 24 33 654321000

User · Answer

One option is to use str  and then to datetime  or similar    In  11   str dt64  Out 11    2012-05-01T01 00 00 000000 0100   In  12   pd to datetime str dt64   Out 12   datetime datetime 2012  5  1  1  0  tzinfo tzoffset None  3600     Note  it is not equal to dt because it s become  offset-aware    In  13   pd to datetime str dt64   replace tzinfo None  Out 13   datetime datetime 2012  5  1  1  0    This seems inelegant      Update  this can deal with the  nasty example    In  21   dt64   numpy datetime64  2002-06-28T01 00 00 000000000 0100    In  22   pd to datetime str dt64   replace tzinfo None  Out 22   datetime datetime 2002  6  28  1  0

User · Answer

gt  gt  gt  dt64 tolist   datetime datetime 2012  5  1  0  0    For DatetimeIndex  the tolist returns a list of datetime objects  For a single datetime64 object it returns a single datetime object

User · Answer

If you want to convert an entire pandas series of datetimes to regular python datetimes  you can also use  to pydatetime     pd date range  20110101   20110102  freq  H   to pydatetime     gt   datetime datetime 2011  1  1  0  0  datetime datetime 2011  1  1  1  0     datetime datetime 2011  1  1  2  0  datetime datetime 2011  1  1  3  0            It also supports timezones   pd date range  20110101   20110102  freq  H   tz localize  UTC   tz convert  Australia Sydney   to pydatetime      datetime datetime 2011  1  1  11  0  tzinfo  lt DstTzInfo  Australia Sydney  EST 11 00 00 DST gt    datetime datetime 2011  1  1  12  0  tzinfo  lt DstTzInfo  Australia Sydney  EST 11 00 00 DST gt          NOTE  If you are operating on a Pandas Series you cannot call to pydatetime   on the entire series  You will need to call  to pydatetime   on each individual datetime64 using a list comprehension or something similar    datetimes    val to pydatetime   for val in df problem datetime column

User · Answer

indeed  all of these datetime types can be difficult  and potentially problematic  must keep careful track of timezone information   here s what i have done  though i admit that i am concerned that at least part of it is  quot not by design quot   also  this can be made a bit more compact as needed  starting with a numpy datetime64 dt a   dt a  numpy datetime64  2015-04-24T23 11 26 270000-0700    dt a1   dt a tolist            yields a datetime object in UTC  but without tzinfo dt a1  datetime datetime 2015  4  25  6  11  26  270000    now  make your  quot aware quot  datetime    dt a2 datetime datetime  list dt a1 timetuple    6      dt a1 microsecond   tzinfo pytz timezone  UTC         and of course  that can be compressed into one line as needed

User · Answer

This post has been up for 4 years and I still struggled with this conversion problem - so the issue is still active in 2017 in some sense  I was somewhat shocked that the numpy documentation does not readily offer a simple conversion algorithm but that s another story   I have come across another way to do the conversion that only involves modules numpy and datetime  it does not require pandas to be imported which seems to me to be a lot of code to import for such a simple conversion  I noticed that datetime64 astype datetime datetime  will return a datetime datetime object if the original datetime64 is in micro-second units while other units return an integer timestamp    I use module xarray for data I O from Netcdf files which uses the datetime64 in nanosecond units making the conversion fail unless you first convert to micro-second units  Here is the example conversion code   import numpy as np import datetime  def convert datetime64 to datetime  usert  np datetime64  - gt datetime datetime      t   np datetime64  usert   us   astype datetime datetime  return t   Its only tested on my machine  which is Python 3 6 with a recent 2017 Anaconda distribution  I have only looked at scalar conversion and have not checked array based conversions although I m guessing it will be good  Nor have I looked at the numpy datetime64 source code to see if the operation makes sense or not

User · Answer

You can just use the pd Timestamp constructor  The following diagram may be useful for this and related questions

User · Answer

Some solutions work well for me but numpy will deprecate some parameters   The solution that work better for me is to read the date as a pandas datetime and excract explicitly the year  month and day of a pandas object  The following code works for the most common situation    def format dates dates       dt   pd to datetime dates      try  return  datetime date x year  x month  x day  for x in dt          except TypeError  return datetime date dt year  dt month  dt day

User · Answer

Welcome to hell   You can just pass a datetime64 object to pandas Timestamp   In  16   Timestamp numpy datetime64  2012-05-01T01 00 00 000000    Out 16    lt Timestamp  2012-05-01 01 00 00 gt    I noticed that this doesn t work right though in NumPy 1 6 1   numpy datetime64  2012-05-01T01 00 00 000000 0100     Also  pandas to datetime can be used  this is off of the dev version  haven t checked v0 9 1    In  24   pandas to datetime  2012-05-01T01 00 00 000000 0100   Out 24   datetime datetime 2012  5  1  1  0  tzinfo tzoffset None  3600

User · Answer

I ve come back to this answer more times than I can count  so I decided to throw together a quick little class  which converts a Numpy datetime64 value to Python datetime value  I hope it helps others out there     from datetime import datetime import pandas as pd  class NumpyConverter object        classmethod     def to datetime cls  dt64  tzinfo None                       Converts a Numpy datetime64 to a Python datetime           param dt64  A Numpy datetime64 variable          type dt64  numpy datetime64          param tzinfo  The timezone the date   time value is in          type tzinfo  pytz timezone          return  A Python datetime variable          rtype  datetime                     ts   pd to datetime dt64          if tzinfo is not None              return datetime ts year  ts month  ts day  ts hour  ts minute  ts second  tzinfo tzinfo          return datetime ts year  ts month  ts day  ts hour  ts minute  ts second    I m gonna keep this in my tool bag  something tells me I ll need it again

[python] Converting between datetime, Timestamp and datetime64

Examples related to python

Examples related to datetime

Examples related to numpy

Examples related to pandas