I have a command line program in Python that takes a while to finish. I want to know the exact time it takes to finish running.
I've looked at the timeit module, but it seems it's only for small snippets of code. I want to time the whole program.
This question is related to
python
time
execution-time
The answer is
Timeit is a class in Python used to calculate the execution time of small blocks of code.
Default_timer is a method in this class which is used to measure the wall clock timing, not CPU execution time. Thus other process execution might interfere with this. Thus it is useful for small blocks of code.
A sample of the code is as follows:
from timeit import default_timer as timer
start= timer()
# Some logic
end = timer()
print("Time taken:", end-start)
import time
start_time = time.clock()
main()
print time.clock() - start_time, "seconds"
time.clock() returns the processor time, which allows us to calculate only the time used by this process (on Unix anyway). The documentation says "in any case, this is the function to use for benchmarking Python or timing algorithms"
This is the simplest way to get the elapsed time for the program:
Write the following code at the end of your program.
import time
print(time.clock())
In Linux or Unix:
$ time python yourprogram.py
In Windows, see this StackOverflow question: How do I measure execution time of a command on the Windows command line?
For more verbose output,
$ time -v python yourprogram.py
Command being timed: "python3 yourprogram.py"
User time (seconds): 0.08
System time (seconds): 0.02
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.10
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 9480
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1114
Voluntary context switches: 0
Involuntary context switches: 22
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
In IPython, "timeit" any script:
def foo():
%run bar.py
timeit foo()
I tried and found time difference using the following scripts.
import time
start_time = time.perf_counter()
[main code here]
print (time.perf_counter() - start_time, "seconds")
I used a very simple function to time a part of code execution:
import time
def timing():
start_time = time.time()
return lambda x: print("[{:.2f}s] {}".format(time.time() - start_time, x))
And to use it, just call it before the code to measure to retrieve function timing, and then call the function after the code with comments. The time will appear in front of the comments. For example:
t = timing()
train = pd.read_csv('train.csv',
dtype={
'id': str,
'vendor_id': str,
'pickup_datetime': str,
'dropoff_datetime': str,
'passenger_count': int,
'pickup_longitude': np.float64,
'pickup_latitude': np.float64,
'dropoff_longitude': np.float64,
'dropoff_latitude': np.float64,
'store_and_fwd_flag': str,
'trip_duration': int,
},
parse_dates = ['pickup_datetime', 'dropoff_datetime'],
)
t("Loaded {} rows data from 'train'".format(len(train)))
Then the output will look like this:
[9.35s] Loaded 1458644 rows data from 'train'
I put this timing.py module into my own site-packages directory, and just insert import timing at the top of my module:
import atexit
from time import clock
def secondsToStr(t):
return "%d:%02d:%02d.%03d" % \
reduce(lambda ll,b : divmod(ll[0],b) + ll[1:],
[(t*1000,),1000,60,60])
line = "="*40
def log(s, elapsed=None):
print line
print secondsToStr(clock()), '-', s
if elapsed:
print "Elapsed time:", elapsed
print line
print
def endlog():
end = clock()
elapsed = end-start
log("End Program", secondsToStr(elapsed))
def now():
return secondsToStr(clock())
start = clock()
atexit.register(endlog)
log("Start Program")
I can also call timing.log from within my program if there are significant stages within the program I want to show. But just including import timing will print the start and end times, and overall elapsed time. (Forgive my obscure secondsToStr function, it just formats a floating point number of seconds to hh:mm:ss.sss form.)
Note: A Python 3 version of the above code can be found here or here.
Just use the timeit module. It works with both Python 2 and Python 3.
import timeit
start = timeit.default_timer()
# All the program statements
stop = timeit.default_timer()
execution_time = stop - start
print("Program Executed in "+str(execution_time)) # It returns time in seconds
It returns in seconds and you can have your execution time. It is simple, but you should write these in thew main function which starts program execution. If you want to get the execution time even when you get an error then take your parameter "Start" to it and calculate there like:
def sample_function(start,**kwargs):
try:
# Your statements
except:
# except statements run when your statements raise an exception
stop = timeit.default_timer()
execution_time = stop - start
print("Program executed in " + str(execution_time))
time.clock()
Deprecated since version 3.3: The behavior of this function depends on the platform: use perf_counter() or process_time() instead, depending on your requirements, to have a well-defined behavior.
time.perf_counter()
Return the value (in fractional seconds) of a performance counter, i.e. a clock with the highest available resolution to measure a short duration. It does include time elapsed during sleep and is system-wide.
time.process_time()
Return the value (in fractional seconds) of the sum of the system and user CPU time of the current process. It does not include time elapsed during sleep.
start = time.process_time()
... do something
elapsed = (time.process_time() - start)
To use metakermit's updated answer for Python 2.7, you will require the monotonic package.
The code would then be as follows:
from datetime import timedelta
from monotonic import monotonic
start_time = monotonic()
end_time = monotonic()
print(timedelta(seconds=end_time - start_time))
The time of a Python program's execution measure could be inconsistent depending on:
- Same program can be evaluated using different algorithms
- Running time varies between algorithms
- Running time varies between implementations
- Running time varies between computers
- Running time is not predictable based on small inputs
This is because the most effective way is using the "Order of Growth" and learn the Big "O" notation to do it properly.
Anyway, you can try to evaluate the performance of any Python program in specific machine counting steps per second using this simple algorithm: adapt this to the program you want to evaluate
import time
now = time.time()
future = now + 10
step = 4 # Why 4 steps? Because until here already four operations executed
while time.time() < future:
step += 3 # Why 3 again? Because a while loop executes one comparison and one plus equal statement
step += 4 # Why 3 more? Because one comparison starting while when time is over plus the final assignment of step + 1 and print statement
print(str(int(step / 10)) + " steps per second")
I really like Paul McGuire's answer, but I use Python 3. So for those who are interested: here's a modification of his answer that works with Python 3 on *nix (I imagine, under Windows, that clock() should be used instead of time()):
#python3
import atexit
from time import time, strftime, localtime
from datetime import timedelta
def secondsToStr(elapsed=None):
if elapsed is None:
return strftime("%Y-%m-%d %H:%M:%S", localtime())
else:
return str(timedelta(seconds=elapsed))
def log(s, elapsed=None):
line = "="*40
print(line)
print(secondsToStr(), '-', s)
if elapsed:
print("Elapsed time:", elapsed)
print(line)
print()
def endlog():
end = time()
elapsed = end-start
log("End Program", secondsToStr(elapsed))
start = time()
atexit.register(endlog)
log("Start Program")
If you find this useful, you should still up-vote his answer instead of this one, as he did most of the work ;).
Similar to the response from @rogeriopvl I added a slight modification to convert to hour minute seconds using the same library for long running jobs.
import time
start_time = time.time()
main()
seconds = time.time() - start_time
print('Time Taken:', time.strftime("%H:%M:%S",time.gmtime(seconds)))
Sample Output
Time Taken: 00:00:08
You do this simply in Python. There is no need to make it complicated.
import time
start = time.localtime()
end = time.localtime()
"""Total execution time in minutes$ """
print(end.tm_min - start.tm_min)
"""Total execution time in seconds$ """
print(end.tm_sec - start.tm_sec)
You can use the Python profiler cProfile to measure CPU time and additionally how much time is spent inside each function and how many times each function is called. This is very useful if you want to improve performance of your script without knowing where to start. This answer to another Stack Overflow question is pretty good. It's always good to have a look in the documentation too.
Here's an example how to profile a script using cProfile from a command line:
$ python -m cProfile euler048.py
1007 function calls in 0.061 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.061 0.061 <string>:1(<module>)
1000 0.051 0.000 0.051 0.000 euler048.py:2(<lambda>)
1 0.005 0.005 0.061 0.061 euler048.py:2(<module>)
1 0.000 0.000 0.061 0.061 {execfile}
1 0.002 0.002 0.053 0.053 {map}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler objects}
1 0.000 0.000 0.000 0.000 {range}
1 0.003 0.003 0.003 0.003 {sum}
I was having the same problem in many places, so I created a convenience package horology. You can install it with pip install horology and then do it in the elegant way:
from horology import Timing
with Timing(name='Important calculations: '):
prepare()
do_your_stuff()
finish_sth()
will output:
Important calculations: 12.43 ms
Or even simpler (if you have one function):
from horology import timed
@timed
def main():
...
will output:
main: 7.12 h
It takes care of units and rounding. It works with python 3.6 or newer.
The following snippet prints elapsed time in a nice human readable <HH:MM:SS> format.
import time
from datetime import timedelta
start_time = time.time()
#
# Perform lots of computations.
#
elapsed_time_secs = time.time() - start_time
msg = "Execution took: %s secs (Wall clock time)" % timedelta(seconds=round(elapsed_time_secs))
print(msg)
This is Paul McGuire's answer that works for me. Just in case someone was having trouble running that one.
import atexit
from time import clock
def reduce(function, iterable, initializer=None):
it = iter(iterable)
if initializer is None:
value = next(it)
else:
value = initializer
for element in it:
value = function(value, element)
return value
def secondsToStr(t):
return "%d:%02d:%02d.%03d" % \
reduce(lambda ll,b : divmod(ll[0],b) + ll[1:],
[(t*1000,),1000,60,60])
line = "="*40
def log(s, elapsed=None):
print (line)
print (secondsToStr(clock()), '-', s)
if elapsed:
print ("Elapsed time:", elapsed)
print (line)
def endlog():
end = clock()
elapsed = end-start
log("End Program", secondsToStr(elapsed))
def now():
return secondsToStr(clock())
def main():
start = clock()
atexit.register(endlog)
log("Start Program")
Call timing.main() from your program after importing the file.
For the data folks using Jupyter Notebook
In a cell, you can use Jupyter's %%time magic command to measure the execution time:
%%time
[ x**2 for x in range(10000)]
Output
CPU times: user 4.54 ms, sys: 0 ns, total: 4.54 ms
Wall time: 4.12 ms
This will only capture the execution time of a particular cell. If you'd like to capture the execution time of the whole notebook (i.e. program), you can create a new notebook in the same directory and in the new notebook execute all cells:
Suppose the notebook above is called example_notebook.ipynb. In a new notebook within the same directory:
# Convert your notebook to a .py script:
!jupyter nbconvert --to script example_notebook.ipynb
# Run the example_notebook with -t flag for time
%run -t example_notebook
Output
IPython CPU timings (estimated):
User : 0.00 s.
System : 0.00 s.
Wall time: 0.00 s.
I've looked at the timeit module, but it seems it's only for small snippets of code. I want to time the whole program.
$ python -mtimeit -n1 -r1 -t -s "from your_module import main" "main()"
It runs your_module.main() function one time and print the elapsed time using time.time() function as a timer.
To emulate /usr/bin/time in Python see Python subprocess with /usr/bin/time: how to capture timing info but ignore all other output?.
To measure CPU time (e.g., don't include time during time.sleep()) for each function, you could use profile module (cProfile on Python 2):
$ python3 -mprofile your_module.py
You could pass -p to timeit command above if you want to use the same timer as profile module uses.
There is a timeit module which can be used to time the execution times of Python code.
It has detailed documentation and examples in Python documentation, 26.6. timeit — Measure execution time of small code snippets.
If you want to measure time in microseconds, then you can use the following version, based completely on the answers of Paul McGuire and Nicojo - it's Python 3 code. I've also added some colour to it:
import atexit
from time import time
from datetime import timedelta, datetime
def seconds_to_str(elapsed=None):
if elapsed is None:
return datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")
else:
return str(timedelta(seconds=elapsed))
def log(txt, elapsed=None):
colour_cyan = '\033[36m'
colour_reset = '\033[0;0;39m'
colour_red = '\033[31m'
print('\n ' + colour_cyan + ' [TIMING]> [' + seconds_to_str() + '] ----> ' + txt + '\n' + colour_reset)
if elapsed:
print("\n " + colour_red + " [TIMING]> Elapsed time ==> " + elapsed + "\n" + colour_reset)
def end_log():
end = time()
elapsed = end-start
log("End Program", seconds_to_str(elapsed))
start = time()
atexit.register(end_log)
log("Start Program")
log() => function that prints out the timing information.
txt ==> first argument to log, and its string to mark timing.
atexit ==> Python module to register functions that you can call when the program exits.
I liked Paul McGuire's answer too and came up with a context manager form which suited my needs more.
import datetime as dt
import timeit
class TimingManager(object):
"""Context Manager used with the statement 'with' to time some execution.
Example:
with TimingManager() as t:
# Code to time
"""
clock = timeit.default_timer
def __enter__(self):
"""
"""
self.start = self.clock()
self.log('\n=> Start Timing: {}')
return self
def __exit__(self, exc_type, exc_val, exc_tb):
"""
"""
self.endlog()
return False
def log(self, s, elapsed=None):
"""Log current time and elapsed time if present.
:param s: Text to display, use '{}' to format the text with
the current time.
:param elapsed: Elapsed time to display. Dafault: None, no display.
"""
print s.format(self._secondsToStr(self.clock()))
if(elapsed is not None):
print 'Elapsed time: {}\n'.format(elapsed)
def endlog(self):
"""Log time for the end of execution with elapsed time.
"""
self.log('=> End Timing: {}', self.now())
def now(self):
"""Return current elapsed time as hh:mm:ss string.
:return: String.
"""
return str(dt.timedelta(seconds = self.clock() - self.start))
def _secondsToStr(self, sec):
"""Convert timestamp to h:mm:ss string.
:param sec: Timestamp.
"""
return str(dt.datetime.fromtimestamp(sec))
For functions, I suggest using this simple decorator I created.
def timeit(method):
def timed(*args, **kw):
ts = time.time()
result = method(*args, **kw)
te = time.time()
if 'log_time' in kw:
name = kw.get('log_name', method.__name__.upper())
kw['log_time'][name] = int((te - ts) * 1000)
else:
print('%r %2.22f ms' % (method.__name__, (te - ts) * 1000))
return result
return timed
@timeit
def foo():
do_some_work()
# 'foo' 0.000953 ms
First, install humanfriendly package by opening Command Prompt (CMD) as administrator and type there -
pip install humanfriendly
Code:
from humanfriendly import format_timespan
import time
begin_time = time.time()
# Put your code here
end_time = time.time() - begin_time
print("Total execution time: ", format_timespan(end_time))
Output:
Later answer, but I use the built-in timeit:
import timeit
code_to_test = """
a = range(100000)
b = []
for i in a:
b.append(i*2)
"""
elapsed_time = timeit.timeit(code_to_test, number=500)
print(elapsed_time)
# 10.159821493085474
- Wrap all your code, including any imports you may have, inside
code_to_test. numberargument specifies the amount of times the code should repeat.- Demo
Even better for Linux: time
$ time -v python rhtest2.py
Command being timed: "python rhtest2.py"
User time (seconds): 4.13
System time (seconds): 0.07
Percent of CPU this job got: 91%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:04.58
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 0
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 15
Minor (reclaiming a frame) page faults: 5095
Voluntary context switches: 27
Involuntary context switches: 279
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
from time import time
start_time = time()
...
end_time = time()
time_taken = end_time - start_time # time_taken is in seconds
hours, rest = divmod(time_taken,3600)
minutes, seconds = divmod(rest, 60)
Use line_profiler.
line_profiler will profile the time individual lines of code take to execute. The profiler is implemented in C via Cython in order to reduce the overhead of profiling.
from line_profiler import LineProfiler
import random
def do_stuff(numbers):
s = sum(numbers)
l = [numbers[i]/43 for i in range(len(numbers))]
m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()
The results will be:
Timer unit: 1e-06 s
Total time: 0.000649 s
File: <ipython-input-2-2e060b054fea>
Function: do_stuff at line 4
Line # Hits Time Per Hit % Time Line Contents
==============================================================
4 def do_stuff(numbers):
5 1 10 10.0 1.5 s = sum(numbers)
6 1 186 186.0 28.7 l = [numbers[i]/43 for i in range(len(numbers))]
7 1 453 453.0 69.8 m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
Following this answer created a simple but convenient instrument.
import time
from datetime import timedelta
def start_time_measure(message=None):
if message:
print(message)
return time.monotonic()
def end_time_measure(start_time, print_prefix=None):
end_time = time.monotonic()
if print_prefix:
print(print_prefix + str(timedelta(seconds=end_time - start_time)))
return end_time
Usage:
total_start_time = start_time_measure()
start_time = start_time_measure('Doing something...')
# Do something
end_time_measure(start_time, 'Done in: ')
start_time = start_time_measure('Doing something else...')
# Do something else
end_time_measure(start_time, 'Done in: ')
end_time_measure(total_start_time, 'Total time: ')
The output:
Doing something...
Done in: 0:00:01.218000
Doing something else...
Done in: 0:00:01.313000
Total time: 0:00:02.672000
I like the output the datetime module provides, where time delta objects show days, hours, minutes, etc. as necessary in a human-readable way.
For example:
from datetime import datetime
start_time = datetime.now()
# do your work here
end_time = datetime.now()
print('Duration: {}'.format(end_time - start_time))
Sample output e.g.
Duration: 0:00:08.309267
or
Duration: 1 day, 1:51:24.269711
As J.F. Sebastian mentioned, this approach might encounter some tricky cases with local time, so it's safer to use:
import time
from datetime import timedelta
start_time = time.monotonic()
end_time = time.monotonic()
print(timedelta(seconds=end_time - start_time))
Source: Stackoverflow.com