[python] How to redirect stdout to both file and console with scripting?

I want to run a python script and capture the output on a text file as well as want to show on console.

I want to specify it as a property of the python script itself. NOT to use the command echo "hello world" | tee test.txt on command prompt every time.

Within script I tried:

sys.stdout = open('log.txt','w')

But this does not show the stdout output on screen.

I have heard about logging module but I could not get luck using that module to do the job.

To redirect output to a file and a terminal without modifying how your Python script is used outside, you could use pty.spawn(itself):

#!/usr/bin/env python
"""Redirect stdout to a file and a terminal inside a script."""
import os
import pty
import sys

def main():
    print('put your code here')

if __name__=="__main__":
    sentinel_option = '--dont-spawn'
    if sentinel_option not in sys.argv:
        # run itself copying output to the log file
        with open('script.log', 'wb') as log_file:
            def read(fd):
                data = os.read(fd, 1024)
                log_file.write(data)
                return data

            argv = [sys.executable] + sys.argv + [sentinel_option]
            rc = pty.spawn(argv, read)
    else:
        sys.argv.remove(sentinel_option)
        rc = main()
    sys.exit(rc)

If pty module is not available (on Windows) then you could replace it with teed_call() function that is more portable but it provides ordinary pipes instead of a pseudo-terminal -- it may change behaviour of some programs.

The advantage of pty.spawn and subprocess.Popen -based solutions over replacing sys.stdout with a file-like object is that they can capture the output at a file descriptor level e.g., if the script starts other processes that can also produce output on stdout/stderr. See my answer to the related question: Redirect stdout to a file in Python?

Based on Brian Burns edited answer, I created a single class that is easier to call:

class Logger(object):

    """
    Class to log output of the command line to a log file

    Usage:
    log = Logger('logfile.log')
    print("inside file")
    log.stop()
    print("outside file")
    log.start()
    print("inside again")
    log.stop()
    """

    def __init__(self, filename):
        self.filename = filename

    class Transcript:
        def __init__(self, filename):
            self.terminal = sys.stdout
            self.log = open(filename, "a")
        def __getattr__(self, attr):
            return getattr(self.terminal, attr)
        def write(self, message):
            self.terminal.write(message)
            self.log.write(message)
        def flush(self):
            pass

    def start(self):
        sys.stdout = self.Transcript(self.filename)

    def stop(self):
        sys.stdout.log.close()
        sys.stdout = sys.stdout.terminal

Here is a simple context manager that prints to the console and writes the same output to an file. It also writes any exceptions to the file.

import traceback
import sys

# Context manager that copies stdout and any exceptions to a log file
class Tee(object):
    def __init__(self, filename):
        self.file = open(filename, 'w')
        self.stdout = sys.stdout

    def __enter__(self):
        sys.stdout = self

    def __exit__(self, exc_type, exc_value, tb):
        sys.stdout = self.stdout
        if exc_type is not None:
            self.file.write(traceback.format_exc())
        self.file.close()

    def write(self, data):
        self.file.write(data)
        self.stdout.write(data)

    def flush(self):
        self.file.flush()
        self.stdout.flush()

To use the context manager:

print("Print")
with Tee('test.txt'):
    print("Print+Write")
    raise Exception("Test")
print("Print")

I tried this:

"""
Transcript - direct print output to a file, in addition to terminal.

Usage:
    import transcript
    transcript.start('logfile.log')
    print("inside file")
    transcript.stop()
    print("outside file")
"""

import sys

class Transcript(object):

    def __init__(self, filename):
        self.terminal = sys.stdout, sys.stderr
        self.logfile = open(filename, "a")

    def write(self, message):
        self.terminal.write(message)
        self.logfile.write(message)

    def flush(self):
        # this flush method is needed for python 3 compatibility.
        # this handles the flush command by doing nothing.
        # you might want to specify some extra behavior here.
        pass

def start(filename):
    """Start transcript, appending print output to given filename"""
    sys.stdout = Transcript(filename)

def stop():
    """Stop transcript and return print functionality to normal"""
    sys.stdout.logfile.close()
    sys.stdout = sys.stdout.terminal
    sys.stderr = sys.stderr.terminal

you can redirect the output to a file by using >> python with print rint's "chevron" syntax as indicated in the docs

let see,

fp=open('test.log','a')   # take file  object reference 
print >> fp , "hello world"            #use file object with in print statement.
print >> fp , "every thing will redirect to file "
fp.close()    #close the file 

checkout the file test.log you will have the data and to print on console just use plain print statement .

Use logging module to debug and follow your app

Here is how I managed to log to file and to console / stdout

import logging
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s - %(levelname)s - %(message)s',
                    filename='logs_file',
                    filemode='w')
# Until here logs only to file: 'logs_file'

# define a new Handler to log to console as well
console = logging.StreamHandler()
# optional, set the logging level
console.setLevel(logging.INFO)
# set a format which is the same for console use
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
# add the handler to the root logger
logging.getLogger('').addHandler(console)

# Now, we can log to both ti file and console
logging.info('Jackdaws love my big sphinx of quartz.')
logging.info('Hello world')

read it from source: https://docs.python.org/2/howto/logging-cookbook.html

Based on Amith Koujalgi's answer, here's a simple module you can use for logging -

transcript.py:

"""
Transcript - direct print output to a file, in addition to terminal.

Usage:
    import transcript
    transcript.start('logfile.log')
    print("inside file")
    transcript.stop()
    print("outside file")
"""

import sys

class Transcript(object):

    def __init__(self, filename):
        self.terminal = sys.stdout
        self.logfile = open(filename, "a")

    def write(self, message):
        self.terminal.write(message)
        self.logfile.write(message)

    def flush(self):
        # this flush method is needed for python 3 compatibility.
        # this handles the flush command by doing nothing.
        # you might want to specify some extra behavior here.
        pass

def start(filename):
    """Start transcript, appending print output to given filename"""
    sys.stdout = Transcript(filename)

def stop():
    """Stop transcript and return print functionality to normal"""
    sys.stdout.logfile.close()
    sys.stdout = sys.stdout.terminal

I got the way to redirect the out put to console as well as to a text file as well simultaneously:

te = open('log.txt','w')  # File where you need to keep the logs

class Unbuffered:

   def __init__(self, stream):

       self.stream = stream

   def write(self, data):

       self.stream.write(data)
       self.stream.flush()
       te.write(data)    # Write the data of stdout here to a text file as well



sys.stdout=Unbuffered(sys.stdout)

You can use shell redirection while executing the python file:

python foo_bar.py > file

This will write all results being printed on stdout from the python source to file to the logfile.

Or if you want logging from within the script:

import sys

class Logger(object):
    def __init__(self):
        self.terminal = sys.stdout
        self.log = open("logfile.log", "a")

    def write(self, message):
        self.terminal.write(message)
        self.log.write(message)  

    def flush(self):
        #this flush method is needed for python 3 compatibility.
        #this handles the flush command by doing nothing.
        #you might want to specify some extra behavior here.
        pass    

sys.stdout = Logger()

Now you can use:

print "Hello"

This will write "Hello" to both stdout and the logfile

I've tried a few solutions here and didn't find the one that writes into file and into console at the same time. So here is what I did (based on this answer)

class Logger(object):
    def __init__(self):
        self.terminal = sys.stdout

    def write(self, message):
        with open ("logfile.log", "a", encoding = 'utf-8') as self.log:            
            self.log.write(message)
        self.terminal.write(message)

    def flush(self):
        #this flush method is needed for python 3 compatibility.
        #this handles the flush command by doing nothing.
        #you might want to specify some extra behavior here.
        pass
sys.stdout = Logger()   

This solution uses more computing power, but reliably saves all of the data from stdout into logger file and uses less memeory. For my needs I've added time stamp into self.log.write(message) aswell. Works great.

from IPython.utils.io import Tee
from contextlib import closing

print('This is not in the output file.')        

with closing(Tee("outputfile.log", "w", channel="stdout")) as outputstream:
    print('This is written to the output file and the console.')
    # raise Exception('The file "outputfile.log" is closed anyway.')
print('This is not written to the output file.')   

# Output on console:
# This is not in the output file.
# This is written to the output file and the console.
# This is not written to the output file.

# Content of file outputfile.txt:
# This is written to the output file and the console.

The Tee class in IPython.utils.io does what you want, but it lacks the __enter__ and __exit__ methods needed to call it in the with-statement. Those are added by contextlib.closing.

I devised an easier solution. Just define a function that will print to file or to screen or to both of them. In the example below I allow the user to input the outputfile name as an argument but that is not mandatory:

OutputFile= args.Output_File
OF = open(OutputFile, 'w')

def printing(text):
    print text
    if args.Output_File:
        OF.write(text + "\n")

After this, all that is needed to print a line both to file and/or screen is: printing(Line_to_be_printed)

This way worked very well in my situation. I just added some modifications based on other code presented in this thread.

import sys, os 

orig_stdout = sys.stdout  # capture original state of stdout

te = open('log.txt','w')  # File where you need to keep the logs

class Unbuffered:
   def __init__(self, stream):
       self.stream = stream

   def write(self, data):
       self.stream.write(data)
       self.stream.flush()
       te.write(data)    # Write the data of stdout here to a text file as well

sys.stdout=Unbuffered(sys.stdout)




#######################################
##  Feel free to use print function  ##
#######################################

print("Here is an Example =)")

#######################################
##  Feel free to use print function  ##
#######################################




# Stop capturing printouts of the application from Windows CMD
sys.stdout = orig_stdout  # put back the original state of stdout
te.flush()  # forces python to write to file
te.close()  # closes the log file

# read all lines at once and capture it to the variable named sys_prints
with open('log.txt', 'r+') as file:
    sys_prints = file.readlines() 

# erase the file contents of log file
open('log.txt', 'w').close()