[python] What does 'killed' mean when a processing of a huge CSV with Python, which suddenly stops?

Exit code 137 (128+9) indicates that your program exited due to receiving signal 9, which is SIGKILL. This also explains the killed message. The question is, why did you receive that signal?

The most likely reason is probably that your process crossed some limit in the amount of system resources that you are allowed to use. Depending on your OS and configuration, this could mean you had too many open files, used too much filesytem space or something else. The most likely is that your program was using too much memory. Rather than risking things breaking when memory allocations started failing, the system sent a kill signal to the process that was using too much memory.

As I commented earlier, one reason you might hit a memory limit after printing finished counting is that your call to counter.items() in your final loop allocates a list that contains all the keys and values from your dictionary. If your dictionary had a lot of data, this might be a very big list. A possible solution would be to use counter.iteritems() which is a generator. Rather than returning all the items in a list, it lets you iterate over them with much less memory usage.

So, I'd suggest trying this, as your final loop:

for key, value in counter.iteritems():
    writer.writerow([key, value])

Note that in Python 3, items returns a "dictionary view" object which does not have the same overhead as Python 2's version. It replaces iteritems, so if you later upgrade Python versions, you'll end up changing the loop back to the way it was.

Examples related to python

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to csv

Pandas: ValueError: cannot convert float NaN to integer Export result set on Dbeaver to CSV Convert txt to csv python script How to import an Excel file into SQL Server? "CSV file does not exist" for a filename with embedded quotes Save Dataframe to csv directly to s3 Python Data-frame Object has no Attribute (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape How to write to a CSV line by line? How to check encoding of a CSV file

Examples related to etl

C# Dictionary get item by index What does 'killed' mean when a processing of a huge CSV with Python, which suddenly stops? How do I fix 'Invalid character value for cast specification' on a date column in flat file? MySQL - Rows to Columns Import Excel spreadsheet columns into SQL Server database

Examples related to kill

How to kill a nodejs process in Linux? What does 'killed' mean when a processing of a huge CSV with Python, which suddenly stops? how to kill hadoop jobs Why number 9 in kill -9 command in unix? How to kill a process on a port on ubuntu How to kill a child process by the parent process? How to kill all processes matching a name? How do I kill an Activity when the Back button is pressed? Terminating idle mysql connections How can I stop a running MySQL query?