[python] How to read pickle file?

I created some data and stored it several times like this:

with open('filename', 'a') as f:
        pickle.dump(data, f)

Every time the size of file increased, but when I open file

with open('filename', 'rb') as f:
    x = pickle.load(f)

I can see only data from the last time. How can I correctly read file?

This question is related to python pickle

The answer is


I developed a software tool that opens (most) Pickle files directly in your browser (nothing is transferred so it's 100% private):

https://pickleviewer.com/


The following is an example of how you might write and read a pickle file. Note that if you keep appending pickle data to the file, you will need to continue reading from the file until you find what you want or an exception is generated by reaching the end of the file. That is what the last function does.

import os
import pickle


PICKLE_FILE = 'pickle.dat'


def main():
    # append data to the pickle file
    add_to_pickle(PICKLE_FILE, 123)
    add_to_pickle(PICKLE_FILE, 'Hello')
    add_to_pickle(PICKLE_FILE, None)
    add_to_pickle(PICKLE_FILE, b'World')
    add_to_pickle(PICKLE_FILE, 456.789)
    # load & show all stored objects
    for item in read_from_pickle(PICKLE_FILE):
        print(repr(item))
    os.remove(PICKLE_FILE)


def add_to_pickle(path, item):
    with open(path, 'ab') as file:
        pickle.dump(item, file, pickle.HIGHEST_PROTOCOL)


def read_from_pickle(path):
    with open(path, 'rb') as file:
        try:
            while True:
                yield pickle.load(file)
        except EOFError:
            pass


if __name__ == '__main__':
    main()

There is a read_pickle function as part of pandas 0.22+

import pandas as pd

object = pd.read_pickle(r'filepath')