How to unpack pkl file

Question

I have a pkl file from MNIST dataset  which consists of handwritten digit images   I d like to take a look at each of those digit images  so I need to unpack the pkl file  except I can t find out how   Is there a way to unpack unzip pkl file

User · Answer

Handy one-liner  pkl       python -c  import pickle sys d pickle load open sys argv 1   rb    print d     1    pkl my pkl   Will print   str   for the pickled object   The generic problem of visualizing an object is of course undefined  so if   str   is not enough  you will need a custom script

User · Answer

In case you want to work with the original MNIST files  here is how you can deserialize them    If you haven t downloaded the files yet  do that first by running the following in the terminal   wget http   yann lecun com exdb mnist train-images-idx3-ubyte gz wget http   yann lecun com exdb mnist train-labels-idx1-ubyte gz wget http   yann lecun com exdb mnist t10k-images-idx3-ubyte gz wget http   yann lecun com exdb mnist t10k-labels-idx1-ubyte gz   Then save the following as deserialize py and run it   import numpy as np import gzip  IMG DIM   28  def decode image file fname       result          n bytes per img   IMG DIM IMG DIM      with gzip open fname   rb   as f          bytes    f read           data   bytes  16            if len data    n bytes per img    0              raise Exception  Something wrong with the file            result   np frombuffer data  dtype np uint8  reshape              len bytes    n bytes per img  n bytes per img       return result  def decode label file fname       result           with gzip open fname   rb   as f          bytes    f read           data   bytes  8            result   np frombuffer data  dtype np uint8       return result  train images   decode image file  train-images-idx3-ubyte gz   train labels   decode label file  train-labels-idx1-ubyte gz    test images   decode image file  t10k-images-idx3-ubyte gz   test labels   decode label file  t10k-labels-idx1-ubyte gz     The script doesn t normalize the pixel values like in the pickled file  To do that  all you have to do is  train images   train images 255 test images   test images 255

User · Answer

Generally  Your pkl file is  in fact  a serialized pickle file  which means it has been dumped using Python s pickle module   To un-pickle the data you can   import pickle   with open  serialized pkl    rb   as f      data   pickle load f    For the MNIST data set  Note gzip is only needed if the file is compressed   import gzip import pickle   with gzip open  mnist pkl gz    rb   as f      train set  valid set  test set   pickle load f    Where each set can be further divided  i e  for the training set    train x  train y   train set   Those would be the inputs  digits  and outputs  labels  of your sets   If you want to display the digits   import matplotlib cm as cm import matplotlib pyplot as plt   plt imshow train x 0  reshape  28  28    cmap cm Greys r  plt show       The other alternative would be to look at the original data   http   yann lecun com exdb mnist   But that will be harder  as you ll need to create a program to read the binary data in those files  So I recommend you to use Python  and load the data with pickle  As you ve seen  it s very easy   -

User · Answer

The pickle  and gzip if the file is compressed  module need to be used  NOTE  These are already in the  standard Python  library   No need to install anything new

[python] How to unpack pkl file?

Examples related to python

Examples related to pickle

Examples related to deep-learning

Examples related to mnist