View RDD contents in Python Spark

Question

Running a simple app in pyspark   f   sc textFile  README md   wc   f flatMap lambda x  x split       map lambda x   x  1   reduceByKey add    I want to view RDD contents using foreach action   wc foreach print    This throws a syntax error    SyntaxError  invalid syntax   What am I missing

User · Answer

By latest document  you can use rdd collect   foreach println  on the driver to display all  but it may cause memory issues on the driver  best is to use rdd take desired number   https   spark apache org docs 2 2 0 rdd-programming-guide html  To print all elements on the driver  one can use the collect   method to first bring the RDD to the driver node thus  rdd collect   foreach println   This can cause the driver to run out of memory  though  because collect   fetches the entire RDD to a single machine  if you only need to print a few elements of the RDD  a safer approach is to use the take    rdd take 100  foreach println

User · Answer

This error is because print isn t a function in Python 2 6   You can either define a helper UDF that performs the print  or use the   future   library to treat print as a function    gt  gt  gt  from operator import add  gt  gt  gt  f   sc textFile  README md    gt  gt  gt  def g x           print x      gt  gt  gt  wc foreach g    or   gt  gt  gt  from   future   import print function  gt  gt  gt  wc foreach print    However  I think it would be better to use collect   to bring the RDD contents back to the driver  because foreach executes on the worker nodes and the outputs may not necessarily appear in your driver   shell  it probably will in local mode  but not when running on a cluster     gt  gt  gt  for x in wc collect            print x

User · Answer

In Spark 2 0  I didn t tested with earlier versions   Simply   print myRDD take n    Where n is the number of lines and myRDD is wc in your case

User · Answer

You can simply collect the entire RDD  which will return a list of rows  and print said list   print wc collect

User · Answer

If you want to see the contents of RDD then yes collect is one option  but it fetches all the data to driver so there can be a problem    lt rdd name gt  take  lt num of elements you want to fetch gt     Better if you want to see just a sample  Running foreach and trying to print  I dont recommend this because if you are running this on cluster then the print logs would be local to the executor and it would print for the data accessible to that executor  print statement is not changing the state hence it is not logically wrong  To get all the logs you will have to do something like    Pseudocode   collect foreach print   But this may result in job failure as collecting all the data on driver may crash it  I would suggest using take command or if u want to analyze it then use sample collect on driver or write to file and then analyze it

User · Answer

Try this   data   f flatMap lambda x  x split       map   data map lambda x   x  1   mapreduce   map reduceByKey lambda x y  x y  result   mapreduce collect     Please note that when you run collect    the RDD - which is a distributed data set is aggregated at the driver node and is essentially converted to a list  So obviously  it won t be a good idea to collect   a 2T data set  If all you need is a couple of samples from your RDD  use take 10

[python] View RDD contents in Python Spark?

Examples related to python

Examples related to apache-spark