How to show full column content in a Spark Dataframe

Question

I am using spark-csv to load data into a DataFrame  I want to do a simple query and display the content   val df   sqlContext read format  com databricks spark csv   option  header    true   load  my csv   df registerTempTable  tasks   results   sqlContext sql  select col from tasks    results show     The col seems truncated   scala gt  results show     --------------------                    col   --------------------   2015-11-16 07 15       2015-11-16 07 15       2015-11-16 07 15       2015-11-16 07 15       2015-11-16 07 15       2015-11-16 07 15       2015-11-16 07 15       2015-11-16 07 15       2015-11-16 07 15       2015-11-16 07 15       2015-11-16 07 15       2015-11-16 07 15       2015-11-16 07 15       2015-11-16 07 15       2015-11-16 07 15       2015-11-06 07 15       2015-11-16 07 15       2015-11-16 07 21       2015-11-16 07 21       2015-11-16 07 21       --------------------    How do I show the full content of the column

User · Answer

The following answer applies to a Spark Streaming application.

By setting the "truncate" option to false, you can tell the output sink to display the full column.

val query = out.writeStream
          .outputMode(OutputMode.Update())
          .format("console")
          .option("truncate", false)
          .trigger(Trigger.ProcessingTime("5 seconds"))
          .start()

User · Answer

Within Databricks you can visualize the dataframe in a tabular format  With the command   display results    It will look like

User · Answer

Try this in scala   df show df count toInt  false    The show method accepts an integer and a Boolean value but df count returns Long   so type casting is required

User · Answer

In c   Option  truncate   false  does not truncate data in the output   StreamingQuery query   spark                      Sql  SELECT   FROM Messages                        WriteStream                        OutputMode  append                        Format  console                        Option  truncate   false                       Start

User · Answer

Below code would help to view all rows without truncation in each column  df show df count    False

User · Answer

The other solutions are good  If these are your goals    No truncation of columns  No loss of rows  Fast and Efficient   These two lines are useful          df persist     df show df count  false     in Scala or  False  in Python   By persisting  the 2 executor actions  count and show  are faster  amp  more efficient when using persist or cache to maintain the interim underlying dataframe structure within the executors  See more about persist and cache

User · Answer

Tried this in pyspark df show truncate 0

User · Answer

results show 20  False  or results show 20  false  depending on whether you are running it on Java Scala Python

User · Answer

PYSPARK In the below code  df is the name of dataframe  1st parameter is to show all rows in the dataframe dynamically rather than hardcoding a numeric value  The 2nd parameter will take care of displaying full column contents since the value is set as False  df show df count   False     SCALA In the below code  df is the name of dataframe  1st parameter is to show all rows in the dataframe dynamically rather than hardcoding a numeric value  The 2nd parameter will take care of displaying full column contents since the value is set as false  df show df count   toInt false

User · Answer

results show 20  false  will not truncate  Check the source 20 is the default number of rows displayed when show   is called without any arguments

User · Answer

results show false  will show you the full column content    Show method by default limit to 20  and adding a number before false will show more rows

User · Answer

I use the plugin Chrome extension works pretty well     https   userstyles org styles 157357 jupyter-notebook-wide  1

User · Answer

If you put results show false    results will not be truncated

User · Answer

try this command        df show df count

User · Answer

results show 20 false  did the trick for me in Scala

[apache-spark] How to show full column content in a Spark Dataframe?

Examples related to apache-spark

Examples related to dataframe

Examples related to spark-csv

Examples related to output-formatting