Spark DataFrame groupBy and sort in the descending order pyspark

Question

I m using pyspark Python 2 7 9 Spark 1 3 1  and have a dataframe GroupObject which I need to filter  amp  sort in the descending order  Trying to achieve it via this piece of code    group by dataframe count   filter   count   gt   10   sort  count   ascending False    But it throws the following error   sort   got an unexpected keyword argument  ascending

User · Answer

Use orderBy  df orderBy  column name   ascending False   Complete answer  group by dataframe count   filter  quot  count   gt   10 quot   orderBy  count   ascending False   http   spark apache org docs 2 0 0 api python pyspark sql html

User · Answer

In PySpark 1 3 sort method doesn t take ascending parameter  You can use desc method instead   from pyspark sql functions import col   group by dataframe      count        filter   count   gt   10        sort col  count   desc       or desc function   from pyspark sql functions import desc   group by dataframe      count        filter   count   gt   10        sort desc  count      Both methods can be used with with Spark    1 3  including Spark 2 x

User · Answer

By far the most convenient way is using this   df orderBy df column name desc      Doesn t require special imports

User · Answer

you can use groupBy and orderBy as follows also  dataFrameWay   df groupBy  firstName   count   withColumnRenamed  count   distinct name   sort desc  count

User · Answer

In pyspark 2 4 4  1  group by dataframe count   filter   count   gt   10   orderBy  count   ascending False   2  from pyspark sql functions import desc    group by dataframe count   filter   count   gt   10   orderBy  count   sort desc  count      No need to import in 1  and 1  is short  amp  easy to read   So I prefer 1  over 2

[python] Spark DataFrame groupBy and sort in the descending order (pyspark)

Examples related to python

Examples related to apache-spark

Examples related to dataframe

Examples related to pyspark

Examples related to apache-spark-sql