PySpark multiple conditions in when clause

Question

I would like to modify the cell values of a dataframe column  Age  where currently it is blank and I would only do it if another column  Survived  has the value 0  for the corresponding row where it is blank for Age  If it is 1 in the Survived column but blank in Age column then I will keep it as null    I tried to use  amp  amp  operator but it didn t work  Here is my code   tdata withColumn  Age    when  tdata Age        amp  amp  tdata Survived     0    mean age 0  otherwise tdata Age   show     Any suggestions how to handle that  Thanks   Error Message   SyntaxError  invalid syntax   File   lt ipython-input-33-3e691784411c gt    line 1     tdata withColumn  Age    when  tdata Age        amp  amp  tdata Survived     0    mean age 0  otherwise tdata Age   show

User · Answer

It should be    when   tdata Age          amp   tdata Survived     0     mean age 0

User · Answer

it should works at least in pyspark 2 4  tdata   tdata withColumn  Age    when  tdata Age         amp   tdata Survived     0      NewValue   otherwise tdata Age

User · Answer

when in pyspark multiple conditions can be built using  amp  for and  and    for or    Note In pyspark t is important to enclose every expressions within parenthesis    that combine to form the condition   pyspark dataDF   spark createDataFrame   66   a    4                                      67   a    0                                      70   b    4                                      71   d    4                                       id    code    amt    dataDF withColumn  new column          when  col  code       a      col  code       d     A          when  col  code       b    amp   col  amt       4     B          otherwise  A1    show     In Spark Scala code   amp  amp   or      conditions can be used within when function    scala val dataDF   Seq         66   a    4     67   a    0     70   b    4     71   d    4           toDF  id    code    amt   dataDF withColumn  new column          when col  code        a     col  code        d    A          when col  code        b   amp  amp  col  amt        4    B          otherwise  A1    show                              Output   --- ---- --- ----------    id code amt new column   --- ---- --- ----------    66    a   4          A    67    a   0          A    70    b   4          B    71    d   4          A   --- ---- --- ----------    This code snippet is copied from sparkbyexamples com

User · Answer

You get SyntaxError error exception because Python has no  amp  amp  operator  It has and and  amp  where the latter one is the correct choice to create   boolean expressions on Column    for a logical disjunction and   for logical negation    Condition you created is also invalid because it doesn t consider operator precedence   amp  in Python has a higher precedence than    so expression has to be parenthesized     col  Age           amp   col  Survived       0      Column lt b   Age     AND  Survived   0    gt    On a side note when function is equivalent to case expression not WHEN clause  Still the same rules apply  Conjunction   df where  col  foo    gt  0   amp   col  bar    lt  0     Disjunction   df where  col  foo    gt  0     col  bar    lt  0     You can of course define conditions separately to avoid brackets   cond1   col  Age          cond2   col  Survived       0   cond1  amp  cond2

[python] PySpark: multiple conditions in when clause

Examples related to python

Examples related to apache-spark

Examples related to dataframe

Examples related to pyspark

Examples related to apache-spark-sql