SyntaxFix

[python] Filter df when values matches part of a string in pyspark

When filtering a DataFrame with string values, I find that the pyspark.sql.functions lower and upper come in handy, if your data could have column entries like "foo" and "Foo":

import pyspark.sql.functions as sql_fun
result = source_df.filter(sql_fun.lower(source_df.col_name).contains("foo"))

Examples related to python

• programming a servo thru a barometer • Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? • python variable NameError • Why my regexp for hyphenated words doesn't work? • Comparing a variable with a string python not working when redirecting from bash script • is it possible to add colors to python output? • Get Public URL for File - Google Cloud Storage - App Engine (Python) • Real time face detection OpenCV, Python • xlrd.biffh.XLRDError: Excel xlsx file; not supported • Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to apache-spark

• Select Specific Columns from Spark DataFrame • Select columns in PySpark dataframe • What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? • How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? • Spark dataframe: collect () vs select () • How does createOrReplaceTempView work in Spark? • Spark difference between reduceByKey vs groupByKey vs aggregateByKey vs combineByKey • Filter df when values matches part of a string in pyspark • Filtering a pyspark dataframe using isin by exclusion • Convert date from String to Date format in Dataframes

Examples related to pyspark

• Pyspark: Filter dataframe based on multiple conditions • How to convert column with string type to int form in pyspark data frame? • Select columns in PySpark dataframe • How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? • Filter df when values matches part of a string in pyspark • Filtering a pyspark dataframe using isin by exclusion • PySpark: withColumn() with two conditions and three outcomes • How to get name of dataframe column in pyspark? • Spark RDD to DataFrame python • PySpark 2.0 The size or shape of a DataFrame

Examples related to apache-spark-sql

• Select Specific Columns from Spark DataFrame • Pyspark: Filter dataframe based on multiple conditions • Select columns in PySpark dataframe • What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? • How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? • Spark dataframe: collect () vs select () • How does createOrReplaceTempView work in Spark? • Filter df when values matches part of a string in pyspark • Convert date from String to Date format in Dataframes • Take n rows from a spark dataframe and pass to toPandas()