SyntaxFix

[python] Filter df when values matches part of a string in pyspark

Spark 2.2 onwards

df.filter(df.location.contains('google.com'))

Spark 2.2 documentation link

Spark 2.1 and before

You can use plain SQL in filter

df.filter("location like '%google.com%'")

or with DataFrame column methods

df.filter(df.location.like('%google.com%'))

Spark 2.1 documentation link

Examples related to python

• programming a servo thru a barometer • Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? • python variable NameError • Why my regexp for hyphenated words doesn't work? • Comparing a variable with a string python not working when redirecting from bash script • is it possible to add colors to python output? • Get Public URL for File - Google Cloud Storage - App Engine (Python) • Real time face detection OpenCV, Python • xlrd.biffh.XLRDError: Excel xlsx file; not supported • Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to apache-spark

• Select Specific Columns from Spark DataFrame • Select columns in PySpark dataframe • What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? • How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? • Spark dataframe: collect () vs select () • How does createOrReplaceTempView work in Spark? • Spark difference between reduceByKey vs groupByKey vs aggregateByKey vs combineByKey • Filter df when values matches part of a string in pyspark • Filtering a pyspark dataframe using isin by exclusion • Convert date from String to Date format in Dataframes

Examples related to pyspark

• Pyspark: Filter dataframe based on multiple conditions • How to convert column with string type to int form in pyspark data frame? • Select columns in PySpark dataframe • How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? • Filter df when values matches part of a string in pyspark • Filtering a pyspark dataframe using isin by exclusion • PySpark: withColumn() with two conditions and three outcomes • How to get name of dataframe column in pyspark? • Spark RDD to DataFrame python • PySpark 2.0 The size or shape of a DataFrame

Examples related to apache-spark-sql

• Select Specific Columns from Spark DataFrame • Pyspark: Filter dataframe based on multiple conditions • Select columns in PySpark dataframe • What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? • How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? • Spark dataframe: collect () vs select () • How does createOrReplaceTempView work in Spark? • Filter df when values matches part of a string in pyspark • Convert date from String to Date format in Dataframes • Take n rows from a spark dataframe and pass to toPandas()