I am using Spark 1 3 0 and Spark Avro 1 0 0 I am working from the example on the repository page This following code works well val df sqlContext read avro src test resources episodes avro df filter doctor gt 5 write avro tmp output But what if I needed to see if the doctor string contains a substring Since we are writing our expression inside of a string What do I do to do a contains

[scala] Filter spark DataFrame on string contains

I am using Spark 1.3.0 and Spark Avro 1.0.0. I am working from the example on the repository page. This following code works well

val df = sqlContext.read.avro("src/test/resources/episodes.avro")
df.filter("doctor > 5").write.avro("/tmp/output")

But what if I needed to see if the doctor string contains a substring? Since we are writing our expression inside of a string. What do I do to do a "contains"?

This question is related to scala apache-spark dataframe apache-spark-sql

The answer is

You can use contains (this works with an arbitrary sequence):

df.filter($"foo".contains("bar"))

like (SQL like with SQL simple regular expression whith _ matching an arbitrary character and % matching an arbitrary sequence):

df.filter($"foo".like("bar"))

or rlike (like with Java regular expressions):

df.filter($"foo".rlike("bar"))

depending on your requirements. LIKE and RLIKE should work with SQL expressions as well.

In pyspark,SparkSql syntax:

where column_n like 'xyz%'

might not work.

Use:

where column_n RLIKE '^xyz'

This works perfectly fine.

Source: Stackoverflow.com

Examples related to scala

• Intermediate language used in scalac? • Why does calling sumr on a stream with 50 tuples not complete • Select Specific Columns from Spark DataFrame • Joining Spark dataframes on the key • Provide schema while reading csv file as a dataframe • how to filter out a null value from spark dataframe • Fetching distinct values on a column using Spark DataFrame • Can't push to the heroku • Spark - Error "A master URL must be set in your configuration" when submitting an app • Add jars to a Spark Job - spark-submit

Examples related to apache-spark

• Select Specific Columns from Spark DataFrame • Select columns in PySpark dataframe • What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? • How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? • Spark dataframe: collect () vs select () • How does createOrReplaceTempView work in Spark? • Spark difference between reduceByKey vs groupByKey vs aggregateByKey vs combineByKey • Filter df when values matches part of a string in pyspark • Filtering a pyspark dataframe using isin by exclusion • Convert date from String to Date format in Dataframes

Examples related to dataframe

• Trying to merge 2 dataframes but get ValueError • How to show all of columns name on pandas dataframe? • Python Pandas - Find difference between two data frames • Pandas get the most frequent values of a column • Display all dataframe columns in a Jupyter Python Notebook • How to convert column with string type to int form in pyspark data frame? • Display/Print one column from a DataFrame of Series in Pandas • Binning column with python pandas • Selection with .loc in python • Set value to an entire column of a pandas dataframe

Examples related to apache-spark-sql

• Select Specific Columns from Spark DataFrame • Pyspark: Filter dataframe based on multiple conditions • Select columns in PySpark dataframe • What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? • How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? • Spark dataframe: collect () vs select () • How does createOrReplaceTempView work in Spark? • Filter df when values matches part of a string in pyspark • Convert date from String to Date format in Dataframes • Take n rows from a spark dataframe and pass to toPandas()