SyntaxFix

[java] Spark read file from S3 using sc.textFile ("s3n://...)

Confirmed that this is related to the Spark build against Hadoop 2.60. Just installed Spark 1.4.0 "Pre built for Hadoop 2.4 and later" (instead of Hadoop 2.6). And the code now works OK.

sc.textFile("s3n://bucketname/Filename") now raises another error:

java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties (respectively).

The code below uses the S3 URL format to show that Spark can read S3 file. Using dev machine (no Hadoop libs).

scala> val lyrics = sc.textFile("s3n://MyAccessKeyID:MySecretKey@zpub01/SafeAndSound_Lyrics.txt")
lyrics: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[3] at textFile at <console>:21

scala> lyrics.count
res1: Long = 9

Even Better: the code above with AWS credentials inline in the S3N URI will break if the AWS Secret Key has a forward "/". Configuring AWS Credentials in SparkContext will fix it. Code works whether the S3 file is public or private.

sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "BLABLA")
sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", "....") // can contain "/"
val myRDD = sc.textFile("s3n://myBucket/MyFilePattern")
myRDD.count

Examples related to java

• Under what circumstances can I call findViewById with an Options Menu / Action Bar item? • How much should a function trust another function • How to implement a simple scenario the OO way • Two constructors • How do I get some variable from another class in Java? • this in equals method • How to split a string in two and store it in a field • How to do perspective fixing? • String index out of range: 4 • My eclipse won't open, i download the bundle pack it keeps saying error log

Examples related to scala

• Intermediate language used in scalac? • Why does calling sumr on a stream with 50 tuples not complete • Select Specific Columns from Spark DataFrame • Joining Spark dataframes on the key • Provide schema while reading csv file as a dataframe • how to filter out a null value from spark dataframe • Fetching distinct values on a column using Spark DataFrame • Can't push to the heroku • Spark - Error "A master URL must be set in your configuration" when submitting an app • Add jars to a Spark Job - spark-submit

Examples related to apache-spark

• Select Specific Columns from Spark DataFrame • Select columns in PySpark dataframe • What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? • How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? • Spark dataframe: collect () vs select () • How does createOrReplaceTempView work in Spark? • Spark difference between reduceByKey vs groupByKey vs aggregateByKey vs combineByKey • Filter df when values matches part of a string in pyspark • Filtering a pyspark dataframe using isin by exclusion • Convert date from String to Date format in Dataframes

Examples related to rdd

• How to create a DataFrame from a text file in Spark • Spark - repartition() vs coalesce() • Difference between DataFrame, Dataset, and RDD in Spark • Spark specify multiple column conditions for dataframe join • Spark read file from S3 using sc.textFile ("s3n://...) • Spark: subtract two DataFrames • How to convert rdd object to dataframe in spark • What is the difference between cache and persist? • Apache Spark: map vs mapPartitions?

Examples related to hortonworks-data-platform

• Spark read file from S3 using sc.textFile ("s3n://...)