There's also JCommander (disclaimer: I created it):
object Main {
object Args {
@Parameter(
names = Array("-f", "--file"),
description = "File to load. Can be specified multiple times.")
var file: java.util.List[String] = null
}
def main(args: Array[String]): Unit = {
new JCommander(Args, args.toArray: _*)
for (filename <- Args.file) {
val f = new File(filename)
printf("file: %s\n", f.getName)
}
}
}
val array2 = array :+ 4
//Array(1, 2, 3, 4)
Works also "reversed":
val array2 = 4 +: array
Array(4, 1, 2, 3)
There is also an "in-place" version:
var array = Array( 1, 2, 3 )
array +:= 4
//Array(4, 1, 2, 3)
array :+= 0
//Array(4, 1, 2, 3, 0)
So to start with some kind of answer : ) - You can't
I am not an expert, but as far as I understand DataFrames, they are not equal to rdd and DataFrame has no such thing as Partitioner.
Generally DataFrame's idea is to provide another level of abstraction that handles such problems itself. The queries on DataFrame are translated into logical plan that is further translated to operations on RDDs. The partitioning you suggested will probably be applied automatically or at least should be.
If you don't trust SparkSQL that it will provide some kind of optimal job, you can always transform DataFrame to RDD[Row] as suggested in of the comments.
For those how are interested, here are some times for the suggested solutions...
Rounding
Java Formatter: Elapsed Time: 105
Scala Formatter: Elapsed Time: 167
BigDecimal Formatter: Elapsed Time: 27
Truncation
Scala custom Formatter: Elapsed Time: 3
Truncation is the fastest, followed by BigDecimal. Keep in mind these test were done running norma scala execution, not using any benchmarking tools.
object TestFormatters {
val r = scala.util.Random
def textFormatter(x: Double) = new java.text.DecimalFormat("0.##").format(x)
def scalaFormatter(x: Double) = "$pi%1.2f".format(x)
def bigDecimalFormatter(x: Double) = BigDecimal(x).setScale(2, BigDecimal.RoundingMode.HALF_UP).toDouble
def scalaCustom(x: Double) = {
val roundBy = 2
val w = math.pow(10, roundBy)
(x * w).toLong.toDouble / w
}
def timed(f: => Unit) = {
val start = System.currentTimeMillis()
f
val end = System.currentTimeMillis()
println("Elapsed Time: " + (end - start))
}
def main(args: Array[String]): Unit = {
print("Java Formatter: ")
val iters = 10000
timed {
(0 until iters) foreach { _ =>
textFormatter(r.nextDouble())
}
}
print("Scala Formatter: ")
timed {
(0 until iters) foreach { _ =>
scalaFormatter(r.nextDouble())
}
}
print("BigDecimal Formatter: ")
timed {
(0 until iters) foreach { _ =>
bigDecimalFormatter(r.nextDouble())
}
}
print("Scala custom Formatter (truncation): ")
timed {
(0 until iters) foreach { _ =>
scalaCustom(r.nextDouble())
}
}
}
}
Another solution (might not work for all types)
import scala.collection.breakOut
val m:Map[P, T] = c.map(t => (t.getP, t))(breakOut)
this avoids the creation of the intermediary list, more info here: Scala 2.8 breakOut
Just to give more perspective to the answers
Spark-shell is a scala repl
You can type :help to see the list of operation that are possible inside the scala shell
scala> :help
All commands can be abbreviated, e.g., :he instead of :help.
:edit <id>|<line> edit history
:help [command] print this summary or command-specific help
:history [num] show the history (optional num is commands to show)
:h? <string> search the history
:imports [name name ...] show import history, identifying sources of names
:implicits [-v] show the implicits in scope
:javap <path|class> disassemble a file or class name
:line <id>|<line> place line(s) at the end of history
:load <path> interpret lines in a file
:paste [-raw] [path] enter paste mode or paste a file
:power enable power user mode
:quit exit the interpreter
:replay [options] reset the repl and replay all previous commands
:require <path> add a jar to the classpath
:reset [options] reset the repl to its initial state, forgetting all session entries
:save <path> save replayable session to a file
:sh <command line> run a shell command (result is implicitly => List[String])
:settings <options> update compiler options, if possible; see reset
:silent disable/enable automatic printing of results
:type [-v] <expr> display the type of an expression without evaluating it
:kind [-v] <expr> display the kind of expression's type
:warnings show the suppressed warnings from the most recent line which had any
:load interpret lines in a file
The map
function is a transformation, which means that Spark will not actually evaluate your RDD until you run an action on it.
To print it, you can use foreach
(which is an action):
linesWithSessionId.foreach(println)
To write it to disk you can use one of the saveAs...
functions (still actions) from the RDD API
And if you didn't want to use strict equality, you could use exists:
myFunction(strings.exists { x => customPredicate(x) })
I wanted to add a concrete example to accompany the other answers
You need to specify the name of the class that you want to test, so if you have the following project (this is a Play project):
You can test just the Login
tests by running the following command from the SBT console:
test:testOnly *LoginServiceSpec
If you are running the command from outside the SBT console, you would do the following:
sbt "test:testOnly *LoginServiceSpec"
df.sort($"ColumnName".desc).show()
Map:
Map transformation.
The map works on a single Row at a time.
Map returns after each input Row.
The map doesn’t hold the output result in Memory.
Map no way to figure out then to end the service.
// map example
val dfList = (1 to 100) toList
val df = dfList.toDF()
val dfInt = df.map(x => x.getInt(0)+2)
display(dfInt)
MapPartition:
MapPartition transformation.
MapPartition works on a partition at a time.
MapPartition returns after processing all the rows in the partition.
MapPartition output is retained in memory, as it can return after processing all the rows in a particular partition.
MapPartition service can be shut down before returning.
// MapPartition example
Val dfList = (1 to 100) toList
Val df = dfList.toDF()
Val df1 = df.repartition(4).rdd.mapPartition((int) => Iterator(itr.length))
Df1.collec()
//display(df1.collect())
For more details, please refer to the Spark map vs mapPartitions transformation article.
Hope this is helpful!
sumr
is implemented in terms of foldRight
:
final def sumr(implicit A: Monoid[A]): A = F.foldRight(self, A.zero)(A.append)
foldRight
is not always tail recursive, so you can overflow the stack if the collection is too long. See Why foldRight and reduceRight are NOT tail recursive? for some more discussion of when this is or isn't true.
To convert the year from string to int, you can add the following option to the csv reader: "inferSchema" -> "true", see DataBricks documentation
As an alternative approach, you can save the SBT Extras script to a file called sbt.sh and set the permission to executable. Then add this file to your path, or just put it under your ~/bin directory.
The bonus here, is that it will download and use the correct version of SBT depending on your project properties. This is a nice convenience if you tend to compile open source projects that you pull from GitHub and other.
You should be using where
, select
is a projection that returns the output of the statement, thus why you get boolean values. where
is a filter that keeps the structure of the dataframe, but only keeps data where the filter works.
Along the same line though, per the documentation, you can write this in 3 different ways
// The following are equivalent:
peopleDf.filter($"age" > 15)
peopleDf.where($"age" > 15)
peopleDf($"age" > 15)
Java 8+
import java.nio.charset.StandardCharsets
import java.nio.file.{Files, Paths}
val path = Paths.get("file.txt")
new String(Files.readAllBytes(path), StandardCharsets.UTF_8)
Java 11+
import java.nio.charset.StandardCharsets
import java.nio.file.{Files, Path}
val path = Path.of("file.txt")
Files.readString(path, StandardCharsets.UTF_8)
These offer control over character encoding, and no resources to clean up. It's also faster than other patterns (e.g. getLines().mkString("\n")
) due to more efficient allocation patterns.
If you need to mutate stuff, use ArrayBuffer
or LinkedBuffer
instead. However, it would be better to address this statement:
I need to declare empty list or empty maps and some where later in the code need to fill them.
Instead of doing that, fill the list with code that returns the elements. There are many ways of doing that, and I'll give some examples:
// Fill a list with the results of calls to a method
val l = List.fill(50)(scala.util.Random.nextInt)
// Fill a list with the results of calls to a method until you get something different
val l = Stream.continually(scala.util.Random.nextInt).takeWhile(x => x > 0).toList
// Fill a list based on its index
val l = List.tabulate(5)(x => x * 2)
// Fill a list of 10 elements based on computations made on the previous element
val l = List.iterate(1, 10)(x => x * 2)
// Fill a list based on computations made on previous element, until you get something
val l = Stream.iterate(0)(x => x * 2 + 1).takeWhile(x => x < 1000).toList
// Fill list based on input from a file
val l = (for (line <- scala.io.Source.fromFile("filename.txt").getLines) yield line.length).toList
We can use the rank() window function (where you would choose the rank = 1) rank just adds a number for every row of a group (in this case it would be the hour)
here's an example. ( from https://github.com/jaceklaskowski/mastering-apache-spark-book/blob/master/spark-sql-functions.adoc#rank )
val dataset = spark.range(9).withColumn("bucket", 'id % 3)
import org.apache.spark.sql.expressions.Window
val byBucket = Window.partitionBy('bucket).orderBy('id)
scala> dataset.withColumn("rank", rank over byBucket).show
+---+------+----+
| id|bucket|rank|
+---+------+----+
| 0| 0| 1|
| 3| 0| 2|
| 6| 0| 3|
| 1| 1| 1|
| 4| 1| 2|
| 7| 1| 3|
| 2| 2| 1|
| 5| 2| 2|
| 8| 2| 3|
+---+------+----+
RDDs extend the Serialisable interface, so this is not what's causing your task to fail. Now this doesn't mean that you can serialise an RDD
with Spark and avoid NotSerializableException
Spark is a distributed computing engine and its main abstraction is a resilient distributed dataset (RDD), which can be viewed as a distributed collection. Basically, RDD's elements are partitioned across the nodes of the cluster, but Spark abstracts this away from the user, letting the user interact with the RDD (collection) as if it were a local one.
Not to get into too many details, but when you run different transformations on a RDD (map
, flatMap
, filter
and others), your transformation code (closure) is:
You can of course run this locally (as in your example), but all those phases (apart from shipping over network) still occur. [This lets you catch any bugs even before deploying to production]
What happens in your second case is that you are calling a method, defined in class testing
from inside the map function. Spark sees that and since methods cannot be serialized on their own, Spark tries to serialize the whole testing
class, so that the code will still work when executed in another JVM. You have two possibilities:
Either you make class testing serializable, so the whole class can be serialized by Spark:
import org.apache.spark.{SparkContext,SparkConf}
object Spark {
val ctx = new SparkContext(new SparkConf().setAppName("test").setMaster("local[*]"))
}
object NOTworking extends App {
new Test().doIT
}
class Test extends java.io.Serializable {
val rddList = Spark.ctx.parallelize(List(1,2,3))
def doIT() = {
val after = rddList.map(someFunc)
after.collect().foreach(println)
}
def someFunc(a: Int) = a + 1
}
or you make someFunc
function instead of a method (functions are objects in Scala), so that Spark will be able to serialize it:
import org.apache.spark.{SparkContext,SparkConf}
object Spark {
val ctx = new SparkContext(new SparkConf().setAppName("test").setMaster("local[*]"))
}
object NOTworking extends App {
new Test().doIT
}
class Test {
val rddList = Spark.ctx.parallelize(List(1,2,3))
def doIT() = {
val after = rddList.map(someFunc)
after.collect().foreach(println)
}
val someFunc = (a: Int) => a + 1
}
Similar, but not the same problem with class serialization can be of interest to you and you can read on it in this Spark Summit 2013 presentation.
As a side note, you can rewrite rddList.map(someFunc(_))
to rddList.map(someFunc)
, they are exactly the same. Usually, the second is preferred as it's less verbose and cleaner to read.
EDIT (2015-03-15): SPARK-5307 introduced SerializationDebugger and Spark 1.3.0 is the first version to use it. It adds serialization path to a NotSerializableException. When a NotSerializableException is encountered, the debugger visits the object graph to find the path towards the object that cannot be serialized, and constructs information to help user to find the object.
In OP's case, this is what gets printed to stdout:
Serialization stack:
- object not serializable (class: testing, value: testing@2dfe2f00)
- field (class: testing$$anonfun$1, name: $outer, type: class testing)
- object (class testing$$anonfun$1, <function1>)
Some more ways to iterate:
scala> xs.foreach (println)
first
second
third
foreach, and similar, map, which would return something (the results of the function, which is, for println, Unit, so a List of Units)
scala> val lens = for (x <- xs) yield (x.length)
lens: Array[Int] = Array(5, 6, 5)
work with the elements, not the index
scala> ("" /: xs) (_ + _)
res21: java.lang.String = firstsecondthird
folding
for(int i=0, j=0; i+j<100; i+=j*2, j+=i+2) {...}
can be done with recursion:
def ijIter (i: Int = 0, j: Int = 0, carry: Int = 0) : Int =
if (i + j >= 100) carry else
ijIter (i+2*j, j+i+2, carry / 3 + 2 * i - 4 * j + 10)
The carry-part is just some example, to do something with i and j. It needn't be an Int.
for simpler stuff, closer to usual for-loops:
scala> (1 until 4)
res43: scala.collection.immutable.Range with scala.collection.immutable.Range.ByOne = Range(1, 2, 3)
scala> (0 to 8 by 2)
res44: scala.collection.immutable.Range = Range(0, 2, 4, 6, 8)
scala> (26 to 13 by -3)
res45: scala.collection.immutable.Range = Range(26, 23, 20, 17, 14)
or without order:
List (1, 3, 2, 5, 9, 7).foreach (print)
Apart from my above answer I tried to demonstrate all the spark joins with same case classes using spark 2.x here is my linked in article with full examples and explanation .
All join types : Default inner
. Must be one of:
inner
, cross
, outer
, full
, full_outer
, left
, left_outer
, right
, right_outer
, left_semi
, left_anti
.
import org.apache.spark.sql._
import org.apache.spark.sql.functions._
/**
* @author : Ram Ghadiyaram
*/
object SparkJoinTypesDemo extends App {
private[this] implicit val spark = SparkSession.builder().master("local[*]").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
case class Person(name: String, age: Int, personid: Int)
case class Profile(profileName: String, personid: Int, profileDescription: String)
/**
* * @param joinType Type of join to perform. Default `inner`. Must be one of:
* * `inner`, `cross`, `outer`, `full`, `full_outer`, `left`, `left_outer`,
* * `right`, `right_outer`, `left_semi`, `left_anti`.
*/
val joinTypes = Seq(
"inner"
, "outer"
, "full"
, "full_outer"
, "left"
, "left_outer"
, "right"
, "right_outer"
, "left_semi"
, "left_anti"
//, "cross"
)
val df1 = spark.sqlContext.createDataFrame(
Person("Nataraj", 45, 2)
:: Person("Srinivas", 45, 5)
:: Person("Ashik", 22, 9)
:: Person("Deekshita", 22, 8)
:: Person("Siddhika", 22, 4)
:: Person("Madhu", 22, 3)
:: Person("Meghna", 22, 2)
:: Person("Snigdha", 22, 2)
:: Person("Harshita", 22, 6)
:: Person("Ravi", 42, 0)
:: Person("Ram", 42, 9)
:: Person("Chidananda Raju", 35, 9)
:: Person("Sreekanth Doddy", 29, 9)
:: Nil)
val df2 = spark.sqlContext.createDataFrame(
Profile("Spark", 2, "SparkSQLMaster")
:: Profile("Spark", 5, "SparkGuru")
:: Profile("Spark", 9, "DevHunter")
:: Profile("Spark", 3, "Evangelist")
:: Profile("Spark", 0, "Committer")
:: Profile("Spark", 1, "All Rounder")
:: Nil
)
val df_asPerson = df1.as("dfperson")
val df_asProfile = df2.as("dfprofile")
val joined_df = df_asPerson.join(
df_asProfile
, col("dfperson.personid") === col("dfprofile.personid")
, "inner")
println("First example inner join ")
// you can do alias to refer column name with aliases to increase readability
joined_df.select(
col("dfperson.name")
, col("dfperson.age")
, col("dfprofile.profileName")
, col("dfprofile.profileDescription"))
.show
println("all joins in a loop")
joinTypes foreach { joinType =>
println(s"${joinType.toUpperCase()} JOIN")
df_asPerson.join(right = df_asProfile, usingColumns = Seq("personid"), joinType = joinType)
.orderBy("personid")
.show()
}
println(
"""
|Till 1.x cross join is : df_asPerson.join(df_asProfile)
|
| Explicit Cross Join in 2.x :
| http://blog.madhukaraphatak.com/migrating-to-spark-two-part-4/
| Cartesian joins are very expensive without an extra filter that can be pushed down.
|
| cross join or cartesian product
|
|
""".stripMargin)
val crossJoinDf = df_asPerson.crossJoin(right = df_asProfile)
crossJoinDf.show(200, false)
println(crossJoinDf.explain())
println(crossJoinDf.count)
println("createOrReplaceTempView example ")
println(
"""
|Creates a local temporary view using the given name. The lifetime of this
| temporary view is tied to the [[SparkSession]] that was used to create this Dataset.
""".stripMargin)
df_asPerson.createOrReplaceTempView("dfperson");
df_asProfile.createOrReplaceTempView("dfprofile")
val sql =
s"""
|SELECT dfperson.name
|, dfperson.age
|, dfprofile.profileDescription
| FROM dfperson JOIN dfprofile
| ON dfperson.personid == dfprofile.personid
""".stripMargin
println(s"createOrReplaceTempView sql $sql")
val sqldf = spark.sql(sql)
sqldf.show
println(
"""
|
|**** EXCEPT DEMO ***
|
""".stripMargin)
println(" df_asPerson.except(df_asProfile) Except demo")
df_asPerson.except(df_asProfile).show
println(" df_asProfile.except(df_asPerson) Except demo")
df_asProfile.except(df_asPerson).show
}
Result :
First example inner join +---------------+---+-----------+------------------+ | name|age|profileName|profileDescription| +---------------+---+-----------+------------------+ | Nataraj| 45| Spark| SparkSQLMaster| | Srinivas| 45| Spark| SparkGuru| | Ashik| 22| Spark| DevHunter| | Madhu| 22| Spark| Evangelist| | Meghna| 22| Spark| SparkSQLMaster| | Snigdha| 22| Spark| SparkSQLMaster| | Ravi| 42| Spark| Committer| | Ram| 42| Spark| DevHunter| |Chidananda Raju| 35| Spark| DevHunter| |Sreekanth Doddy| 29| Spark| DevHunter| +---------------+---+-----------+------------------+ all joins in a loop INNER JOIN +--------+---------------+---+-----------+------------------+ |personid| name|age|profileName|profileDescription| +--------+---------------+---+-----------+------------------+ | 0| Ravi| 42| Spark| Committer| | 2| Snigdha| 22| Spark| SparkSQLMaster| | 2| Meghna| 22| Spark| SparkSQLMaster| | 2| Nataraj| 45| Spark| SparkSQLMaster| | 3| Madhu| 22| Spark| Evangelist| | 5| Srinivas| 45| Spark| SparkGuru| | 9| Ram| 42| Spark| DevHunter| | 9| Ashik| 22| Spark| DevHunter| | 9|Chidananda Raju| 35| Spark| DevHunter| | 9|Sreekanth Doddy| 29| Spark| DevHunter| +--------+---------------+---+-----------+------------------+ OUTER JOIN +--------+---------------+----+-----------+------------------+ |personid| name| age|profileName|profileDescription| +--------+---------------+----+-----------+------------------+ | 0| Ravi| 42| Spark| Committer| | 1| null|null| Spark| All Rounder| | 2| Nataraj| 45| Spark| SparkSQLMaster| | 2| Snigdha| 22| Spark| SparkSQLMaster| | 2| Meghna| 22| Spark| SparkSQLMaster| | 3| Madhu| 22| Spark| Evangelist| | 4| Siddhika| 22| null| null| | 5| Srinivas| 45| Spark| SparkGuru| | 6| Harshita| 22| null| null| | 8| Deekshita| 22| null| null| | 9| Ashik| 22| Spark| DevHunter| | 9| Ram| 42| Spark| DevHunter| | 9|Chidananda Raju| 35| Spark| DevHunter| | 9|Sreekanth Doddy| 29| Spark| DevHunter| +--------+---------------+----+-----------+------------------+ FULL JOIN +--------+---------------+----+-----------+------------------+ |personid| name| age|profileName|profileDescription| +--------+---------------+----+-----------+------------------+ | 0| Ravi| 42| Spark| Committer| | 1| null|null| Spark| All Rounder| | 2| Nataraj| 45| Spark| SparkSQLMaster| | 2| Meghna| 22| Spark| SparkSQLMaster| | 2| Snigdha| 22| Spark| SparkSQLMaster| | 3| Madhu| 22| Spark| Evangelist| | 4| Siddhika| 22| null| null| | 5| Srinivas| 45| Spark| SparkGuru| | 6| Harshita| 22| null| null| | 8| Deekshita| 22| null| null| | 9| Ashik| 22| Spark| DevHunter| | 9| Ram| 42| Spark| DevHunter| | 9|Sreekanth Doddy| 29| Spark| DevHunter| | 9|Chidananda Raju| 35| Spark| DevHunter| +--------+---------------+----+-----------+------------------+ FULL_OUTER JOIN +--------+---------------+----+-----------+------------------+ |personid| name| age|profileName|profileDescription| +--------+---------------+----+-----------+------------------+ | 0| Ravi| 42| Spark| Committer| | 1| null|null| Spark| All Rounder| | 2| Nataraj| 45| Spark| SparkSQLMaster| | 2| Meghna| 22| Spark| SparkSQLMaster| | 2| Snigdha| 22| Spark| SparkSQLMaster| | 3| Madhu| 22| Spark| Evangelist| | 4| Siddhika| 22| null| null| | 5| Srinivas| 45| Spark| SparkGuru| | 6| Harshita| 22| null| null| | 8| Deekshita| 22| null| null| | 9| Ashik| 22| Spark| DevHunter| | 9| Ram| 42| Spark| DevHunter| | 9|Chidananda Raju| 35| Spark| DevHunter| | 9|Sreekanth Doddy| 29| Spark| DevHunter| +--------+---------------+----+-----------+------------------+ LEFT JOIN +--------+---------------+---+-----------+------------------+ |personid| name|age|profileName|profileDescription| +--------+---------------+---+-----------+------------------+ | 0| Ravi| 42| Spark| Committer| | 2| Snigdha| 22| Spark| SparkSQLMaster| | 2| Meghna| 22| Spark| SparkSQLMaster| | 2| Nataraj| 45| Spark| SparkSQLMaster| | 3| Madhu| 22| Spark| Evangelist| | 4| Siddhika| 22| null| null| | 5| Srinivas| 45| Spark| SparkGuru| | 6| Harshita| 22| null| null| | 8| Deekshita| 22| null| null| | 9| Ram| 42| Spark| DevHunter| | 9| Ashik| 22| Spark| DevHunter| | 9|Chidananda Raju| 35| Spark| DevHunter| | 9|Sreekanth Doddy| 29| Spark| DevHunter| +--------+---------------+---+-----------+------------------+ LEFT_OUTER JOIN +--------+---------------+---+-----------+------------------+ |personid| name|age|profileName|profileDescription| +--------+---------------+---+-----------+------------------+ | 0| Ravi| 42| Spark| Committer| | 2| Nataraj| 45| Spark| SparkSQLMaster| | 2| Meghna| 22| Spark| SparkSQLMaster| | 2| Snigdha| 22| Spark| SparkSQLMaster| | 3| Madhu| 22| Spark| Evangelist| | 4| Siddhika| 22| null| null| | 5| Srinivas| 45| Spark| SparkGuru| | 6| Harshita| 22| null| null| | 8| Deekshita| 22| null| null| | 9|Chidananda Raju| 35| Spark| DevHunter| | 9|Sreekanth Doddy| 29| Spark| DevHunter| | 9| Ashik| 22| Spark| DevHunter| | 9| Ram| 42| Spark| DevHunter| +--------+---------------+---+-----------+------------------+ RIGHT JOIN +--------+---------------+----+-----------+------------------+ |personid| name| age|profileName|profileDescription| +--------+---------------+----+-----------+------------------+ | 0| Ravi| 42| Spark| Committer| | 1| null|null| Spark| All Rounder| | 2| Snigdha| 22| Spark| SparkSQLMaster| | 2| Meghna| 22| Spark| SparkSQLMaster| | 2| Nataraj| 45| Spark| SparkSQLMaster| | 3| Madhu| 22| Spark| Evangelist| | 5| Srinivas| 45| Spark| SparkGuru| | 9|Sreekanth Doddy| 29| Spark| DevHunter| | 9|Chidananda Raju| 35| Spark| DevHunter| | 9| Ram| 42| Spark| DevHunter| | 9| Ashik| 22| Spark| DevHunter| +--------+---------------+----+-----------+------------------+ RIGHT_OUTER JOIN +--------+---------------+----+-----------+------------------+ |personid| name| age|profileName|profileDescription| +--------+---------------+----+-----------+------------------+ | 0| Ravi| 42| Spark| Committer| | 1| null|null| Spark| All Rounder| | 2| Meghna| 22| Spark| SparkSQLMaster| | 2| Snigdha| 22| Spark| SparkSQLMaster| | 2| Nataraj| 45| Spark| SparkSQLMaster| | 3| Madhu| 22| Spark| Evangelist| | 5| Srinivas| 45| Spark| SparkGuru| | 9|Sreekanth Doddy| 29| Spark| DevHunter| | 9| Ashik| 22| Spark| DevHunter| | 9|Chidananda Raju| 35| Spark| DevHunter| | 9| Ram| 42| Spark| DevHunter| +--------+---------------+----+-----------+------------------+ LEFT_SEMI JOIN +--------+---------------+---+ |personid| name|age| +--------+---------------+---+ | 0| Ravi| 42| | 2| Nataraj| 45| | 2| Meghna| 22| | 2| Snigdha| 22| | 3| Madhu| 22| | 5| Srinivas| 45| | 9|Chidananda Raju| 35| | 9|Sreekanth Doddy| 29| | 9| Ram| 42| | 9| Ashik| 22| +--------+---------------+---+ LEFT_ANTI JOIN +--------+---------+---+ |personid| name|age| +--------+---------+---+ | 4| Siddhika| 22| | 6| Harshita| 22| | 8|Deekshita| 22| +--------+---------+---+ Till 1.x Cross join is : `df_asPerson.join(df_asProfile)` Explicit Cross Join in 2.x : http://blog.madhukaraphatak.com/migrating-to-spark-two-part-4/ Cartesian joins are very expensive without an extra filter that can be pushed down. Cross join or Cartesian product +---------------+---+--------+-----------+--------+------------------+ |name |age|personid|profileName|personid|profileDescription| +---------------+---+--------+-----------+--------+------------------+ |Nataraj |45 |2 |Spark |2 |SparkSQLMaster | |Nataraj |45 |2 |Spark |5 |SparkGuru | |Nataraj |45 |2 |Spark |9 |DevHunter | |Nataraj |45 |2 |Spark |3 |Evangelist | |Nataraj |45 |2 |Spark |0 |Committer | |Nataraj |45 |2 |Spark |1 |All Rounder | |Srinivas |45 |5 |Spark |2 |SparkSQLMaster | |Srinivas |45 |5 |Spark |5 |SparkGuru | |Srinivas |45 |5 |Spark |9 |DevHunter | |Srinivas |45 |5 |Spark |3 |Evangelist | |Srinivas |45 |5 |Spark |0 |Committer | |Srinivas |45 |5 |Spark |1 |All Rounder | |Ashik |22 |9 |Spark |2 |SparkSQLMaster | |Ashik |22 |9 |Spark |5 |SparkGuru | |Ashik |22 |9 |Spark |9 |DevHunter | |Ashik |22 |9 |Spark |3 |Evangelist | |Ashik |22 |9 |Spark |0 |Committer | |Ashik |22 |9 |Spark |1 |All Rounder | |Deekshita |22 |8 |Spark |2 |SparkSQLMaster | |Deekshita |22 |8 |Spark |5 |SparkGuru | |Deekshita |22 |8 |Spark |9 |DevHunter | |Deekshita |22 |8 |Spark |3 |Evangelist | |Deekshita |22 |8 |Spark |0 |Committer | |Deekshita |22 |8 |Spark |1 |All Rounder | |Siddhika |22 |4 |Spark |2 |SparkSQLMaster | |Siddhika |22 |4 |Spark |5 |SparkGuru | |Siddhika |22 |4 |Spark |9 |DevHunter | |Siddhika |22 |4 |Spark |3 |Evangelist | |Siddhika |22 |4 |Spark |0 |Committer | |Siddhika |22 |4 |Spark |1 |All Rounder | |Madhu |22 |3 |Spark |2 |SparkSQLMaster | |Madhu |22 |3 |Spark |5 |SparkGuru | |Madhu |22 |3 |Spark |9 |DevHunter | |Madhu |22 |3 |Spark |3 |Evangelist | |Madhu |22 |3 |Spark |0 |Committer | |Madhu |22 |3 |Spark |1 |All Rounder | |Meghna |22 |2 |Spark |2 |SparkSQLMaster | |Meghna |22 |2 |Spark |5 |SparkGuru | |Meghna |22 |2 |Spark |9 |DevHunter | |Meghna |22 |2 |Spark |3 |Evangelist | |Meghna |22 |2 |Spark |0 |Committer | |Meghna |22 |2 |Spark |1 |All Rounder | |Snigdha |22 |2 |Spark |2 |SparkSQLMaster | |Snigdha |22 |2 |Spark |5 |SparkGuru | |Snigdha |22 |2 |Spark |9 |DevHunter | |Snigdha |22 |2 |Spark |3 |Evangelist | |Snigdha |22 |2 |Spark |0 |Committer | |Snigdha |22 |2 |Spark |1 |All Rounder | |Harshita |22 |6 |Spark |2 |SparkSQLMaster | |Harshita |22 |6 |Spark |5 |SparkGuru | |Harshita |22 |6 |Spark |9 |DevHunter | |Harshita |22 |6 |Spark |3 |Evangelist | |Harshita |22 |6 |Spark |0 |Committer | |Harshita |22 |6 |Spark |1 |All Rounder | |Ravi |42 |0 |Spark |2 |SparkSQLMaster | |Ravi |42 |0 |Spark |5 |SparkGuru | |Ravi |42 |0 |Spark |9 |DevHunter | |Ravi |42 |0 |Spark |3 |Evangelist | |Ravi |42 |0 |Spark |0 |Committer | |Ravi |42 |0 |Spark |1 |All Rounder | |Ram |42 |9 |Spark |2 |SparkSQLMaster | |Ram |42 |9 |Spark |5 |SparkGuru | |Ram |42 |9 |Spark |9 |DevHunter | |Ram |42 |9 |Spark |3 |Evangelist | |Ram |42 |9 |Spark |0 |Committer | |Ram |42 |9 |Spark |1 |All Rounder | |Chidananda Raju|35 |9 |Spark |2 |SparkSQLMaster | |Chidananda Raju|35 |9 |Spark |5 |SparkGuru | |Chidananda Raju|35 |9 |Spark |9 |DevHunter | |Chidananda Raju|35 |9 |Spark |3 |Evangelist | |Chidananda Raju|35 |9 |Spark |0 |Committer | |Chidananda Raju|35 |9 |Spark |1 |All Rounder | |Sreekanth Doddy|29 |9 |Spark |2 |SparkSQLMaster | |Sreekanth Doddy|29 |9 |Spark |5 |SparkGuru | |Sreekanth Doddy|29 |9 |Spark |9 |DevHunter | |Sreekanth Doddy|29 |9 |Spark |3 |Evangelist | |Sreekanth Doddy|29 |9 |Spark |0 |Committer | |Sreekanth Doddy|29 |9 |Spark |1 |All Rounder | +---------------+---+--------+-----------+--------+------------------+ == Physical Plan == BroadcastNestedLoopJoin BuildRight, Cross :- LocalTableScan [name#0, age#1, personid#2] +- BroadcastExchange IdentityBroadcastMode +- LocalTableScan [profileName#7, personid#8, profileDescription#9] () 78 createOrReplaceTempView example Creates a local temporary view using the given name. The lifetime of this temporary view is tied to the [[SparkSession]] that was used to create this Dataset. createOrReplaceTempView sql SELECT dfperson.name , dfperson.age , dfprofile.profileDescription FROM dfperson JOIN dfprofile ON dfperson.personid == dfprofile.personid +---------------+---+------------------+ | name|age|profileDescription| +---------------+---+------------------+ | Nataraj| 45| SparkSQLMaster| | Srinivas| 45| SparkGuru| | Ashik| 22| DevHunter| | Madhu| 22| Evangelist| | Meghna| 22| SparkSQLMaster| | Snigdha| 22| SparkSQLMaster| | Ravi| 42| Committer| | Ram| 42| DevHunter| |Chidananda Raju| 35| DevHunter| |Sreekanth Doddy| 29| DevHunter| +---------------+---+------------------+ **** EXCEPT DEMO *** df_asPerson.except(df_asProfile) Except demo +---------------+---+--------+ | name|age|personid| +---------------+---+--------+ | Ashik| 22| 9| | Harshita| 22| 6| | Madhu| 22| 3| | Ram| 42| 9| | Ravi| 42| 0| |Chidananda Raju| 35| 9| | Siddhika| 22| 4| | Srinivas| 45| 5| |Sreekanth Doddy| 29| 9| | Deekshita| 22| 8| | Meghna| 22| 2| | Snigdha| 22| 2| | Nataraj| 45| 2| +---------------+---+--------+ df_asProfile.except(df_asPerson) Except demo +-----------+--------+------------------+ |profileName|personid|profileDescription| +-----------+--------+------------------+ | Spark| 5| SparkGuru| | Spark| 9| DevHunter| | Spark| 2| SparkSQLMaster| | Spark| 3| Evangelist| | Spark| 0| Committer| | Spark| 1| All Rounder| +-----------+--------+------------------+
As discussed above these are the venn diagrams of all the joins.
I know I am quite late to answer this but I have come up with a different answer:
val rdd = sc.textFile("/home/training/mydata/file.txt")
val text = rdd.map(lines=lines.split(",")).map(arrays=>(ararys(0),arrays(1))).toDF("id","name").show
tow table join not rename the joined key
// method 1: create a new DF
day1 = day1.toDF(day1.columns.map(x => if (x.equals(key)) x else s"${x}_d1"): _*)
// method 2: use withColumnRenamed
for ((x, y) <- day1.columns.filter(!_.equals(key)).map(x => (x, s"${x}_d1"))) {
day1 = day1.withColumnRenamed(x, y)
}
works!
I think overall all the answers have given a semantic explanation about classes and case classes. This could be very much relevant, but every newbie in scala should know what happens when you create a case class. I have written this answer, which explains case class in a nutshell.
Every programmer should know that if they are using any pre-built functions, then they are writing a comparatively less code, which is enabling them by giving the power to write most optimized code, but power comes with great responsibilities. So, use prebuilt functions with very cautions.
Some developers avoid writing case classes due to additional 20 methods, which you can see by disassembling class file.
Please refer this link if you want to check all the methods inside a case class.
Java and C:
You may find more information in Android developers site.
While using embedded jetty the property webapp.resource.loader.path should starts with slash:
webapp.resource.loader.path=/templates
otherwise templates will not be found in ../webapp/templates
1) Download winutils.exe from https://github.com/steveloughran/winutils
2) Create a directory In windows "C:\winutils\bin
3) Copy the winutils.exe inside the above bib folder .
4) Set the environmental property in the code
System.setProperty("hadoop.home.dir", "file:///C:/winutils/");
5) Create a folder "file:///C:/temp" and give 777 permissions.
6) Add config property in spark Session ".config("spark.sql.warehouse.dir", "file:///C:/temp")"
There is one more way to use Java
import java.io._
def printToFile(f: java.io.File)(op: java.io.PrintWriter => Unit)
{
val p = new java.io.PrintWriter(f);
try { op(p) }
finally { p.close() }
}
printToFile(new File("C:/TEMP/df.csv")) { p => df.collect().foreach(p.println)}
use further as
rs.getString(1);
rs.getInt(2);
1, 2 is the column number of table and set int or string as per data-type of coloumn
If you want to split you dataframe into two different ones, do two selects on it with the different columns you want.
val sourceDf = spark.read.csv(...)
val df1 = sourceDF.select("first column", "second column", "third column")
val df2 = sourceDF.select("first column", "second column", "third column")
Note that this of course means that the sourceDf would be evaluated twice, so if it can fit into distributed memory and you use most of the columns across both dataframes it might be a good idea to cache it. It it has many extra columns that you don't need, then you can do a select on it first to select on the columns you will need so it would store all that extra data in memory.
This should return the collection containing single list:
dataFrame.select("YOUR_COLUMN_NAME").rdd.map(r => r(0)).collect()
Without the mapping, you just get a Row object, which contains every column from the database.
Keep in mind that this will probably get you a list of Any type. Ïf you want to specify the result type, you can use .asInstanceOf[YOUR_TYPE] in r => r(0).asInstanceOf[YOUR_TYPE]
mapping
P.S. due to automatic conversion you can skip the .rdd
part.
You must first import the functions:
import org.apache.spark.sql.functions._
Then you can use them like this:
val df = CSV.load(args(0))
val sumSteps = df.agg(sum("steps")).first.get(0)
You can also cast the result if needed:
val sumSteps: Long = df.agg(sum("steps").cast("long")).first.getLong(0)
Edit:
For multiple columns (e.g. "col1", "col2", ...), you could get all aggregations at once:
val sums = df.agg(sum("col1").as("sum_col1"), sum("col2").as("sum_col2"), ...).first
Edit2:
For dynamically applying the aggregations, the following options are available:
df.groupBy().sum()
val columnNames = List("col1", "col2")
df.groupBy().sum(columnNames: _*)
val cols = List("col1", "col2")
val sums = cols.map(colName => sum(colName).cast("double").as("sum_" + colName))
df.groupBy().agg(sums.head, sums.tail:_*).show()
I hope it's not a "suicide note", but I can see your point. You hit on what is at the same time both a strength and a problem of Scala: its extensibility. This lets us implement most major functionality in libraries. In some other languages, sequences with something like map
or collect
would be built in, and nobody has to see all the hoops the compiler has to go through to make them work smoothly. In Scala, it's all in a library, and therefore out in the open.
In fact the functionality of map
that's supported by its complicated type is pretty advanced. Consider this:
scala> import collection.immutable.BitSet
import collection.immutable.BitSet
scala> val bits = BitSet(1, 2, 3)
bits: scala.collection.immutable.BitSet = BitSet(1, 2, 3)
scala> val shifted = bits map { _ + 1 }
shifted: scala.collection.immutable.BitSet = BitSet(2, 3, 4)
scala> val displayed = bits map { _.toString + "!" }
displayed: scala.collection.immutable.Set[java.lang.String] = Set(1!, 2!, 3!)
See how you always get the best possible type? If you map Int
s to Int
s you get again a BitSet
, but if you map Int
s to String
s, you get a general Set
. Both the static type and the runtime representation of map's result depend on the result type of the function that's passed to it. And this works even if the set is empty, so the function is never applied! As far as I know there is no other collection framework with an equivalent functionality. Yet from a user perspective this is how things are supposed to work.
The problem we have is that all the clever technology that makes this happen leaks into the type signatures which become large and scary. But maybe a user should not be shown by default the full type signature of map
? How about if she looked up map
in BitSet
she got:
map(f: Int => Int): BitSet (click here for more general type)
The docs would not lie in that case, because from a user perspective indeed map has the type (Int => Int) => BitSet
. But map
also has a more general type which can be inspected by clicking on another link.
We have not yet implemented functionality like this in our tools. But I believe we need to do this, to avoid scaring people off and to give more useful info. With tools like that, hopefully smart frameworks and libraries will not become suicide notes.
In Java 1.8 This code snippet perfectly working to read CSV files
POM.xml
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>2.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.8</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-csv_2.10</artifactId>
<version>1.4.0</version>
</dependency>
Java
SparkConf conf = new SparkConf().setAppName("JavaWordCount").setMaster("local");
// create Spark Context
SparkContext context = new SparkContext(conf);
// create spark Session
SparkSession sparkSession = new SparkSession(context);
Dataset<Row> df = sparkSession.read().format("com.databricks.spark.csv").option("header", true).option("inferSchema", true).load("hdfs://localhost:9000/usr/local/hadoop_data/loan_100.csv");
//("hdfs://localhost:9000/usr/local/hadoop_data/loan_100.csv");
System.out.println("========== Print Schema ============");
df.printSchema();
System.out.println("========== Print Data ==============");
df.show();
System.out.println("========== Print title ==============");
df.select("title").show();
df.where(df.col("friend_id").isNull)
A lazy val is most easily understood as a "memoized (no-arg) def".
Like a def, a lazy val is not evaluated until it is invoked. But the result is saved so that subsequent invocations return the saved value. The memoized result takes up space in your data structure, like a val.
As others have mentioned, the use cases for a lazy val are to defer expensive computations until they are needed and store their results, and to solve certain circular dependencies between values.
Lazy vals are in fact implemented more or less as memoized defs. You can read about the details of their implementation here:
http://docs.scala-lang.org/sips/pending/improved-lazy-val-initialization.html
i have tested that and it worked
val x = 9
def printType[T](x:T) :Unit = {println(x.getClass.toString())}
An object has exactly one instance (you can not call new MyObject
). You can have multiple instances of a class.
Object serves the same (and some additional) purposes as the static methods and fields in Java.
To create a list of string, use the following:
val l = List("is", "am", "are", "if")
In pyspark,SparkSql syntax:
where column_n like 'xyz%'
might not work.
Use:
where column_n RLIKE '^xyz'
This works perfectly fine.
Also, in the above case there should be only one
implicit function whose type is double => Int
. Otherwise, the compiler gets confused and won't compile properly.
//this won't compile
implicit def doubleToInt(d: Double) = d.toInt
implicit def doubleToIntSecond(d: Double) = d.toInt
val x: Int = 42.0
This code works perfectly from Spark 2.x with Scala 2.11
Import necessary classes
import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.types.{DoubleType, StringType, StructField, StructType}
Create SparkSession
Object, and Here it's spark
val spark: SparkSession = SparkSession.builder.master("local").getOrCreate
val sc = spark.sparkContext // Just used to create test RDDs
Let's an RDD
to make it DataFrame
val rdd = sc.parallelize(
Seq(
("first", Array(2.0, 1.0, 2.1, 5.4)),
("test", Array(1.5, 0.5, 0.9, 3.7)),
("choose", Array(8.0, 2.9, 9.1, 2.5))
)
)
Using SparkSession.createDataFrame(RDD obj)
.
val dfWithoutSchema = spark.createDataFrame(rdd)
dfWithoutSchema.show()
+------+--------------------+
| _1| _2|
+------+--------------------+
| first|[2.0, 1.0, 2.1, 5.4]|
| test|[1.5, 0.5, 0.9, 3.7]|
|choose|[8.0, 2.9, 9.1, 2.5]|
+------+--------------------+
Using SparkSession.createDataFrame(RDD obj)
and specifying column names.
val dfWithSchema = spark.createDataFrame(rdd).toDF("id", "vals")
dfWithSchema.show()
+------+--------------------+
| id| vals|
+------+--------------------+
| first|[2.0, 1.0, 2.1, 5.4]|
| test|[1.5, 0.5, 0.9, 3.7]|
|choose|[8.0, 2.9, 9.1, 2.5]|
+------+--------------------+
This way requires the input rdd
should be of type RDD[Row]
.
val rowsRdd: RDD[Row] = sc.parallelize(
Seq(
Row("first", 2.0, 7.0),
Row("second", 3.5, 2.5),
Row("third", 7.0, 5.9)
)
)
create the schema
val schema = new StructType()
.add(StructField("id", StringType, true))
.add(StructField("val1", DoubleType, true))
.add(StructField("val2", DoubleType, true))
Now apply both rowsRdd
and schema
to createDataFrame()
val df = spark.createDataFrame(rowsRdd, schema)
df.show()
+------+----+----+
| id|val1|val2|
+------+----+----+
| first| 2.0| 7.0|
|second| 3.5| 2.5|
| third| 7.0| 5.9|
+------+----+----+
Thanks to the answer by @Nulu, it works for pyspark with minimal tweaking
from pyspark.sql.types import LongType, StringType, StructField, StructType, BooleanType, ArrayType, IntegerType
customSchema = StructType(Array(
StructField("project", StringType, true),
StructField("article", StringType, true),
StructField("requests", IntegerType, true),
StructField("bytes_served", DoubleType, true)))
pagecount = sc.read.format("com.databricks.spark.csv")
.option("delimiter"," ")
.option("quote","")
.option("header", "false")
.schema(customSchema)
.load("dbfs:/databricks-datasets/wikipedia-datasets/data-001/pagecounts/sample/pagecounts-20151124-170000")
The keyword yield
in Scala is simply syntactic sugar which can be easily replaced by a map
, as Daniel Sobral already explained in detail.
On the other hand, yield
is absolutely misleading if you are looking for generators (or continuations) similar to those in Python. See this SO thread for more information: What is the preferred way to implement 'yield' in Scala?
The nearest equivalents would be icode and bcode as used by scalac, view Miguel Garcia's site on the Scalac optimiser for more information, here: http://magarciaepfl.github.io/scala/
You might also consider Java bytecode itself to be your intermediate representation, given that bytecode is the ultimate output of scalac.
Or perhaps the true intermediate is something that the JIT produces before it finally outputs native instructions?
Ultimately though... There's no single place that you can point at an claim "there's the intermediate!". Scalac works in phases that successively change the abstract syntax tree, every single phase produces a new intermediate. The whole thing is like an onion, and it's very hard to try and pick out one layer as somehow being more significant than any other.
You have to call close()
on the GZIPOutputStream
before you attempt to read it. The final bytes of the file will only be written when the file is actually closed. (This is irrespective of any explicit buffering in the output stack. The stream only knows to compress and write the last bytes when you tell it to close. A flush()
probably won't help ... though calling finish()
instead of close()
should work. Look at the javadocs.)
Here's the correct code (in Java);
package test;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
public class GZipTest {
public static void main(String[] args) throws
FileNotFoundException, IOException {
String name = "/tmp/test";
GZIPOutputStream gz = new GZIPOutputStream(new FileOutputStream(name));
gz.write(10);
gz.close(); // Remove this to reproduce the reported bug
System.out.println(new GZIPInputStream(new FileInputStream(name)).read());
}
}
(I've not implemented resource management or exception handling / reporting properly as they are not relevant to the purpose of this code. Don't treat this as an example of "good code".)
Other configurable Spark option relating to jars and classpath, in case of yarn
as deploy mode are as follows
From the spark documentation,
spark.yarn.jars
List of libraries containing Spark code to distribute to YARN containers. By default, Spark on YARN will use Spark jars installed locally, but the Spark jars can also be in a world-readable location on HDFS. This allows YARN to cache it on nodes so that it doesn't need to be distributed each time an application runs. To point to jars on HDFS, for example, set this configuration to hdfs:///some/path. Globs are allowed.
spark.yarn.archive
An archive containing needed Spark jars for distribution to the YARN cache. If set, this configuration replaces spark.yarn.jars and the archive is used in all the application's containers. The archive should contain jar files in its root directory. Like with the previous option, the archive can also be hosted on HDFS to speed up file distribution.
Users can configure this parameter to specify their jars, which inturn gets included in Spark driver's classpath.
I think the biggest advantage of having case classes
over enumerations
is that you can use type class pattern a.k.a ad-hoc polymorphysm. Don't need to match enums like:
someEnum match {
ENUMA => makeThis()
ENUMB => makeThat()
}
instead you'll have something like:
def someCode[SomeCaseClass](implicit val maker: Maker[SomeCaseClass]){
maker.make()
}
implicit val makerA = new Maker[CaseClassA]{
def make() = ...
}
implicit val makerB = new Maker[CaseClassB]{
def make() = ...
}
Just adding to the other excellent answers. Scala offers two often criticized symbolic operators, /:
(foldLeft
) and :\
(foldRight
) operators, the first being right-associative. So the following three statements are the equivalent:
( 1 to 100 ).foldLeft( 0, _+_ )
( 1 to 100 )./:( 0 )( _+_ )
( 0 /: ( 1 to 100 ) )( _+_ )
As are these three:
( 1 to 100 ).foldRight( 0, _+_ )
( 1 to 100 ).:\( 0 )( _+_ )
( ( 1 to 100 ) :\ 0 )( _+_ )
If there were just one header line in the first record, then the most efficient way to filter it out would be:
rdd.mapPartitionsWithIndex {
(idx, iter) => if (idx == 0) iter.drop(1) else iter
}
This doesn't help if of course there are many files with many header lines inside. You can union three RDDs you make this way, indeed.
You could also just write a filter
that matches only a line that could be a header. This is quite simple, but less efficient.
Python equivalent:
from itertools import islice
rdd.mapPartitionsWithIndex(
lambda idx, it: islice(it, 1, None) if idx == 0 else it
)
Here is a small example for those who want to peruse quickly
object ApplyExample01 extends App {
class Greeter1(var message: String) {
println("A greeter-1 is being instantiated with message " + message)
}
class Greeter2 {
def apply(message: String) = {
println("A greeter-2 is being instantiated with message " + message)
}
}
val g1: Greeter1 = new Greeter1("hello")
val g2: Greeter2 = new Greeter2()
g2("world")
}
output
A greeter-1 is being instantiated with message hello
A greeter-2 is being instantiated with message world
How about something like this?
val newDF = df.filter($"B" === "").take(1) match {
case Array() => df
case _ => df.withColumn("D", $"B" === "")
}
Using take(1)
should have a minimal hit
Follow the below steps:
Firstly I must say that it's much much simpler if you put your headers in separate files - this is the convention in big data.
Anyway Daniel's answer is pretty good, but it has an inefficiency and a bug, so I'm going to post my own. The inefficiency is that you don't need to check every record to see if it's the header, you just need to check the first record for each partition. The bug is that by using .split(",")
you could get an exception thrown or get the wrong column when entries are the empty string and occur at the start or end of the record - to correct that you need to use .split(",", -1)
. So here is the full code:
val header =
scala.io.Source.fromInputStream(
hadoop.fs.FileSystem.get(new java.net.URI(filename), sc.hadoopConfiguration)
.open(new hadoop.fs.Path(path)))
.getLines.head
val columnIndex = header.split(",").indexOf(columnName)
sc.textFile(path).mapPartitions(iterator => {
val head = iterator.next()
if (head == header) iterator else Iterator(head) ++ iterator
})
.map(_.split(",", -1)(columnIndex))
Final points, consider Parquet if you want to only fish out certain columns. Or at least consider implementing a lazily evaluated split function if you have wide rows.
I like Lift ;-)
Play is my second choice for Scala-friendly web frameworks.
Wicket is my third choice.
Well to obtain all different values in a Dataframe
you can use distinct. As you can see in the documentation that method returns another DataFrame
. After that you can create a UDF
in order to transform each record.
For example:
val df = sc.parallelize(Array((1, 2), (3, 4), (1, 6))).toDF("age", "salary")
// I obtain all different values. If you show you must see only {1, 3}
val distinctValuesDF = df.select(df("age")).distinct
// Define your udf. In this case I defined a simple function, but they can get complicated.
val myTransformationUDF = udf(value => value / 10)
// Run that transformation "over" your DataFrame
val afterTransformationDF = distinctValuesDF.select(myTransformationUDF(col("age")))
hadoop-aws
jar from maven repository matching your hadoop version.$SPARK_HOME/jars
location.Now in your Pyspark script, setup AWS Access Key & Secret Access Key.
spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3.awsAccessKeyId", "ACCESS_KEY")
spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3.awsSecretAccessKey", "YOUR_SECRET_ACCESSS_KEY")
// where spark is SparkSession instance
For Spark scala:
spark.sparkContext.hadoopConfiguration.set("fs.s3.awsAccessKeyId", "ACCESS_KEY")
spark.sparkContext.hadoopConfiguration.set("fs.s3.awsSecretAccessKey", "YOUR_SECRET_ACCESSS_KEY")
val
is final, that is, cannot be set. Think final
in java.
import scala.util.control._
object demo_brk_963
{
def main(args: Array[String])
{
var a = 0;
var b = 0;
val numList1 = List(1,2,3,4,5,6,7,8,9,10);
val numList2 = List(11,12,13);
val outer = new Breaks; //object for break
val inner = new Breaks; //object for break
outer.breakable // Outer Block
{
for( a <- numList1)
{
println( "Value of a: " + a);
inner.breakable // Inner Block
{
for( b <- numList2)
{
println( "Value of b: " + b);
if( b == 12 )
{
println( "break-INNER;");
inner.break;
}
}
} // inner breakable
if( a == 6 )
{
println( "break-OUTER;");
outer.break;
}
}
} // outer breakable.
}
}
Basic method to break the loop, using Breaks class. By declaring the loop as breakable.
Please use parenthesis () to access the list elements list_name(index)
import scala.reflect.runtime.{universe => ru}
def createEmptyDataFrame[T: ru.TypeTag] =
hiveContext.createDataFrame(sc.emptyRDD[Row],
ScalaReflection.schemaFor(ru.typeTag[T].tpe).dataType.asInstanceOf[StructType]
)
case class RawData(id: String, firstname: String, lastname: String, age: Int)
val sourceDF = createEmptyDataFrame[RawData]
You can use this;
String.format("%1$s %2$s %2$s %3$s", "a", "b", "c");
Output:
a b b c
There has to be a .git
directory in the root of your project.
If you don't see that directory run git init
and then re-associate your remote.
Like so:
heroku git:remote -a herokuAppName
git push heroku master
If this was working before, it means the PATH
isn't correct anymore.
That can happen when the PATH
becomes too long and gets truncated.
All posts (like this one) suggest updating the PATH
, which you can test first in a separate DOS session, by setting a minimal path and see if java works again there.
Finally the OP Highland Mark concludes:
Finally fixed by uninstalling java, removing all references to it from the registry, and then re-installing.
scary ;)
You should have a look at the scalax library : http://scalax.scalaforge.org/ In this library, there is a Logging trait, using sl4j as backend. By using this trait, you can log quite easily (just use the logger field in the class inheriting the trait).
The default value of "spark.master" is spark://HOST:PORT, and the following code tries to get a session from the standalone cluster that is running at HOST:PORT, and expects the HOST:PORT value to be in the spark config file.
SparkSession spark = SparkSession
.builder()
.appName("SomeAppName")
.getOrCreate();
"org.apache.spark.SparkException: A master URL must be set in your configuration" states that HOST:PORT is not set in the spark configuration file.
To not bother about value of "HOST:PORT", set spark.master as local
SparkSession spark = SparkSession
.builder()
.appName("SomeAppName")
.config("spark.master", "local")
.getOrCreate();
Here is the link for list of formats in which master URL can be passed to spark.master
Reference : Spark Tutorial - Setup Spark Ecosystem
How about mkString ?
theStrings.mkString(",")
A variant exists in which you can specify a prefix and suffix too.
See here for an implementation using foldLeft, which is much more verbose, but perhaps worth looking at for education's sake.
Starting from Scala 3, there is now enum keyword which can represent a set of constants (and other use cases)
enum Color:
case Red, Green, Blue
scala> val red = Color.Red
val red: Color = Red
scala> red.ordinal
val res0: Int = 0
val source_html = Source.fromResource("file.html").mkString
Although I'm sure @Ben Jackson's asnwer with Option(getObject).foreach
is the preferred way of doing it, I like to use an AnyRef
pimp that allows me to write:
getObject ifNotNull ( QueueManager.add(_) )
I find it reads better.
And, in a more general way, I sometimes write
val returnVal = getObject ifNotNull { obj =>
returnSomethingFrom(obj)
} otherwise {
returnSomethingElse
}
... replacing ifNotNull with ifSome if I'm dealing with an Option
. I find it clearer than first wrapping in an option and then pattern-matching it.
(For the implementation, see Implementing ifTrue, ifFalse, ifSome, ifNone, etc. in Scala to avoid if(...) and simple pattern matching and the Otherwise0
/Otherwise1
classes.)
The point is that the first line of your codes is not what you expected.
You should use:
val map = scala.collection.mutable.Map[A,B]()
You then have multiple equivalent alternatives to add items:
scala> val map = scala.collection.mutable.Map[String,String]()
map: scala.collection.mutable.Map[String,String] = Map()
scala> map("k1") = "v1"
scala> map
res1: scala.collection.mutable.Map[String,String] = Map((k1,v1))
scala> map += "k2" -> "v2"
res2: map.type = Map((k1,v1), (k2,v2))
scala> map.put("k3", "v3")
res3: Option[String] = None
scala> map
res4: scala.collection.mutable.Map[String,String] = Map((k3,v3), (k1,v1), (k2,v2))
And starting Scala 2.13
:
scala> map.addOne("k4" -> "v4")
res5: map.type = HashMap(k1 -> v1, k2 -> v2, k3 -> v3, k4 -> v4)
For Hive external tables I use this function in PySpark:
def save_table(sparkSession, dataframe, database, table_name, save_format="PARQUET"):
print("Saving result in {}.{}".format(database, table_name))
output_schema = "," \
.join(["{} {}".format(x.name.lower(), x.dataType) for x in list(dataframe.schema)]) \
.replace("StringType", "STRING") \
.replace("IntegerType", "INT") \
.replace("DateType", "DATE") \
.replace("LongType", "INT") \
.replace("TimestampType", "INT") \
.replace("BooleanType", "BOOLEAN") \
.replace("FloatType", "FLOAT")\
.replace("DoubleType","FLOAT")
output_schema = re.sub(r'DecimalType[(][0-9]+,[0-9]+[)]', 'FLOAT', output_schema)
sparkSession.sql("DROP TABLE IF EXISTS {}.{}".format(database, table_name))
query = "CREATE EXTERNAL TABLE IF NOT EXISTS {}.{} ({}) STORED AS {} LOCATION '/user/hive/{}/{}'" \
.format(database, table_name, output_schema, save_format, database, table_name)
sparkSession.sql(query)
dataframe.write.insertInto('{}.{}'.format(database, table_name),overwrite = True)
The method you are looking for is .limit.
Returns a new Dataset by taking the first n rows. The difference between this function and head is that head returns an array while limit returns a new Dataset.
Example usage:
df.limit(1000)
scala> val list = List(1,2,4,2,4,7,3,2,4)
list: List[Int] = List(1, 2, 4, 2, 4, 7, 3, 2, 4)
scala> println(list.filter(_ == 2).size)
3
So, is that a behavior to expect, a bug
Truth be told I am not sure. It looks like parser is interpreting count
not as a column name but a function and expects following parentheses. Looks like a bug or at least a serious limitation of the parser.
is there a canonical way to go around?
Some options have been already mentioned by Herman and mattinbits so here more SQLish approach from me:
import org.apache.spark.sql.functions.count
df.groupBy("x").agg(count("*").alias("cnt")).where($"cnt" > 2)
Legacy. List was originally defined to be functional-languages-looking:
1 :: 2 :: Nil // a list
list1 ::: list2 // concatenation of two lists
list match {
case head :: tail => "non-empty"
case Nil => "empty"
}
Of course, Scala evolved other collections, in an ad-hoc manner. When 2.8 came out, the collections were redesigned for maximum code reuse and consistent API, so that you can use ++
to concatenate any two collections -- and even iterators. List, however, got to keep its original operators, aside from one or two which got deprecated.
They can be differentiated with where they are coming from or which developers they're targeting mainly.
Groovy is a bit like scripting version of Java. Long time Java programmers feel at home when building agile applications backed by big architectures. Groovy on Grails is, as the name suggests similar to the Rails framework. For people who don't want to bother with Java's verbosity all the time.
Scala is an object oriented and functional programming language and Ruby or Python programmers may feel more closer to this one. It employs quite a lot of common good ideas found in these programming languages.
Clojure is a dialect of the Lisp programming language so Lisp, Scheme or Haskell developers may feel at home while developing with this language.
From (my entry) in the FAQ, which I certainly do not guarantee to be complete (I added two entries just two days ago):
import scala._ // Wild card -- all of Scala is imported
import scala.{ Predef => _, _ } // Exception, everything except Predef
def f[M[_]] // Higher kinded type parameter
def f(m: M[_]) // Existential type
_ + _ // Anonymous function placeholder parameter
m _ // Eta expansion of method into method value
m(_) // Partial function application
_ => 5 // Discarded parameter
case _ => // Wild card pattern -- matches anything
val (a, _) = (1, 2) // same thing
for (_ <- 1 to 10) // same thing
f(xs: _*) // Sequence xs is passed as multiple parameters to f(ys: T*)
case Seq(xs @ _*) // Identifier xs is bound to the whole matched sequence
var i: Int = _ // Initialization to the default value
def abc_<>! // An underscore must separate alphanumerics from symbols on identifiers
t._2 // Part of a method name, such as tuple getters
1_000_000 // Numeric literal separator (Scala 2.13+)
This is also part of this question.
Use ammonite ops library. The syntax is very minimal, but the breadth of the library is almost as wide as what one would expect from attempting such a task in a shell scripting language like bash.
On the page I linked to, it shows numerous operations one can do with the library, but to answer this question, this is an example of writing to a file
import ammonite.ops._
write(pwd/'"file.txt", "file contents")
Apache Spark does not support native CSV output on disk.
You have four available solutions though:
You can convert your Dataframe into an RDD :
def convertToReadableString(r : Row) = ???
df.rdd.map{ convertToReadableString }.saveAsTextFile(filepath)
This will create a folder filepath. Under the file path, you'll find partitions files (e.g part-000*)
What I usually do if I want to append all the partitions into a big CSV is
cat filePath/part* > mycsvfile.csv
Some will use coalesce(1,false)
to create one partition from the RDD. It's usually a bad practice, since it may overwhelm the driver by pulling all the data you are collecting to it.
Note that df.rdd
will return an RDD[Row]
.
With Spark <2, you can use databricks spark-csv library:
Spark 1.4+:
df.write.format("com.databricks.spark.csv").save(filepath)
Spark 1.3:
df.save(filepath,"com.databricks.spark.csv")
With Spark 2.x the spark-csv
package is not needed as it's included in Spark.
df.write.format("csv").save(filepath)
You can convert to local Pandas data frame and use to_csv
method (PySpark only).
Note: Solutions 1, 2 and 3 will result in CSV format files (part-*
) generated by the underlying Hadoop API that Spark calls when you invoke save
. You will have one part-
file per partition.
List(1,2,3) :+ 4
Results in List[Int] = List(1, 2, 3, 4)
Note that this operation has a complexity of O(n). If you need this operation frequently, or for long lists, consider using another data type (e.g. a ListBuffer).
In Java terms, Scala's Seq
would be Java's List
, and Scala's List
would be Java's LinkedList
.
Note that Seq
is a trait
, which is equivalent to Java's interface
, but with the equivalent of up-and-coming defender methods. Scala's List
is an abstract class that is extended by Nil
and ::
, which are the concrete implementations of List
.
So, where Java's List
is an interface
, Scala's List
is an implementation.
Beyond that, Scala's List
is immutable, which is not the case of LinkedList
. In fact, Java has no equivalent to immutable collections (the read only thing only guarantees the new object cannot be changed, but you still can change the old one, and, therefore, the "read only" one).
Scala's List
is highly optimized by compiler and libraries, and it's a fundamental data type in functional programming. However, it has limitations and it's inadequate for parallel programming. These days, Vector
is a better choice than List
, but habit is hard to break.
Seq
is a good generalization for sequences, so if you program to interfaces, you should use that. Note that there are actually three of them: collection.Seq
, collection.mutable.Seq
and collection.immutable.Seq
, and it is the latter one that is the "default" imported into scope.
There's also GenSeq
and ParSeq
. The latter methods run in parallel where possible, while the former is parent to both Seq
and ParSeq
, being a suitable generalization for when parallelism of a code doesn't matter. They are both relatively newly introduced, so people doesn't use them much yet.
This is the way I do the pattern match:
val result = JSON.parseFull(jsonStr)
result match {
// Matches if jsonStr is valid JSON and represents a Map of Strings to Any
case Some(map: Map[String, Any]) => println(map)
case None => println("Parsing failed")
case other => println("Unknown data structure: " + other)
}
This below code snippet for scala users :
Option 1 :
Below snippet you can add at the file level
import org.apache.log4j.{Level, Logger}
Logger.getLogger("org").setLevel(Level.WARN)
Option 2 :
Note : which will be applicable for all the application which is using spark session.
import org.apache.spark.sql.SparkSession
private[this] implicit val spark = SparkSession.builder().master("local[*]").getOrCreate()
spark.sparkContext.setLogLevel("WARN")
Option 3 :
Note : This configuration should be added to your log4j.properties.. (could be like /etc/spark/conf/log4j.properties (where the spark installation is there) or your project folder level log4j.properties) since you are changing at module level. This will be applicable for all the application.
log4j.rootCategory=ERROR, console
IMHO, Option 1 is wise way since it can be switched off at file level.
Let me also give you the SON of JSON version:
import nl.typeset.sonofjson._
arr(
obj(id = 1, name = "John)
obj(id = 2, name = "Dani)
)
Use import scala.collection.mutable.MutableList or similar if you really need mutation.
import scala.collection.mutable.MutableList
val x = MutableList(1, 2, 3, 4, 5)
x += 6 // MutableList(1, 2, 3, 4, 5, 6)
x ++= MutableList(7, 8, 9) // MutableList(1, 2, 3, 4, 5, 6, 7, 8, 9)
As Kintaro already says, person is a method (because of def) and always returns a new Person instance. As you found out it would work if you change the method to a var or val:
val person = new Person("Kumar",12)
Another possibility would be:
def person = new Person("Kumar",12)
val p = person
p.age=20
println(p.age)
However, person.age=20
in your code is allowed, as you get back a Person
instance from the person
method, and on this instance you are allowed to change the value of a var
. The problem is, that after that line you have no more reference to that instance (as every call to person
will produce a new instance).
This is nothing special, you would have exactly the same behavior in Java:
class Person{
public int age;
private String name;
public Person(String name; int age) {
this.name = name;
this.age = age;
}
public String name(){ return name; }
}
public Person person() {
return new Person("Kumar", 12);
}
person().age = 20;
System.out.println(person().age); //--> 12
Since version 2.10, one can use Scala's string interpolation feature:
implicit class RegexOps(sc: StringContext) {
def r = new util.matching.Regex(sc.parts.mkString, sc.parts.tail.map(_ => "x"): _*)
}
scala> "123" match { case r"\d+" => true case _ => false }
res34: Boolean = true
Even better one can bind regular expression groups:
scala> "123" match { case r"(\d+)$d" => d.toInt case _ => 0 }
res36: Int = 123
scala> "10+15" match { case r"(\d\d)${first}\+(\d\d)${second}" => first.toInt+second.toInt case _ => 0 }
res38: Int = 25
It is also possible to set more detailed binding mechanisms:
scala> object Doubler { def unapply(s: String) = Some(s.toInt*2) }
defined module Doubler
scala> "10" match { case r"(\d\d)${Doubler(d)}" => d case _ => 0 }
res40: Int = 20
scala> object isPositive { def unapply(s: String) = s.toInt >= 0 }
defined module isPositive
scala> "10" match { case r"(\d\d)${d @ isPositive()}" => d.toInt case _ => 0 }
res56: Int = 10
An impressive example on what's possible with Dynamic
is shown in the blog post Introduction to Type Dynamic:
object T {
class RegexpExtractor(params: List[String]) {
def unapplySeq(str: String) =
params.headOption flatMap (_.r unapplySeq str)
}
class StartsWithExtractor(params: List[String]) {
def unapply(str: String) =
params.headOption filter (str startsWith _) map (_ => str)
}
class MapExtractor(keys: List[String]) {
def unapplySeq[T](map: Map[String, T]) =
Some(keys.map(map get _))
}
import scala.language.dynamics
class ExtractorParams(params: List[String]) extends Dynamic {
val Map = new MapExtractor(params)
val StartsWith = new StartsWithExtractor(params)
val Regexp = new RegexpExtractor(params)
def selectDynamic(name: String) =
new ExtractorParams(params :+ name)
}
object p extends ExtractorParams(Nil)
Map("firstName" -> "John", "lastName" -> "Doe") match {
case p.firstName.lastName.Map(
Some(p.Jo.StartsWith(fn)),
Some(p.`.*(\\w)$`.Regexp(lastChar))) =>
println(s"Match! $fn ...$lastChar")
case _ => println("nope")
}
}
To surpass samthebest and the contributors before him, I have improved the naming and conciseness:
def using[A <: {def close() : Unit}, B](resource: A)(f: A => B): B =
try f(resource) finally resource.close()
def writeStringToFile(file: File, data: String, appending: Boolean = false) =
using(new FileWriter(file, appending))(_.write(data))
Can also do more dynamic inits with fill, e.g.
Array.fill(10){scala.util.Random.nextInt(5)}
==>
Array[Int] = Array(0, 1, 0, 0, 3, 2, 4, 1, 4, 3)
We are using Akka in a large scale Telco project (unfortunately I can't disclose a lot of details). Akka actors are deployed and accessed remotely by a web application. In this way, we have a simplified RPC model based on Google protobuffer and we achieve parallelism using Akka Futures. So far, this model has worked brilliantly. One note: we are using the Java API.
It's not as simple as just omitting the return
keyword. In Scala, if there is no return
then the last expression is taken to be the return value. So, if the last expression is what you want to return, then you can omit the return
keyword. But if what you want to return is not the last expression, then Scala will not know that you wanted to return it.
An example:
def f() = {
if (something)
"A"
else
"B"
}
Here the last expression of the function f
is an if/else expression that evaluates to a String. Since there is no explicit return
marked, Scala will infer that you wanted to return the result of this if/else expression: a String.
Now, if we add something after the if/else expression:
def f() = {
if (something)
"A"
else
"B"
if (somethingElse)
1
else
2
}
Now the last expression is an if/else expression that evaluates to an Int. So the return type of f
will be Int. If we really wanted it to return the String, then we're in trouble because Scala has no idea that that's what we intended. Thus, we have to fix it by either storing the String to a variable and returning it after the second if/else expression, or by changing the order so that the String part happens last.
Finally, we can avoid the return
keyword even with a nested if-else expression like yours:
def f() = {
if(somethingFirst) {
if (something) // Last expression of `if` returns a String
"A"
else
"B"
}
else {
if (somethingElse)
1
else
2
"C" // Last expression of `else` returns a String
}
}
Try this SQLite Database Browser
See full document here. This is very simple and fast database browser for SQLite.
Spent a day on finding the easiest way to do this. The purpose was to find the fastest way to achieve this goal. I couldn't make it as fast as running javac command from terminal or compiling from netbeans or sublime text 3. But still got a good speed with android studio.
This looks ruff and tuff way but since we don't initiate projects on daily bases that is why I am okay to do this.
I downloaded IntelliJ IDEA community version and created a simply java project. I added a main class and tested a run. Then simply closed IntelliJ IDEA and opened Android Studio and opened the same project there. Then I had to simply attach JDK where IDE helped me by showing a list of available JDKs and I selected 1.8 and then it compiled well. I can now open any main file and press Control+Shift+R to run that main file.
Then I copied all my Java files into src folder by Mac OS Finder. And I am able to compile anything I want to.
There is nothing related to Gradle or Android and compile speed is pretty good.
Thanks buddies
If it helps anyone, I found that the problem was I already had an android:onClick event in my layout file (which I inflated for the ListView rows). This was superseding the onItemClick event.
From the horse's mouth:
Foreign keys allow key values that are all NULL, even if there are no matching PRIMARY or UNIQUE keys
No Constraints on the Foreign Key
When no other constraints are defined on the foreign key, any number of rows in the child table can reference the same parent key value. This model allows nulls in the foreign key. ...
NOT NULL Constraint on the Foreign Key
When nulls are not allowed in a foreign key, each row in the child table must explicitly reference a value in the parent key because nulls are not allowed in the foreign key.
Any number of rows in the child table can reference the same parent key value, so this model establishes a one-to-many relationship between the parent and foreign keys. However, each row in the child table must have a reference to a parent key value; the absence of a value (a null) in the foreign key is not allowed. The same example in the previous section can be used to illustrate such a relationship. However, in this case, employees must have a reference to a specific department.
UNIQUE Constraint on the Foreign Key
When a UNIQUE constraint is defined on the foreign key, only one row in the child table can reference a given parent key value. This model allows nulls in the foreign key.
This model establishes a one-to-one relationship between the parent and foreign keys that allows undetermined values (nulls) in the foreign key. For example, assume that the employee table had a column named MEMBERNO, referring to an employee membership number in the company insurance plan. Also, a table named INSURANCE has a primary key named MEMBERNO, and other columns of the table keep respective information relating to an employee insurance policy. The MEMBERNO in the employee table must be both a foreign key and a unique key:
To enforce referential integrity rules between the EMP_TAB and INSURANCE tables (the FOREIGN KEY constraint)
To guarantee that each employee has a unique membership number (the UNIQUE key constraint)
UNIQUE and NOT NULL Constraints on the Foreign Key
When both UNIQUE and NOT NULL constraints are defined on the foreign key, only one row in the child table can reference a given parent key value, and because NULL values are not allowed in the foreign key, each row in the child table must explicitly reference a value in the parent key.
See this:
Take a look at Blockspring - you do need to install the plugin, but then it's just another function you call like this:
=BLOCKSPRING("twodee-array-reduce","input_array",D5:F7)
The source code and other details are here.
If this doesn't suit and/or you want to build off my solution, you can fork
my function (Python) or use another supported scripting language (Ruby
, R
, JS
, etc...).
Like this:
>>> text = 'a,b,c'
>>> text = text.split(',')
>>> text
[ 'a', 'b', 'c' ]
Alternatively, you can use eval()
if you trust the string to be safe:
>>> text = 'a,b,c'
>>> text = eval('[' + text + ']')
I don't know of any way doing this in SQL Developer. You can see all the other schemas and their objects (if you have the correct privileges) when looking in "Other Users" -> "< Schemaname >".
In your case, either use the method described above or create a new connection for the schema in which you want to work or create synonyms for all the tables you wish to access.
If you would work in SQL*Plus, issuing ALTER SESSION SET CURRENT_SCHEMA=MY_NAME
would set your current schema (This is probably what your DBA means).
Do you want to fetch and execute the script again and again, every time this component is rendered, or just once when this component is mounted into the DOM?
Perhaps try something like this:
componentDidMount () {
const script = document.createElement("script");
script.src = "https://use.typekit.net/foobar.js";
script.async = true;
document.body.appendChild(script);
}
However, this is only really helpful if the script you want to load isn't available as a module/package. First, I would always:
npm install typekit
)import
the package where I need it (import Typekit from 'typekit';
)This is likely how you installed the packages react
and react-document-title
from your example, and there is a Typekit package available on npm.
Now that we have hooks, a better approach might be to use useEffect
like so:
useEffect(() => {
const script = document.createElement('script');
script.src = "https://use.typekit.net/foobar.js";
script.async = true;
document.body.appendChild(script);
return () => {
document.body.removeChild(script);
}
}, []);
Which makes it a great candidate for a custom hook (eg: hooks/useScript.js
):
import { useEffect } from 'react';
const useScript = url => {
useEffect(() => {
const script = document.createElement('script');
script.src = url;
script.async = true;
document.body.appendChild(script);
return () => {
document.body.removeChild(script);
}
}, [url]);
};
export default useScript;
Which can be used like so:
import useScript from 'hooks/useScript';
const MyComponent = props => {
useScript('https://use.typekit.net/foobar.js');
// rest of your component
}
There are a couple ways to do this.
First, instead of going into openssl command prompt mode, just enter everything on one command line from the Windows prompt:
E:\> openssl x509 -pubkey -noout -in cert.pem > pubkey.pem
If for some reason, you have to use the openssl command prompt, just enter everything up to the ">". Then OpenSSL will print out the public key info to the screen. You can then copy this and paste it into a file called pubkey.pem.
openssl> x509 -pubkey -noout -in cert.pem
Output will look something like this:
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAryQICCl6NZ5gDKrnSztO
3Hy8PEUcuyvg/ikC+VcIo2SFFSf18a3IMYldIugqqqZCs4/4uVW3sbdLs/6PfgdX
7O9D22ZiFWHPYA2k2N744MNiCD1UE+tJyllUhSblK48bn+v1oZHCM0nYQ2NqUkvS
j+hwUU3RiWl7x3D2s9wSdNt7XUtW05a/FXehsPSiJfKvHJJnGOX0BgTvkLnkAOTd
OrUZ/wK69Dzu4IvrN4vs9Nes8vbwPa/ddZEzGR0cQMt0JBkhk9kU/qwqUseP1QRJ
5I1jR4g8aYPL/ke9K35PxZWuDp3U0UPAZ3PjFAh+5T+fc7gzCs9dPzSHloruU+gl
FQIDAQAB
-----END PUBLIC KEY-----
Change the Platform of your C++ project to "x64" (or whichever platform you are targeting) instead of "Win32". This can be found in Visual Studio under Build -> Configuration Manager. Find your project in the list and change the Platform column. Don't forget to do this for all solution configurations.
This is also possible:
a {
all: unset;
}
unset: This keyword indicates to change all the properties applying to the element or the element's parent to their parent value if they are inheritable or to their initial value if not. unicode-bidi and direction values are not affected.
Source: Mozilla description of all
Current system language is retrieved using :
CultureInfo.InstalledUICulture
"Gets the CultureInfo that represents the culture installed with the operating system."
To set it as default language for thread use :
System.Globalization.CultureInfo.DefaultThreadCurrentCulture=CultureInfo.InstalledUICulture;
At the package stage, the plugin allows configuration of the imported file names via file mapping:
http://maven.apache.org/plugins/maven-ear-plugin/examples/customize-file-name-mapping.html
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-ear-plugin</artifactId>
<version>2.7</version>
<configuration>
[...]
<fileNameMapping>full</fileNameMapping>
</configuration>
</plugin>
http://maven.apache.org/plugins/maven-war-plugin/war-mojo.html#outputFileNameMapping
If you have configured your version to be 'testing' via a profile or something, this would work for a war package:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.2</version>
<configuration>
<encoding>UTF-8</encoding>
<outputFileNameMapping>@{groupId}@-@{artifactId}@-@{baseVersion}@@{dashClassifier?}@.@{extension}@</outputFileNameMapping>
</configuration>
</plugin>
I translated the sample from Michael Borgwardt. This is the result:
public static bool NearlyEqual(float a, float b, float epsilon){
float absA = Math.Abs (a);
float absB = Math.Abs (b);
float diff = Math.Abs (a - b);
if (a == b) {
return true;
} else if (a == 0 || b == 0 || diff < float.Epsilon) {
// a or b is zero or both are extremely close to it
// relative error is less meaningful here
return diff < epsilon;
} else { // use relative error
return diff / (absA + absB) < epsilon;
}
}
Feel free to improve this answer.
You have to escape each \
to be \\
:
var ttt = "aa ///\\\\\\";
Updated: I think this question is not about the escape character in string at all. The asker doesn't seem to explain the problem correctly.
because you had to show a message to user that user can't give a name which has (\) character.
I think the scenario is like:
var user_input_name = document.getElementById('the_name').value;
Then the asker wants to check if user_input_name
contains any [\
]. If so, then alert the user.
If user enters [aa ///\
] in HTML input box, then if you alert(user_input_name)
, you will see [aaa ///\
]. You don't need to escape, i.e. replace [\
] to be [\\
] in JavaScript code. When you do escaping, that is because you are trying to make of a string which contain special characters in JavaScript source code. If you don't do it, it won't be parsed correct. Since you already get a string, you don't need to pass it into an escaping function. If you do so, I am guessing you are generating another JavaScript code from a JavaScript code, but it's not the case here.
I am guessing asker wants to simulate the input, so we can understand the problem. Unfortunately, asker doesn't understand JavaScript well. Therefore, a syntax error code being supplied to us:
var ttt = "aa ///\";
Hence, we assume the asker having problem with escaping.
If you want to simulate, you code must be valid at first place.
var ttt = "aa ///\\"; // <- This is correct
// var ttt = "aa ///\"; // <- This is not.
alert(ttt); // You will see [aa ///\] in dialog, which is what you expect, right?
Now, you only need to do is
var user_input_name = document.getElementById('the_name').value;
if (user_input_name.indexOf("\\") >= 0) { // There is a [\] in the string
alert("\\ is not allowed to be used!"); // User reads [\ is not allowed to be used]
do_something_else();
}
Edit: I used []
to quote text to be shown, so it would be less confused than using ""
.
Short form:
.zip
is an archive format using, usually, the Deflate compression method. The .gz
gzip format is for single files, also using the Deflate compression method. Often gzip is used in combination with tar to make a compressed archive format, .tar.gz
. The zlib library provides Deflate compression and decompression code for use by zip, gzip, png (which uses the zlib wrapper on deflate data), and many other applications.
Long form:
The ZIP format was developed by Phil Katz as an open format with an open specification, where his implementation, PKZIP, was shareware. It is an archive format that stores files and their directory structure, where each file is individually compressed. The file type is .zip
. The files, as well as the directory structure, can optionally be encrypted.
The ZIP format supports several compression methods:
0 - The file is stored (no compression)
1 - The file is Shrunk
2 - The file is Reduced with compression factor 1
3 - The file is Reduced with compression factor 2
4 - The file is Reduced with compression factor 3
5 - The file is Reduced with compression factor 4
6 - The file is Imploded
7 - Reserved for Tokenizing compression algorithm
8 - The file is Deflated
9 - Enhanced Deflating using Deflate64(tm)
10 - PKWARE Data Compression Library Imploding (old IBM TERSE)
11 - Reserved by PKWARE
12 - File is compressed using BZIP2 algorithm
13 - Reserved by PKWARE
14 - LZMA
15 - Reserved by PKWARE
16 - IBM z/OS CMPSC Compression
17 - Reserved by PKWARE
18 - File is compressed using IBM TERSE (new)
19 - IBM LZ77 z Architecture
20 - deprecated (use method 93 for zstd)
93 - Zstandard (zstd) Compression
94 - MP3 Compression
95 - XZ Compression
96 - JPEG variant
97 - WavPack compressed data
98 - PPMd version I, Rev 1
99 - AE-x encryption marker (see APPENDIX E)
Methods 1 to 7 are historical and are not in use. Methods 9 through 98 are relatively recent additions and are in varying, small amounts of use. The only method in truly widespread use in the ZIP format is method 8, Deflate, and to some smaller extent method 0, which is no compression at all. Virtually every .zip
file that you will come across in the wild will use exclusively methods 8 and 0, likely just method 8. (Method 8 also has a means to effectively store the data with no compression and relatively little expansion, and Method 0 cannot be streamed whereas Method 8 can be.)
The ISO/IEC 21320-1:2015 standard for file containers is a restricted zip format, such as used in Java archive files (.jar), Office Open XML files (Microsoft Office .docx, .xlsx, .pptx), Office Document Format files (.odt, .ods, .odp), and EPUB files (.epub). That standard limits the compression methods to 0 and 8, as well as other constraints such as no encryption or signatures.
Around 1990, the Info-ZIP group wrote portable, free, open-source implementations of zip
and unzip
utilities, supporting compression with the Deflate format, and decompression of that and the earlier formats. This greatly expanded the use of the .zip
format.
In the early '90s, the gzip format was developed as a replacement for the Unix compress
utility, derived from the Deflate code in the Info-ZIP utilities. Unix compress
was designed to compress a single file or stream, appending a .Z
to the file name. compress
uses the LZW compression algorithm, which at the time was under patent and its free use was in dispute by the patent holders. Though some specific implementations of Deflate were patented by Phil Katz, the format was not, and so it was possible to write a Deflate implementation that did not infringe on any patents. That implementation has not been so challenged in the last 20+ years. The Unix gzip
utility was intended as a drop-in replacement for compress
, and in fact is able to decompress compress
-compressed data (assuming that you were able to parse that sentence). gzip
appends a .gz
to the file name. gzip
uses the Deflate compressed data format, which compresses quite a bit better than Unix compress
, has very fast decompression, and adds a CRC-32 as an integrity check for the data. The header format also permits the storage of more information than the compress
format allowed, such as the original file name and the file modification time.
Though compress
only compresses a single file, it was common to use the tar
utility to create an archive of files, their attributes, and their directory structure into a single .tar
file, and to then compress it with compress
to make a .tar.Z
file. In fact, the tar
utility had and still has an option to do the compression at the same time, instead of having to pipe the output of tar
to compress
. This all carried forward to the gzip format, and tar
has an option to compress directly to the .tar.gz
format. The tar.gz
format compresses better than the .zip
approach, since the compression of a .tar
can take advantage of redundancy across files, especially many small files. .tar.gz
is the most common archive format in use on Unix due to its very high portability, but there are more effective compression methods in use as well, so you will often see .tar.bz2
and .tar.xz
archives.
Unlike .tar
, .zip
has a central directory at the end, which provides a list of the contents. That and the separate compression provides random access to the individual entries in a .zip
file. A .tar
file would have to be decompressed and scanned from start to end in order to build a directory, which is how a .tar
file is listed.
Shortly after the introduction of gzip, around the mid-1990s, the same patent dispute called into question the free use of the .gif
image format, very widely used on bulletin boards and the World Wide Web (a new thing at the time). So a small group created the PNG losslessly compressed image format, with file type .png
, to replace .gif
. That format also uses the Deflate format for compression, which is applied after filters on the image data expose more of the redundancy. In order to promote widespread usage of the PNG format, two free code libraries were created. libpng and zlib. libpng handled all of the features of the PNG format, and zlib provided the compression and decompression code for use by libpng, as well as for other applications. zlib was adapted from the gzip
code.
All of the mentioned patents have since expired.
The zlib library supports Deflate compression and decompression, and three kinds of wrapping around the deflate streams. Those are: no wrapping at all ("raw" deflate), zlib wrapping, which is used in the PNG format data blocks, and gzip wrapping, to provide gzip routines for the programmer. The main difference between zlib and gzip wrapping is that the zlib wrapping is more compact, six bytes vs. a minimum of 18 bytes for gzip, and the integrity check, Adler-32, runs faster than the CRC-32 that gzip uses. Raw deflate is used by programs that read and write the .zip
format, which is another format that wraps around deflate compressed data.
zlib is now in wide use for data transmission and storage. For example, most HTTP transactions by servers and browsers compress and decompress the data using zlib, specifically HTTP header Content-Encoding: deflate
means deflate compression method wrapped inside the zlib data format.
Different implementations of deflate can result in different compressed output for the same input data, as evidenced by the existence of selectable compression levels that allow trading off compression effectiveness for CPU time. zlib and PKZIP are not the only implementations of deflate compression and decompression. Both the 7-Zip archiving utility and Google's zopfli library have the ability to use much more CPU time than zlib in order to squeeze out the last few bits possible when using the deflate format, reducing compressed sizes by a few percent as compared to zlib's highest compression level. The pigz utility, a parallel implementation of gzip, includes the option to use zlib (compression levels 1-9) or zopfli (compression level 11), and somewhat mitigates the time impact of using zopfli by splitting the compression of large files over multiple processors and cores.
This is an Oracle-specific notation for an outer join. It means that it will include all rows from t1, and use NULLS in the t0 columns if there is no corresponding row in t0.
In standard SQL one would write:
SELECT t0.foo, t1.bar
FROM FIRST_TABLE t0
RIGHT OUTER JOIN SECOND_TABLE t1;
Oracle recommends not to use those joins anymore if your version supports ANSI joins (LEFT/RIGHT JOIN) :
Oracle recommends that you use the FROM clause OUTER JOIN syntax rather than the Oracle join operator. Outer join queries that use the Oracle join operator (+) are subject to the following rules and restrictions […]
import matplotlib.pyplot as plt
plt.rcParams.update({'figure.max_open_warning': 0})
If you use this, you won’t get that error, and it is the simplest way to do that.
You can increase the List View Threshold beyond the 5,000 default, but it is highly recommended that you don't, as it has performance implications. The recommended fix is to add an index to the field or fields used in the query (usually the ID field for a list or the Title field for a library).
When there is an index, that is used to retrieve the item(s); when there is no index the whole list is opened for a scan (and therefore hits the threshold). You create the index on the List (or Library) settings page.
This article is a good overview: http://office.microsoft.com/en-us/sharepoint-foundation-help/manage-lists-and-libraries-with-many-items-HA010377496.aspx
Try wrapping the spans in an anchor tag and apply the background image to that.
HTML:
<div class="header">
<a href="/">
<span class="header-title">My gray sea design</span><br />
<span class="header-title-two">A beautiful design</span>
</a>
</div>
CSS:
.header {
border-bottom:1px solid #eaeaea;
}
.header a {
display: block;
background-image: url("./images/embouchure.jpg");
background-repeat: no-repeat;
height:160px;
padding-left:280px;
padding-top:50px;
width:470px;
color: #eaeaea;
}
Here is kotlin version
override fun onResume() {
super.onResume()
val params:ViewGroup.LayoutParams = dialog.window.attributes
params.width = LinearLayout.LayoutParams.MATCH_PARENT
params.height = LinearLayout.LayoutParams.MATCH_PARENT
dialog.window.attributes = params as android.view.WindowManager.LayoutParams
}
Just in case you just need to show a picture in response to a camera request, there is image-to-camera.
Just download, build, install, copy an image of your choice to the device, and you can select it via the app, which is an alternative to the built-in camera.
Here is the conversion from aioobe converted to Kotlin:
/**
* https://stackoverflow.com/a/3758880/1006741
*/
fun Long.humanReadableByteCountBinary(): String {
val b = when (this) {
Long.MIN_VALUE -> Long.MAX_VALUE
else -> abs(this)
}
return when {
b < 1024L -> "$this B"
b <= 0xfffccccccccccccL shr 40 -> "%.1f KiB".format(Locale.UK, this / 1024.0)
b <= 0xfffccccccccccccL shr 30 -> "%.1f MiB".format(Locale.UK, this / 1048576.0)
b <= 0xfffccccccccccccL shr 20 -> "%.1f GiB".format(Locale.UK, this / 1.073741824E9)
b <= 0xfffccccccccccccL shr 10 -> "%.1f TiB".format(Locale.UK, this / 1.099511627776E12)
b <= 0xfffccccccccccccL -> "%.1f PiB".format(Locale.UK, (this shr 10) / 1.099511627776E12)
else -> "%.1f EiB".format(Locale.UK, (this shr 20) / 1.099511627776E12)
}
}
If the were checked as plugin-projects, than you just need to check them out. But do not select the "trunk"(for example) to speed it up. You must select all the projects you want to check out and proceed. Eclipse will than recognize them as such.
The function JSON.stringify will turn your json object into a string:
var jsonAsString = JSON.stringify(obj);
In case the browser does not implement it (IE6/IE7), use the JSON2.js script. It's safe as it uses the native implementation if it exists.
DPAPI is just for this purpose. Use DPAPI to encrypt the password the first time the user enters is, store it in a secure location (User's registry, User's application data directory, are some choices). Whenever the app is launched, check the location to see if your key exists, if it does use DPAPI to decrypt it and allow access, otherwise deny it.
Yes, that is fully possible (i.e. I do exactly this); you just need to reference the right dll (System.ServiceProcess.dll) and add an installer class...
[RunInstaller(true)]
public sealed class MyServiceInstallerProcess : ServiceProcessInstaller
{
public MyServiceInstallerProcess()
{
this.Account = ServiceAccount.NetworkService;
}
}
[RunInstaller(true)]
public sealed class MyServiceInstaller : ServiceInstaller
{
public MyServiceInstaller()
{
this.Description = "Service Description";
this.DisplayName = "Service Name";
this.ServiceName = "ServiceName";
this.StartType = System.ServiceProcess.ServiceStartMode.Automatic;
}
}
static void Install(bool undo, string[] args)
{
try
{
Console.WriteLine(undo ? "uninstalling" : "installing");
using (AssemblyInstaller inst = new AssemblyInstaller(typeof(Program).Assembly, args))
{
IDictionary state = new Hashtable();
inst.UseNewContext = true;
try
{
if (undo)
{
inst.Uninstall(state);
}
else
{
inst.Install(state);
inst.Commit(state);
}
}
catch
{
try
{
inst.Rollback(state);
}
catch { }
throw;
}
}
}
catch (Exception ex)
{
Console.Error.WriteLine(ex.Message);
}
}
yes, using and
, like:
@media screen and (max-width: 800px),
screen and (max-height: 600px) {
...
}
I would suggest using following
req.param('<param_name>')
req.param("") works as following
Lookup is performed in the following order:
req.params
req.body
req.query
Direct access to req.body, req.params, and req.query should be favoured for clarity - unless you truly accept input from each object.
If you're pasting into Word you can use the Paste Special command.
In my case, I restart the computer and enable the virtualization technology in BIOS. Then start up computer, open VM Virtual Box
, choose a virtual device, go to Settings
-General
-Basic
-Version
, choose ubuntu(64 bit)
, save the settings then start virtual device from genymotion, everything is ok now.
You can use native JS so you don't have to rely on external libraries.
(I will use some ES2015 syntax, a.k.a ES6, modern javascript) What is ES2015?
fetch('/api/rest/abc')
.then(response => response.json())
.then(data => {
// Do what you want with your data
});
You can also capture errors if any:
fetch('/api/rest/abc')
.then(response => response.json())
.then(data => {
// Do what you want with your data
})
.catch(err => {
console.error('An error ocurred', err);
});
By default it uses GET
and you don't have to specify headers, but you can do all that if you want. For further reference: Fetch API reference
public class NonKeyboardEditText extends AppCompatEditText {
public NonKeyboardEditText(Context context, AttributeSet attrs) {
super(context, attrs);
}
@Override
public boolean onCheckIsTextEditor() {
return false;
}
}
and add
NonKeyboardEditText.setTextIsSelectable(true);
wrap your entire code block in this:
//<![CDATA[
//code here
//]]>
also make sure to specify the type of script to be text/javascript
try that and let me know how it goes
Kotlin has a built-in function for this, removeSuffix
(Documentation)
var text = "filename.xml"
text = text.removeSuffix(".xml") // "filename"
If the suffix does not exist in the string, it just returns the original
var text = "not_a_filename"
text = text.removeSuffix(".xml") // "not_a_filename"
You can also check out removePrefix
and removeSurrounding
which are similar
rem = lambda x, unwanted : ''.join([ c for i, c in enumerate(x) if i != unwanted])
rem('1230004', 4)
'123004'
Changing the port in Device Manager works for me. I was also able to fix it by finding the port that Arduino was using and then select it from the Adruion IDE from tools menu Tools>Port>Com Port
I've been struggling all morning to get ONLY landscape left/right supported properly. I discovered something really annoying; although the "General" tab allows you to deselect "Portrait" for device orientation, you have to edit the plist itself to disable Portrait and PortraitUpsideDown INTERFACE orientations - it's the last key in the plist: "Supported Interface Orientations".
The other thing is that it seems you must use the "mask" versions of the enums (e.g., UIInterfaceOrientationMask.LandscapeLeft), not just the orientation one. The code that got it working for me (in my main viewController):
override func shouldAutorotate() -> Bool {
return true
}
override func supportedInterfaceOrientations() -> Int {
return Int(UIInterfaceOrientationMask.LandscapeLeft.rawValue) | Int(UIInterfaceOrientationMask.LandscapeRight.rawValue)
}
Making this combination of plist changes and code is the only way I've been able to get it working properly.
The params
parameter modifier gives callers a shortcut syntax for passing multiple arguments to a method. There are two ways to call a method with a params
parameter:
1) Calling with an array of the parameter type, in which case the params
keyword has no effect and the array is passed directly to the method:
object[] array = new[] { "1", "2" };
// Foo receives the 'array' argument directly.
Foo( array );
2) Or, calling with an extended list of arguments, in which case the compiler will automatically wrap the list of arguments in a temporary array and pass that to the method:
// Foo receives a temporary array containing the list of arguments.
Foo( "1", "2" );
// This is equivalent to:
object[] temp = new[] { "1", "2" );
Foo( temp );
In order to pass in an object array to a method with a "params object[]
" parameter, you can either:
1) Create a wrapper array manually and pass that directly to the method, as mentioned by lassevk:
Foo( new object[] { array } ); // Equivalent to calling convention 1.
2) Or, cast the argument to object
, as mentioned by Adam, in which case the compiler will create the wrapper array for you:
Foo( (object)array ); // Equivalent to calling convention 2.
However, if the goal of the method is to process multiple object arrays, it may be easier to declare it with an explicit "params object[][]
" parameter. This would allow you to pass multiple arrays as arguments:
void Foo( params object[][] arrays ) {
foreach( object[] array in arrays ) {
// process array
}
}
...
Foo( new[] { "1", "2" }, new[] { "3", "4" } );
// Equivalent to:
object[][] arrays = new[] {
new[] { "1", "2" },
new[] { "3", "4" }
};
Foo( arrays );
Edit: Raymond Chen describes this behavior and how it relates to the C# specification in a new post.
Extending on Bumptious Q Bangwhistle's answer:
$('#select2').append($('<option>', {
value: item,
text : item
})).select2();
This would add the new options into the <select>
tags and lets select2 re-render it.
If you still get this error later after setting the permissions you may need to modify your creation mask. We found our new commits (folders under objects) were still being created with no group write permission, hence only the person who committed them could push into the repository.
We fixed this by setting the umask of the SSH users to 002 with an appropriate group shared by all users.
e.g.
umask 002
where the middle 0 is allowing group write by default.
Do not use regular expressions to parse HTML.
But if you ever need to find all regexp matches in a string, use the findall
function.
import re
line = 'bla bla bla<form>Form 1</form> some text...<form>Form 2</form> more text?'
matches = re.findall('<form>(.*?)</form>', line, re.DOTALL)
print(matches)
# Output: ['Form 1', 'Form 2']
Hm, combining LEFT and RIGHT JOIN with UNION could do this:
SELECT p.LastName, p.FirstName, o.OrderNo
FROM persons AS p
LEFT JOIN
orders AS o
ON p.P_Id = Orders.P_Id
UNION ALL
SELECT p.LastName, p.FirstName, o.OrderNo
FROM persons AS p
RIGHT JOIN
orders AS o
ON p.P_Id = Orders.P_Id
WHERE p.P_Id IS NULL
JSON doesn't require you to do that, it allows you to do that. It also allows you to use "\u0061" for "A", but it's not required. Allowing \/
helps when embedding JSON in a <script>
tag, which doesn't allow </
inside strings, like Seb points out.
Some of Microsoft's ASP.NET Ajax/JSON API's use this loophole to add extra information, e.g., a datetime will be sent as "\/Date(milliseconds)\/"
. (Yuck)
This is the best solution I could come up with, NOT using CSS 3. And it works great on Firefox, Chrome, and Internet Explorer as far as I can see.
Put a container div
and two children div
s at the same level, one for content, one for the background.
And using CSS, auto-size the background to fit the content and put the background actually in the back using z-index.
.container {_x000D_
position: relative;_x000D_
}_x000D_
.content {_x000D_
position: relative;_x000D_
color: White;_x000D_
z-index: 5;_x000D_
}_x000D_
.background {_x000D_
position: absolute;_x000D_
top: 0px;_x000D_
left: 0px;_x000D_
width: 100%;_x000D_
height: 100%;_x000D_
background-color: Black;_x000D_
z-index: 1;_x000D_
/* These three lines are for transparency in all browsers. */_x000D_
-ms-filter: "progid:DXImageTransform.Microsoft.Alpha(Opacity=50)";_x000D_
filter: alpha(opacity=50);_x000D_
opacity: .5;_x000D_
}
_x000D_
<div class="container">_x000D_
<div class="content">_x000D_
Here is the content._x000D_
<br/>Background should grow to fit._x000D_
</div>_x000D_
<div class="background"></div>_x000D_
</div>
_x000D_
$ alias gpuom='git push origin master'
$ alias
hit Enter.$ vim ~/.bashrc
and hit Enter (I'm guessing you are familiar with vim).#My custom aliases
alias gpuom='git push origin master'
alias gplom='git pull origin master'
$ alias
hit Enter.This might help someone in the future, you can style the label for the input as you like and put anything you want inside it and hide the input with display none.
It works perfectly on cordova with iOS
<link href="https://cdnjs.cloudflare.com/ajax/libs/ratchet/2.0.2/css/ratchet.css" rel="stylesheet"/>_x000D_
<label for="imageUpload" class="btn btn-primary btn-block btn-outlined">Seleccionar imagenes</label>_x000D_
<input type="file" id="imageUpload" accept="image/*" style="display: none">
_x000D_
Here is a very simple method. There are a different ways to do it, like you can get it by Expanded
, Sizedbox
or Container
and it should be used according to needs.
Use Expanded
: A widget that expands a child of a Row
, Column
, or Flex
so that the child fills the available space.
Expanded(
child: ListView(scrollDirection: Axis.horizontal,
children: <Widget>[
OutlineButton(onPressed: null,
child: Text("Facebook")),
Padding(padding: EdgeInsets.all(5.00)),
OutlineButton(onPressed: null,
child: Text("Google")),
Padding(padding: EdgeInsets.all(5.00)),
OutlineButton(onPressed: null,
child: Text("Twitter"))
]),
),
Using an Expanded
widget makes a child of a Row
, Column
, or Flex
expand to fill the available space along the main axis (e.g., horizontally for a Row or vertically for a Column).
Use SizedBox
: A box with a specified size.
SizedBox(
height: 100,
child: ListView(scrollDirection: Axis.horizontal,
children: <Widget>[
OutlineButton(
color: Colors.white,
onPressed: null,
child: Text("Amazon")
),
Padding(padding: EdgeInsets.all(5.00)),
OutlineButton(onPressed: null,
child: Text("Instagram")),
Padding(padding: EdgeInsets.all(5.00)),
OutlineButton(onPressed: null,
child: Text("SoundCloud"))
]),
),
If given a child, this widget forces its child to have a specific width and/or height (assuming values are permitted by this widget's parent).
Use Container
: A convenience widget that combines common painting, positioning, and sizing widgets.
Container(
height: 80.0,
child: ListView(scrollDirection: Axis.horizontal,
children: <Widget>[
OutlineButton(onPressed: null,
child: Text("Shopify")),
Padding(padding: EdgeInsets.all(5.00)),
OutlineButton(onPressed: null,
child: Text("Yahoo")),
Padding(padding: EdgeInsets.all(5.00)),
OutlineButton(onPressed: null,
child: Text("LinkedIn"))
]),
),
The output to all three would be something like this
To set output directory and output file format and more, try the following:
INSERT OVERWRITE [LOCAL] DIRECTORY directory1
[ROW FORMAT row_format] [STORED AS file_format]
SELECT ... FROM ...
Example:
INSERT OVERWRITE DIRECTORY '/path/to/output/dir'
ROW FORMAT DELIMITED
STORED AS PARQUET
SELECT * FROM table WHERE id > 100;
You can only use switch-case on types castable to an int.
You could, however, define a std::map<std::string, std::function> dispatcher
and use it like dispatcher[str]()
to achieve same effect.
Whether you use currentframe().f_back
depends on whether you are using a
function or not.
Calling inspect directly:
from inspect import currentframe, getframeinfo
cf = currentframe()
filename = getframeinfo(cf).filename
print "This is line 5, python says line ", cf.f_lineno
print "The filename is ", filename
Calling a function that does it for you:
from inspect import currentframe
def get_linenumber():
cf = currentframe()
return cf.f_back.f_lineno
print "This is line 7, python says line ", get_linenumber()
Why don't you place the account number in a div. Style it as you please and then have a hidden input in the form that also contains the account number. Then when the form gets submitted, the value should come through and not be null.
As @Travis Solution works but in some cases
var millisecondsSince1970:Int
WILL CAUSE CRASH APPLICATION ,
with error
Double value cannot be converted to Int because the result would be greater than Int.max if it occurs Please update your answer with Int64
Here is Updated Answer
extension Date {
var millisecondsSince1970:Int64 {
return Int64((self.timeIntervalSince1970 * 1000.0).rounded())
//RESOLVED CRASH HERE
}
init(milliseconds:Int) {
self = Date(timeIntervalSince1970: TimeInterval(milliseconds / 1000))
}
}
On 32-bit platforms, Int is the same size as Int32, and on 64-bit platforms, Int is the same size as Int64.
Generally, I encounter this problem in iPhone 5
, which runs in 32-bit env. New devices run 64-bit env now. Their Int
will be Int64
.
Hope it is helpful to someone who also has same problem
Here's a plain implementation without using any third-party library. Main advantage over retainAll
, removeAll
and addAll
is that these methods don't modify the original lists input to the methods.
public class Test {
public static void main(String... args) throws Exception {
List<String> list1 = new ArrayList<String>(Arrays.asList("A", "B", "C"));
List<String> list2 = new ArrayList<String>(Arrays.asList("B", "C", "D", "E", "F"));
System.out.println(new Test().intersection(list1, list2));
System.out.println(new Test().union(list1, list2));
}
public <T> List<T> union(List<T> list1, List<T> list2) {
Set<T> set = new HashSet<T>();
set.addAll(list1);
set.addAll(list2);
return new ArrayList<T>(set);
}
public <T> List<T> intersection(List<T> list1, List<T> list2) {
List<T> list = new ArrayList<T>();
for (T t : list1) {
if(list2.contains(t)) {
list.add(t);
}
}
return list;
}
}
Below is another solution with a focus on simplicity for anyone else (like me) who get here from Google. Logging should be easy! Only for 3.2 or higher.
import multiprocessing
import logging
from logging.handlers import QueueHandler, QueueListener
import time
import random
def f(i):
time.sleep(random.uniform(.01, .05))
logging.info('function called with {} in worker thread.'.format(i))
time.sleep(random.uniform(.01, .05))
return i
def worker_init(q):
# all records from worker processes go to qh and then into q
qh = QueueHandler(q)
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logger.addHandler(qh)
def logger_init():
q = multiprocessing.Queue()
# this is the handler for all log records
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(levelname)s: %(asctime)s - %(process)s - %(message)s"))
# ql gets records from the queue and sends them to the handler
ql = QueueListener(q, handler)
ql.start()
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# add the handler to the logger so records from this process are handled
logger.addHandler(handler)
return ql, q
def main():
q_listener, q = logger_init()
logging.info('hello from main thread')
pool = multiprocessing.Pool(4, worker_init, [q])
for result in pool.map(f, range(10)):
pass
pool.close()
pool.join()
q_listener.stop()
if __name__ == '__main__':
main()
@Xaqq has provided a really nice type-safe way to use enum flags here by a flag_set
class.
I published the code in GitHub, usage is as follows:
#include "flag_set.hpp"
enum class AnimalFlags : uint8_t {
HAS_CLAWS,
CAN_FLY,
EATS_FISH,
ENDANGERED,
_
};
int main()
{
flag_set<AnimalFlags> seahawkFlags(AnimalFlags::HAS_CLAWS
| AnimalFlags::EATS_FISH
| AnimalFlags::ENDANGERED);
if (seahawkFlags & AnimalFlags::ENDANGERED)
cout << "Seahawk is endangered";
}
To show both string.Split
and Regex
usage:
string input = "abc][rfd][5][,][.";
string[] parts1 = input.Split(new string[] { "][" }, StringSplitOptions.None);
string[] parts2 = Regex.Split(input, @"\]\[");
Be sure to include the $route service into your scope and do this:
$route.reload();
See this:
Restarting Xcode did the trick for me. :)
Code:
x1 = 5.63
x2 = 5.65
print(float('%.2f' % round(x1,1))) # gives you '5.6'
print(float('%.2f' % round(x2,1))) # gives you '5.7'
Output:
5.6
5.7
I had the same problem - also make sure you add name=""
in the input button. Well, that fix worked for me.
if($_SERVER['REQUEST_METHOD'] == 'POST' && !empty($_POST['add'])){
echo "stuff is happening now";
}
<input type="submit" name="add" value="Submit">
Yes, you can switch on the name...
switch (obj.GetType().Name)
{
case "TextBox":...
}
For Multiple file input : Try this code :
<form name="form" id="form" method="post" enctype="multipart/form-data">
<input type="file" name="file[]">
<input type="file" name="file[]" >
<input type="text" name="name" id="name">
<input type="text" name="name1" id="name1">
<input type="button" name="submit" value="upload" id="upload">
</form>
$('#upload').on('click', function() {
var fd = new FormData();
var c=0;
var file_data;
$('input[type="file"]').each(function(){
file_data = $('input[type="file"]')[c].files; // for multiple files
for(var i = 0;i<file_data.length;i++){
fd.append("file_"+c, file_data[i]);
}
c++;
});
var other_data = $('form').serializeArray();
$.each(other_data,function(key,input){
fd.append(input.name,input.value);
});
$.ajax({
url: 'work.php',
data: fd,
contentType: false,
processData: false,
type: 'POST',
success: function(data){
console.log(data);
}
});
});
You might also try
Application.CalculateFull
or
Application.CalculateFullRebuild
if you don't mind rebuilding all open workbooks, rather than just the active worksheet. (CalculateFullRebuild
rebuilds dependencies as well.)
As per the documentation on the return
statement, return
may only occur syntactically nested in a function definition. The same is true for yield
.
this will do it with just javascript - you can also put the function in a .js file and call it with onclick
//button
<div onclick="
document.forms['name_of_the_form']['name_of_the_input'].value += 'text you want to add to it'"
>button</div>
I found this component. https://github.com/jsdf/react-native-htmlview
This component takes HTML content and renders it as native views, with customisable style and handling of links, etc.
If by binary you mean bytes
type, you can just use encode
method of the string object that encodes your string as a bytes object using the passed encoding type. You just need to make sure you pass a proper encoding to encode
function.
In [9]: "hello world".encode('ascii')
Out[9]: b'hello world'
In [10]: byte_obj = "hello world".encode('ascii')
In [11]: byte_obj
Out[11]: b'hello world'
In [12]: byte_obj[0]
Out[12]: 104
Otherwise, if you want them in form of zeros and ones --binary representation-- as a more pythonic way you can first convert your string to byte array then use bin
function within map
:
>>> st = "hello world"
>>> map(bin,bytearray(st))
['0b1101000', '0b1100101', '0b1101100', '0b1101100', '0b1101111', '0b100000', '0b1110111', '0b1101111', '0b1110010', '0b1101100', '0b1100100']
Or you can join it:
>>> ' '.join(map(bin,bytearray(st)))
'0b1101000 0b1100101 0b1101100 0b1101100 0b1101111 0b100000 0b1110111 0b1101111 0b1110010 0b1101100 0b1100100'
Note that in python3 you need to specify an encoding for bytearray
function :
>>> ' '.join(map(bin,bytearray(st,'utf8')))
'0b1101000 0b1100101 0b1101100 0b1101100 0b1101111 0b100000 0b1110111 0b1101111 0b1110010 0b1101100 0b1100100'
You can also use binascii
module in python 2:
>>> import binascii
>>> bin(int(binascii.hexlify(st),16))
'0b110100001100101011011000110110001101111001000000111011101101111011100100110110001100100'
hexlify
return the hexadecimal representation of the binary data then you can convert to int by specifying 16 as its base then convert it to binary with bin
.
In case that you want to repeat a character n times being n a VARIABLE number of times depending on, say, the length of a string you can do:
#!/bin/bash
vari='AB'
n=$(expr 10 - length $vari)
echo 'vari equals.............................: '$vari
echo 'Up to 10 positions I must fill with.....: '$n' equal signs'
echo $vari$(perl -E 'say "=" x '$n)
It displays:
vari equals.............................: AB
Up to 10 positions I must fill with.....: 8 equal signs
AB========
Standard troubleshooting steps for Eclipse should include deleting and re-importing the project at some point, which when I have dealt with this error has worked.
Old question ... nonetheless, just add a checkbox somewhere in the sheet. Checking or unchecking it will refresh the cell formulae.
I recently open sourced Piper (https://github.com/creactiviti/piper) a distributed and very light weight, Spring-based, workflow engine.
<Grid >
<Grid.ColumnDefinitions>
<ColumnDefinition Width="*"/>
</Grid.ColumnDefinitions>
<Button Command="{Binding ClickCommand}" Width="100" Height="100" Content="wefwfwef"/>
</Grid>
the code behind for the window:
public partial class MainWindow : Window
{
public MainWindow()
{
InitializeComponent();
DataContext = new ViewModelBase();
}
}
The ViewModel:
public class ViewModelBase
{
private ICommand _clickCommand;
public ICommand ClickCommand
{
get
{
return _clickCommand ?? (_clickCommand = new CommandHandler(() => MyAction(), ()=> CanExecute));
}
}
public bool CanExecute
{
get
{
// check if executing is allowed, i.e., validate, check if a process is running, etc.
return true/false;
}
}
public void MyAction()
{
}
}
Command Handler:
public class CommandHandler : ICommand
{
private Action _action;
private Func<bool> _canExecute;
/// <summary>
/// Creates instance of the command handler
/// </summary>
/// <param name="action">Action to be executed by the command</param>
/// <param name="canExecute">A bolean property to containing current permissions to execute the command</param>
public CommandHandler(Action action, Func<bool> canExecute)
{
_action = action;
_canExecute = canExecute;
}
/// <summary>
/// Wires CanExecuteChanged event
/// </summary>
public event EventHandler CanExecuteChanged
{
add { CommandManager.RequerySuggested += value; }
remove { CommandManager.RequerySuggested -= value; }
}
/// <summary>
/// Forcess checking if execute is allowed
/// </summary>
/// <param name="parameter"></param>
/// <returns></returns>
public bool CanExecute(object parameter)
{
return _canExecute.Invoke();
}
public void Execute(object parameter)
{
_action();
}
}
I hope this will give you the idea.
set nu
set ai
set tabstop=4
set ls=2
set autoindent
Add the above code in your .vimrc file. if .vimrc file is not present please create in your home directory (/home/name of user)
set nu -> This makes Vim display line numbers
set ai -> This makes Vim enable auto-indentation
set ls=2 -> This makes Vim show a status line
set tabstop=4 -> This makes Vim set tab of length 4 spaces (it is 8 by default)
The filename will also be displayed.
Let's try this way:
select
a.ip,
a.os,
a.hostname,
a.port,
a.protocol,
b.state
from a
left join b
on a.ip = b.ip
and a.port = b.port /*if you has to filter by columns from right table , then add this condition in ON clause*/
where a.somecolumn = somevalue /*if you have to filter by some column from left table, then add it to where condition*/
So, in where
clause you can filter result set by column from right table only on this way:
...
where b.somecolumn <> (=) null
open C:\myfile.txt for append as #1
write #1, text1.text, text2.text
close()
This is the code I use in Visual Basic 6.0. It helps me to create a txt file on my drive, write two pieces of data into it, and then close the file... Give it a try...
You can get a client
with new session directly like below.
s3_client = boto3.client('s3',
aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY,
aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY,
region_name=REGION_NAME
)
.Net Core has this method built-in:
Replace(String, String, StringComparison)
Doc. Now we can simply write:
"...".Replace("oldValue", "newValue", StringComparison.OrdinalIgnoreCase)
You can use the jQuery ajax method link if you want to pass data from client to server.
In this case you can use $_SERVER['HTTP_USER_AGENT']
variable to found browser user agent.
You are single quoting your SQL statement which is making the variables text instead of variables.
$sql = "SELECT *
FROM $usertable
WHERE PartNumber = $partid";
While I found the two top-upvoted answers useful on other occasions, today, the simplest way to resolve the issue was to realize that PyCharm might be keeping a session open, and if I clicked Stop
in PyCharm, that might help. With pgAdmin4 open in the browser, I did so, and almost immediately saw the Database sessions stats drop to 0, at which point I was able to drop the database.
You mean the tellItLikeItIs
method? Yes, you have to specify void
to specify that the method doesn't return anything. All methods have to have a return type specified, even if it's void
.
It certainly doesn't return a string - look, there are no return statements anywhere. It's not really clear why you think it is returning a string. It's printing strings to the console, but that's not the same thing as returning one from the method.
Joran suggested to first remove the NAs before running the model. Thus, I removed the NAs, run the model and obtained the residuals. When I updated model2 by inclusion of the lagged residuals, the error message did not appear again.
df2<-df1[complete.cases(df1),]
model2<-gam(death ~ pm10 + s(trend,k=14*7)+ s(temp,k=5), data=df2, family=poisson)
resid2 <- residuals(model2,type="deviance")
model2_1 <- update(model2,.~.+ Lag(resid2,1), na.action=na.omit)
Date
has the time part, so we only need to extract it from Date
I personally prefer the default format
parameter of the Date
when date and time needs to be separated instead of using the extra SimpleDateFormat
Date date = new Date()
String datePart = date.format("dd/MM/yyyy")
String timePart = date.format("HH:mm:ss")
println "datePart : " + datePart + "\ttimePart : " + timePart
Maybe some AJAX page content injection could be used as an alternative, though I still can't get around why your teacher would refuse to rid the website of frames.
Additionally, is there any specific reason you personally want to us HTML5?
But if not, I believe <iframe>
s are still around.
As long as you specify a width on the element, it should wrap itself without needing anything else.
Take care to strip out newlines from any string you pass to setData().
Otherwise an exception gets thrown.
Also note that even if you do that, then subsequently get that data again using getData(),
CKEditor puts the line breaks back in.
Put your script in a file foo.py
starting with
#!/usr/bin/python
Then give execute permission to that script using
chmod a+x foo.py
and use the full path of your foo.py
file in your crontab
.
See documentation of execve(2) which is handling the shebang.
The string has a substring method that returns the string at the specified position.
String name="123456789";
System.out.println(name.substring(0,1));
DECIMAL(18,0)
will allow 0 digits after the decimal point.
Use something like DECIMAL(18,4)
instead that should do just fine!
That gives you a total of 18 digits, 4 of which after the decimal point (and 14 before the decimal point).
In Angular 2+, you can set route to default page by adding this route to your route module. In this case login is my target route for the default page.
{path:'',redirectTo:'login', pathMatch: 'full' },
You can't upload nested structures like that through the online tool. I'd recommend using something like Bucket Explorer for more complicated uploads.
var text = $(".title").contents().filter(function() {
return this.nodeType == Node.TEXT_NODE;
}).text();
This gets the contents
of the selected element, and applies a filter function to it. The filter function returns only text nodes (i.e. those nodes with nodeType == Node.TEXT_NODE
).
I think you need to do is to transform your data from object not to JSON string, but to url params.
By default, the $http service will transform the outgoing request by serializing the data as JSON and then posting it with the content- type, "application/json". When we want to post the value as a FORM post, we need to change the serialization algorithm and post the data with the content-type, "application/x-www-form-urlencoded".
Example from here.
$http({
method: 'POST',
url: url,
headers: {'Content-Type': 'application/x-www-form-urlencoded'},
transformRequest: function(obj) {
var str = [];
for(var p in obj)
str.push(encodeURIComponent(p) + "=" + encodeURIComponent(obj[p]));
return str.join("&");
},
data: {username: $scope.userName, password: $scope.password}
}).then(function () {});
To use new services added with AngularJS V1.4, see
You can easily crop a Mat using opencv funtions.
setMouseCallback("Original",mouse_call);
The mouse_call
is given below:
void mouse_call(int event,int x,int y,int,void*)
{
if(event==EVENT_LBUTTONDOWN)
{
leftDown=true;
cor1.x=x;
cor1.y=y;
cout <<"Corner 1: "<<cor1<<endl;
}
if(event==EVENT_LBUTTONUP)
{
if(abs(x-cor1.x)>20&&abs(y-cor1.y)>20) //checking whether the region is too small
{
leftup=true;
cor2.x=x;
cor2.y=y;
cout<<"Corner 2: "<<cor2<<endl;
}
else
{
cout<<"Select a region more than 20 pixels"<<endl;
}
}
if(leftDown==true&&leftup==false) //when the left button is down
{
Point pt;
pt.x=x;
pt.y=y;
Mat temp_img=img.clone();
rectangle(temp_img,cor1,pt,Scalar(0,0,255)); //drawing a rectangle continuously
imshow("Original",temp_img);
}
if(leftDown==true&&leftup==true) //when the selection is done
{
box.width=abs(cor1.x-cor2.x);
box.height=abs(cor1.y-cor2.y);
box.x=min(cor1.x,cor2.x);
box.y=min(cor1.y,cor2.y);
Mat crop(img,box); //Selecting a ROI(region of interest) from the original pic
namedWindow("Cropped Image");
imshow("Cropped Image",crop); //showing the cropped image
leftDown=false;
leftup=false;
}
}
For details you can visit the link Cropping the Image using Mouse
While using a std::map
is fine or using a 256-sized char table would be fine, you could save yourself an enormous amount of space agony by simply using an enum
. If you have C++11 features, you can use enum class
for strong-typing:
// First, we define base-pairs. Because regular enums
// Pollute the global namespace, I'm using "enum class".
enum class BasePair {
A,
T,
C,
G
};
// Let's cut out the nonsense and make this easy:
// A is 0, T is 1, C is 2, G is 3.
// These are indices into our table
// Now, everything can be so much easier
BasePair Complimentary[4] = {
T, // Compliment of A
A, // Compliment of T
G, // Compliment of C
C, // Compliment of G
};
Usage becomes simple:
int main (int argc, char* argv[] ) {
BasePair bp = BasePair::A;
BasePair complimentbp = Complimentary[(int)bp];
}
If this is too much for you, you can define some helpers to get human-readable ASCII characters and also to get the base pair compliment so you're not doing (int)
casts all the time:
BasePair Compliment ( BasePair bp ) {
return Complimentary[(int)bp]; // Move the pain here
}
// Define a conversion table somewhere in your program
char BasePairToChar[4] = { 'A', 'T', 'C', 'G' };
char ToCharacter ( BasePair bp ) {
return BasePairToChar[ (int)bp ];
}
It's clean, it's simple, and its efficient.
Now, suddenly, you don't have a 256 byte table. You're also not storing characters (1 byte each), and thus if you're writing this to a file, you can write 2 bits per Base pair instead of 1 byte (8 bits) per base pair. I had to work with Bioinformatics Files that stored data as 1 character each. The benefit is it was human-readable. The con is that what should have been a 250 MB file ended up taking 1 GB of space. Movement and storage and usage was a nightmare. Of coursse, 250 MB is being generous when accounting for even Worm DNA. No human is going to read through 1 GB worth of base pairs anyhow.
There's the %
sign. It's not just for the remainder, it is the modulo operation.
This bit of code from the above link did the trick in my case :)
protected override void OnMouseDown(MouseEventArgs e)
{
base.OnMouseDown(e);
if (e.Button == MouseButtons.Left)
{
this.Capture = false;
Message msg = Message.Create(this.Handle, 0XA1, new IntPtr(2), IntPtr.Zero);
this.WndProc(ref msg);
}
}
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html
mean_squared_error(y_true, y_pred)
So, final code would be something like:
from sklearn.metrics import mean_squared_error from math import sqrt
RMSD = sqrt(mean_squared_error(testing_y, prediction))
print(RMSD)
You must just install jdk1.8.0 and then right click on my computer icon and then select properties,then in left panel, select advanced system settings, then in dialog bog select Environment Variables, then in that's dialog box,in section user variables create new variable that's name must be JAVA_HOME
and path is C:\Program Files\Java\jdk1.8.0
(in my pc) then sytem variable section, select PATH
variable and append it's end this path C:\Program Files\Java\jdk1.8.0\bin
and then select ok for all dialog box and after this steps run Android studio. And for test, run cmd in windows and run this command java -version
if returned a java version and ... it is installed correctly.
Note: I get answer in windows 8.1 64 bit.
The $(this).html().replace(',', '')
shouldn't actually modify the page. Are you sure the commas are being removed in the page?
If it is, this addCommas function should do the trick.
function addCommas(nStr) {
nStr += '';
var x = nStr.split('.');
var x1 = x[0];
var x2 = x.length > 1 ? '.' + x[1] : '';
var rgx = /(\d+)(\d{3})/;
while (rgx.test(x1)) {
x1 = x1.replace(rgx, '$1' + ',' + '$2');
}
return x1 + x2;
}
You don't need initialization lists for that:
std::vector<int> vector1(length, 0);
std::vector<double> vector2(length, 0.0);
<script type="text/javascript">
function hide(){
document.getElementById('id').hidden = true;
}
function show(){
document.getElementById('id').hidden = false;
}
</script>
something like:
p
{
display:inline;
}
in your stylesheet would do it for all p tags.
c# solution to reverse words in a sentence
using System;
class helloworld {
public void ReverseString(String[] words) {
int end = words.Length-1;
for (int start = 0; start < end; start++) {
String tempc;
if (start < end ) {
tempc = words[start];
words[start] = words[end];
words[end--] = tempc;
}
}
foreach (String s1 in words) {
Console.Write("{0} ",s1);
}
}
}
class reverse {
static void Main() {
string s= "beauty lies in the heart of the peaople";
String[] sent_char=s.Split(' ');
helloworld h1 = new helloworld();
h1.ReverseString(sent_char);
}
}
output: peaople the of heart the in lies beauty Press any key to continue . . .
Mockito has limitations testing final, static, and private methods.
with jMockit testing library, you can do few stuff very easy and straight-forward as below:
Mock constructor of a java.io.File class:
new MockUp<File>(){
@Mock
public void $init(String pathname){
System.out.println(pathname);
// or do whatever you want
}
};
Mock a static method:
Try like below with Gson
Library.
Earlier Conversion List format were:
[Product [Id=1, City=Bengalore, Category=TV, Brand=Samsung, Name=Samsung LED, Type=LED, Size=32 inches, Price=33500.5, Stock=17.0], Product [Id=2, City=Bengalore, Category=TV, Brand=Samsung, Name=Samsung LED, Type=LED, Size=42 inches, Price=41850.0, Stock=9.0]]
and here the conversion source begins.
//** Note I have created the method toString() in Product class.
//Creating and initializing a java.util.List of Product objects
List<Product> productList = (List<Product>)productRepository.findAll();
//Creating a blank List of Gson library JsonObject
List<JsonObject> entities = new ArrayList<JsonObject>();
//Simply printing productList size
System.out.println("Size of productList is : " + productList.size());
//Creating a Iterator for productList
Iterator<Product> iterator = productList.iterator();
//Run while loop till Product Object exists.
while(iterator.hasNext()){
//Creating a fresh Gson Object
Gson gs = new Gson();
//Converting our Product Object to JsonElement
//Object by passing the Product Object String value (iterator.next())
JsonElement element = gs.fromJson (gs.toJson(iterator.next()), JsonElement.class);
//Creating JsonObject from JsonElement
JsonObject jsonObject = element.getAsJsonObject();
//Collecting the JsonObject to List
entities.add(jsonObject);
}
//Do what you want to do with Array of JsonObject
System.out.println(entities);
Converted Json Result is :
[{"Id":1,"City":"Bengalore","Category":"TV","Brand":"Samsung","Name":"Samsung LED","Type":"LED","Size":"32 inches","Price":33500.5,"Stock":17.0}, {"Id":2,"City":"Bengalore","Category":"TV","Brand":"Samsung","Name":"Samsung LED","Type":"LED","Size":"42 inches","Price":41850.0,"Stock":9.0}]
Hope this would help many guys!
If you don't need to do it programatically, but just want to manage your keys, then I've used IBM's free KeyMan tool for a long time now. Very nice for exporting a private key to a PFX file (then you can easily use OpenSSL to manipulate it, extract it, change pwds, etc).
Select your keystore, select the private key entry, then File->Save to a pkcs12 file (*.pfx, typically). You can then view the contents with:
$ openssl pkcs12 -in mykeyfile.pfx -info
Suppose I have the following table T
:
a b
--------
1 abc
1 def
1 ghi
2 jkl
2 mno
2 pqr
And I do the following query:
SELECT a, b
FROM T
GROUP BY a
The output should have two rows, one row where a=1
and a second row where a=2
.
But what should the value of b show on each of these two rows? There are three possibilities in each case, and nothing in the query makes it clear which value to choose for b in each group. It's ambiguous.
This demonstrates the single-value rule, which prohibits the undefined results you get when you run a GROUP BY query, and you include any columns in the select-list that are neither part of the grouping criteria, nor appear in aggregate functions (SUM, MIN, MAX, etc.).
Fixing it might look like this:
SELECT a, MAX(b) AS x
FROM T
GROUP BY a
Now it's clear that you want the following result:
a x
--------
1 ghi
2 pqr
In this world of flashy new native functions, we sometimes forget the basics.
for (let i = 0; i < arr.length; i++) {
console.log('index:', i, 'element:', arr[i]);
}
Clean, efficient, and you can still break
the loop. Bonus! You can also start from the end and go backwards with i--
!
Additional note: If you're using the value a lot within the loop, you may wish to do const value = arr[i];
at the top of the loop for an easy, readable reference.
It is a viable way to use std::vector as an intermediate container:
QString dataSrc("FooBar");
QString databa = dataSrc.toUtf8();
std::vector<char> data(databa.begin(), databa.end());
char* pDataChar = data.data();
One more JAVA 8 way:
LocalDateTime localDateTime = LocalDateTime.now().truncatedTo(ChronoUnit.HOURS);
But it's a lot more useful to edit the date that already exists.
Try:
float number mkm = (((((amountdrug/fluidvol)*1000f)/60f)*infrate)/ptwt)*1000f;
int newNum = (int) mkm;
mkm = newNum/1000f; // Will return 3 decimal places
Swift "Button factory" extension for UIButton (and while we're at it) also for UILabel like so:
extension UILabel
{
// A simple UILabel factory function
// returns instance of itself configured with the given parameters
// use example (in a UIView or any other class that inherits from UIView):
// addSubview( UILabel().make( x: 0, y: 0, w: 100, h: 30,
// txt: "Hello World!",
// align: .center,
// fnt: aUIFont,
// fntColor: UIColor.red) )
//
func make(x: CGFloat, y: CGFloat, w: CGFloat, h: CGFloat,
txt: String,
align: NSTextAlignment,
fnt: UIFont,
fntColor: UIColor)-> UILabel
{
frame = CGRect(x: x, y: y, width: w, height: h)
adjustsFontSizeToFitWidth = true
textAlignment = align
text = txt
textColor = fntColor
font = fnt
return self
}
// Of course, you can make more advanced factory functions etc.
// Also one could subclass UILabel, but this seems to be a convenient case for an extension.
}
extension UIButton
{
// UIButton factory returns instance of UIButton
//usage example:
// addSubview(UIButton().make(x: btnx, y:100, w: btnw, h: btnh,
// title: "play", backColor: .red,
// target: self,
// touchDown: #selector(play), touchUp: #selector(stopPlay)))
func make( x: CGFloat,y: CGFloat,
w: CGFloat,h: CGFloat,
title: String, backColor: UIColor,
target: UIView,
touchDown: Selector,
touchUp: Selector ) -> UIButton
{
frame = CGRect(x: x, y: y, width: w, height: h)
backgroundColor = backColor
setTitle(title, for: .normal)
addTarget(target, action: touchDown, for: .touchDown)
addTarget(target, action: touchUp , for: .touchUpInside)
addTarget(target, action: touchUp , for: .touchUpOutside)
return self
}
}
Tested in Swift in Xcode Version 9.2 (9C40b) Swift 4.x
In Eclipse, eventually I had to add Compatibility/Support Library by right-clicking on my project and selecting:
Android Tools -> Add Support Library
Once it was added, then I was able to use LocalBroadcastManager
class in my code.
this helped me
git checkout -b newbranch
git checkout master
git merge newbranch
git branch -d newbranch
Here is a functional ES6 way of iterating over a NodeList
. This method uses the Array
's forEach
like so:
Array.prototype.forEach.call(element.childNodes, f)
Where f
is the iterator function that receives a child nodes as it's first parameter and the index as the second.
If you need to iterate over NodeLists more than once you could create a small functional utility method out of this:
const forEach = f => x => Array.prototype.forEach.call(x, f);
// For example, to log all child nodes
forEach((item) => { console.log(item); })(element.childNodes)
// The functional forEach is handy as you can easily created curried functions
const logChildren = forEach((childNode) => { console.log(childNode); })
logChildren(elementA.childNodes)
logChildren(elementB.childNodes)
(You can do the same trick for map()
and other Array functions.)
Both the options can be correct depending on your situation.
A very simple example would be: If you have multiple constructors all of which initialize the variable the same way(int x=2 for each one of them). It makes sense to initialize the variable at declaration to avoid redundancy.
It also makes sense to consider final variables in such a situation. If you know what value a final variable will have at declaration, it makes sense to initialize it outside the constructors. However, if you want the users of your class to initialize the final variable through a constructor, delay the initialization until the constructor.
Make your size a factor in your dataframe by:
temp$size_f = factor(temp$size, levels=c('50%','100%','150%','200%'))
Then change the facet_grid(.~size)
to facet_grid(.~size_f)
Then plot:
The graphs are now in the correct order.
Changing the NPM repo URL to HTTP works as a quick-fix, but I wanted to use HTTPS.
In my case, the proxy at my employer (ZScaler) was causing issues (as it acts as a MITM, causing certification verification issues)
I forgot I found a script that helps with this and Git (for cloning GitHub repos via HTTPS had the same issue) and forked it for my use
Basically, it does the following for git:
git config --global http.proxy http://gateway.zscaler.net:80/
git config --system http.proxy http://gateway.zscaler.net:80/
and for Node, it adds proxy=http://gateway.zscaler.net:80/
to the end of c:\Users\$USERNAME\npm\.npmrc
That solved the issue for me.
If you are using graph for navigation between fragments you can do this: From fragment A:
Bundle bundle = new Bundle();
bundle.putSerializable(KEY, yourObject);
Navigation.findNavController(view).navigate(R.id.fragment, bundle);
To fragment B:
Bundle bundle = getArguments();
object = (Object) bundle.getSerializable(KEY);
Of course your object must implement Serializable
Here's another solution, similar to adding bbox_extra_artists
and bbox_inches
, where you don't have to have your extra artists in the scope of your savefig
call. I came up with this since I generate most of my plot inside functions.
Instead of adding all your additions to the bounding box when you want to write it out, you can add them ahead of time to the Figure
's artists. Using something similar to Franck Dernoncourt's answer above:
import matplotlib.pyplot as plt
# data
all_x = [10,20,30]
all_y = [[1,3], [1.5,2.9],[3,2]]
# plotting function
def gen_plot(x, y):
fig = plt.figure(1)
ax = fig.add_subplot(111)
ax.plot(all_x, all_y)
lgd = ax.legend( [ "Lag " + str(lag) for lag in all_x], loc="center right", bbox_to_anchor=(1.3, 0.5))
fig.artists.append(lgd) # Here's the change
ax.set_title("Title")
ax.set_xlabel("x label")
ax.set_ylabel("y label")
return fig
# plotting
fig = gen_plot(all_x, all_y)
# No need for `bbox_extra_artists`
fig.savefig("image_output.png", dpi=300, format="png", bbox_inches="tight")
None of the existing answers here has provided a complete example of how to do this from within a script (not a one-liner). Here is what I did:
rename($file, $file.'.bak');
open(IN, '<'.$file.'.bak') or die $!;
open(OUT, '>'.$file) or die $!;
while(<IN>)
{
$_ =~ s/blue/red/g;
print OUT $_;
}
close(IN);
close(OUT);
Actually, for functional component the best practice is like below, I create a sample Spinner component:
import React from 'react';
import { ActivityIndicator } from 'react-native';
import { colors } from 'helpers/theme';
import type { FC } from 'types';
interface SpinnerProps {
color?: string;
size?: 'small' | 'large' | 1 | 0;
animating?: boolean;
hidesWhenStopped?: boolean;
}
const Spinner: FC<SpinnerProps> = ({
color,
size,
animating,
hidesWhenStopped,
}) => (
<ActivityIndicator
color={color}
size={size}
animating={animating}
hidesWhenStopped={hidesWhenStopped}
/>
);
Spinner.defaultProps = {
animating: true,
color: colors.primary,
hidesWhenStopped: true,
size: 'small',
};
export default Spinner;
for all windows os
cd C:\Program Files (x86)\Git\bin
ssh-keygen
I see it hinted here and there, but you could also do a faster calculation first, like, compare the bounding boxes for overlap, and THEN do a radius-based overlap if that first test passes.
The addition/difference math is much faster for a bounding box than all the trig for the radius, and most times, the bounding box test will dismiss the possibility of a collision. But if you then re-test with trig, you're getting the accurate results that you're seeking.
Yes, it's two tests, but it will be faster overall.
It would help if you showed the code you are using for the simple test program. Put directly into the interpreter this seems to work.
>>> def pyth_test (x1, x2):
... print x1 + x2
...
>>> pyth_test(1, 2)
3
>>>
pkill NAMEofAPP
Non rooted marshmallow, termux & terminal emulator.
This worked for me.
In php.ini
file, change the following property.
Before:
mbstring.encoding_transration = On
After:
mbstring.encoding_transration = Off
I had a case where Process.HasExited
didn't change after closing the window belonging to the process. So Process.WaitForExit()
also didn't work. I had to monitor Process.Responding
that went to false after closing the window like that:
while (!_process.HasExited && _process.Responding) {
Thread.Sleep(100);
}
...
Perhaps this helps someone.
Hadley's tidyr
just got this amazing function drop_na
library(tidyr)
DF %>% drop_na(y)
x y z
1 1 0 NA
2 2 10 33
from __future__ import with_statement
with open('file.txt','r+') as f:
counter = str(int(f.read().strip())+1)
f.seek(0)
f.write(counter)
Just a helpful hint, there is a company called Yodlee.com who provides this data. They do charge for the API. Companies like Mint.com use this API to gather bank and financial account data.
Also, checkout https://plaid.com/, they are a similar company Yodlee.com and provide both authentication API for several banks and REST-based transaction fetching endpoints.
This creates the branch with one command:
git push origin <sha1-of-commit>:refs/heads/<branch-name>
I prefer this way better than the ones published above, because it creates the branch immediately (does not require an extra push command afterwards).
Yes you could use an <input type="hidden" />
and set the value of that hidden field in your javascript code so it gets posted with your other form data.
so Laravel actually returns a collection when just using Model::all();
you don't want a collection you want an array so you can type set it.
(array)Model::all();
then you can use array_filter to return the results
$models = (array)Model::all()
$models = array_filter($models);
if(empty($models))
{
do something
}
this will also allow you to do things like count()
.
How about sourcing the profile before running the command?
ssh user@host "source /etc/profile; /path/script.sh"
You might find it best to change that to ~/.bash_profile
, ~/.bashrc
, or whatever.
Following looks very simple to me:
model.layers[idx].output
Above is a tensor object, so you can modify it using operations that can be applied to a tensor object.
For example, to get the shape model.layers[idx].output.get_shape()
idx
is the index of the layer and you can find it from model.summary()
I use [^\t\r\n\x20-\x7E]+
and that seems to be working fine.
You should use the express framework.
npm install express
and then
var express = require('express');
var app = express();
app.use(express.static(__dirname + '/public'));
app.listen(8080);
and then the URL localhost:8080/images/logo.gif should work.
Here is a modified version of the chosen answer:
getMonth("Feb")
function getMonth(month) {
d = new Date().toString().split(" ")
d[1] = month
d = new Date(d.join(' ')).getMonth()+1
if(!isNaN(d)) {
return d
}
return -1;
}
You either need to run "update-database" from the package manager console to push your changes to the database OR you can delete the pending migration file ([201203170856167_left]) from your Migrations folder and then re-run "add-migration" to create a brand new migration based off of your edits.
You confuse how browsers renders the elements when there are floating elements. If one block element is floating (your inner div in your case), other block elements will ignore it because browser removes floating elements from the normal flow of the web page. Then, because the floated div has been removed from the normal flow, the outside div is filled in, like the inner div isn't there. However, inline elements (images, links, text, blackquotes) will respect the boundaries of the floating element. If you introduce text in the outside div, the text will place arround de inner div.
In other words, block elements (headers, paragraphs, divs, etc) ignore floating elements and fill in, and inline elements (images, links, text, etc) respect boundaries of floating elements.
<body>
<div style="float:right; background-color:blue;width:200px;min-height:400px;margin-right:20px">
floating element
</div>
<h1 style="background-color:red;"> this is a big header</h1>
<p style="background-color:green"> this is a parragraph with text and a big image. The text places arrounds the floating element. Because of the image is wider than space between paragrah and floating element places down the floating element. Try to make wider the viewport and see what happens :D
<img src="http://2.bp.blogspot.com/_nKxzQGcCLtQ/TBYPAJ6xM4I/AAAAAAAAAC8/lG6XemOXosU/s1600/css.png">
</p>
Wxruby is a great framework, simple and clean. Try it or use glade with ruby (the simpliest option)
List myList = new ArrayList();
Collections.addAll(myList, filesOrig);
If the indexes are not in sequence, or if the indexes are not consecutive, the code in the other solutions listed here will break. A solution that would work somewhat better might be:
function in_array(needle, haystack) {
for(var i in haystack) {
if(haystack[i] == needle) return true;
}
return false;
}
And, as a bonus, here's the equivalent to PHP's array_search (for finding the key of the element in the array:
function array_search(needle, haystack) {
for(var i in haystack) {
if(haystack[i] == needle) return i;
}
return false;
}
erlswf is an opensource project written in erlang for decompiling .swf files.
Here's the site: https://github.com/bef/erlswf
As for how to represent a single apostrophe as a string in Python, you can simply surround it with double quotes ("'"
) or you can escape it inside single quotes ('\''
).
To remove apostrophes from a string, a simple approach is to just replace the apostrophe character with an empty string:
>>> "didn't".replace("'", "")
'didnt'
Generally you can't change styles programmatically; you can set the look of a screen, or part of a layout, or individual button in your XML layout using themes or styles. Themes can, however, be applied programmatically.
There is also such a thing as a StateListDrawable
which lets you define different drawables for each state the your Button
can be in, whether focused, selected, pressed, disabled and so on.
For example, to get your button to change colour when it's pressed, you could define an XML file called res/drawable/my_button.xml
directory like this:
<?xml version="1.0" encoding="utf-8"?>
<selector xmlns:android="http://schemas.android.com/apk/res/android">
<item
android:state_pressed="true"
android:drawable="@drawable/btn_pressed" />
<item
android:state_pressed="false"
android:drawable="@drawable/btn_normal" />
</selector>
You can then apply this selector to a Button
by setting the property android:background="@drawable/my_button"
.
You can also install simplejson.
If you have pip (see https://pypi.python.org/pypi/pip) as your Python package manager you can install simplejson with:
pip install simplejson
This is similar to the comment of installing with easy_install, but I prefer pip to easy_install as you can easily uninstall in pip with "pip uninstall package".
Tables are iffy (at least, in IE) when it comes to fixing heights and not wrapping text. I think you'll find that the only solution is to put the text inside a div
element, like so:
td.container > div {_x000D_
width: 100%;_x000D_
height: 100%;_x000D_
overflow:hidden;_x000D_
}_x000D_
td.container {_x000D_
height: 20px;_x000D_
}
_x000D_
<table>_x000D_
<tr>_x000D_
<td class="container">_x000D_
<div>This is a long line of text designed not to wrap _x000D_
when the container becomes too small.</div>_x000D_
</td>_x000D_
</tr>_x000D_
</table>
_x000D_
This way, the div
's height is that of the containing cell and the text cannot grow the div
, keeping the cell/row the same height no matter what the window size is.
If you prefer significant digits to fixed digits then, the signif command might be useful:
> signif(1.12345, digits = 3)
[1] 1.12
> signif(12.12345, digits = 3)
[1] 12.1
> signif(12345.12345, digits = 3)
[1] 12300
The problem is that '_' underscores are not valid in header attribute. If removing the underscore is not an option you can add to the server block:
underscores_in_headers on;
This is basically a copy and paste from @kishorer747 comment on @Fleshgrinder answer, and solution is from: https://serverfault.com/questions/586970/nginx-is-not-forwarding-a-header-value-when-using-proxy-pass/586997#586997
I added it here as in my case the application behind nginx was working perfectly fine, but as soon ngix was between my flask app and the client, my flask app would not see the headers any longer. It was kind of time consuming to debug.
This resolved my problem.
Hope it helps
The best way to do this is to have your files zipped and link to that:
The other solution can be found here: How to make a link open multiple pages when clicked
Which states the following:
HTML:
<a href="#" class="yourlink">Download</a>
JS:
$('a.yourlink').click(function(e) {
e.preventDefault();
window.open('mysite.com/file1');
window.open('mysite.com/file2');
window.open('mysite.com/file3');
});
Having said this, I would still go with zipping the file, as this implementation requires JavaScript and can also sometimes be blocked as popups.
Version 25.2.3 (and higher) of Android SDK Tools
package contains new tool - sdkmanager - which simplifies this task of installing build-tools from the command line.
It is located in android_sdk/tools/bin
folder.
Usage (from documentation):
sdkmanager packages [options]
The
packages
argument is an SDK-style path, wrapped in quotes (for example,"build-tools;25.0.0"
or"platforms;android-25"
). You can pass multiple package paths, separated with a space, but they must each be wrapped in their own set of quotes.
Example usage (on my Mac):
alex@mbpro:~/sdk/tools/bin$ ls ../../build-tools/
25.0.0/
alex@mbpro:~/sdk/tools/bin$ ./sdkmanager "build-tools;25.0.2"
done
alex@mbpro:~/sdk/tools/bin$ ls ../../build-tools/
25.0.0/ 25.0.2/
You can also specify various options, for example to force all connections to use HTTP (--no_https
), or in order to use proxy server (--proxy_host=address
and --proxy_port=port
).
To check the available options, use the --help
flag. On my machine (Mac), the output is as following:
alex@mbpro:~/sdk/tools/bin$ ./sdkmanager --help
Usage:
sdkmanager [--uninstall] [<common args>] \
[--package_file <package-file>] [<packages>...]
sdkmanager --update [<common args>]
sdkmanager --list [<common args>]
In its first form, installs, or uninstalls, or updates packages.
<package> is a sdk-style path (e.g. "build-tools;23.0.0" or
"platforms;android-23").
<package-file> is a text file where each line is a sdk-style path
of a package to install or uninstall.
Multiple --package_file arguments may be specified in combination
with explicit paths.
In its second form (with --update), currently installed packages are
updated to the latest version.
In its third form, all installed and available packages are printed out.
Common Arguments:
--sdk_root=<sdkRootPath>: Use the specified SDK root instead of the SDK containing this tool
--channel=<channelId>: Include packages in channels up to <channelId>.
Common channels are:
0 (Stable), 1 (Beta), 2 (Dev), and 3 (Canary).
--include_obsolete: With --list, show obsolete packages in the
package listing. With --update, update obsolete
packages as well as non-obsolete.
--no_https: Force all connections to use http rather than https.
--proxy=<http | socks>: Connect via a proxy of the given type.
--proxy_host=<IP or DNS address>: IP or DNS address of the proxy to use.
--proxy_port=<port #>: Proxy port to connect to.
* If the env var REPO_OS_OVERRIDE is set to "windows",
"macosx", or "linux", packages will be downloaded for that OS.
If you have seaborn
installed, an easier method that does not require you to perform pivot
:
import seaborn as sns
sns.lineplot(data=df, x='x', y='y', hue='color')
pandas
uses matplotlib
for basic dataframe plots. So, if you are using pandas
for basic plot you can use matplotlib for plot customization. However, I propose an alternative method here using seaborn
which allows more customization of the plot while not going into the basic level of matplotlib
.
Working Code:
import pandas as pd
import seaborn as sns
values = [[1, 2], [2, 5]]
df2 = pd.DataFrame(values, columns=['Type A', 'Type B'],
index=['Index 1', 'Index 2'])
ax= sns.lineplot(data=df2, markers= True)
ax.set(xlabel='xlabel', ylabel='ylabel', title='Video streaming dropout by category')
Other answers are great, I just want to add... a different approach usable in PowerShell: Install GNUWin32 utils and use grep to view the lines / redirect the output to file http://gnuwin32.sourceforge.net/
This overwrites the new file every time:
grep "step[49]" logIn.log > logOut.log
This appends the log output, in case you overwrite the logIn file and want to keep the data:
grep "step[49]" logIn.log >> logOut.log
Note: to be able to use GNUWin32 utils globally you have to add the bin folder to your system path.
The [] operator is called an indexer. You can provide indexers that take an integer, a string, or any other type you want to use as a key. The syntax is straightforward, following the same principles as property accessors.
For example, in your case where an int
is the key or index:
public int this[int index]
{
get => GetValue(index);
}
You can also add a set accessor so that the indexer becomes read and write rather than just read-only.
public int this[int index]
{
get => GetValue(index);
set => SetValue(index, value);
}
If you want to index using a different type, you just change the signature of the indexer.
public int this[string index]
...
I was finally able to get this to work for my needs.
The old voted up code does not run on windows 10 system (at least not mine). The referenced MS library link below provides current examples on how to make this work. My example uses them with late bindings.
https://docs.microsoft.com/en-us/windows/win32/shell/folder-getdetailsof.
The attribute codes were different on my computer and like someone mentioned above most return blank values even if they are not. I used a for loop to cycle through all of them and found out that Title and Subject can still be accessed which is more then enough for my purposes.
Private Sub MySubNamek()
Dim objShell As Object 'Shell
Dim objFolder As Object 'Folder
Set objShell = CreateObject("Shell.Application")
Set objFolder = objShell.NameSpace("E:\MyFolder")
If (Not objFolder Is Nothing) Then
Dim objFolderItem As Object 'FolderItem
Set objFolderItem = objFolder.ParseName("Myfilename.txt")
For i = 0 To 288
szItem = objFolder.GetDetailsOf(objFolderItem, i)
Debug.Print i & " - " & szItem
Next
Set objFolderItem = Nothing
End If
Set objFolder = Nothing
Set objShell = Nothing
End Sub
//defines an array of 280 pointers (1120 or 2240 bytes)
int *pointer1 [280];
//defines a pointer (4 or 8 bytes depending on 32/64 bits platform)
int (*pointer2)[280]; //pointer to an array of 280 integers
int (*pointer3)[100][280]; //pointer to an 2D array of 100*280 integers
Using pointer2
or pointer3
produce the same binary except manipulations as ++pointer2
as pointed out by WhozCraig.
I recommend using typedef
(producing same binary code as above pointer3
)
typedef int myType[100][280];
myType *pointer3;
Note: Since C++11, you can also use keyword using
instead of typedef
using myType = int[100][280];
myType *pointer3;
in your example:
myType *pointer; // pointer creation
pointer = &tab1; // assignation
(*pointer)[5][12] = 517; // set (write)
int myint = (*pointer)[5][12]; // get (read)
Note: If the array tab1
is used within a function body => this array will be placed within the call stack memory. But the stack size is limited. Using arrays bigger than the free memory stack produces a stack overflow crash.
The full snippet is online-compilable at gcc.godbolt.org
int main()
{
//defines an array of 280 pointers (1120 or 2240 bytes)
int *pointer1 [280];
static_assert( sizeof(pointer1) == 2240, "" );
//defines a pointer (4 or 8 bytes depending on 32/64 bits platform)
int (*pointer2)[280]; //pointer to an array of 280 integers
int (*pointer3)[100][280]; //pointer to an 2D array of 100*280 integers
static_assert( sizeof(pointer2) == 8, "" );
static_assert( sizeof(pointer3) == 8, "" );
// Use 'typedef' (or 'using' if you use a modern C++ compiler)
typedef int myType[100][280];
//using myType = int[100][280];
int tab1[100][280];
myType *pointer; // pointer creation
pointer = &tab1; // assignation
(*pointer)[5][12] = 517; // set (write)
int myint = (*pointer)[5][12]; // get (read)
return myint;
}
#Single line
'''
multi-line
comment
'''
"""
also,
multi-line comment
"""
https://github.com/koffsyrup/FileSaver.js#examples
Saving text(All Browsers)
saveTextAs("Hi,This,is,a,CSV,File", "test.csv");
saveTextAs("<div>Hello, world!</div>", "test.html");
Saving text(HTML 5)
var blob = new Blob(["Hello, world!"], {type: "text/plain;charset=utf-8"});
saveAs(blob, "hello world.txt");
You can remove item from list view like this: or you can choose on your Button event which item have to be removed
public class Third extends ListActivity {
private ArrayAdapter<String> adapter;
private List<String> liste;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_third);
String[] values = new String[] { "Android", "iPhone", "WindowsMobile",
"Blackberry", "WebOS", "Ubuntu", "Windows7", "Max OS X",
"Linux", "OS/2" };
liste = new ArrayList<String>();
Collections.addAll(liste, values);
adapter = new ArrayAdapter<String>(this,
android.R.layout.simple_list_item_1, liste);
setListAdapter(adapter);
}
@Override
protected void onListItemClick(ListView l, View v, int position, long id) {
liste.remove(position);
adapter.notifyDataSetChanged();
}
}
A C++ identifier is a name used to identify a variable, function, class, module, or any other user-defined item. In C++ all names have to be declared before they are used. If you try to use the name of a such that hasn't been declared you will get an "undeclared identifier" compile-error.
According to the documentation, the declaration of printf()
is in cstdio
i.e. you have to include it, before using the function.
The history limit is a pane attribute that is fixed at the time of pane creation and cannot be changed for existing panes. The value is taken from the history-limit
session option (the default value is 2000).
To create a pane with a different value you will need to set the appropriate history-limit
option before creating the pane.
To establish a different default, you can put a line like the following in your .tmux.conf
file:
set-option -g history-limit 3000
Note: Be careful setting a very large default value, it can easily consume lots of RAM if you create many panes.
For a new pane (or the initial pane in a new window) in an existing session, you can set that session’s history-limit
. You might use a command like this (from a shell):
tmux set-option history-limit 5000 \; new-window
For (the initial pane of the initial window in) a new session you will need to set the “global” history-limit
before creating the session:
tmux set-option -g history-limit 5000 \; new-session
Note: If you do not re-set the history-limit
value, then the new value will be also used for other panes/windows/sessions created in the future; there is currently no direct way to create a single new pane/window/session with its own specific limit without (at least temporarily) changing history-limit
(though show-option
(especially in 1.7 and later) can help with retrieving the current value so that you restore it later).
The bottom line is not all browsers will actually look for your favicon.ico file. Some browsers allow users to choose whether or not it should automatically look. Therefore, in order to ensure that it will always appear and get looked at, you do have to define it.
in React v16.8+
function component, you can use useState()
to create a function state that lets you update the parent state, then pass it on to child as a props attribute, then inside the child component you can trigger the parent state function, the following is a working snippet:
const { useState , useEffect } = React;_x000D_
_x000D_
function Timer({ setParentCounter }) {_x000D_
const [counter, setCounter] = React.useState(0);_x000D_
_x000D_
useEffect(() => {_x000D_
let countersystem;_x000D_
countersystem = setTimeout(() => setCounter(counter + 1), 1000);_x000D_
_x000D_
return () => {_x000D_
clearTimeout(countersystem);_x000D_
};_x000D_
}, [counter]);_x000D_
_x000D_
return (_x000D_
<div className="App">_x000D_
<button_x000D_
onClick={() => {_x000D_
setParentCounter(counter);_x000D_
}}_x000D_
>_x000D_
Set parent counter value_x000D_
</button>_x000D_
<hr />_x000D_
<div>Child Counter: {counter}</div>_x000D_
</div>_x000D_
);_x000D_
}_x000D_
_x000D_
function App() {_x000D_
const [parentCounter, setParentCounter] = useState(0);_x000D_
_x000D_
return (_x000D_
<div className="App">_x000D_
Parent Counter: {parentCounter}_x000D_
<hr />_x000D_
<Timer setParentCounter={setParentCounter} />_x000D_
</div>_x000D_
);_x000D_
}_x000D_
_x000D_
ReactDOM.render(<App />, document.getElementById('react-root'));
_x000D_
<script src="https://cdnjs.cloudflare.com/ajax/libs/react/16.8.4/umd/react.production.min.js"></script>_x000D_
<script src="https://cdnjs.cloudflare.com/ajax/libs/react-dom/16.8.4/umd/react-dom.production.min.js"></script>_x000D_
<div id="react-root"></div>
_x000D_
Don't forget that if you are concerned about reading in a file that might have huge lines that could swamp your RAM during runtime, you can always read the file piece-meal. See "Why slurping a file is bad".
File.open('file_path', 'rb') do |io|
while chunk = io.read(16 * 1024) do
something_with_the chunk
# like stream it across a network
# or write it to another file:
# other_io.write chunk
end
end
A 400 means that the request was malformed. In other words, the data stream sent by the client to the server didn't follow the rules.
In the case of a REST API with a JSON payload, 400's are typically, and correctly I would say, used to indicate that the JSON is invalid in some way according to the API specification for the service.
By that logic, both the scenarios you provided should be 400s.
Imagine instead this were XML rather than JSON. In both cases, the XML would never pass schema validation--either because of an undefined element or an improper element value. That would be a bad request. Same deal here.
Per material design guidelines, you need to use the style like below code
<style name="MyButton" parent="Theme.AppCompat.Light>
<item name="colorControlHighlight">#F36F21</item>
<item name="colorControlHighlight">#FF8D00</item>
</style>
and in layout add this property to your button
android:theme="@style/MyButton"
It appears this can be done. I'm unable to determine the version of GCC that it was added, but it was sometime before June 2010.
Here's an example:
#pragma GCC diagnostic error "-Wuninitialized"
foo(a); /* error is given for this one */
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wuninitialized"
foo(b); /* no diagnostic for this one */
#pragma GCC diagnostic pop
foo(c); /* error is given for this one */
#pragma GCC diagnostic pop
foo(d); /* depends on command line options */
You can do this:
<div style="text-align: center">
<a href="contact.html" class="button large hpbottom">Get Started</a>
</div>
In short just replace the content of config.inc.php from line 50-69 with...
$cfg['Servers'][$i]['pma__bookmark'] = 'pma__bookmark';
$cfg['Servers'][$i]['pma__relation'] = 'pma__relation';
$cfg['Servers'][$i]['pma__table_info'] = 'pma__table_info';
$cfg['Servers'][$i]['pma__table_coords'] = 'pma__table_coords';
$cfg['Servers'][$i]['pma__pdf_pages'] = 'pma__pdf_pages';
$cfg['Servers'][$i]['pma__column_info'] = 'pma__column_info';
$cfg['Servers'][$i]['pma__table_uiprefs'] = 'pma__history';
$cfg['Servers'][$i]['pma__table_uiprefs'] = 'pma__table_uiprefs';
$cfg['Servers'][$i]['pma__tracking'] = 'pma__tracking';
$cfg['Servers'][$i]['pma__userconfig'] = 'pma__userconfig';
$cfg['Servers'][$i]['pma__recent'] = 'pma__recent';
$cfg['Servers'][$i]['pma__users'] = 'pma__users';
$cfg['Servers'][$i]['pma__usergroups'] = 'pma__usergroups';
$cfg['Servers'][$i]['pma__navigationhiding'] = 'pma__navigationhiding';
$cfg['Servers'][$i]['pma__savedsearches'] = 'pma__savedsearches';
$cfg['Servers'][$i]['pma__central_columns'] = 'pma__central_columns';
$cfg['Servers'][$i]['pma__designer_coords'] = 'pma__designer_coords';
$cfg['Servers'][$i]['pma__designer_settings'] = 'pma__designer_settings';
$cfg['Servers'][$i]['pma__export_templates'] = 'pma__export_templates';
$cfg['Servers'][$i]['pma__favorite'] = 'pma__favorite';
Python might be overkill for this.
for file in *; do mencoder -some options $file; rm -f $file ; done
It is not necessary for the table to be sorted for the following formula to return a 1 for each unique value present.
assuming the table range for the data presented in the question is A1:B7 enter the following formula in Cell C1:
=IF(COUNTIF($B$1:$B1,B1)>1,0,COUNTIF($B$1:$B1,B1))
Copy that formula to all rows and the last row will contain:
=IF(COUNTIF($B$1:$B7,B7)>1,0,COUNTIF($B$1:$B7,B7))
This results in a 1 being returned the first time a record is found and 0 for all times afterwards.
Simply sum the column in your pivot table
for any1 using ie8 and dont want to use a plugin i've made something inspired by Rohit Azad and Bacotasan's blog, i just added a span using JS to show the selected value.
the html:
<div class="styled-select">
<select>
<option>Here is the first option</option>
<option>The second option</option>
</select>
<span>Here is the first option</span>
</div>
the css (i used only an arrow for BG but you could put a full image and drop the positioning):
.styled-select div
{
display:inline-block;
border: 1px solid darkgray;
width:100px;
background:url("/Style Library/Nifgashim/Images/drop_arrrow.png") no-repeat 10px 10px;
position:relative;
}
.styled-select div select
{
height: 30px;
width: 100px;
font-size:14px;
font-family:ariel;
-moz-opacity: 0.00;
opacity: .00;
filter: alpha(opacity=00);
}
.styled-select div span
{
position: absolute;
right: 10px;
top: 6px;
z-index: -5;
}
the js:
$(".styled-select select").change(function(e){
$(".styled-select span").html($(".styled-select select").val());
});
You can use the he library, available from https://github.com/mathiasbynens/he
Example:
console.log(he.decode("Jörg & Jürgen rocked to & fro "));
// Logs "Jörg & Jürgen rocked to & fro"
I challenged the library's author on the question of whether there was any reason to use this library in clientside code in favour of the <textarea>
hack provided in other answers here and elsewhere. He provided a few possible justifications:
If you're using node.js serverside, using a library for HTML encoding/decoding gives you a single solution that works both clientside and serverside.
Some browsers' entity decoding algorithms have bugs or are missing support for some named character references. For example, Internet Explorer will both decode and render non-breaking spaces (
) correctly but report them as ordinary spaces instead of non-breaking ones via a DOM element's innerText
property, breaking the <textarea>
hack (albeit only in a minor way). Additionally, IE 8 and 9 simply don't support any of the new named character references added in HTML 5. The author of he also hosts a test of named character reference support at http://mathias.html5.org/tests/html/named-character-references/. In IE 8, it reports over one thousand errors.
If you want to be insulated from browser bugs related to entity decoding and/or be able to handle the full range of named character references, you can't get away with the <textarea>
hack; you'll need a library like he.
He just darn well feels like doing things this way is less hacky.
Beamer just delegates responsibility for managing layout of itemize
environments back to the base LaTeX packages, so there's nothing funky you need to do in Beamer itself to alter the apperaance / layout of your lists.
Since Beamer redefines itemize, item, etc., the fully proper way to manipulate things like indentation is to redefine the Beamer templates. I get the impression that you're not looking to go that far, but if that's not the case, let me know and I'll elaborate.
There are at least three ways of accomplishing your goal from within your document, without mussing about with Beamer templates.
itemize
In the following code snippet, you can change the value of \itemindent
from 0em
to whatever you please, including negative values. 0em
is the default item indentation.
The advantage of this method is that the list is styled normally. The disadvantage is that Beamer's redefinition of itemize
and \item
means that the number of paramters that can be manipulated to change the list layout is limited. It can be very hard to get the spacing right with multi-line items.
\begin{itemize}
\setlength{\itemindent}{0em}
\item This is a normally-indented item.
\end{itemize}
list
In the following code snippet, the second parameter to \list
is the bullet to use, and the third parameter is a list of layout parameters to change. The \leftmargin
parameter adjusts the indentation of the entire list item and all of its rows; \itemindent
alters the indentation of subsequent lines.
The advantage of this method is that you have all of the flexibility of lists in non-Beamer LaTeX. The disadvantage is that you have to setup the bullet style (and other visual elements) manually (or identify the right command for the template you're using). Note that if you leave the second argument empty, no bullet will be displayed and you'll save some horizontal space.
\begin{list}{$\square$}{\leftmargin=1em \itemindent=0em}
\item This item uses the margin and indentation provided above.
\end{list}
customlist
environmentThe shortcomings of the list
solution can be ameliorated by defining a new customlist
environment that basically redefines the itemize
environment from Beamer but also incorporates the \leftmargin
and \itemindent
(etc.) parameters. Put the following in your preamble:
\makeatletter
\newenvironment{customlist}[2]{
\ifnum\@itemdepth >2\relax\@toodeep\else
\advance\@itemdepth\@ne%
\beamer@computepref\@itemdepth%
\usebeamerfont{itemize/enumerate \beameritemnestingprefix body}%
\usebeamercolor[fg]{itemize/enumerate \beameritemnestingprefix body}%
\usebeamertemplate{itemize/enumerate \beameritemnestingprefix body begin}%
\begin{list}
{
\usebeamertemplate{itemize \beameritemnestingprefix item}
}
{ \leftmargin=#1 \itemindent=#2
\def\makelabel##1{%
{%
\hss\llap{{%
\usebeamerfont*{itemize \beameritemnestingprefix item}%
\usebeamercolor[fg]{itemize \beameritemnestingprefix item}##1}}%
}%
}%
}
\fi
}
{
\end{list}
\usebeamertemplate{itemize/enumerate \beameritemnestingprefix body end}%
}
\makeatother
Now, to use an itemized list with custom indentation, you can use the following environment. The first argument is for \leftmargin
and the second is for \itemindent
. The default values are 2.5em and 0em respectively.
\begin{customlist}{2.5em}{0em}
\item Any normal item can go here.
\end{customlist}
A custom bullet style can be incorporated into the customlist
solution using the standard Beamer mechanism of \setbeamertemplate
. (See the answers to this question on the TeX Stack Exchange for more information.)
Alternatively, the bullet style can just be modified directly within the environment, by replacing \usebeamertemplate{itemize \beameritemnestingprefix item}
with whatever bullet style you'd like to use (e.g. $\square$
).
Either modify the pattern beforehand so that it only matches the entire string:
var r = /^a$/
or check afterward whether the pattern matched the whole string:
function matchExact(r, str) {
var match = str.match(r);
return match && str === match[0];
}
If you want a list of columns of a certain type, you can use groupby
:
>>> df = pd.DataFrame([[1, 2.3456, 'c', 'd', 78]], columns=list("ABCDE"))
>>> df
A B C D E
0 1 2.3456 c d 78
[1 rows x 5 columns]
>>> df.dtypes
A int64
B float64
C object
D object
E int64
dtype: object
>>> g = df.columns.to_series().groupby(df.dtypes).groups
>>> g
{dtype('int64'): ['A', 'E'], dtype('float64'): ['B'], dtype('O'): ['C', 'D']}
>>> {k.name: v for k, v in g.items()}
{'object': ['C', 'D'], 'int64': ['A', 'E'], 'float64': ['B']}
Seems like you need to strip the URL from the URL, so you can do it in a following way:
request.getRequestURL().toString().replace(request.getRequestURI(), "")
To be complete, a short introduction for "shell noobs":
First of all, start your shell - you can find it inside the common desktop environments under the name "Terminal" or "Shell" somewhere in the desktops application menu.
You can also try using the key combo CTRL+F2, followed by one of those commands (depending on the desktop envrionment you're using) and the ENTER key:
xfce4-terminal
gnome-console
terminal
rxvt
konsole
If all of the above fail, try using xterm
- it'll work in most cases.
Hint for the following commands: Execute the commands without the $
- it's just a marker identifying that you're on the shell.
After that just fire up mongod
with the --version
flag:
$ mongod --version
It shows you then something like
$ mongod --version
db version v2.4.6
Wed Oct 16 16:17:00.241 git version: nogitversion
To update it just execute
$ sudo apt-get update
and then
$ sudo apt-get install mongodb
// Get a notification builder that's compatible with platform versions
// >= 4
NotificationCompat.Builder builder = new NotificationCompat.Builder(
this);
builder.setSound(soundUri);
builder.setAutoCancel(true);
this works if you are using a notification builder...
Handles either type of line break
str.replace(new RegExp('\r?\n','g'), '<br />');
Now dstat provides the feature to find out in your running system which process is candidate for getting killed by oom mechanism
dstat --top-oom
--out-of-memory---
kill score
java 77
java 77
java 77
and as per man page
--top-oom
show process that will be killed by OOM the first
You need a dict
:
my_dict = {'cheese': 'cake'}
Example code (from the docs):
>>> a = dict(one=1, two=2, three=3)
>>> b = {'one': 1, 'two': 2, 'three': 3}
>>> c = dict(zip(['one', 'two', 'three'], [1, 2, 3]))
>>> d = dict([('two', 2), ('one', 1), ('three', 3)])
>>> e = dict({'three': 3, 'one': 1, 'two': 2})
>>> a == b == c == d == e
True
You can read more about dictionaries here.
use this command mode option
ggguG
gg - Goto the first line
g - start to converting from current line
u - Convert into lower case for all characters
G - To end of the file.
To reset your settings
Click Finish.
The Reset Complete page alerts you to any problems encountered during the reset.
You can use require as well:
const moduleHolder = []
function loadModules(path) {
let stat = fs.lstatSync(path)
if (stat.isDirectory()) {
// we have a directory: do a tree walk
const files = fs.readdirSync(path)
let f,
l = files.length
for (var i = 0; i < l; i++) {
f = pathModule.join(path, files[i])
loadModules(f)
}
} else {
// we have a file: load it
var controller = require(path)
moduleHolder.push(controller)
}
}
Then use your moduleHolder with dynamically loaded controllers:
loadModules(DIR)
for (const controller of moduleHolder) {
controller(app, db)
}
Call list()
on the dictionary instead:
keys = list(test)
In Python 3, the dict.keys()
method returns a dictionary view object, which acts as a set. Iterating over the dictionary directly also yields keys, so turning a dictionary into a list results in a list of all the keys:
>>> test = {'foo': 'bar', 'hello': 'world'}
>>> list(test)
['foo', 'hello']
>>> list(test)[0]
'foo'
Try using Ctrl-click on the multiple places you want the cursors. Ctrl-D is for multiple incremental finds.
I was looking similar but I wanted the difference in either list (uncommon elements between the 2 lists).
Let say I have:
List<String> oldKeys = Arrays.asList("key0","key1","key2","key5");
List<String> newKeys = Arrays.asList("key0","key2","key5", "key6");
And I wanted to know which key has been added and which key is removed i.e I wanted to get (key1, key6)
Using org.apache.commons.collections.CollectionUtils
List<String> list = new ArrayList<>(CollectionUtils.disjunction(newKeys, oldKeys));
Result
["key1", "key6"]
readonly="readonly"
is xhtml syntax. In xhtml boolean attributes are written this way. In xhtml 'attribute minimization' (<input type="checkbox" checked>
) isn't allowed, so this is the valid way to include boolean attributes in xhtml. See this page for more.information.
If your document type is xhtml transitional or strict and you want to validate it, use readonly="readonly
otherwise readonly
is sufficient.
Addition to the accepted answer:
struct immutableint { immutableint(int i) : i_(i) {} const int& get() const { return i_; } private: int i_; };
I'd argue that this example is not okay and should be avoided if possible. Why? It is very easy to end-up with a dangling reference.
To illustrate the point with an example:
struct Foo
{
Foo(int i = 42) : boo_(i) {}
immutableint boo()
{
return boo_;
}
private:
immutableint boo_;
};
entering the danger-zone:
Foo foo;
const int& dangling = foo.boo().get(); // dangling reference!
When you use a web service you have a client and a server:
When you use a message queue like RabbitMQ, Beanstalkd, ActiveMQ, IBM MQ Series, Tuxedo you expect different and more fault tolerant results:
Message Queues has a lot more features but this is some rule of thumb to decide if you want to handle error conditions yourself or leave them to the message queue.