and boolean vs bitwise - Why difference in behavior with lists vs numpy arrays

Question

What explains the difference in behavior of boolean and bitwise operations on lists vs NumPy arrays    I m confused about the appropriate use of  amp  vs and in Python  illustrated in the following examples   mylist1    True   True   True  False   True  mylist2    False  True  False   True  False    gt  gt  gt  len mylist1     len mylist2  True    ---- Example 1 ----  gt  gt  gt  mylist1 and mylist2  False  True  False  True  False    I would have expected  False  True  False  False  False     ---- Example 2 ----  gt  gt  gt  mylist1  amp  mylist2 TypeError  unsupported operand type s  for  amp    list  and  list    Why not just like example 1    gt  gt  gt  import numpy as np    ---- Example 3 ----  gt  gt  gt  np array mylist1  and np array mylist2  ValueError  The truth value of an array with more than one element is ambiguous  Use a any   or a all     Why not just like Example 4     ---- Example 4 ----  gt  gt  gt  np array mylist1   amp  np array mylist2  array  False   True  False  False  False   dtype bool    This is the output I was expecting    This answer and this answer helped me understand that and is a boolean operation but  amp  is a bitwise operation    I read about bitwise operations to better understand the concept  but I am struggling to use that information to make sense of my above 4 examples   Example 4 led me to my desired output  so that is fine  but I am still confused about when how why I should use and vs  amp   Why do lists and NumPy arrays behave differently with these operators    Can anyone help me understand the difference between boolean and bitwise operations to explain why they handle lists and NumPy arrays differently

User · Answer

Example 1   This is how the and operator works    x and y      if x is false  then x  else y  So in other words  since mylist1 is not False  the result of the expression is mylist2   Only empty lists evaluate to False    Example 2   The  amp  operator is for a bitwise and  as you mention  Bitwise operations only work on numbers  The result of a  amp  b is a number composed of 1s in bits that are 1 in both a and b  For example     gt  gt  gt  3  amp  1 1   It s easier to see what s happening using a binary literal  same numbers as above      gt  gt  gt  0b0011  amp  0b0001 0b0001   Bitwise operations are similar in concept to boolean  truth  operations  but they work only on bits    So  given a couple statements about my car   My car is red My car has wheels   The logical  and  of these two statements is        is my car red   and  does car have wheels      logical true of false value   Both of which are true  for my car at least  So the value of the statement as a whole is logically true    The bitwise  and  of these two statements is a little more nebulous       the numeric value of the statement  my car is red    amp   the numeric value of the statement  my car has wheels      number   If python knows how to convert the statements to numeric values  then it will do so and compute the bitwise-and of the two values  This may lead you to believe that  amp  is interchangeable with and  but as with the above example they are different things  Also  for the objects that can t be converted  you ll just get a TypeError    Example 3 and 4   Numpy implements arithmetic operations for arrays       Arithmetic and comparison operations on ndarrays are defined as element-wise operations  and generally yield ndarray objects as results    But does not implement logical operations for arrays  because you can t overload logical operators in python  That s why example three doesn t work  but example four does    So to answer your and vs  amp  question  Use and    The bitwise operations are used for examining the structure of a number  which bits are set  which bits aren t set   This kind of information is mostly used in low-level operating system interfaces  unix permission bits  for example   Most python programs won t need to know that    The logical operations  and  or  not   however  are used all the time

User · Answer

For the first example and base on the django s doc It will always return the second list  indeed a non empty list is see as a True value for Python thus python return the  last  True value so the second list  In  74   mylist1    False  In  75   mylist2    False  True  False   True  False  In  76   mylist1 and mylist2 Out 76    False  True  False  True  False  In  77   mylist2 and mylist1 Out 77    False

User · Answer

The short-circuiting boolean operators  and  or  can t be overriden because there is no satisfying way to do this without introducing new language features or sacrificing short circuiting  As you may or may not know  they evaluate the first operand for its truth value  and depending on that value  either evaluate and return the second argument  or don t evaluate the second argument and return the first   something true and x - gt  x something false and x - gt  something false something true or x - gt  something true something false or x - gt  x   Note that the  result of evaluating the  actual operand is returned  not truth value thereof   The only way to customize their behavior is to override   nonzero    renamed to   bool   in Python 3   so you can affect which operand gets returned  but not return something different  Lists  and other collections  are defined to be  truthy  when they contain anything at all  and  falsey  when they are empty   NumPy arrays reject that notion  For the use cases they aim at  two different notions of truth are common   1  Whether any element is true  and  2  whether all elements are true  Since these two are completely  and silently  incompatible  and neither is clearly more correct or more common  NumPy refuses to guess and requires you to explicitly use  any   or  all      amp  and    and not  by the way  can be fully overriden  as they don t short circuit  They can return anything at all when overriden  and NumPy makes good use of that to do element-wise operations  as they do with practically any other scalar operation  Lists  on the other hand  don t broadcast operations across their elements  Just as mylist1 - mylist2 doesn t mean anything and mylist1   mylist2 means something completely different  there is no  amp  operator for lists

User · Answer

In Python an expression of X and Y returns Y  given that bool X     True or any of X or Y evaluate to False  e g    True and 20   gt  gt  gt  20  False and 20  gt  gt  gt  False  20 and     gt  gt  gt      Bitwise operator is simply not defined for lists  But it is defined for integers - operating over the binary representation of the numbers  Consider 16  01000  and 31  11111    16  amp  31  gt  gt  gt  16  NumPy is not a psychic  it does not know  whether you mean that e g   False  False  should be equal to True in a logical expression  In this it overrides a standard Python behaviour  which is   Any empty collection with len collection     0 is False   Probably an expected behaviour of NumPy s arrays s  amp  operator

User · Answer

and tests whether both expressions are logically True while  amp   when used with True False values  tests if both are True   In Python  empty built-in objects are typically treated as logically False while non-empty built-ins are logically True  This facilitates the common use case where you want to do something if a list is empty and something else if the list is not  Note that this means that the list  False  is logically True    gt  gt  gt  if  False          print  True      True   So in Example 1  the first list is non-empty and therefore logically True  so the truth value of the and is the same as that of the second list   In our case  the second list is non-empty and therefore logically True  but identifying that would require an unnecessary step of calculation    For example 2  lists cannot meaningfully be combined in a bitwise fashion because they can contain arbitrary unlike elements  Things that can be combined bitwise include  Trues and Falses  integers   NumPy objects  by contrast  support vectorized calculations  That is  they let you perform the same operations on multiple pieces of data   Example 3 fails because NumPy arrays  of length   1  have no truth value as this prevents vector-based logic confusion   Example 4 is simply a vectorized bit and operation   Bottom Line   If you are not dealing with arrays and are not performing math manipulations of integers  you probably want and  If you have vectors of truth values that you wish to combine  use numpy with  amp

User · Answer

Operations with a Python list operate on the list  list1 and list2 will check if list1 is empty  and return list1 if it is  and list2 if it isn t  list1   list2 will append list2 to list1  so you get a new list with len list1    len list2  elements    Operators that only make sense when applied element-wise  such as  amp   raise a TypeError  as element-wise operations aren t supported without looping through the elements   Numpy arrays support element-wise operations  array1  amp  array2 will calculate the bitwise or for each corresponding element in array1 and array2  array1   array2 will calculate the sum for each corresponding element in array1 and array2    This does not work for and and or   array1 and array2 is essentially a short-hand for the following code   if bool array1       return array2 else      return array1   For this you need a good definition of bool array1   For global operations like used on Python lists  the definition is that bool list     True if list is not empty  and False if it is empty  For numpy s element-wise operations  there is some disambiguity whether to check if any element evaluates to True  or all elements evaluate to True  Because both are arguably correct  numpy doesn t guess and raises a ValueError when bool   is  indirectly  called on an array

User · Answer

About list  First a very important point  from which everything will follow  I hope    In ordinary Python  list is not special in any way  except having cute syntax for constructing  which is mostly a historical accident   Once a list  3 2 6  is made  it is for all intents and purposes just an ordinary Python object  like a number 3  set  3 7   or a function lambda x  x 5    Yes  it supports changing its elements  and it supports iteration  and many other things  but that s just what a type is  it supports some operations  while not supporting some others  int supports raising to a power  but that doesn t make it very special - it s just what an int is  lambda supports calling  but that doesn t make it very special - that s what lambda is for  after all     About and  and is not an operator  you can call it  operator   but you can call  for  an operator too    Operators in Python are  implemented through  methods called on objects of some type  usually written as part of that type  There is no way for a method to hold an evaluation of some of its operands  but and can  and must  do that   The consequence of that is that and cannot be overloaded  just like for cannot be overloaded  It is completely general  and communicates through a specified protocol  What you can do is customize your part of the protocol  but that doesn t mean you can alter the behavior of and completely  The protocol is   Imagine Python interpreting  a and b   this doesn t happen literally this way  but it helps understanding   When it comes to  and   it looks at the object it has just evaluated  a   and asks it  are you true   NOT  are you True   If you are an author of a s class  you can customize this answer  If a answers  no   and  skips b completely  it is not evaluated at all  and  says  a is my result  NOT  False is my result    If a doesn t answer  and asks it  what is your length   Again  you can customize this as an author of a s class   If a answers 0  and does the same as above - considers it false  NOT False   skips b  and gives a as result   If a answers something other than 0 to the second question   what is your length    or it doesn t answer at all  or it answers  yes  to the first one   are you true    and evaluates b  and says  b is my result  Note that it does NOT ask b any questions   The other way to say all of this is that a and b is almost the same as b if a else a  except a is evaluated only once   Now sit for a few minutes with a pen and paper  and convince yourself that when  a b  is a subset of  True False   it works exactly as you would expect of Boolean operators  But I hope I have convinced you it is much more general  and as you ll see  much more useful this way   Putting those two together  Now I hope you understand your example 1  and doesn t care if mylist1 is a number  list  lambda or an object of a class Argmhbl  It just cares about mylist1 s answer to the questions of the protocol  And of course  mylist1 answers 5 to the question about length  so  and returns mylist2  And that s it  It has nothing to do with elements of mylist1 and mylist2 - they don t enter the picture anywhere   Second example   amp  on list  On the other hand   amp  is an operator like any other  like   for example  It can be defined for a type by defining a special method on that class  int defines it as bitwise  and   and bool defines it as logical  and   but that s just one option  for example  sets and some other objects like dict keys views define it as a set intersection  list just doesn t define it  probably because Guido didn t think of any obvious way of defining it   numpy  On the other leg -D  numpy arrays are special  or at least they are trying to be  Of course  numpy array is just a class  it cannot override and in any way  so it does the next best thing  when asked  are you true   numpy array raises a ValueError  effectively saying  please rephrase the question  my view of truth doesn t fit into your model    Note that the ValueError message doesn t speak about and - because numpy array doesn t know who is asking it the question  it just speaks about truth    For  amp   it s completely different story  numpy array can define it as it wishes  and it defines  amp  consistently with other operators  pointwise  So you finally get what you want   HTH

User · Answer

Good question  Similar to the observation you have about examples 1 and 4  or should I say 1  amp  4      over logical and bitwise  amp  operators  I experienced on sum operator  The numpy sum and py sum behave differently as well  For example   Suppose  mat  is a numpy 5x5 2d array such as   array    1   2   3   4   5            6   7   8   9  10           11  12  13  14  15           16  17  18  19  20           21  22  23  24  25      Then numpy sum mat  gives total sum of the entire matrix  Whereas the built-in sum from Python such as sum mat  totals along the axis only  See below   np sum mat      -- gt  gives 325 sum mat         -- gt  gives array  55  60  65  70  75

[python] 'and' (boolean) vs '&' (bitwise) - Why difference in behavior with lists vs numpy arrays?

Examples related to python

Examples related to numpy

Examples related to bit-manipulation

Examples related to boolean-expression

Examples related to ampersand