Concatenate a list of pandas dataframes together

Question

I have a list of Pandas dataframes that I would like to combine into one Pandas dataframe   I am using Python 2 7 10 and Pandas 0 16 2  I created the list of dataframes from   import pandas as pd dfs      sqlall    select   from mytable   for chunk in pd read sql query sqlall   cnxn  chunksize 10000       dfs append chunk    This returns a list of dataframes  type dfs 0   Out 6   pandas core frame DataFrame  type dfs  Out 7   list  len dfs  Out 8   408   Here is some sample data    sample dataframes d1   pd DataFrame   one     1   2   3   4     two     4   3   2   1     d2   pd DataFrame   one     5   6   7   8     two     9   10   11   12     d3   pd DataFrame   one     15   16   17   18     two     19   10   11   12        list of dataframes mydfs    d1  d2  d3    I would like to combine d1  d2  and d3 into one pandas dataframe   Alternatively  a method of reading a large-ish table directly into a dataframe when using the chunksize option would be very helpful

User · Answer

If the dataframes DO NOT all have the same columns try the following   df   pd DataFrame from dict map dict df list

User · Answer

concat also works nicely with a list comprehension pulled using the  loc  command against an existing dataframe   df   pd read csv    data csv     ie  Dataframe pulled from csv file with a  userID  column  review ids     1   2   3     ie  ID values to grab from DataFrame    Gets rows in df where IDs match in the userID column and combines them   dfa   pd concat  df loc df  userID      x  for x in review ids

User · Answer

You also can do it with functional programming  from functools import reduce reduce lambda df1  df2  df1 merge df2   quot outer quot    mydfs

User · Answer

Given that all the dataframes have the same columns  you can simply concat them   import pandas as pd df   pd concat list of dataframes

[python] Concatenate a list of pandas dataframes together

Examples related to python

Examples related to pandas

Examples related to dataframe

Examples related to concat