[python] How to merge a Series and DataFrame

If you came here looking for information on how to merge a DataFrame and Series on the index, please look at this answer.

The OP's original intention was to ask how to assign series elements as columns to another DataFrame. If you are interested in knowing the answer to this, look at the accepted answer by EdChum.


Best I can come up with is

df = pd.DataFrame({'a':[1, 2], 'b':[3, 4]})  # see EDIT below
s = pd.Series({'s1':5, 's2':6})

for name in s.index:
    df[name] = s[name]

   a  b  s1  s2
0  1  3   5   6
1  2  4   5   6

Can anybody suggest better syntax / faster method?

My attempts:

df.merge(s)
AttributeError: 'Series' object has no attribute 'columns'

and

df.join(s)
ValueError: Other Series must have a name

EDIT The first two answers posted highlighted a problem with my question, so please use the following to construct df:

df = pd.DataFrame({'a':[np.nan, 2, 3], 'b':[4, 5, 6]}, index=[3, 5, 6])

with the final result

    a  b  s1  s2
3 NaN  4   5   6
5   2  5   5   6
6   3  6   5   6

This question is related to python pandas dataframe

The answer is


If I could suggest setting up your dataframes like this (auto-indexing):

df = pd.DataFrame({'a':[np.nan, 1, 2], 'b':[4, 5, 6]})

then you can set up your s1 and s2 values thus (using shape() to return the number of rows from df):

s = pd.DataFrame({'s1':[5]*df.shape[0], 's2':[6]*df.shape[0]})

then the result you want is easy:

display (df.merge(s, left_index=True, right_index=True))

Alternatively, just add the new values to your dataframe df:

df = pd.DataFrame({'a':[nan, 1, 2], 'b':[4, 5, 6]})
df['s1']=5
df['s2']=6
display(df)

Both return:

     a  b  s1  s2
0  NaN  4   5   6
1  1.0  5   5   6
2  2.0  6   5   6

If you have another list of data (instead of just a single value to apply), and you know it is in the same sequence as df, eg:

s1=['a','b','c']

then you can attach this in the same way:

df['s1']=s1

returns:

     a  b s1
0  NaN  4  a
1  1.0  5  b
2  2.0  6  c

You can easily set a pandas.DataFrame column to a constant. This constant can be an int such as in your example. If the column you specify isn't in the df, then pandas will create a new column with the name you specify. So after your dataframe is constructed, (from your question):

df = pd.DataFrame({'a':[np.nan, 2, 3], 'b':[4, 5, 6]}, index=[3, 5, 6])

You can just run:

df['s1'], df['s2'] = 5, 6

You could write a loop or comprehension to make it do this for all the elements in a list of tuples, or keys and values in a dictionary depending on how you have your real data stored.


Here's one way:

df.join(pd.DataFrame(s).T).fillna(method='ffill')

To break down what happens here...

pd.DataFrame(s).T creates a one-row DataFrame from s which looks like this:

   s1  s2
0   5   6

Next, join concatenates this new frame with df:

   a  b  s1  s2
0  1  3   5   6
1  2  4 NaN NaN

Lastly, the NaN values at index 1 are filled with the previous values in the column using fillna with the forward-fill (ffill) argument:

   a  b  s1  s2
0  1  3   5   6
1  2  4   5   6

To avoid using fillna, it's possible to use pd.concat to repeat the rows of the DataFrame constructed from s. In this case, the general solution is:

df.join(pd.concat([pd.DataFrame(s).T] * len(df), ignore_index=True))

Here's another solution to address the indexing challenge posed in the edited question:

df.join(pd.DataFrame(s.repeat(len(df)).values.reshape((len(df), -1), order='F'), 
        columns=s.index, 
        index=df.index))

s is transformed into a DataFrame by repeating the values and reshaping (specifying 'Fortran' order), and also passing in the appropriate column names and index. This new DataFrame is then joined to df.


If df is a pandas.DataFrame then df['new_col']= Series list_object of length len(df) will add the or Series list_object as a column named 'new_col'. df['new_col']= scalar (such as 5 or 6 in your case) also works and is equivalent to df['new_col']= [scalar]*len(df)

So a two-line code serves the purpose:

df = pd.DataFrame({'a':[1, 2], 'b':[3, 4]})
s = pd.Series({'s1':5, 's2':6})
for x in s.index:    
    df[x] = s[x]

Output: 
   a  b  s1  s2
0  1  3   5   6
1  2  4   5   6

Update
From v0.24.0 onwards, you can merge on DataFrame and Series as long as the Series is named.

df.merge(s.rename('new'), left_index=True, right_index=True)
# If series is already named,
# df.merge(s, left_index=True, right_index=True)

Nowadays, you can simply convert the Series to a DataFrame with to_frame(). So (if joining on index):

df.merge(s.to_frame(), left_index=True, right_index=True)

Examples related to python

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to pandas

xlrd.biffh.XLRDError: Excel xlsx file; not supported Pandas Merging 101 How to increase image size of pandas.DataFrame.plot in jupyter notebook? Trying to merge 2 dataframes but get ValueError Python Pandas User Warning: Sorting because non-concatenation axis is not aligned How to show all of columns name on pandas dataframe? Pandas/Python: Set value of one column based on value in another column Python Pandas - Find difference between two data frames Pandas get the most frequent values of a column Python convert object to float

Examples related to dataframe

Trying to merge 2 dataframes but get ValueError How to show all of columns name on pandas dataframe? Python Pandas - Find difference between two data frames Pandas get the most frequent values of a column Display all dataframe columns in a Jupyter Python Notebook How to convert column with string type to int form in pyspark data frame? Display/Print one column from a DataFrame of Series in Pandas Binning column with python pandas Selection with .loc in python Set value to an entire column of a pandas dataframe