Pandas: How to merge two data frames and fill NaN values using values from the second data frame

Cave :

I have a pandas dataframe (df1) that looks like this:

No     car          pl.       Value      Expected      
1      Toyota       HK        0.1        0.12      
1      Toyota       NY        0.2        NaN     
2      Saab         LOS       0.3        NaN      
2      Saab         UK        0.4        0.6       
2      Saab         HK        0.5        0.51     
3      Audi         NYU       0.6        NaN      
3      Audi         LOS       0.7        NaN      
4      VW           UK        0.8        NaN   
5      Audi         HK        0.9        NaN    

And I have another dataframe (df2) that looks like this:

No        pl.             Expected              
2         LOS              0.35      
3         NYU              0.62   
3         LOS              0.76    
5         HK               0.91     

I would like my final dataframe to look like this:

No     car          pl.       Value      Expected      
1      Toyota       HK        0.1        0.12      
1      Toyota       NY        0.2        NaN     
2      Saab         LOS       0.3        0.35      
2      Saab         UK        0.4        0.6       
2      Saab         HK        0.5        0.51     
3      Audi         NYU       0.6        0.62      
3      Audi         LOS       0.7        0.76      
4      VW           UK        0.8        NaN   
5      Audi         HK        0.9        0.91    

I tried this:

df = df1.fillna(df1.merge(df2, on=['No','pl.']))

But df1 remains unchanged in the output

The questions that I have seen here have been of dataframes with the same shape. Is there a way to do this when the shapes are different?

Thanks in advance!

jezrael :

Use left join with suffixes parameter and then replace missing values by Series.fillna with DataFrame.pop for use and drop column Expected_:

df = df1.merge(df2, on=['No','pl.'], how='left', suffixes=('_',''))
df['Expected'] = df.pop('Expected_').fillna(df['Expected'])
print (df)
   No     car  pl.  Value  Expected
0   1  Toyota   HK    0.1      0.12
1   1  Toyota   NY    0.2       NaN
2   2    Saab  LOS    0.3      0.35
3   2    Saab   UK    0.4      0.60
4   2    Saab   HK    0.5      0.51
5   3    Audi  NYU    0.6      0.62
6   3    Audi  LOS    0.7      0.76
7   4      VW   UK    0.8       NaN
8   5    Audi   HK    0.9      0.91

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

How to fill NaN values according to the data type in pandas data frame?

How to forward fill NaN values in pandas data frame?

How can I fill data frames with NAN with same values of previous data frames from the same list

How to merge two data frames with missing values?

Nan values when I merge these data frames

How to merge two pandas data.frame into one, by time index, keeping all values from both of them

Merge two data frames based on common column values in Pandas

Python Pandas merge two data frames based on multiple values field

How to compare columns from two different Data Frames and keep the values from the first Data Frame?

How to merge two data frames based on one column in one data frame and two column in second dataframe

Merge two data frames on multiple values

Concat or Merge Two Data Frames with same values

How to merge two data frames based on similar values in R

Python: How to merge two data frames where the values are not unique

how to get the corresponding values in two data frames and merge them?

Merge on pandas data frames with multiple values

Using match (not merge) to fill column values from another bigger data frame

How to fill NaN with a value in same row of the data frame using Pandas

Pandas: Merge two data frames and keep non-intersecting data from a single data frame

Compare values from two pandas data frames, order-independent

How do I merge two data frames if a column and it's values exist in both data frames?

How to compare two different Data frames on simile column values and put values to other data frame

Pandas data frame fill null values with index

Replacing nan values in a Pandas data frame with lists

How to replace a range of values with NaN in Pandas data-frame?

How to merge to two pandas data frames?

How to merge two data frames in pandas?

Fill NaN in second level of multi indexed pandas data frame

How to use the index values returned by nlargest() in Pandas in a second data frame?