在这种情况下，如何通过匹配公共值来生成新的数据帧？

首先DataFrame.merge针对两列的常见行，对列重命名，并通过Series.value_counts和添加计数Series.map：

df = (data_2019.merge(data_2020, on=['state','industry'])
               .rename(columns={'industry':'common_industry'}))
df['common_industry_count'] = df['state'].map(df['state'].value_counts())
df = df[['state','common_industry','common_industry_count']]
print (df)
       state common_industry  common_industry_count
0      Texas     Agriculture                      1
1     Nevada        Medicine                      2
2  Louisiana     Manufacture                      1
3     Nevada     Manufacture                      2

然后通过concat删除所有重复项Series.drop_duplicates并通过删除一列DataFrame获得所有状态Series.to_frame：

both = pd.concat([data_2019['state'], data_2020['state']]).drop_duplicates().to_frame()
print (both)
          state
0          Ohio
1         Texas
2  Pennsylvania
3        Nevada
4      New York
7      Virginia
8     Louisiana
9       Florida
0        Kansas
2    California
3         Idaho

最后与leftjoin合并，并用替换缺少的值Series.fillna：

df = both.merge(df, how='left')
df['common_industry_count'] = df['common_industry_count'].fillna(0).astype(int)
df['common_industry'] = df['common_industry'].fillna('')
print (df)
           state common_industry  common_industry_count
0           Ohio                                      0
1          Texas     Agriculture                      1
2   Pennsylvania                                      0
3         Nevada        Medicine                      2
4         Nevada     Manufacture                      2
5       New York                                      0
6       Virginia                                      0
7      Louisiana     Manufacture                      1
8        Florida                                      0
9         Kansas                                      0
10    California                                      0
11         Idaho                                      0