Concatenate two columns of spark dataframe with null values

Padfoot13288

I have two columns in my spark dataframe

First_name  Last_name
Shiva       Kumar
Karthik     kumar
Shiva       Null
Null        Shiva

My requirement is to add a new column to dataframe by concatenating the above 2 columns with a comma and handle null values too.

I have tried using concat and coalesce but I can't get the output with comma delimiter only when both columns are available

Expected output

Full_name
Shiva,kumar
Karthik,kumar
Shiva
Shiva
RudyVerboven

concat_ws concats and handles null values for you.

df.withColumn('Full_Name', F.concat_ws(',', F.col('First_name'), F.col('Last_name'))

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

How to concatenate two columns of spark dataframe with null values but get one value

How to concatenate null columns in spark dataframe in java?

subtract two columns with null in spark dataframe

Spark dataframe not adding columns with null values

Concatenate multiple columns of dataframe with a seperating character for Non-null values

How to concatenate two columns that might contain NULL values in a select statement?

concatenate two columns in pandas dataframe

Concatenate multiple columns, with null values

concatenate two columns values pandas

Concatenate columns to list of columns in Apache Spark DataFrame

Spark / Scala - Compare Two Columns In a Dataframe when one is NULL

Extract values from column in spark dataframe and to two new columns

Concatenate distinct values from several columns to one column in Java spark dataframe

`AttributeError`when trying to concatenate two columns in a dataframe

How do you concatenate multiple columns in a DataFrame into a another column when some values are null?

Concatenate all values between two columns

How to concatenate spark dataframe columns using Spark sql in databricks

Replace null values in Spark DataFrame

Concat two columns values of dataframe

Map values on DataFrame by two columns

concatenate values in dataframe if a column has specific values and None or Null values

Jolt concatenate only if my two columns are not Null, default value if it is NULL

how can concatenate two string columns in one column in spark python

merge rows in a dataframe by id trying to avoid null values in columns (Spark scala)

How to find the list of columns which are all having null or NA values in spark scala Dataframe?

Spark DataFrame Get Null Count For All Columns

Spark DataFrame Aggregation based on two or more Columns

How to check for intersection of two DataFrame columns in Spark

Sort Spark Dataframe with two columns in different order