Explode multiple columns to rows in pyspark

Padfoot123

I have the below spark dataframe.

Name     age     subject        parts
xxxx     21      Maths,Physics  I
yyyy     22      English,French I,II

I am trying to explode the above dataframe in both subject and parts like below.

Expected output:

Name     age     subject        parts
xxxx     21      Maths          I
xxxx     21      Physics        I
yyyy     22      English        I
yyyy     22      English        II
yyyy     22      French         I
yyyy     22      French         II

I tried using array.zip for subject and parts and then tried to explode using the temp column, but I am getting null values in the place where there is only one part.

Is there a way to achieve this in Pyspark.

Padfoot123

I did this by passing columns as list to a for loop and exploded the dataframe for every element in list

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

Explode 2 columns into multiple columns in pyspark dataframe

PySpark - Explode columns into rows based on the type of the column

Spark explode multiple columns of row in multiple rows

Explode column values into multiple columns in pyspark

Pyspark explode multiple columns with sliding window

How to explode multiple columns of a dataframe in pyspark

Pyspark: explode json in column to multiple columns

PySpark explode list into multiple columns based on name

PySpark Explode JSON String into Multiple Columns

Dataframe explode list columns in multiple rows

Scala Spark Explode multiple columns pairs into rows

pyspark: Explode struct into columns

Pyspark: Filtering rows on multiple columns

PySpark: Create new rows (explode) based on a number in a column and multiple conditions

Explode multiple list columns pairs to more rows in Pandas

Explode multiple columns

Explode on multiple columns in Hive

Explode multiple columns in Pandas

Pyspark: explode columns to new dataframe

Pyspark: Split multiple array columns into rows

Pyspark exploding nested JSON into multiple columns and rows

pyspark aggregation across columns via explode on columns?

Efficiently transpose/explode spark dataframe columns into rows in a new table/dataframe format [pyspark]

pyspark : How to explode a column of string type into rows and columns of a spark data frame

How to filter multiple rows based on rows and columns condition in pyspark

Explode column of list to multiple rows

Python pandas: explode multiple rows

Explode Cell into Multiple Rows in Pandas

Explode multiple uneven rows in Pandas