How to convert from Pandas' DatetimeIndex to DataFrame in PySpark?

user1761806

I have the following code:

# Get the min and max dates
minDate, maxDate = df2.select(f.min("MonthlyTransactionDate"), f.max("MonthlyTransactionDate")).first()
d = pd.date_range(start=minDate, end=maxDate, freq='MS')    

tmp = pd.Series(d)
df3 = spark.createDataFrame(tmp)

I have checked tmp and a I have a pandas dataframe of a list of dates. I then check df3 but it looks like lit's just an empty list:

++ 
|| 
++ 
|| 
|| 
|| 
|| 
|| 
|| 
|| 
||

What's happening?

neeraj bhadani

In your case d is DatetimeIndex. What you can do is create pandas DataFrame from DatetimeIndex and then convert Pandas DF to spark DF. PFB Sample code.

1. Create DatetimeIndex

import pandas as pd
d = pd.date_range('2018-12-01', '2019-01-02', freq='MS')

2. Create Pandas DF.

p_df = pd.DataFrame(d)

3. Create Spark DataFrame.

spark.createDataFrame(p_df).show()

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at2020-12-3

Comments

0 comments

How to convert a dictionary to dataframe in PySpark?

How do I properly set the Datetimeindex for a Pandas datetime object in a dataframe?

How do i slice pandas Series with DatetimeIndex and put it in a DataFrame by rows?

Pandas: How to extract Datetime ranges from DatetimeIndex

How to convert from Pandas' DatetimeIndex to DataFrame in PySpark?

How to convert from Pandas' DatetimeIndex to DataFrame in PySpark?

1. Create DatetimeIndex

2. Create Pandas DF.

3. Create Spark DataFrame.

Loopback Error: connect ECONNREFUSED 127.0.0.1:3306 (MAMP)

Can't pre-populate phone number and message body in SMS link on iPhones when SMS app is not running in the background

pump.io port in URL

How to import an asset in swift using Bundle.main.path() in a react-native native module

Failed to listen on localhost:8000 (reason: Cannot assign requested address)

Spring Boot JPA PostgreSQL Web App - Internal Authentication Error

Emulator wrong screen resolution in Android Studio 1.3

3D Touch Peek Swipe Like Mail

Double spacing in rmarkdown pdf

Svchost high CPU from Microsoft.BingWeather app errors

How to how increase/decrease compared to adjacent cell

Using Response.Redirect with Friendly URLS in ASP.NET

java.lang.NullPointerException: Cannot read the array length because "<local3>" is null

BigQuery - concatenate ignoring NULL

How to fix "pickle_module.load(f, **pickle_load_args) _pickle.UnpicklingError: invalid load key, '<'" using YOLOv3?

ngClass error (Can't bind ngClass since it isn't a known property of div) in Angular 11.0.3

Can a 32-bit antivirus program protect you from 64-bit threats

Make a B+ Tree concurrent thread safe

Bootstrap 5 Static Modal Still Closes when I Click Outside

Vector input in shiny R and then use it

Assembly definition can't resolve namespaces from external packages