Fetch multiple folders data into dataframe for given dates

497367209:
user1403789

I have storage container where data is in multiple folders with date appended at the end. (below)

    "dbfs:/mnt/input/raw/extract/pro_2023-01-01/parquet files here"
    "dbfs:/mnt/input/raw/extract/pro_2023-01-02/parquet files here"
    "dbfs:/mnt/input/raw/extract/pro_2023-01-03/parquet files here"
    "dbfs:/mnt/input/raw/extract/pro_2023-01-04/parquet files here"
    "dbfs:/mnt/input/raw/extract/pro_2023-01-05/parquet files here"

It works fine with good performance if I try to read the data from one folder. example:

df = spark.read.parquet("dbfs:/mnt/input/raw/extract/pro_2023-01-05/")

But I have situation when I need to load the data for multiple days into dataframe (mostly weekly basis). For that I pull all data and then use temp view to filter based on folderLoadDate which one of the column in parquet files. In that case, it works but takes forever to run while it scans all the folder and do transformations, I think. Example:

   df = spark.read.parquet("dbfs:/mnt/input/raw/extract/pro_2023-*/")

   df = there are few transformations and new columns added to df before I create temp view below

   df.createOrReplaceTempView ("Alldata")

then run spark sql

   %sql select * from Alldata where cast(FolderDate as date) BETWEEN '2023-01-01' AND '2023-01-07'

Is there a way I can pull the data for needed dates into df at very first step? something like

df = spark.read.parquet("dbfs:/mnt/input/raw/extract/BETWEEN(pro_2023-2023-01-01 and pro_2023-01-07")

Any help is appreciated..

notNull

Try by using regular expression for this case.

To read from 01-07 dates use 0[1-7] and spark will read 01,02,03,04,05,06,07 dates into dataframe.

Example:

spark.read.parquet("dbfs:/mnt/input/raw/extract/pro_2023-01-0[1-7]/")

UPDATE:

Define the empty dataframe and unionAll the data with the empty dataframe in every iteration.

#define empty dataframe with the schema of parquet files
schema = StructType([
    StructField("k", StringType(), True), StructField("v", IntegerType(), False)
])

df = spark.createDataFrame([], schema)

list_directories = ['2021-02-03']
for i in list_directories:
    df1 = spark.read.parquet('')
    df = df.unionAll(df1)

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

Filter data where date is within +/-30 days of multiple given dates

Filtering dataframe given a list of dates

How to update multiple columns of given dataframe with new corresponding scraped data?

creating a matrix of flags for given dates in a pandas dataframe

Given a pandas dataframe that contains multiple dates and multiple times per date how can I select the times for each date?

Grouping multiple dates from dataframe

List of dates between two given dates in pandas dataframe

Writing data to multiple folders in Python

Arrange data in Dataframe based on dates

Fetch data from another dataframe

Cumulative loan data with dates for a given firm in SAS

How to fetch data from api given in Newline

Fetch data after given createdAt date

fetch data multiple times in one fetch

How to get elements from a dataframe given dates in another dataframe

Append csv files in multiple folders into one dataframe

How to plot the Data For given DataFrame?

fetch data from multiple tabels

How to multiple fetch data expressJS

php fetch data multiple tables

Need a regexr to fetch the multiple data

useeffect function fetch multiple data

How to remove multiple folders with given pattern from repository index?

Filtering data for multiple years by date range given by months and days in pandas dataframe

Pandas - Split dataframe into multiple dataframes based on dates?

Fill in missing dates for a pandas dataframe with multiple series

Pandas fill in missing dates in DataFrame with multiple columns

Spliting DataFrame into Multiple Frames by Dates Python

Selecting multiple ranges of dates from dataframe

TOP Ranking

  1. 1

    Failed to listen on localhost:8000 (reason: Cannot assign requested address)

  2. 2

    Loopback Error: connect ECONNREFUSED 127.0.0.1:3306 (MAMP)

  3. 3

    How to import an asset in swift using Bundle.main.path() in a react-native native module

  4. 4

    pump.io port in URL

  5. 5

    Compiler error CS0246 (type or namespace not found) on using Ninject in ASP.NET vNext

  6. 6

    BigQuery - concatenate ignoring NULL

  7. 7

    ngClass error (Can't bind ngClass since it isn't a known property of div) in Angular 11.0.3

  8. 8

    ggplotly no applicable method for 'plotly_build' applied to an object of class "NULL" if statements

  9. 9

    Spring Boot JPA PostgreSQL Web App - Internal Authentication Error

  10. 10

    How to remove the extra space from right in a webview?

  11. 11

    java.lang.NullPointerException: Cannot read the array length because "<local3>" is null

  12. 12

    Jquery different data trapped from direct mousedown event and simulation via $(this).trigger('mousedown');

  13. 13

    flutter: dropdown item programmatically unselect problem

  14. 14

    How to use merge windows unallocated space into Ubuntu using GParted?

  15. 15

    Change dd-mm-yyyy date format of dataframe date column to yyyy-mm-dd

  16. 16

    Nuget add packages gives access denied errors

  17. 17

    Svchost high CPU from Microsoft.BingWeather app errors

  18. 18

    Can't pre-populate phone number and message body in SMS link on iPhones when SMS app is not running in the background

  19. 19

    12.04.3--- Dconf Editor won't show com>canonical>unity option

  20. 20

    Any way to remove trailing whitespace *FOR EDITED* lines in Eclipse [for Java]?

  21. 21

    maven-jaxb2-plugin cannot generate classes due to two declarations cause a collision in ObjectFactory class

HotTag

Archive