How to create a 5-minute interval datetime index using incomplete dataset (Python)

Bhajmeister

First time posting on here, so hopefully it will be fruitful :)

Basically, I have a CSV file which contains timeseries data. I'm using pandas.read_csv to create a dataframe from the CSV. It is organised in an annoying way however, with the first column used for dates only, and the second column used for intervals of 5 mins for each day (e.g. 0, 5, 10, 15, up to 1435). To further complicate the problem, some days have incomplete data, i.e. rows missing. So for a particular date, the time intervals may go something like 5, 10, 60, 505, etc. This means that I can't simply create a datetime index from scratch with 5 minute intervals.

I therefore have to have some way of using the data in column B and combining it with the date in column A to give me the timeseries I want. I have to attach this resulting column to the same dataframe and then use it as the index to allow me to produce hourly means of my data.

After much blood, sweat and tears, I came up with this so far, which generates a new list of times. The first part is correct as far as I can tell.

df = pd.read_csv(myfile)

newtime = []

for r in df['Time']:
    if r // 60 < 10:
        if r % 60 < 10:
            r = "0" + str(r // 60) + ":0" + str(r % 60) + ":00"
        else:
            r = "0" + str(r // 60) + ":" + str(r % 60) + ":00"
    else:
        if r % 60 < 10:
            r = str(r // 60) + ":0" + str(r % 60) + ":00"
        else:
            r = str(r // 60) + ":" + str(r % 60) + ":00"

    newtime.append(r)

datetimes = []

for r in range(len(df['Date'])+1):
    v = str(df['Date'][r]) + newtime[r]
    datetimes.append(v)

print datetimes

However, when I try concatenating it with the dates (the last bit of code), I get a very random error (KeyError: 203591L). Weirdly, it works fine if I replace the v = with just print and remove the line with the append statement.

What I'm hoping is that someone can either help me develop the existing code into a solution (to the point where it's ready for df.resample('1H', how={columnX: np.mean}) ), or tell me how I can do the whole thing a different way.

Thanks very much in advance!

Seb

R. Max

Looks like you are looking for this:

In [17]: df
Out[17]: 
         Date  Time
0  2014-05-01    60
1  2014-05-02   505

In [18]: pd.to_datetime(df['Date']) + pd.to_timedelta(df['Time'], unit='m')
Out[18]: 
0   2014-05-01 01:00:00
1   2014-05-02 08:25:00
dtype: datetime64[ns]

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

Filter dataframe using the most recent datetime entry on a 5 minute interval

Create a dataframe with two columns (hour and minute) using a datetime index

How to add new 5-minute interval

How to get every 5 minute interval record?

Create 5-minute interval between two timestamp

How to create datetime index from string in python?

How to break down datetime object or interval object into minute by minute per row in R

How to round datetime previous 10 minute with python

Create list from incomplete dataset

Is it possible to inject a number into excel at every 1 minute interval using Python?

How to set hour range and minute interval using APScheduler

How to load different pages in interval every minute using javascript?

SSRS Create temporary datetime table updated with 15 minute interval - parameter not showing

Roof a DateTime to next value of a specific minute interval

how to roll pandas datetime 5 minute into the previous business day

how to extract missing datetime interval in python

How to generate a random datetime interval in python?

How do I label a Pandas datetime row (in a new column) for every 15 minute interval?

Create an array of times with 15 minute interval (Swift)

How to generate Fixed Minute based DateTime using Pandas

How to extract hour:minute from a datetime stamp in Python

How to create a composite datetime with fixed date + random time interval in Postgresql

How to round off to the closest 5 minute interval if it is 1 or 2 minutes ahead or behind?

How to round it down to the closest 5 minute interval if it is 1,2,3,4 minutes ahead?

How to create an animated GIF using FFMPEG with an interval?

Sending notifications with interval 1 minute using resque

How to set '5 day' (datetime interval) in jdbc for PostgreSQL?

How to partition datetime with 5 minutes interval in SQL Server?

Generating 15 minute time interval array in python