Tutorial 2a: Managing data sets¶
(c) 2016 Justin Bois. This work is licensed under a Creative Commons Attribution License CC-BY 4.0. All code contained herein is licensed under an MIT license.
This tutorial was generated from an Jupyter notebook. You can download the notebook here.
import numpy as np
import pandas as pd
# Import pyplot for plotting
import matplotlib.pyplot as plt
# Some pretty Seaborn settings
import seaborn as sns
rc={'lines.linewidth': 2, 'axes.labelsize': 14, 'axes.titlesize': 14}
sns.set(rc=rc)
# Make Matplotlib plots appear inline
%matplotlib inline
In Tutorial 1, we used data on frog tongue adhesion to learn about loading and plotting data. That data set was relatively small and fairly easy to plot and investigate. That is not to say it was trivial!
In this tutorial, we will work with a more complicated data set, which can be downloaded here.
The data we are investigating come from David Prober's lab. A description of their work on the genetic regulation of sleep can be found on the research page of the lab website. There is a movie of the moving/sleeping larvae similar to the one used to produce the data set we are using in this tutorial. The work based on this data set was published in (Gandhi et al., Neuron, 85, 1193–1199, 2015).
In the experiment we are analyzing in this tutorial, we study the effect of a deletion in the gene coding for arylalkylamine N-acetyltransferase (aanat), which is a key enzyme in the rhythmic production of melatonin. Melatonin is a hormone responsible for regulation of circadian rhythms. It is often taken as a drug to treat sleep disorders. The goal of this study is to investigate the effects of aanat deletion on sleep pattern in 5+ day old zebrafish larvae.
In our data set, we have data on the movement of zebrafish larvae over time for wild type, heterozygous mutant, and homozygous mutant zebrafish.
To understand how the data were taken, here is the text of the Methods section of Prober, et al., J. Neurosci., 26, 13400-13410, 2006.
Larvae were raised on a 14/10 h light/dark (LD) cycle at 28.5°C. On the fourth day of development, single larva were placed in each of 80 wells of a 96-well plate (7701-1651; Whatman, Clifton, NJ), which allowed simultaneous tracking of each larva and prevented the larvae from interfering with the activity of each other. Locomotor activity was monitored for several days using an automated video-tracking system (Videotrack; ViewPoint Life Sciences, Montreal, Quebec, Canada) with a Dinion one-third inch Monochrome camera (model LTC0385; Bosch, Fairport, NY) fitted with a fixed-angle megapixel lens (M5018-MP; Computar) and infrared filter, and the movement of each larva was recorded using Videotrack quantization mode. The 96-well plate and camera were housed inside a custom-modified Zebrabox (ViewPoint Life Sciences) that was continuously illuminated with infrared lights and was illuminated with white lights from 9:00 A.M. to 11:00 P.M. The 96-well plate was housed in a chamber filled with circulating water to maintain a constant temperature of 28.5°C. The Videotrack threshold parameters for detection were matched to visual observation of the locomotion of single larva. The Videotrack quantization parameters were set as follows: detection threshold, 40; burst (threshold for very large movement), 25; freeze (threshold for no movement), 4; bin size, 60 s. The data were further analyzed using custom PERL software and Visual Basic Macros for Microsoft (Seattle, WA) Excel. Any 1 min bin with zero detectable movement was considered 1 min of rest because this duration of inactivity was correlated with an increased arousal threshold; a rest bout was defined as a continuous string of rest minutes. Sleep latency was defined as the length of time from lights out to the start of the first rest bout. An active minute was defined as a 1 min bin with any detectable activity. An active bout was considered any continuous stretch of 1 min bins with detectable movement.
The data set comes exactly (except for the comments) as I received it from Avni Gandhi, the first author of the paper. These are the raw data that come from the automated instrument/camera system (Videotrack) that measures the fish. In this tutorial, you will parse these raw data into tidy DataFrames and then save them as CSV files. (In Tutorial 2b, we will use these DataFrames to define and estimate some parameters.) This may seem like a mundane task, and maybe not very instructive, but this type of thing is encountered time and time again while doing real data analysis. Hopefully in this tutorial, you will learn the tricks of the trade on simple ways to parse complicated data files that are not already in tidy format. As a reminder, we are using "tidy" as a technical term.
Data file names¶
First, we need to unzip the data set using your favorite unzipper. You will have two data files. One is 130315_1A_genotypes.txt, which specifies the genotype of the larva in each well. The other is 130315_01_rawDATA.txt, which contains (you guessed it) the raw data from each well, as well as the time points for acquisition.
I pause for a moment to comment that these are the file names that I received from the Prober lab. Their method of naming their files is a good one, I think, and one that I use in my own research. By naming first with a six-digit year-month-day code, they will appear in chronological order on your computer, which makes it easier to reference against your lab notebook. Because these are long experiments, the time is not also included. In cases where I do several experiments in a day, I use a 10-digit code, where the last four digits are the 24-hour-style time that the experiment was done. The file name after the time-stamp code should contain a description of the experiment itself.
The genotype data¶
We'll first load the genotype file. Each column in this file contains a list of wells in the 96 well plate corresponding to each genotype. Looking at this with a text editor reveals that it is a tab delimited file. We can parse this using the delimiter keyword argument of pd.read_csv(). When specifying the delimiter, tabs are denoted as \t.
We also see that there are two header rows. The first is really redundant, so we can skip it. Nonetheless, we will read both in as headers using the header kwarg of pd.read_csv().
# Load in the genotype file, call it df_gt for genotype DataFrame
fname = 'data/130315_1A_genotypes.txt'
df_gt = pd.read_csv(fname, delimiter='\t', comment='#', header=[0, 1])
# Take a look at it
df_gt
We notice that we have two header rows for the columns. If we look at the column names, we see that they are a MultiIndex instance. This is part of Pandas's slick multi-indexing functionality, which is not something we need if we have tidy data. (However, messy data using multi-indexing may result in performance boosts for accessing data. This may become important for very large data sets. In general, tidy data are easier to conceptualize and syntactically much simpler to access.)
df_gt.columns
We do not need the multi-indexing, and level zero of indexing ('Genotype1', etc.) is not necessary. So, we just want level one index. This is easily obtained using the get_level_values() method of Pandas MultiIndex objects.
# Reset the columns to be the second level of indexing
df_gt.columns = df_gt.columns.get_level_values(1)
# Check out the new columns
df_gt.columns
The numbers at the end of these column headings say how many of each type were encountered. These are dispensable, so let's clean up the column names.
df_gt.columns = ['wt', 'het', 'mut']
Note that we could have done this in the first place instead of messing around with the MultiIndex, but I wanted you to be aware of multi-indexing in Pandas and to recognize it when you encounter it. Many data sets you will encounter feature it, and you will need to know how to tidy data sets in this format.
Tidying the genotype data¶
As they are, the data are not tidy. For tidy data, we would have two columns, 'fish', which give the well number for each embryo, and 'genotype', which with either 'wt' for wild type, 'het' for heterozygote, or 'mut' for mutant.
A useful tool for tidying data is the pd.melt() function. For this simple data set, it takes the column headings and makes them into a column (with repeated entries) and puts the data accordingly in the correct order. We just need to specify the name of the "variable" column, in this case the genotype, and the name of the "value" column, in this case the fish ID.
# Tidy the DataFrame
df_gt = pd.melt(df_gt, var_name='genotype', value_name='fish')
# Take a look
df_gt
The NaN entries were still preserved. We can drop them using the dropna() method of DataFrames.
# Drop all rows that have a NaN in them
df_gt = df_gt.dropna()
# Take a look
df_gt
Note, though, that the indices do have some skips. This happened when we dropped the NaN's. This is not really of concern, since we will not use them. If it annoys you, though, you can use the reset_index() method.
df_gt = df_gt.reset_index(drop=True)
We now have a tidy DataFrame. There is a problem, though. The fish numbers are now floats. This happened because NaN is a float, so the column got converted to floats. We can reset the data type of the column to ints.
df_gt.loc[:,'fish'] = df_gt.loc[:, 'fish'].astype(int)
Initial cleaning of the raw activity data¶
Let's now load the raw data. Again, but looking at the file in a text editor, we can see that it is tab delimited (and quite large) with two header rows. As a reminder, each entry in this data set is the number of seconds in a one-minute interval during with the larva was moving.
# Load raw data into a DataFrame
fname = 'data/130315_01_rawDATA.txt'
df = pd.read_csv(fname, delimiter='\t', comment='#', header=[0, 1])
# Take a look
df.head()
We see that we have over 5000 time points for each of 96 fish. The file also contains some redundant headers and two blank columns. Unfortunately, level 1 for the first two columns is appropriate for the column name and level 0 is appropriate for the remaining columns. We can deal with this by explicitly taking the first two columns at level 1 and the remaining at level 0.
# Make list of columns (use type conversion to allow list concatenation)
df.columns = list(df.columns.get_level_values(1)[:2]) \
+ list(df.columns.get_level_values(0)[2:])
df.head()
Now, we wish to drop any column that has the string 'Unnamed' in its name. We can use Pandas's convenient str methods. We can get a list of the columns we want to drop using the str.contains() method.
# Columns we want to drop
cols_to_drop = df.columns[df.columns.str.contains('Unnamed')]
Now, we can use the drop() method to drop them.
# Drop the columns (use axis=1 to drop columns, axis=0 to drop rows)
df = df.drop(cols_to_drop, axis=1)
df.head()
The start and end times are dispensable because the stop point of one entry is the start point of the next. We can therefore delete those columns.
df = df.drop(['start', 'end'], axis=1)
Finally, we note that the genotype of some of the fish is unknown. To check, we need to convert entries like 'FISH42' to just 42. This is easily done using Python's built-in string methods and type conversion.
int('FISH42'.lstrip('FISH'))
Now, we just loop through the FISH columns of the DataFrame and drop the ones that do not have genotypes.
# Find columns to drop
cols_to_drop = []
for col in df.columns:
if 'FISH' in col and int(col.lstrip('FISH')) not in df_gt['fish'].values:
cols_to_drop.append(col)
# Drop 'em!
df = df.drop(cols_to_drop, axis=1)
df.head()
We've now cleaned up the headers and gotten rid of the blank columns. Time to tidy!
Tidying time series data¶
In general, the best way to tidy time series data of multiple measurements is difficult to define. For now, let's consider a simplified version of our data set were we just have a set of time points for each fish with an activity measured at each time point. This is analogous to any generic time series for multiple measurements.
I can think of three options on how to tidy a generic set of time series data.
- Each row represents a single fish at a single time interval. For this style of tidying, the columns are then
['fish', 'time', 'activity']. - Each row represents a single time point, considering all fish. For this data set, the columns are then
['time', 'activity_1', 'activity_2', ...], where the integers correspond to IDs of the fish we're studying. - Each row represents a single fish. In this case, the columns are
['fish', t_0, t_1, t_2, ...], wheret_0,t_1, etc., are the time points. We could less ambiguously call these column headings'activity_t_0','activity_t_1', etc. We would also need to name our indices after the fish, with with a number or string.
One could make arguments for any of these. Option 1 is obviously tidy: one observation, one row. Option 2 could be considered tidy. The 96-well plate is observed at each time point. The data from each observation are the activities of every fish. Similarly, Option 3 could also be tidy. Each observation is the time course for an individual fish.
In our case, we also want need to include information about the genotype of each fish. If we want to add these data, the column headings are adjusted as follows.
['fish', 'genotype', 'time', 'activity'].['time', 'activity_%s_1', 'activity_%s_2', ...], where the strings represent the genotype.['fish', 'genotype', t_0, t_1, ...].
For option 2, instead of having string that we need to search in the column headings, we could instead use a multi-index for the activities for a given fish ID and genotype. If we did want to use the column headings with the strings describing genotype as shown above, we could use the str methods of DataFrames to select which columns to use.
We also want to incorporate convenient information about the time points, such as what day it is or if the light is on or not. Adding this information, the new column headings are
['fish', 'genotype', 'day', 'light', 'time', 'activity'].['day', 'light', 'time', 'activity_%s_1', 'activity_%s_2', ...], where the strings are the genotype.['fish', 'genotype', '%s_%d_%f', ...], where the string formats represent'light'or'dark', the day, and the time stamp.
Now it becomes clear that multi-indexing is useful for 2 and 3. The headings get to be unruly and difficult to search.
It is also clear that if we want to include any information about the individual subjects or about time points that option 1 is the only one that can be kept tidy (and index-able using only column slicing and Boolean indexing).
So, for time series data, if we do not need any other information about the subjects (besides their IDs) nor about the time points (besides their values), tidying options 1, 2, and 3 are all reasonable. I would argue that option 3 is more difficult to work with because of the necessary labeling of indices and the use of the values in the column heading (the times) for plotting and analysis. It is generally not a great idea to use floats as column headings of indices because of precision issues. So, in my opinion, options 1 and 2 are preferred.
However, if other information about the subjects (such as genotype in this case) are needed, multi-indexing is required for option 2. Option 1 always provides tidy data.
We will ultimately have a Zeitgeber time, as well as information giving the day and whether or not the light was on for a given time point. (These latter two are for convenience.) We also want to have the genotype of the fish being considered.
We will use option 1 in our analysis, but I will show how to tidy the data into the shapes of option 2 for demonstration purposes. I do not think option 3 is a very good idea.
Before we tidy the data, however, we need to make sure everything is ok with the clocks. This is more convenient to do now, prior to tidying, because we only have to operate once on a columns; tidying results in repeated entries in the columns.
Setting the Zeitgeber time¶
When inspecting the DataFrame a little more closely, we see that the CLOCK times are not sequential. This is most easily seen by plotting them.
# Plot clock and t_start values
plt.plot(df.CLOCK, '.')
plt.xlabel('index')
plt.ylabel('time (hours)')
It would be better to have continuous time. It is also convenient to work with so-called Zeitgeber time, where 0 corresponds to 9AM when the lights go on, and 14 hours corresponds to 11PM, when the lights go off. We would therefore like to have continuous Zeitgeber time, meaning we go from the start of the experiment continuously through, such that 9h, 33h, etc., all correspond to Zeitgeber time zero. As per the comments at the head of the data set, the df.CLOCK column gives the Zeitgeber time.
Before we do that, let's exploit the values in df.CLOCK to create a column containing Booleans as to whether the light is on. We have to be careful when adding the light column to the DataFrame. This is because it needs to have the indices as df. To do this, we make the light column into a Pandas Series, which is basically a column of a DataFrame. When we do this, we use the index keyword argument to set the indexing of the Series to match that of df. We can then add the Series as a column to the DataFrame.
df['light'] = pd.Series(df.CLOCK < 14.0, index=df.index)
Computing which day it is is a bit more tricky. Remember, these are "days" according to the zeitgeber time, meaning that they go from 9AM to 9AM PST. The experiment was started in the middle of day 0. We have to find when the light goes from off to on (that's the start of a day), and then populate every entry until the next time the light goes from off to on (the start of the next day) with the same index. We can find where we switch from dark to light by using the np.diff() function, which computes the difference between consecutive entries in an array. Wherever we have np.diff(df.light.astype(np.int)) equal to unity, we have switched from dark to light. (Note that we have converted df['light'] from Boolean to integer, because np.diff() treats differences of Booleans as Booleans, meaning that it returns True for transitions from day to night and from night to day.) We can then use the np.where() function to find where the switches took place. Finally, we can loop through these point and populate the day array.
# Find where the lights switch from off to on.
dark_to_light = np.where(np.diff(df['light'].astype(np.int)) == 1)[0]
# Initialize array with day numbers
day = np.zeros_like(df['light'], dtype=np.int)
# Loop through transitions to set days (+1 on indices b/c diff() is offset by 1)
for i in range(len(dark_to_light) - 1):
day[dark_to_light[i]+1:dark_to_light[i+1]+1] = i + 1
day[dark_to_light[-1]+1:] = len(dark_to_light)
# Insert into DataFrame
df['day'] = pd.Series(day, index=df.index)
We now need to compute the continuous Zeitgeber time. This is easily done using the day values we computed. We will call the column with the continuous Zeitgeber time zeit.
# Build ziet
zeit = 24.0 * df['day'] + df['CLOCK']
# Put in DataFrame
df['zeit'] = pd.Series(zeit, index=df.index)
df.head()
We now have all the information in our DataFrame that we need. We can proceed to tidy it.
Tidying the DataFrame¶
As I mentioned before, we will tidy the data using all three of the aforementioned options.
Tidying option 1¶
For this option, we again use the pd.melt() function. It is easiest to incorporate the genotype by making a multi-index for the columns and then using pd.melt(). We first get a list of genotypes for our respective fish from our tidy df_gt.
# Build list of genotypes
genotypes = []
# Check each column, put None for non-FISH column
for col in df.columns:
if 'FISH' in col:
fish_id = int(col.lstrip('FISH'))
genotypes.append(df_gt.genotype[df_gt.fish==fish_id].iloc[0])
else:
genotypes.append(None)
Now that we have the list of genotypes, we can make a multi-index with genotypes. It's as simple as reassigning the columns to be a list of lists.
df.columns = pd.MultiIndex.from_arrays((genotypes, df.columns),
names=['genotype', 'variable'])
Now let's look at our multi-indexed DataFrame.
df.head()
We can now use the pd.melt() function to make a tidy DataFrame. This DataFrame is a bit more complicated than the genotype DataFrame, so we will need some more kwargs for pd.melt(). First, value_vars are brought from column IDs into a data column. id_vars are retained. We wish to retain 'CLOCK', 'zeit', 'light', and 'day', so these are our id_vars. The value variables are all of the fish names and the genotypes.
# Value variables are the ones with FISH
col_1 = df.columns.get_level_values(1)
value_vars = list(df.columns[col_1.str.contains('FISH')])
# ID vars are the non-FISH entries
id_vars = list(df.columns[~col_1.str.contains('FISH')])
# Perform the melt
df1 = pd.melt(df, value_vars=value_vars, id_vars=id_vars, value_name='activity',
var_name=['genotype', 'fish'])
Let's take a look at our newly tidied DataFrame.
df1.head()
This is very nice, but we do have some annoying tuples as column headings, a remnant of our multi-index. We can easily change that.
# Rename any column that is a tuple
for col in df1.columns:
if type(col) is tuple:
df1.rename(columns={col: col[1]}, inplace=True)
Finally, for purely aesthetic purposes, we can put the columns in the order we like.
df1 = df1[['fish', 'genotype', 'day', 'light', 'CLOCK', 'zeit', 'activity']]
Now, let's take a look at our tidy DataFrame.
df1.head()
The 'FISH' string in the fish column is redundant, so we can delete it. One way to do it is to apply a function that converts FISH# to FISH. We can do this using the apply method of a DataFrame. First, we'll write the function.
def fish_to_number(fish_str):
return int(fish_str.lstrip('FISH'))
# Apply the function to each entry in the fish column
df1['fish'] = df1['fish'].apply(fish_to_number)
# Look at the DataFrame
df1.head()
Now we have a nice, tidy DataFrame ready for slicing with Boolean indexing.
Tidying the data with option 2¶
Though we will use option 1 as our tidy data, we will show how to tidy the data with option 2, where each row is a time point. Guess what? After the work we did to df in tidying with option 1, it is already in the format of option 2.
df2 = df.copy()
df2.head()
The only thing we should change is swapping out 'FISH' for 'activity_' to have more descriptive column names.
# Zero-level columns
cols = [col.replace('FISH', 'activity_') \
for col in df2.columns.get_level_values('variable')]
# Rename columns
df2.columns = pd.MultiIndex.from_arrays(
[df2.columns.get_level_values('genotype'), cols],
names=['genotype', 'variable'])
Let's take a look.
df2.head()
To access the data, it is useful to see how to slice with multi-indexing. One option is to pull out a "cross-section" of the data using the xs() method of DataFrames. The kwargs specify which level of the MultiIndex and along which axis (axis=1 for columns).
# Pull out wild type cross-section
df_wt = df2.xs('wt', level='genotype', axis=1)
# Look at it
df_wt.head()
We can also slice out time information using cross sections. The 'zeit', 'day', 'light', and 'CLOCK' columns are in level variable of the MultiIndex.
df2.xs('zeit', level='variable', axis=1).head()
If the index we are after is at level zero, we can use usual column slicing to accomplish the same thing as using df.xs(). (I usually use cross sections just to keep organized.)
df2['wt'].head()
It may be convenient to use this to slice out time data as well. We therefore want to swap the column index levels to do this.
df2.swaplevel('genotype', 'variable', axis=1)['zeit'].head()
We still have a column heading, but this is of little consequence. We know that the column heading for the genotype is np.nan, so we can extract a Series if we want to.
df2.xs('zeit', level='variable', axis=1)[np.nan].head()
Preliminary data exploration¶
While we will do more data exploration in Tutorial 2b, we will take a peek at the data here to help us get it packaged up for more convenient analysis later.
We'll start by plotting the activity of a single wild type fish. For this example, we will not directly use plt to make all of our plots. Instead, we will use plt to generate figure and axis objects. These allow for more direct control of plotting. Once we have the axis objects, generating plots is done with very similar syntax as in the previous tutorial.
# Get view into DataFrame with only fish 2
df_fish = df1[df1['fish']==2]
# Get figure and axis objects using plt.subplots utility function
fig, ax = plt.subplots()
# Plot activity versus time for fish 2. lw denotes the line width
ax.plot(df_fish.zeit, df_fish.activity, '-', lw=0.25)
# Set axis labels
ax.set_xlabel('time (hr)')
ax.set_ylabel('activity (sec / min)')
# Set axis limits
ax.set_xlim((df_fish.zeit.min(), df_fish.zeit.max()))
ax.set_ylim((0, ax.get_ylim()[1]))
# We can overlay day and night. We'll make night shaded.
# The where keyword is useful, and ~df_wt.light means "not light."
ax.fill_between(df_fish.zeit, 0, ax.get_ylim()[1], where=~df_fish.light,
color='gray', alpha=0.3)
print('Fish 2 is', df_fish.genotype.iloc[0])
Since we will want to make many plots like this, we can make a function to conveniently set the axis limits and put in the day/night shading. We will for plots of multiple traces.
def pretty_activity_plot(ax, selector, selection, col, df, xlabel='time (hr)',
ylabel='activity (sec / min)', lw=0.25,
color=None):
"""
Makes a pretty plot of sleep traces. Generates the plot on axes ax,
and then returns the updated ax.
"""
# Make sure selection input is iterable
if type(selection) in [str, int, float]:
selection = [selection]
# Plot time traces of column col for each fish
for sel in selection:
# Pull out record of interest
df_plot = df[df[selector]==sel]
# Generate plot
if color is None:
ax.plot(df_plot.zeit, df_plot[col], '-', lw=lw)
else:
ax.plot(df_plot.zeit, df_plot[col], '-', lw=lw, color=color)
# Label axes
ax.set_xlabel(xlabel)
ax.set_ylabel(ylabel)
# Set axis limits
ax.set_xlim((df_plot.zeit.min(), df_plot.zeit.max()))
ax.set_ylim((0.0, ax.get_ylim()[1]))
# Overlay night shading
ax.fill_between(df_plot.zeit, 0.0, ax.get_ylim()[1],
where=~df_plot.light, color='gray', alpha=0.3, zorder=0)
return ax
We see decidedly more activity during the daylight time. When looking at cumulative activity per minute, we see a lot of "noise" in the data. This is not really noise, as zebrafish embryos tend to move in bursts when they are awake (as you can see in the Prober lab website). The little fish may be moving and stopping, moving and stopping, etc., even though it is awake the whole time. We will discuss at length the effects this has on our decisions regarding how to treat the data. For now, we will following the example of the Prober, et al. paper. There, the authors summed activity over 10-minute intervals. We can do a similar summing to get activity over 10 minute intervals. We can use the the rolling() method of DataFrames to do this. The result is that entry i in a column of a DataFrame is given by the sum of elements i-9 through i.
We will want to do this for each fish. We could use Boolean indexing to slice out each fish record from the DataFrame and then compute the rolling sums. This is cumbersome, and fortunately we use the groupby() method of DataFrames to split the our DataFrame but fish ID. Once we've grouped the DataFrame, we make a rolling object and apply a sum. The result can be stored as a new column, windows in our DataFrame.
# Perform the rolling sum and return it as a new DataFrame
time_win = 10 # in units of no. of indices to sum over, equal to min here
# Make GroupBy object
df_gb = df1.groupby('fish')['activity']
# Compute rolling sum (use .values so we don't have to deal with indices)
df1['window'] = df_gb.rolling(window=time_win).sum().values
# Plot the rolling summed data
fig, ax = plt.subplots()
ax = pretty_activity_plot(ax, 'fish', 2, 'window', df1, lw=1,
ylabel='activity (sec/10 min)')
This is a bit smoother, as we have averaged out some of the on/off nature of the larval movement. However, the windows over which we summed overlap. (This is an example of data smoothing, which we'll discuss later in the course.) Prober, et al., report distinct ten minute intervals. We can easily access these by slicing. Because we have new time intervals, we should create a new DataFrame that instead has activity as the number of active seconds for every 10 minutes. Note that we define the time stamp for each ten-minute interval as the end time of the time minute interval.
# Time points/lightness
t = df1.zeit[df1.fish==df1.fish.unique()[0]]
light = df1.light[df1.fish==df1.fish.unique()[0]]
# Index for right edge of first averaging win. (ensure win. ends at lights out)
start_ind = time_win + np.where(np.diff(light.astype(int)))[0][0] % time_win
# The time points we want
times = np.sort(df1.zeit.unique())[start_ind::time_win]
# The indicies of the tidy DataFrame we want
inds = df1.zeit.isin(times)
# New DataFrame
new_cols = ['fish', 'genotype', 'day', 'light', 'CLOCK', 'zeit', 'window']
df1_10 = df1[new_cols][inds]
# Rename window column to activity
df1_10 = df1_10.rename(columns={'window': 'activity'})
We can now plot the results of our non-overlapping intervals.
# Plot the results for non-overlapping ten minute intervals
fig, ax = plt.subplots()
ax = pretty_activity_plot(ax, 'fish', 2, 'activity', df1_10, lw=1,
ylabel='activity (sec/10 min)')
Saving the data to CSV files¶
Now that we have parsed the data into a more readily usable format, we would like to save the tidy DataFrame as a CSV file so we can easily load it when we get back to work at a later time. Pandas's to_csv() method is a very convenient way to do this.
# Save our new DataFrame to CSV file, using kwarg index=False to suppress
# writing of (trivial) indices
df_gt.to_csv('130315_genotypes.csv', index=False)
df1.to_csv('130315_1_minute_intervals.csv', index=False)
df1_10.to_csv('130315_10_minute_intervals.csv', index=False)