This tutorial was generated from an IPython notebook. You can download the notebook here.
In this tutorial, we will learn how to load data stored on disk into a Python data structure. We will use pandas to read in CSV (comma separated value) files.
The data set we will use comes from a fun paper about the adhesive properties of frog tongues. The reference is Kleinteich and Gorb, Tongue adhesion in the horned frog Ceratophrys sp., Sci. Rep., 4, 5225, 2014. You can download the paper here. You might also want to check out a New York Times feature on the paper here.
In this paper, the authors investigated various properties of the adhesive characteristics of the tongues of horned frogs when they strike prey. The authors had a striking pad connected to a cantilever to measure forces. They also used high speed cameras to capture the strike and record relevant data.
As I mentioned in the last tutorial, we need to import modules we need for data analysis. I generally like to import everything we'll need at the beginning. We will use __future__, numpy, and matplotlib as in the last tutorial, but also pandas.
from __future__ import division, absolute_import, \
print_function, unicode_literals
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
# Necessary to display plots in this IPython notebook
%matplotlib inline
The data from the paper are contained in the file frog_tongue_adhesion.csv, which you can download here. We can look at its contents.
# Use ! to invoke a shell command. Use head to look at top 20 lines of file.
!head -n 20 ../data/kleinteich_and_gorb/frog_tongue_adhesion.csv
The first lines all begin with # signs, signifying that they are comments and not data. They do give important information, though, such as the meaning of the ID data. The ID refers to which specific frog was tested.
Immediately after the comments, we have a row of comma-separated headers. This row sets the number of columns in this data set and labels the meaning of the columns. So, we see that the first column is the date of the experiment, the second column is the ID of the frog, the third is the trial number, and so on.
After this row, each row repesents a single experiment where the frog struck the target.
CSV files are generally a good way to store data. Commas make better delimiters than white space (such as tabs) because they have no portability issues. Delimiter collision is avoided by putting the data fields in double quotes when necessary. There are other good ways to store data, such as JSON, but we will almost exclusively use CSV files in this class.
We will use pd.read_csv to load the data set. The data are stored in a DataFrame, which is one of the data types that makes pandas so convenient for use in data analysis. DataFrames offer mixed data types, including incomplete columns, and convenient slicing, among many, many other convenient features. We will use the DataFrame to look at the data, at the same time demonstrating some of the power of DataFrames. They are like spreadsheets, only a lot better.
# Use pd.read_csv to read in the data and store in a DataFrame
# I am using my relative path of the data file; adjust as needed.
fname = '../data/kleinteich_and_gorb/frog_tongue_adhesion.csv'
df = pd.read_csv(fname, comment='#')
Notice that we used the kwarg comment to specify that lines that begin with # are comments and are to be ignored. If you check out the doc string for pd.read_csv, you will see there are lots of options for reading in the data.
Let's jump right in and look at the contents of the DataFrame.
# Look at the contents (first 10 indices)
df[:10]
We see that the column headings were automatically assigned. pandas also automatically defined the indices (names of the rows) as integers going up from zero. We could have defined the indices to be any of the columns of data.
To access a column of data, we use the following syntax.
# Slicing a column out of a DataFrame is achieved by using the column name
df['impact force (mN)']
The indexing of the rows is preserved, and we can see that we can easily extract all of the impact forces. Note, though, that pd.read_csv interpreted the data to be integer (dtype = int64), so we may want to convert these to floats.
# Use .astype method to convert it to a NumPy float64 data type.
df['impact force (mN)'] = df['impact force (mN)'].astype(np.float64)
# Check that it worked
print('dtype = ', df['impact force (mN)'].dtype)
Now let's say we only want the impact force of strikes above one Newton.
# We can use a Boolean in the indexing of the DataFrame
df_big_force = df[df['impact force (mN)'] > 1000.0]
# Let's look at it; there will be only a few high-force values
df_big_force
So now we only have the strikes of high force. Note, though, that the original indexing of rows was retained! In our new DataFrame with only the big force strikes, there is no index 4, for example.
# Executing the below will result in an exception
df_big_force['impact force (mN)'][4]
This might seem counterintuitive, but it is actually a good idea. Remember, indices do not have to be integers!
There is a way around this, though. We can use the iloc attribute of a DataFrame. This gives indexing with sequential integers.
# Using iloc enables indexing by the corresponding sequence of integers
df_big_force['impact force (mN)'].iloc[4]
One column represents many experiments of the same kind of measurement (e.g., impact force). One row represents a single experiment and many kinds of measurements. We have extracted a column out of the DataFrame, but how do we extract a row? This is also very easy with DataFrames using their ix method.
# Let's slice out experiment with index 42
exp_42 = df.ix[42]
# And now let's look at it
exp_42
Notice how clever the DataFrame is. We sliced a row out, and now the indices describing its elements are the column headings.
You may take issue with the rather lengthy syntax of access column names. I.e., if you were trying to access the ratio of the contact area with mucus to the contact area without mucus for trial number 3 on May 27, 2013, you would do the following.
# Set up criteria for our seach of the DataFrame
date = (df['date'] == '2013_05_27')
trial = (df['trial number'] == 3)
ID = (df['ID'] == 'III')
# When indexing DataFrames, use & for Boolean and (and | for or; ~ for not)
df['contact area with mucus / contact area without mucus'][date & trial & ID]
Yeesh. That syntax is clunky. But many would argue that this is prefered because there is no ambiguity about what you are asking for. However, you may want to use shorter names. Conveniently, when your column names do not have spaces, you can use attribute access. For example....
# Attribute access
df.date[:10]
Attribute access is really for convenience and is not the default, but if can make writing the code much less cumbersome. So, let's change the name of the colums as
\begin{align} \texttt{trial number} &\to \texttt{trial} \\ \texttt{contact area with mucus / contact area without mucus} &\to \texttt{ca_ratio}. \end{align}
DataFrames have a nice rename method to do this.
# Make a dictionary to rename columns
rename_dict = {'trial number' : 'trial',
'contact area with mucus / contact area without mucus' : 'ca_ratio'}
# Rename in-place; otherwise returns a new DataFrame
df.rename(columns=rename_dict, inplace=True)
# Try out our new column name
df.ca_ratio[42]
Notice that we have introduced a new Python data structure, the dictionary. A dictionary is a collection of objects, each one with a key to use for indexing. For example, in rename_dict, we could get what we wanted to rename 'trial number'.
# Indexing of dictionaries looks syntactially similar to cols in DataFrames
rename_dict['trial number']
We can go on and on about indexing, but we will stop here. For much more on all of the clever ways you can access data and subsets thereor in DataFrames, see the pandas docs on indexing.
"Exploratory data analysis" is the time during data analysis where you explore your data. You look at numbers, plot various quantities, and think about what analysis techniques you would like to use. pandas DataFrames help you do this.
As we go through the interactive analysis, we will learn about syntax for various matplotlib plot styles.
The first thing we'll do is look at strike forces. We'll rename the column for impact force for convenient attribute access because we'll access it many times.
df.rename(columns={'impact force (mN)': 'impf'}, inplace=True)
Now, let's start plotting the impact forces to see what we're dealing with. We'll start with the most naive plot.
# Just make a scatter plot of forces
plt.plot(df.impf, 'k.')
plt.margins(x=0.02, y=0.02)
plt.xlabel('order in DataFrame')
plt.ylabel('impact force (mN)')
The $x$-axis is pretty meaningless. So, instead let's plot a histogram of impact forces.
# Make a histogram plot; bins kwarg gives the number of bars
n, bin_edges, patches = plt.hist(df.impf, bins=20, normed=False)
plt.grid(True, axis='y')
plt.xlabel('impact force (mN)')
plt.ylabel('number of observations')
This is a better way to look at the impact force measurements. We see that there are a few high-force impacts, but that most of the impacts are about 500 mN or so.
This is still only part of the story. We would like to know how the impacts vary from frog to frog. First, let's see how many trials we have for each frog.
# This is a fancy way to do string formatting; unnecessary, but I thought
# I'd show how to do it.
print("""
Frog ID Number of samples
======= =================
I {0:d}
II {1:d}
III {2:d}
IV {3:d}
""".format(df.ID[df.ID=='I'].count(), df.ID[df.ID=='II'].count(),
df.ID[df.ID=='III'].count(), df.ID[df.ID=='IV'].count()))
So, we only have 20 samples for each frog. That's a bit few to construct a meaningful histogram for each frog. So, maybe we can make a bar graph showing the mean and standard deviation of each sample.
# Compute the mean impacts
mean_impacts = df.groupby('ID').impf.mean()
# We will use standard deviation as error bar
# Default DataFrame std is normalized by N-1; more on that in lecture
std_impacts = df.groupby('ID').impf.std()
# Bar locations
x = np.arange(4)
# Bar widths
bar_width = 0.5
# How to label each bar
bar_labels = df.ID.unique()
# We use matplotlib's bar function to make plot
plt.bar(x, mean_impacts, yerr=std_impacts, width=bar_width, align='center',
color='lightgray', error_kw={'ecolor' : 'black'})
plt.grid(True, axis='y')
plt.xticks(x, bar_labels)
plt.ylabel('impact force (mN)')
I had you do the plot that way because I want you to get comfortable with matplotlib, which is really diverse in its uses. However, you can make essentially the same figure in one or two lines using some of the awesomeness of DataFrames.
# Compute standard deviations grouped by ID
yerr = df.groupby('ID').impf.std()
# Plot the bar graphs
df.groupby('ID').impf.mean().plot(kind='bar', yerr=yerr, color='lightgray')
plt.ylabel('impact force (mN)')
We only have 20 measurements for each frog. Wouldn't it be better just to plot all 20 instead of trying to distill it down to a mean and a standard deviation in the bar graph? After all, the impact force measurements might not be Gaussian distributed; they may be bimodal or something else. So, we would like to generate a column scatter plot. We will plot each data point for each frog in a column, and "jitter" its position along the $x$-axis. We make the points somewhat transparent to allow visualization of overlap.
# Column positions
x_pos = np.arange(4)
# Column labels
x_labels = ['I', 'II', 'III', 'IV']
# Jitter by adding normally distributed random number and plot with
# alpha < 1, which gives transparency
for i in range(len(x_pos)):
y = df[df.ID==x_labels[i]].impf
x = np.random.normal(x_pos[i], 0.04, size=len(y))
plt.plot(x, y, 'ko', alpha=0.3)
plt.xticks(x_pos, x_labels)
plt.ylabel('impact force (mN)')
Very nice! Now we can see that frog I, an adult, strikes with a wide range of impact forces, and can strike really hard. Frog II, also an adult, tends to strike at around 500 mN, but occassionally will strike harder. Juvenile frog III is a pretty consistent striker, while frog IV can have some pretty weak strikes.
The column scatter plot is not difficult to look at. The informational content of the data does not need to be distilled into a bar with a standard deviation.
Plots like these in which the dots to not overlap are called beeswarm plots. For large numbers of data points, these are not favored over the column scatter plot, but for small numbers of data points, beeswarm plots tend to be easier to read, as each data point can be clearly seen. Unfortunately, matplotlib does not natively have a clean way of doing this. It is definitely possible, but is cumbersome.
Fortunately, there is a package to do it, pybeeswarm, written by Melissa Gyrmek. It is not available through Canopy, so we need to install it by hand, which we will occassionally need to do. To install Python packages, pip is the most commonly used tool. To do it, we simply invoke the shell from IPython and ask pip to install pybeeswarm.
# This should work with Mac OS X and Linux. For Windows, you'll have to use
# PowerShell or cygwin.
!pip install pybeeswarm
Now that we have pybeeswarm, we can put it to use!
# Import it first (will do this at the beginning of future tutorials)
import beeswarm as bs
# Separate impact forces by frog
list_of_impfs = [df.impf[df.ID=='I'], df.impf[df.ID=='II'],
df.impf[df.ID=='III'], df.impf[df.ID=='IV']]
# Generate a beeswarm plot
bs_plot, ax = bs.beeswarm(list_of_impfs, labels=['I', 'II', 'III', 'IV'])
plt.ylabel('impact force (mN)')
We can get even more information. We might be interested to see if the variability in impact force is day-to-day or time-independent. So, we would like to make the beeswarm plot with different colors on different days. This requires a bit more code, but it is not too bad. (As is the case in many of the examples in the tutorials, there is likely a more concise way to do this.)
# We need to import the colormap module from matplotlib
from matplotlib import cm
# Get list of dates; the unique method from a DataFrame works well
dates = df.date.unique()
# Number of unique dates
n_dates = len(dates)
# Assign colors to date names, calling colormap with value between 0 and 1
# We will use the qualitative colormap "Accent."
colors = []
for i in range(n_dates):
colors.append(cm.Set1(float(i) / float(n_dates)))
# Make a dictionary of dates and colors
color_dict = dict(zip(dates, colors))
# Sort the DataFrame by ID to make sure color labels are in correct order
# in the beeswarm plot
df.sort('ID', inplace=True)
# Make a list of colors
colors = []
for i in range(len(df)):
colors.append(color_dict[df.date.iloc[i]])
# Make beeswarm with date coloring
bs_plot, ax = bs.beeswarm(list_of_impfs, labels=['I', 'II', 'III', 'IV'],
col=colors)
plt.ylabel('impact force (mN)')
We do see some clustering of colors, so there may be some correlation between the day of the measurement and the impact force of the frog's tongue. I am careful to say may be some correlation, since we have not carefully quantified it.
Now let's look at correlations. We may be curious: does a hard strike result in better adhesion? We can plot the adhesive force versus the impact force.
# Rename the column for convenience in future use
df.rename(columns={'adhesive force (mN)' : 'adhf'}, inplace=True)
# Plot adhesive force vs. impact force
plt.plot(df.impf, df.adhf, 'k.', alpha=0.5)
plt.xlabel('impact force (mN)')
plt.ylabel('adhesive force (mN)')
Later in the course, we will learn how to do regressions to test the relationship between two variables.
For the time remaining in this tutorial, explore the frog tongue adhesion data set, generating plots of various quantities.
In this tutorial, we have learned how to load data from CSV files into pandas DataFrames. DataFrames are useful objects for looking at data from many angles. Together with matplotlib, you can start to look at qualitative features of your data and think about how you will go about doing your detailed analysis.