Setting up computing resources¶
In this lesson, you will set up a GitHub account, learn about Google Colab, and set up a Python computing environment for scientific computing.
It is advantageous to learn how to set up a Python distribution and manage packages on your own machine, as each person can have different needs. That said, we will make some use of Google Colab in this class, mainly because it is easier for remote collaborators to work together using Colab. It is also FREE! (Thanks, Google!) We will first introduce Google Colab, and then proceed with instructions for installation on your own machine using Anaconda.
Getting a GitHub account¶
We will make extensive use of Git during the course. We will use GitHub to host the repositories. You need to set up a GitHub account. Go to http://github.com/ to get an account. You should register with your academic email address so you get free private repositories as academics. You should also think carefully about picking your user name. There is a good chance other people in your professional life will see this.
Once you have a GitHub account, send an email to bois at caltech dot edu with your account ID to get access to the BE/Bi 103 Group on GitHub. Within this group, you will form a team. Your team consists of your partners for homework submission.
Setting up Google Colab¶
In order to use Google Colab, you must have a Google account. Caltech students and employees have an account through Caltech’s G Suite. Many of you may have a personal Google account, usually set up for things like GMail, YouTube, etc. For your work in this class, use your Caltech account. This will facilitate collaboration with your teammates in the course, as well as with course staff.
Many of you probably use your personal Google account on your machine, so it can get annoying to log in and out of it. A trick that I find useful is to use one browser, e.g., Safari or Microsoft Edge, for your personal use, web browsing, etc., and a different browser for your scientific work, including the work in this class. Google Colab are most tested for Chrome, Firefox, and Safari (in fact JupyterLab, which you will use on your own machine, only supports these three browsers).
Once you have either logged out of all of your personal accounts or have a different browser open, you can launch a Colab notebook by simply navigating to https://colab.research.google.com/. Alternatively, you can click the “Launch in Colab” badge at the top right of this page, and you will launch this notebook in Colab. That badge will appear in the top right of all pages in the course content generated from notebooks.
Watchouts when using Colab¶
If you do run a notebook in Colab, you are doing your computing on one of Google’s computers via a virtual machine. You get two CPU cores and 12 GB of RAM. You can also get GPUs and TPUs (Google’s tensor processing units), but we will not use those in this course. The computing resources should be enough for all of our calculations this term (though you will need more computing power in the sequel of this course). However, there are some limitations you should be aware of.
If your notebook is idle for too long, you will get disconnected from your notebook. “Idle” means that cells are not being edited or executed. The idle timeout varies depending on the load on Google’s computers; I find that I almost always get disconnected if idle for an hour.
Your virtual machine will disconnect if it is being used for too long. It typically will only available for 12 hours before disconnecting, though times can vary, again based on load.
These limitations are in place so that Google can offer Colab for free. If you want more cores, longer timeouts, etc., you might want to check out Colab Pro. However, the free tier should work well for you in the course. You of course can always run on your own machine, and in fact are encouraged to do so except where collaboration is necessary.
There are additional software-specific watchouts when using Colab.
Later in the course, we will use HoloViews for high-level plotting. Colab will not render HoloViews plots unless
hv.extension('bokeh')is called in each cell that has a HoloViews plot.Colab does not allow for full functionality Bokeh apps and some Panel functionality that we will use later in the course when we do dashboarding.
Colab instances have specific software installed, so you will need to install anything else you need in your notebook. This is not a major burden, and is discussed in the next section.
I recommend reading the Colab FAQs for more information about Colab.
Software in Colab¶
When you launch a Google Colab notebook, much of the software we will use in class is already installed. It is not always the latest version of the software, however. In fact, as of early September 2020, Colab is running Python 3.6, whereas you will run Python 3.8 on your machine through your Anaconda installation. Nonetheless, most (but not all) of the analyses we do for this class will work just fine in Colab. We will make every effort to let you know when Colab will not be able to handle activities in class, the most important example being some dashboarding applications.
Because the notebooks in Colab have software preinstalled, and no more, you will often need to install software before you can run the rest of the code in a notebook. To enable this, when necessary, in the first code cell of each notebook in this class, we will have the following code (or a variant thereof depending on what is needed or if the default installations of Colab change). Running this code will not affect running your notebook on your local machine; the same notebook will work on your local machine or on Colab.
[1]:
# Colab setup ------------------
import os, sys, subprocess
if "google.colab" in sys.modules:
cmd = "pip install --upgrade iqplot colorcet datashader bebi103 watermark"
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = process.communicate()
data_path = "https://s3.amazonaws.com/bebi103.caltech.edu/data/"
else:
data_path = "../data/"
# ------------------------------
In addition to installing the necessary software on a Colab instance, this also sets the relative path to data sets we will use in the course. When running in Colab, the data set is fetched from cloud storage on AWS. When running on your local machine, the path to the data is one directory up from where you are working. We will discuss relative paths in a future lesson.
Collaborating with Colab¶
If you want to collaborate with another student or with the course staff on a notebook, you can click “Share” on the top right corner of the Colab window and choose with whom and how (the defaults are fine) you want to share.
When we talk about Git in a future lesson, we will discuss Colab’s GitHub support, which will be necessary for version control and submitting and sharing your homework.
Installation on your own machine¶
We now proceed to discuss installation of the necessary software on your own machine. Before we get into that, there are some preliminaries to configure your computer.
macOS users: Install XCode¶
If you are using macOS, you should install XCode, if you haven’t already. It’s a large piece of software, taking up about 5GB on your hard drive, so make sure you have enough space. You can install it through the App Store.
After installing it, you need to open the program. Be sure to do that, for example by clicking on the XCode icon in your Applications folder. Upon opening XCode, it may perform more installations. After these are completed, you can close XCode.
Windows users: Install Git and Chrome or Firefox¶
We will be using JupyterLab in this course. It is browser-based, and Chrome, Firefox, and Safari are supported. Microsoft Edge is not. Therefore, if you are a Windows user, you need to be sure you have either Chrome of Firefox installed.
Git is installed on Macs with XCode. For Windows users, you need to install Git. You can do this by following the instructions here.
Uninstalling Anaconda¶
Unless you have experience with Anaconda and know how to set up environments, if you have previously installed Anaconda with a version of Python other than 3.8, you need to uninstall it, removing it completely from your computer. You can find instructions on how to do that from the official uninstallation documentation.
Downloading and installing Anaconda¶
That’s it! After you do that, you will have a functioning Python distribution.
Launching JupyterLab and a terminal¶
After installing the Anaconda distribution, you should be able to launch the Anaconda Navigator. If you are using macOS, this is available in your Applications menu. If you are using Windows, you can do this from the Start menu. Launch Anaconda Navigator.
We will be using JupyterLab throughout the course (more on that in Lesson 0b). You should see an option to launch JupyterLab. When you do that, a new browser window or tab will open with JupyterLab running. Within the JupyterLab window, you will have the option to launch a notebook, a console, a terminal, or a text editor. We will use all of these during the course. For the updating and installation of necessary packages, click on Terminal to launch a
terminal. You will get a terminal window (probably black) with a bash prompt. We refer to this text interface in the terminal as the command line.
The conda package manager¶
conda is a package manager for keeping all of your packages up-to-date. It has plenty of functionality beyond our basic usage in class, which you can learn more about by reading the docs. We will primarily be using conda to install and update packages.
conda works from the command line. Now that you know how to get a command line prompt, you can start using conda. The first thing we’ll do is update conda itself. Enter the following on the command line
conda update conda
You will be prompted to continue this operation, so press y to continue. Next, we’ll update the packages that came with the Anaconda distribution. To do this, enter the following on the command line:
conda update --all
If anything is out of date, you will be prompted to perform the updates, and press y to continue. (If everything is up to date, you will just see a list of all the installed packages.) They may even be some downgrades. This happens when there are package conflicts where one package requires an earlier version of another. conda is very smart and figures all of this out for you, so you can almost always say “yes” (or “y”) to conda when it prompts you.
Installations¶
There are several additional installations you need to do. We will first install some plotting packages we need.
conda install holoviews param panel colorcet hvplot datashader
These packages are part of the HoloViz suite of packages. This also includes GeoViews, which is an excellent plotting package for geographical data. Because we will not be using geographical data sets in class and because as of early September 2020 GeoViews has conflicts with other packages, we are not installing it. If your research involves geographical data, you may with to install it as well and accept the package downgrades that conda
enforces. The easiest way to do an installation of the HoloViz packages including GeoViews (and accepting downgrades) is to not do the above installation, but instead do conda install -c pyviz holoviz. If you are not going to be using GeoViews, do not do this, but rather execute the suggested installation above.
Next, we need to install a few packages we will use indirectly during the course that are handy to have.
conda install nodejs netcdf4 black
We will also install watermark, which enables us to conveniently display version numbers of the software we are using. For this installation, we will use pip. There are a few other packages from pip we will need, so we can go ahead and install those now.
pip install watermark blackcellmagic multiprocess jupytext cmdstanpy arviz iqplot bebi103
Finally, we need to configure JupyterLab to work with the plotting packages we will use.
jupyter labextension install --no-build @pyviz/jupyterlab_pyviz
You may also wish to install a spell-checker (this one isn’t necessary).
jupyter labextension install --no-build @ijmbarr/jupyterlab_spellchecker
After installing all of these extensions, you can rebuild JupyterLab.
jupyter lab build
You should close your JupyterLab session and terminate Anaconda Navigator after you have completed the build. Relaunch Anaconda Navigator and launch a fresh JupyterLab instance. As before, after JupyterLab launches, launch a new terminal window so that you can proceed with setting up Git.
Checking your distribution¶
We’ll now run a quick test to make sure things are working properly. We will make a quick plot that requires some of the scientific libraries we will use.
Use the JupyterLab launcher (you can get a new launcher by clicking on the + icon on the left pane of your JupyterLab window) to launch a notebook. In the first cell (the box next to the [ ]: prompt), paste the code below. To run the code, press Shift+Enter while the cursor is active inside the cell. You should see a plot that looks like the one below. If you do, you have a functioning Python environment for scientific computing!
You can also test this in Colab (and it should work with no problems).
[2]:
import numpy as np
import bokeh.plotting
import bokeh.io
bokeh.io.output_notebook()
# Generate plotting values
t = np.linspace(0, 2*np.pi, 200)
x = 16 * np.sin(t)**3
y = 13 * np.cos(t) - 5 * np.cos(2*t) - 2 * np.cos(3*t) - np.cos(4*t)
p = bokeh.plotting.figure(height=250, width=275)
p.line(x, y, color='red', line_width=3)
text = bokeh.models.Label(x=0, y=0, text='BE/Bi 103 a', text_align='center')
p.add_layout(text)
bokeh.io.show(p)
Computing environment¶
[3]:
%load_ext watermark
%watermark -v -p numpy,bokeh,jupyterlab
CPython 3.8.3
IPython 7.16.1
numpy 1.18.5
bokeh 2.1.1
jupyterlab 2.1.5