Setting up computing resources

In this lesson, you will set up a GitHub account, learn about Google Colab, and set up a Python computing environment for scientific computing.

It is advantageous to learn how to set up a Python distribution and manage packages on your own machine, as each person can have different needs. That said, Google Colab is a nice, free resource to run Jupyter Notebooks on Google’s computers without any local installations necessary.

Getting a GitHub account

We will make extensive use of Git during the course. We will use GitHub to host the repositories. You need to set up a GitHub account. Go to http://github.com/ to get an account. You should register with your academic email address so you get free private repositories as academics. You should also think carefully about picking your user name. There is a good chance other people in your professional life will see this.

Setting up Google Colab

In order to use Google Colab, you must have a Google account. Caltech students and employees have an account through Caltech’s G Suite. Many of you may have a personal Google account, usually set up for things like GMail, YouTube, etc. For your work in this class, use your Caltech account. This will facilitate collaboration with your teammates in the course, as well as with course staff.

Many of you probably use your personal Google account on your machine, so it can get annoying to log in and out of it. A trick that I find useful is to use one browser, e.g., Safari or Microsoft Edge, for your personal use, web browsing, etc., and a different browser for your scientific work, including the work in this class. Google Colab are most tested for Chrome, Firefox, and Safari (in fact JupyterLab, which you will use on your own machine, only supports these three browsers).

Once you have either logged out of all of your personal accounts or have a different browser open, you can launch a Colab notebook by simply navigating to https://colab.research.google.com/. Alternatively, you can click the “Launch in Colab” badge at the top right of this page, and you will launch this notebook in Colab. That badge will appear in the top right of all pages in the course content generated from notebooks.

Watchouts when using Colab

If you do run a notebook in Colab, you are doing your computing on one of Google’s computers via a virtual machine. You get two CPU cores and 12 GB of RAM. You can also get GPUs and TPUs (Google’s tensor processing units), but we will not use those in this course. The computing resources should be enough for all of our calculations this term (though you will need more computing power in the sequel of this course). However, there are some limitations you should be aware of.

  • If your notebook is idle for too long, you will get disconnected from your notebook. “Idle” means that cells are not being edited or executed. The idle timeout varies depending on the load on Google’s computers; I find that I almost always get disconnected if idle for an hour.

  • Your virtual machine will disconnect if it is being used for too long. It typically will only available for 12 hours before disconnecting, though times can vary, again based on load.

These limitations are in place so that Google can offer Colab for free. If you want more cores, longer timeouts, etc., you might want to check out Colab Pro. However, the free tier should work well for you in the course. You of course can always run on your own machine, and in fact are encouraged to do so except where collaboration is necessary.

There are additional software-specific watchouts when using Colab.

  • Later in the course, we will use HoloViews for high-level plotting. Colab will not render HoloViews plots unless hv.extension('bokeh') is called in each cell that has a HoloViews plot.

  • Colab does not allow for full functionality Bokeh apps and some Panel functionality that we will use when we do dashboarding.

  • Colab instances have specific software installed, so you will need to install anything else you need in your notebook. This is not a major burden, and is discussed in the next section.

I recommend reading the Colab FAQs for more information about Colab.

Software in Colab

When you launch a Google Colab notebook, much of the software we will use in class is already installed. It is not always the latest version of the software, however. In fact, as of mid-August 2021, Colab is running Python 3.7, whereas you will run Python 3.8 on your machine through your Anaconda installation. Nonetheless, most (but not all) of the analyses we do for this class will work just fine in Colab. We will make every effort to let you know when Colab will not be able to handle activities in class, the most important example being some dashboarding applications.

Because the notebooks in Colab have software preinstalled, and no more, you will often need to install software before you can run the rest of the code in a notebook. To enable this, when necessary, in the first code cell of each notebook in this class, we will have the following code (or a variant thereof depending on what is needed or if the default installations of Colab change). Running this code will not affect running your notebook on your local machine; the same notebook will work on your local machine or on Colab.

# Colab setup ------------------
import os, sys, subprocess
if "google.colab" in sys.modules:
    cmd = "pip install --upgrade iqplot colorcet datashader bebi103 watermark"
    process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = process.communicate()
    data_path = "https://s3.amazonaws.com/bebi103.caltech.edu/data/"
else:
    data_path = "../data/"
# ------------------------------

In addition to installing the necessary software on a Colab instance, this also sets the relative path to data sets we will use in the course. When running in Colab, the data set is fetched from cloud storage on AWS. When running on your local machine, the path to the data is one directory up from where you are working.

In most notebooks, the Colab and data path setup code cells are hidden in the HTML rendering to avoid clutter, but will be present when you download the notebooks.

Collaborating with Colab

If you want to collaborate with another student or with the course staff on a notebook, you can click “Share” on the top right corner of the Colab window and choose with whom and how (the defaults are fine) you want to share.

When we talk about Git in a future lesson, we will discuss Colab’s GitHub support, which will be necessary for version control and submitting and sharing your homework.

Installation on your own machine

We now proceed to discuss installation of the necessary software on your own machine. Before we get into that, there are some preliminaries to configure your computer.

macOS users: Install XCode

If you are using macOS, you should install XCode, if you haven’t already. It’s a large piece of software, taking up about 5GB on your hard drive, so make sure you have enough space. You can install it through the App Store.

After installing it, you need to open the program. Be sure to do that, for example by clicking on the XCode icon in your Applications folder. Upon opening XCode, it may perform more installations. After these are completed, you can close XCode.

Windows users: Install Git and Chrome or Firefox

We will be using JupyterLab in this course. It is browser-based, and Chrome, Firefox, and Safari are supported. Microsoft Edge is not. Therefore, if you are a Windows user, you need to be sure you have either Chrome of Firefox installed.

Git is installed on Macs with XCode. For Windows users, you need to install Git by itself. You can do this by following the instructions here.

Uninstalling Anaconda

Unless you have experience with Anaconda and know how to set up environments, if you have previously installed Anaconda with a version of Python other than 3.8, you need to uninstall it, removing it completely from your computer. You can find instructions on how to do that from the official uninstallation documentation.

Downloading and installing Anaconda

Downloading and installing Anaconda is simple.

  1. Go to the Anaconda distribution homepage and download the graphical installer.

  2. Be sure to download Anaconda for Python 3.8 for the appropriate operating system.

  3. Follow the on-screen instructions for installation. When prompted, be sure to “Install for me only.”

  4. You may be prompted for optional installations, like PyCharm. You will not need these for the course.

That’s it! After you do that, you will have a functioning Python distribution.

Install node.js

node.js is a platform that enables you to run JavaScript outside of the browser. We will not use it directly, but it needs to be installed for some of the more sophisticated JupyterLab functionality. Install node.js by downloading the appropriate installer for your machine here.

Launching JupyterLab and a terminal

You can alternatively launch JupyterLab via the Anaconda Navigator or via your operating system’s terminal program (Terminal on macOS and PowerShell on Windows). If you wish to launch using the latter (which I prefer), skip to the next section.

After installing the Anaconda distribution, you should be able to launch the Anaconda Navigator. If you are using macOS, this is available in your Applications menu. If you are using Windows, you can do this from the Start menu. Launch Anaconda Navigator.

We will be using JupyterLab throughout the course (more on that in the next part of this lesson). You should see an option to launch JupyterLab. When you do that, a new browser window or tab will open with JupyterLab running. Within the JupyterLab window, you will have the option to launch a notebook, a console, a terminal, or a text editor. We will use all of these during the course. For the updating and installation of necessary packages, click on Terminal to launch a terminal. You will get a terminal window (probably black) with a bash prompt. We refer to this text interface in the terminal as the command line.

You can use this terminal for the package installations that follow. Alternatively, you can directly use your operating system’s terminal application.

Launching JupyterLab from the command line

While launching JupyterLab from the Anaconda Navigator is fine, I generally prefer to launch it from the command line on my own machine. If you are on a Mac, open the Terminal program. You can do this hitting Command + space bar and searching for “terminal.” Using Windows, you should launch PowerShell. You can do this by hitting Windows + R and typing “powershell” in the text box.

Once you have a terminal or PowerShell window open, you will have a prompt. At the prompt, type

jupyter lab

and you will have an instance of JupyterLab running in your browser. If you want to specify the browser, you can, for example, type

jupyter lab --browser=firefox

on the command line.

It is up to you if you want to launch JupyterLab from the Anaconda Navigator or command line.

The conda package manager

conda is a package manager for keeping all of your packages up-to-date. It has plenty of functionality beyond our basic usage in class, which you can learn more about by reading the docs. We will primarily be using conda to install and update packages.

conda works from the command line. Now that you know how to get a command line prompt, you can start using conda. The first thing we’ll do is update conda itself. Enter the following on the command line

conda update conda

You will be prompted to continue this operation, so press y to continue. You should do this once more, again entering

conda update conda

on the command line.

Next, we’ll update the packages that came with the Anaconda distribution. To do this, enter the following on the command line:

conda update --all

If anything is out of date, you will be prompted to perform the updates, and press y to continue. (If everything is up to date, you will just see a list of all the installed packages.) There may even be some downgrades. This happens when there are package conflicts where one package requires an earlier version of another. conda is very smart and figures all of this out for you, so you can almost always say “yes” (or “y”) to conda when it prompts you.

Package installations

There are several additional installations of Python packages you need to do. Many of these packages are available through conda. First, we need to install jupyter_bokeh, which allows Bokeh plots to be displayed withing Jupyter notebooks. Do the following on the command line.

conda install -c bokeh jupyter_bokeh

Next, we will first install some plotting packages we need.

conda install holoviews param panel colorcet hvplot datashader selenium

With the exception of Selenium, which is used to render graphics as SGV, these packages are part of the HoloViz suite of packages. This also includes GeoViews, which is an excellent plotting package for geographical data. Because we will not be using geographical data sets in class and because as of mid-August 2021 GeoViews has conflicts with other packages, we are not installing it. If your research involves geographical data, you may with to install it as well and accept the package downgrades that conda enforces. The easiest way to do an installation of the HoloViz packages including GeoViews (and accepting downgrades) is to not do the above installation, but instead do conda install -c pyviz holoviz. If you are not going to be using GeoViews, do not do this, but rather execute the suggested installation above.

Next, we need to install a few packages we will use indirectly during the course that are handy to have.

conda install netcdf4 black

We will also install watermark, which enables us to conveniently display version numbers of the software we are using. For this installation, we will use pip. There are a few other packages from pip we will need, so we can go ahead and install those now.

pip install watermark blackcellmagic jupyterlab-spellchecker multiprocess jupytext cmdstanpy arviz iqplot bebi103

You should close your JupyterLab session and terminate Anaconda Navigator after you have completed the installations. Relaunch Anaconda Navigator and launch a fresh JupyterLab instance.

Checking your distribution

We’ll now run a quick test to make sure things are working properly. We will make a quick plot that requires some of the scientific libraries we will use.

Use the JupyterLab launcher (you can get a new launcher by clicking on the + icon on the left pane of your JupyterLab window) to launch a notebook. In the first cell (the box next to the [ ]: prompt), paste the code below. To run the code, press Shift+Enter while the cursor is active inside the cell. You should see a plot that looks like the one below. If you do, you have a functioning Python environment for scientific computing!

You can also test this in Colab (and it should work with no problems).

[2]:

import numpy as np
import bokeh.plotting
import bokeh.io

bokeh.io.output_notebook()

# Generate plotting values
t = np.linspace(0, 2*np.pi, 200)
x = 16 * np.sin(t)**3
y = 13 * np.cos(t) - 5 * np.cos(2*t) - 2 * np.cos(3*t) - np.cos(4*t)

p = bokeh.plotting.figure(height=250, width=275)
p.line(x, y, color='red', line_width=3)
text = bokeh.models.Label(x=0, y=0, text='BE/Bi 103 a', text_align='center')
p.add_layout(text)

bokeh.io.show(p)
Loading BokehJS ...

Computing environment

[3]:

%load_ext watermark
%watermark -v -p numpy,bokeh,jupyterlab

Python implementation: CPython
Python version       : 3.8.11
IPython version      : 7.27.0

numpy     : 1.20.3
bokeh     : 2.3.3
jupyterlab: 3.1.7