File I/O¶
[1]:
# Colab setup ------------------
import os, sys, subprocess
if "google.colab" in sys.modules:
cmd = "pip install --upgrade watermark"
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
# ------------------------------
data_path = "../data"
import glob
Reading data in from files and then writing your results out again is one of the most common practices in scientific computing. In this tutorial, we will learn about some of Python’s File I/O capabilities. We will use a PDB file as an example. The PDB file contains the crystal structure for the tetramerization domain of p53. Note that 1OLG is its unique Protein Databank identifier.
You can download the PDB file here. To use the file in this notebook, you will need to have the file in the directory ../data/. This is the directory structure you use when doing your homework.
Note: Using the built-in open function will not work for opening files via URL. Therefore, if you are running this notebook in Google Colab, you will not be able to open the data file.
File objects¶
To open a file, we use the built-in open() function. When opening files, we should do this using context management. I will demonstrate how to open a file and then describe the syntax.
[2]:
with open(os.path.join(data_path, '1OLG.pdb'), 'r') as f:
print(type(f))
<class '_io.TextIOWrapper'>
Python has a wonderful keyword, with. This keyword enables context management. Upon entry into a with block, variables have certain meaning. In this case, the variable f has the meaning of an open file, an instance of the _io.TextIOWrapper class. Upon exit, certain operations take place. For file objects created by opening them, the file is automatically closed upon exit, even if there is an error. This is important. If your program raises an exception before you have a
chance to close the file, it won’t get closed and you could be in trouble. If you use context management, the file will still get closed. So here is an important tip:
Use context management using with when working with files.
Let’s focus for a moment on the variable f in the above code cell. It is a Python file object, which has methods and attributes, just like any other object. We’ll explore those in a moment, but first, let’s look at how we opened the file. The first argument to open() is a string that has the name of the file, with the full path if necessary. The second argument is a string that says what we will be doing with the file. I.e., are we reading or writing to the file? The possible strings
for this second argument are
string |
meaning |
|---|---|
|
open a text file for reading |
|
create and open a text file for writing |
|
append an existing text file |
|
open a text file for reading and writing |
append |
same as above, except for binary files |
We will mostly be working with text files in the bootcamp, so the first three are the most useful. A big warning, though….
Trying to open an existing file with 'w' will wipe it out and create a new file.
Reading data out of the file with file object methods¶
We will focus on the methods f.read() and f.readlines(). What do they do?
method |
task |
|---|---|
|
Read the entire contents of the file into a string |
|
Read the entire file into a list with each item being a string representing a line |
First, we’ll try using the first method to get a single string with the entire contents of the file.
[3]:
# Read file into string
with open('../data/1OLG.pdb', 'r') as f:
f_str = f.read()
# Let's look at the first 1000 characters
f_str[:1000]
[3]:
'HEADER ANTI-ONCOGENE 13-JUN-94 1OLG \nTITLE HIGH-RESOLUTION SOLUTION STRUCTURE OF THE OLIGOMERIZATION \nTITLE 2 DOMAIN OF P53 BY MULTI-DIMENSIONAL NMR \nCOMPND MOL_ID: 1; \nCOMPND 2 MOLECULE: TUMOR SUPPRESSOR P53 (OLIGOMERIZATION DOMAIN); \nCOMPND 3 CHAIN: A, B, C, D; \nCOMPND 4 ENGINEERED: YES \nSOURCE MOL_ID: 1; \nSOURCE 2 ORGANISM_SCIENTIFIC: HOMO SAPIENS; \nSOURCE 3 ORGANISM_COMMON: HUMAN; \nSOURCE 4 ORGANISM_TAXID: 9606 \nKEYWDS ANTI-ONCOGENE \nEXPDTA SOLUTION NMR '
We see lots of \n, which signifies a new line. The backslash is known as an escape character, meaning that the n after it does not signify the letter n, but that \n together means a new line.
Now, let’s try reading it in as a list.
[4]:
# Read contents of the file in as a list
with open('../data/1OLG.pdb', 'r') as f:
f_list = f.readlines()
# Look at the list (first ten entries)
f_list[:10]
[4]:
['HEADER ANTI-ONCOGENE 13-JUN-94 1OLG \n',
'TITLE HIGH-RESOLUTION SOLUTION STRUCTURE OF THE OLIGOMERIZATION \n',
'TITLE 2 DOMAIN OF P53 BY MULTI-DIMENSIONAL NMR \n',
'COMPND MOL_ID: 1; \n',
'COMPND 2 MOLECULE: TUMOR SUPPRESSOR P53 (OLIGOMERIZATION DOMAIN); \n',
'COMPND 3 CHAIN: A, B, C, D; \n',
'COMPND 4 ENGINEERED: YES \n',
'SOURCE MOL_ID: 1; \n',
'SOURCE 2 ORGANISM_SCIENTIFIC: HOMO SAPIENS; \n',
'SOURCE 3 ORGANISM_COMMON: HUMAN; \n']
We see that each entry is a line, including the newline character. To look at lines in files, the rstrip() method for strings can come it handy. It strips all whitespace, including newlines, from the end of a string.
[5]:
f_list[0].rstrip()
[5]:
'HEADER ANTI-ONCOGENE 13-JUN-94 1OLG'
Reading line-by-line¶
What if we do not want to read the entire file into a list? For example, if a file is several gigabytes, we do not want to spend all of our RAM storing a list. Instead, we can read it line-by-line. Conveniently, the file object can be used as an iterator.
[6]:
# Print the first ten lines of the file
with open('../data/1OLG.pdb', 'r') as f:
for i, line in enumerate(f):
print(line.rstrip())
if i >= 10:
break
HEADER ANTI-ONCOGENE 13-JUN-94 1OLG
TITLE HIGH-RESOLUTION SOLUTION STRUCTURE OF THE OLIGOMERIZATION
TITLE 2 DOMAIN OF P53 BY MULTI-DIMENSIONAL NMR
COMPND MOL_ID: 1;
COMPND 2 MOLECULE: TUMOR SUPPRESSOR P53 (OLIGOMERIZATION DOMAIN);
COMPND 3 CHAIN: A, B, C, D;
COMPND 4 ENGINEERED: YES
SOURCE MOL_ID: 1;
SOURCE 2 ORGANISM_SCIENTIFIC: HOMO SAPIENS;
SOURCE 3 ORGANISM_COMMON: HUMAN;
SOURCE 4 ORGANISM_TAXID: 9606
Alternatively, we can use the method f.readline() to read a single line in the file and return it as a string.
[7]:
# Print the first ten lines of the file
with open('../data/1OLG.pdb', 'r') as f:
i = 0
while i < 10:
print(f.readline().rstrip())
i += 1
HEADER ANTI-ONCOGENE 13-JUN-94 1OLG
TITLE HIGH-RESOLUTION SOLUTION STRUCTURE OF THE OLIGOMERIZATION
TITLE 2 DOMAIN OF P53 BY MULTI-DIMENSIONAL NMR
COMPND MOL_ID: 1;
COMPND 2 MOLECULE: TUMOR SUPPRESSOR P53 (OLIGOMERIZATION DOMAIN);
COMPND 3 CHAIN: A, B, C, D;
COMPND 4 ENGINEERED: YES
SOURCE MOL_ID: 1;
SOURCE 2 ORGANISM_SCIENTIFIC: HOMO SAPIENS;
SOURCE 3 ORGANISM_COMMON: HUMAN;
Each subsequent call to f.readline() reads in the next line of the file. (Remember, as we read through a file, we keep moving forward in the bytes of the file and we have to use f.seek() to rewind.)
Writing to a file¶
Writing to a file has similar syntax. We already saw how to open a file for writing. Again, context management is useful. However, before trying to open a file, we should check to make sure a file of the same name does not exist before opening it. The os.path module is useful. The function os.path.isfile() function checks to see if a file exists.
[8]:
os.path.isfile('../data/1OLG.pdb')
[8]:
True
Now that we know how to check existence of a file so we do not overwrite it, we can open and write a file.
[9]:
if os.path.isfile('knowledge.txt'):
raise RuntimeError('knowledge.txt already exists.')
with open('knowledge.txt', 'w') as f:
f.write("...as we know, there are known knowns;")
f.write("there are things we know we know.")
f.write("We also know there are known unknowns;")
f.write("that is to say we know there are some things we do not know.")
f.write("But there are also unknown unknowns,")
f.write("the ones we don't know we don't know.")
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-9-d3c7f41e3b51> in <module>
1 if os.path.isfile('knowledge.txt'):
----> 2 raise RuntimeError('knowledge.txt already exists.')
3
4 with open('knowledge.txt', 'w') as f:
5 f.write("...as we know, there are known knowns;")
RuntimeError: knowledge.txt already exists.
Note that we can use the f.write() method to write strings to a file. Let’s look at the file contents.
[10]:
!cat knowledge.txt
...as we know, there are known knowns;
there are things we know we know.
We also know there are known unknowns;
that is to say we know there are some things we do not know.
But there are also unknown unknowns,
the ones we don't know we don't know.
Ah! There are no newlines! When writing to a file, unlike when you use the print() function, you must include the newline characters. Let’s try again, intentionally obliterating our first attempt.
[11]:
with open('knowledge.txt', 'w') as f:
f.write("...as we know, there are known knowns;\n")
f.write("there are things we know we know.\n")
f.write("We also know there are known unknowns;\n")
f.write("that is to say we know there are some things we do not know.\n")
f.write("But there are also unknown unknowns,\n")
f.write("the ones we don't know we don't know.")
!cat knowledge.txt
...as we know, there are known knowns;
there are things we know we know.
We also know there are known unknowns;
that is to say we know there are some things we do not know.
But there are also unknown unknowns,
the ones we don't know we don't know.
That’s better. Note also that f.write() only takes strings as arguments. You cannot pass numbers. They must be converted to strings first.
[12]:
# This will result in an exception
with open('gimme_phi.txt', 'w') as f:
f.write('The golden ratio is φ = ')
f.write(1.61803398875)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-12-9265e8a96772> in <module>
2 with open('gimme_phi.txt', 'w') as f:
3 f.write('The golden ratio is φ = ')
----> 4 f.write(1.61803398875)
TypeError: write() argument must be str, not float
Yup. It must be a string. Let’s try again.
[13]:
with open('gimme_phi.txt', 'w') as f:
f.write('The golden ratio is φ = ')
f.write('{phi:.8f}'.format(phi=1.61803398875))
!cat gimme_phi.txt
The golden ratio is φ = 1.61803399
That works!
An exercise: extract atomic coordinates for first chain in tetramer¶
As an example on how to do file I/O, we will take the PDB file and extract only the ATOM records for the first chain of the tetramer and write only those entries to a new file.
It is useful to know that according to the PDB format specification, column 21 in the ATOM entry gives the ID of the chain.
We also conveniently use the fact that we can have multiple files open in our with block, separating them with commas.
[14]:
with open('../data/1OLG.pdb', 'r') as f, open('atoms_chain_A.txt', 'w') as f_out:
# Put the ATOM lines from chain A in new file
for line in f:
if len(line) > 21 and line[:4] == 'ATOM' and line[21] == 'A':
f_out.write(line)
Let’s see how we did!
[15]:
!head -10 atoms_chain_A.txt
ATOM 1 N LYS A 319 18.634 25.437 10.685 1.00 4.81 N
ATOM 2 CA LYS A 319 17.984 25.295 9.354 1.00 4.32 C
ATOM 3 C LYS A 319 18.160 23.876 8.818 1.00 3.74 C
ATOM 4 O LYS A 319 19.259 23.441 8.537 1.00 3.67 O
ATOM 5 CB LYS A 319 18.609 26.282 8.371 1.00 4.67 C
ATOM 6 CG LYS A 319 18.003 26.056 6.986 1.00 5.15 C
ATOM 7 CD LYS A 319 16.476 26.057 7.091 1.00 5.90 C
ATOM 8 CE LYS A 319 16.014 27.341 7.784 1.00 6.51 C
ATOM 9 NZ LYS A 319 16.388 28.518 6.952 1.00 7.33 N
ATOM 10 H1 LYS A 319 18.414 24.606 11.281 1.00 5.09 H
[16]:
!tail -10 atoms_chain_A.txt
ATOM 689 HD2 PRO A 359 0.183 25.663 13.542 1.00 4.71 H
ATOM 690 HD3 PRO A 359 0.246 23.956 13.062 1.00 4.53 H
ATOM 691 N GLY A 360 -3.984 26.791 10.832 1.00 5.45 N
ATOM 692 CA GLY A 360 -4.489 28.138 10.445 1.00 5.95 C
ATOM 693 C GLY A 360 -5.981 28.236 10.765 1.00 6.77 C
ATOM 694 O GLY A 360 -6.401 27.621 11.732 1.00 7.24 O
ATOM 695 OXT GLY A 360 -6.679 28.924 10.039 1.00 7.15 O
ATOM 696 H GLY A 360 -4.589 26.020 10.828 1.00 5.72 H
ATOM 697 HA2 GLY A 360 -3.950 28.896 10.995 1.00 5.99 H
ATOM 698 HA3 GLY A 360 -4.341 28.288 9.386 1.00 6.05 H
Nice!
Finding files and with glob¶
In the above snippet of code, we extracted all atom records from a PDB file. We might want to do this (or some other operation) for many files. For example, the directory ~/git/data/ has four PDB files in it. For the present discussion, let’s say we want to pull the sequence of chain A out of each PDB file.
The glob module from the standard library enables us to get a list of all files that match a pattern. In our case, we want all files matching data/*.pdb, where * is a wild card character, meaning that any matches of characters where * appears are allowed. Let’s see what glob.glob() gives us.
[17]:
file_list = glob.glob('../data/*.pdb')
file_list
[17]:
['../data/1OLG.pdb',
'../data/1J6Z.pdb',
'../data/1FAG.pdb',
'../data/2ERK.pdb']
We have the four PDB files. We can now loop over them and pull out the sequences.
[18]:
# Dictionary to hold sequences
seqs = {}
# Loop through all matching files
for file_name in file_list:
# Extract PDB ID
pdb_id = file_name[file_name.rfind('/')+1:file_name.rfind('.')]
# Initialize sequence string, which we build as we go along
seq = ''
with open(file_name, 'r') as f:
for line in f:
if len(line) > 11 and line[:6] == 'SEQRES' and line[11] == 'A':
seq += line[19:].rstrip() + ' '
# Build sequence with dash-joined three letter codes
seq = '-'.join(seq.split())
# Store in the dictionary
seqs[pdb_id] = seq
Let’s take a look at what we got. We’ll look at actin.
[19]:
seqs['1J6Z']
[19]:
'ASP-GLU-ASP-GLU-THR-THR-ALA-LEU-VAL-CYS-ASP-ASN-GLY-SER-GLY-LEU-VAL-LYS-ALA-GLY-PHE-ALA-GLY-ASP-ASP-ALA-PRO-ARG-ALA-VAL-PHE-PRO-SER-ILE-VAL-GLY-ARG-PRO-ARG-HIS-GLN-GLY-VAL-MET-VAL-GLY-MET-GLY-GLN-LYS-ASP-SER-TYR-VAL-GLY-ASP-GLU-ALA-GLN-SER-LYS-ARG-GLY-ILE-LEU-THR-LEU-LYS-TYR-PRO-ILE-GLU-HIC-GLY-ILE-ILE-THR-ASN-TRP-ASP-ASP-MET-GLU-LYS-ILE-TRP-HIS-HIS-THR-PHE-TYR-ASN-GLU-LEU-ARG-VAL-ALA-PRO-GLU-GLU-HIS-PRO-THR-LEU-LEU-THR-GLU-ALA-PRO-LEU-ASN-PRO-LYS-ALA-ASN-ARG-GLU-LYS-MET-THR-GLN-ILE-MET-PHE-GLU-THR-PHE-ASN-VAL-PRO-ALA-MET-TYR-VAL-ALA-ILE-GLN-ALA-VAL-LEU-SER-LEU-TYR-ALA-SER-GLY-ARG-THR-THR-GLY-ILE-VAL-LEU-ASP-SER-GLY-ASP-GLY-VAL-THR-HIS-ASN-VAL-PRO-ILE-TYR-GLU-GLY-TYR-ALA-LEU-PRO-HIS-ALA-ILE-MET-ARG-LEU-ASP-LEU-ALA-GLY-ARG-ASP-LEU-THR-ASP-TYR-LEU-MET-LYS-ILE-LEU-THR-GLU-ARG-GLY-TYR-SER-PHE-VAL-THR-THR-ALA-GLU-ARG-GLU-ILE-VAL-ARG-ASP-ILE-LYS-GLU-LYS-LEU-CYS-TYR-VAL-ALA-LEU-ASP-PHE-GLU-ASN-GLU-MET-ALA-THR-ALA-ALA-SER-SER-SER-SER-LEU-GLU-LYS-SER-TYR-GLU-LEU-PRO-ASP-GLY-GLN-VAL-ILE-THR-ILE-GLY-ASN-GLU-ARG-PHE-ARG-CYS-PRO-GLU-THR-LEU-PHE-GLN-PRO-SER-PHE-ILE-GLY-MET-GLU-SER-ALA-GLY-ILE-HIS-GLU-THR-THR-TYR-ASN-SER-ILE-MET-LYS-CYS-ASP-ILE-ASP-ILE-ARG-LYS-ASP-LEU-TYR-ALA-ASN-ASN-VAL-MET-SER-GLY-GLY-THR-THR-MET-TYR-PRO-GLY-ILE-ALA-ASP-ARG-MET-GLN-LYS-GLU-ILE-THR-ALA-LEU-ALA-PRO-SER-THR-MET-LYS-ILE-LYS-ILE-ILE-ALA-PRO-PRO-GLU-ARG-LYS-TYR-SER-VAL-TRP-ILE-GLY-GLY-SER-ILE-LEU-ALA-SER-LEU-SER-THR-PHE-GLN-GLN-MET-TRP-ILE-THR-LYS-GLN-GLU-TYR-ASP-GLU-ALA-GLY-PRO-SER-ILE-VAL-HIS-ARG-LYS-CYS-PHE'
Computing environment¶
[20]:
%load_ext watermark
%watermark -v -p jupyterlab
CPython 3.8.3
IPython 7.16.1
jupyterlab 2.1.5