Chapter 0  Preface

Python for Informatics: Remixing an Open Book

Preface for ``Think Python''

The strange history of ``Think Python''

• Keep it short. It is better for students to read 10 pages than not read 50 pages.
  
  
• Be careful with vocabulary. I tried to minimize the jargon and define each term at first use.
  
  
• Build gradually. To avoid trap doors, I took the most difficult topics and split them into a series of small steps.
  
  
• Focus on programming, not the programming language. I included the minimum useful subset of Java and left out the rest.

Acknowledgements for ``Think Python''

Chapter 1  Why should you learn to write programs?

python words.py
Enter file:words.txt
to 16

1.1  Creativity and motivation

1.2  Computer hardware architecture

• The Central Processing Unit (or CPU) is that part of the computer that is built to be obsessed with ``what is next?''. If your computer is rated at 3.0 Gigahertz, it means that the CPU will ask ``What next?'' three billion times per second. You are going to have to learn how to talk fast to keep up with the CPU.
  
  
• The Main Memory is used to store information that the CPU needs in a hurry. The main memory is nearly as fast as the CPU. But the information stored in the main memory vanishes when the computer is turned off.
  
  
• The Secondary Memory is also used to store information, but it is much slower than the main memory. The advantage of the secondary memory is that it can store information even when there is no power to the computer. Examples of secondary memory are disk drives or flash memory (typically found in USB sticks and portable music players).
  
  
• The Input and Output Devices are simply our screen, keyboard, mouse, microphone, speaker, touchpad, etc. They are all of the ways we interact with the computer.
  
  
• These days, most computers also have a Network Connection to retrieve information over a network. We can think of the network as a very slow place to store and retrieve data that might not always be ``up''. So in a sense, the network is a slower and at times unreliable form of Secondary Memory

1.3  Understanding programming

• First you need to know the programming language (Python) - you need to know the vocabulary and the grammar. You need to be able spell the words in this new language properly and how to construct well-formed ``sentences'' in this new languages.
  
  
• Second you need to ``tell a story''. In writing a story, you combine words and sentences to convey an idea to the reader. There is a skill and art in constructing the story and skill in story writing is improved by doing some writing and getting some feedback. In programming, our program is the ``story'' and the problem you are trying to solve is the ``idea''.

1.4  Words and sentences

and   del   for   is   raise 
assert   elif   from   lambda   return 
break   else   global   not   try 
class   except   if   or   while 
continue   exec   import   pass   yield 
def   �nally   in   print 

print 'Hello world!'

1.5  Conversing with Python

Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 

>>> I come in peace, please take me to your leader
  File "<stdin>", line 1
    I come in peace, please take me to your leader
         ^
SyntaxError: invalid syntax
>>> 

>>> print 'Hello world!'
Hello world!

>>> print 'You must be the legendary god that comes from the sky'
You must be the legendary god that comes from the sky
>>> print 'We have been waiting for you for a long time'
We have been waiting for you for a long time
>>> print 'Our legend says you will be very tasty with mustard'
Our legend says you will be very tasty with mustard
>>> print 'We will have a feast tonight unless you say
  File "<stdin>", line 1
    print 'We will have a feast tonight unless you say
                                                     ^
SyntaxError: EOL while scanning string literal
>>> 

>>> good-bye
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'good' is not defined

>>> if you don't mind, I need to leave
  File "<stdin>", line 1
    if you don't mind, I need to leave
             ^
SyntaxError: invalid syntax

>>> quit()

1.6  Terminology: interpreter and compiler

01010001110100100101010000001111
11100110000011101010010101101101
...

>>> x = 6
>>> print x
6
>>> y = x * 7
>>> print y
42
>>> 

^?ELF^A^A^A^@^@^@^@^@^@^@^@^@^B^@^C^@^A^@^@^@\xa0\x82
^D^H4^@^@^@\x90^]^@^@^@^@^@^@4^@ ^@^G^@(^@$^@!^@^F^@
^@^@4^@^@^@4\x80^D^H4\x80^D^H\xe0^@^@^@\xe0^@^@^@^E
^@^@^@^D^@^@^@^C^@^@^@^T^A^@^@^T\x81^D^H^T\x81^D^H^S
^@^@^@^S^@^@^@^D^@^@^@^A^@^@^@^A\^D^HQVhT\x83^D^H\xe8
....

C:\Python27\python.exe

1.7  Writing a program

csev$ cat hello.py
print 'Hello world!'
csev$ python hello.py
Hello world!
csev$

1.8  What is a program?

the clown ran after the car and the car ran into the tent 
and the tent fell down on the clown and the car 

name = raw_input('Enter file:')
handle = open(name, 'r')
text = handle.read()
words = text.split()
counts = dict()

for word in words:
   counts[word] = counts.get(word,0) + 1

bigcount = None
bigword = None
for word,count in counts.items():
    if bigcount is None or count > bigcount:
        bigword = word
        bigcount = count

print bigword, bigcount

1.9  The building blocks of programs

input:

Get data from the the ``outside world''. This might be reading data from a file, or even some kind of sensor like a microphone or GPS. In our initial programs, our input will come from the user typing data on the keyboard.

output:

Display the results of the program on a screen or store them in a file or perhaps write them to a device like a speaker to play music or speak text.

sequential execution:

Perform statements one after another in the order they are encountered in the script.

conditional execution:

Check for certain conditions and execute or skip a sequence of statements.

repeated execution:

Perform some set of statements repeatedly, usually with some variation.

reuse:

Write a set of instructions once and give them a name and then reuse those instructions as needed throughout your program.

1.10  What could possibly go wrong?

>>> primt 'Hello world!'
  File "<stdin>", line 1
    primt 'Hello world!'
                       ^
SyntaxError: invalid syntax
>>> primt 'Hello world'
  File "<stdin>", line 1
    primt 'Hello world'
                      ^
SyntaxError: invalid syntax
>>> I hate you Python!
  File "<stdin>", line 1
    I hate you Python!
         ^
SyntaxError: invalid syntax
>>> if you come out of there, I would teach you a lesson
  File "<stdin>", line 1
    if you come out of there, I would teach you a lesson
              ^
SyntaxError: invalid syntax
>>> 

Syntax errors:

These are the first errors you will make and the easiest to fix. A syntax error means that you have violated the ``grammar'' rules of Python. Python does its best to point right at the line and character where it noticed it was confused. The only tricky bit of syntax errors is that sometimes the mistake that needs fixing is actually earlier in the program than where Python noticed it was confused. So the line and character that Python indicates in a syntax error may just be a starting point for your investigation.

Logic errors:

A logic error is when your program has good syntax but there is a mistake in the order of the statements or perhaps a mistake in how the statements relate to one another. A good example of a logic error might be, ``take a drink from your water bottle, put it in your backpack, walk to the library, and then put the top back on the bottle.''

Semantic errors:

A semantic error is when your description of the steps to take is syntactically perfect and in the right order, but there is simply a mistake in the program. The program is perfectly correct but it does not do what you intended for it to do. A simple example would be if you were giving a person directions to a restaurant and said, ``... when you reach the intersection with the gas station, turn left and go one mile and the restaurant is a red building on your left.''. Your friend is very late and calls you to tell you that they are on a farm and walking around behind a barn, with no sign of a restaurant. The you say ``did you turn left or right gas station?'' and they say, ``I followed your directions perfectly, I have them written down, it says turn left and go one mile at the gas station.''. Then you say, ``I am very sorry, because while my instructions were syntactically correct, they sadly contained a small but undetected semantic error.''.

1.11  The learning journey

1.12  Glossary

bug:

An error in a program.

central processing unit:

The heart of any computer. It is what runs the software that we write; also called ``CPU'' or ``the processor''.

compile:

To translate a program written in a high-level language into a low-level language all at once, in preparation for later execution.

high-level language:

A programming language like Python that is designed to be easy for humans to read and write.

interactive mode:

A way of using the Python interpreter by typing commands and expressions at the prompt.

interpret:

To execute a program in a high-level language by translating it one line at a time.

low-level language:

A programming language that is designed to be easy for a computer to execute; also called ``machine code'' or ``assembly language.''

machine code:

The lowest level language for software which is the language that is directly executed by the central processing unit (CPU).

main memory:

Stores programs and data. Main memory loses its information when the power is turned off.

parse:

To examine a program and analyze the syntactic structure.

portability:

A property of a program that can run on more than one kind of computer.

print statement:

An instruction that causes the Python interpreter to display a value on the screen.

problem solving:

The process of formulating a problem, finding a solution, and expressing the solution.

program:

A set of instructions that specifies a computation.

prompt:

When a program displays a message and pauses for the user to type some input to the program.

secondary memory:

Stores programs and data and retains its information even when the power is turned off. Generally slower than main memory. Examples of secondary memory include disk drives and flash memory in USB sticks.

semantics:

The meaning of a program.

semantic error:

An error in a program that makes it do something other than what the programmer intended.

source code:

A program in a high-level language.

1.13  Exercises

>>> primt 'Hello world!'
  File "<stdin>", line 1
    primt 'Hello world!'
                       ^
SyntaxError: invalid syntax
>>> 

x = 123

x = 43
x = x + 1
print x

1

http://xkcd.com/231/

Chapter 2  Variables, expressions and statements

2.1  Values and types

python
>>> print 4
4

>>> type('Hello, World!')
<type 'str'>
>>> type(17)
<type 'int'>

>>> type(3.2)
<type 'float'>

>>> type('17')
<type 'str'>
>>> type('3.2')
<type 'str'>

>>> print 1,000,000
1 0 0

2.2  Variables

>>> message = 'And now for something completely different'
>>> n = 17
>>> pi = 3.1415926535897931

>>> print n
17
>>> print pi
3.14159265359

>>> type(message)
<type 'str'>
>>> type(n)
<type 'int'>
>>> type(pi)
<type 'float'>

2.3  Variable names and keywords

>>> 76trombones = 'big parade'
SyntaxError: invalid syntax
>>> more@ = 1000000
SyntaxError: invalid syntax
>>> class = 'Advanced Theoretical Zymurgy'
SyntaxError: invalid syntax

and       del       from      not       while    
as        elif      global    or        with     
assert    else      if        pass      yield    
break     except    import    print              
class     exec      in        raise              
continue  finally   is        return             
def       for       lambda    try

2.4  Statements

print 1
x = 2
print x

1
2

2.5  Operators and operands

20+32   hour-1   hour*60+minute   minute/60   5**2   (5+9)*(15-7)

>>> minute = 59
>>> minute/60
0

>>> minute/60.0
0.98333333333333328

2.6  Expressions

17
x
x + 17

>>> 1 + 1
2

5
x = 5
x + 1

2.7  Order of operations

• Parentheses have the highest precedence and can be used to force an expression to evaluate in the order you want. Since expressions in parentheses are evaluated first, 2 * (3-1) is 4, and (1+1)**(5-2) is 8. You can also use parentheses to make an expression easier to read, as in (minute * 100) / 60, even if it doesn't change the result.
  
  
• Exponentiation has the next highest precedence, so 2**1+1 is 3, not 4, and 3*1**3 is 3, not 27.
  
  
• Multiplication and Division have the same precedence, which is higher than Addition and Subtraction, which also have the same precedence. So 2*3-1 is 5, not 4, and 6+4/2 is 8, not 5.
  
  
• Operators with the same precedence are evaluated from left to right. So in the expression 5-3-1 is 1, not 3 because the 5-3 happens first and then 1 is subtracted from 2.

2.8  Modulus operator

>>> quotient = 7 / 3
>>> print quotient
2
>>> remainder = 7 % 3
>>> print remainder
1

2.9  String operations

>>> first = 10
>>> second = 15
>>> print first+second
25
>>> first = '100'
>>> second = '150'
>>> print first + second
100150

2.10  Asking the user for input

>>> input = raw_input()
Some silly stuff
>>> print input
Some silly stuff

>>> name = raw_input('What is your name?\n')
What is your name?
Chuck
>>> print name
Chuck

>>> prompt = 'What...is the airspeed velocity of an unladen swallow?\n'
>>> speed = raw_input(prompt)
What...is the airspeed velocity of an unladen swallow?
17
>>> int(speed)
17
>>> int(speed) + 5
22

>>> speed = raw_input(prompt)
What...is the airspeed velocity of an unladen swallow?
What do you mean, an African or a European swallow?
>>> int(speed)
ValueError: invalid literal for int()

2.11  Comments

# compute the percentage of the hour that has elapsed
percentage = (minute * 100) / 60

percentage = (minute * 100) / 60     # percentage of an hour

v = 5     # assign 5 to v

v = 5     # velocity in meters/second. 

2.12  Choosing mnemonic variable names

a = 35.0
b = 12.50
c = a * b
print c

hours = 35.0
rate = 12.50
pay = hours * rate
print pay

x1q3z9ahd = 35.0
x1q3z9afd = 12.50
x1q3p9afd = x1q3z9ahd * x1q3z9afd
print x1q3p9afd

for word in words:
    print word

for slice in pizza:
    print slice

2.13  Debugging

>>> bad name = 5
SyntaxError: invalid syntax

>>> principal = 327.68
>>> interest = principle * rate
NameError: name 'principle' is not defined

>>> 1.0 / 2.0 * pi

2.14  Glossary

assignment:

A statement that assigns a value to a variable.

concatenate:

To join two operands end-to-end.

comment:

Information in a program that is meant for other programmers (or anyone reading the source code) and has no effect on the execution of the program.

evaluate:

To simplify an expression by performing the operations in order to yield a single value.

expression:

A combination of variables, operators, and values that represents a single result value.

floating-point:

A type that represents numbers with fractional parts.

floor division:

The operation that divides two numbers and chops off the fraction part.

integer:

A type that represents whole numbers.

keyword:

A reserved word that is used by the compiler to parse a program; you cannot use keywords like if, def, and while as variable names.

mnemonic:

A memory aid. We often give variables mnemonic names to help us remember what is stored in the variable.

modulus operator:

An operator, denoted with a percent sign (%), that works on integers and yields the remainder when one number is divided by another.

operand:

One of the values on which an operator operates.

operator:

A special symbol that represents a simple computation like addition, multiplication, or string concatenation.

rules of precedence:

The set of rules governing the order in which expressions involving multiple operators and operands are evaluated.

statement:

A section of code that represents a command or action. So far, the statements we have seen are assignments and print statements.

string:

A type that represents sequences of characters.

type:

A category of values. The types we have seen so far are integers (type int), floating-point numbers (type float), and strings (type str).

value:

One of the basic units of data, like a number or string, that a program manipulates.

variable:

A name that refers to a value.

2.15  Exercises

Enter your name: Chuck
Hello Chuck

Enter Hours: 35
Enter Rate: 2.75
Pay: 96.25

width = 17
height = 12.0

1

In Python 3.0, exec is no longer a keyword, but nonlocal is.

2

In Python 3.0, the result of this division is a float. In Python 3.0, the new operator // performs integer division.

3

In Python 3.0, this function is named input.

4

See http://en.wikipedia.org/wiki/Mnemonic for an extended description of the word ``mnemonic''.

Chapter 3  Conditional execution

3.1  Boolean expressions

>>> 5 == 5
True
>>> 5 == 6
False

>>> type(True)
<type 'bool'>
>>> type(False)
<type 'bool'>

      x != y               # x is not equal to y
      x > y                # x is greater than y
      x < y                # x is less than y
      x >= y               # x is greater than or equal to y
      x <= y               # x is less than or equal to y
      x is y               # x is the same as y
      x is not y           # x is not the same as y

3.2  Logical operators

>>> 17 and True
True

3.3  Conditional execution

if x > 0 :
    print 'x is positive'

if x < 0 :
    pass          # need to handle negative values!

>>> x = 3
>>> if x < 10:
...    print 'Small'
... 
Small
>>>

3.4  Alternative execution

if x%2 == 0 :
    print 'x is even'
else :
    print 'x is odd'

3.5  Chained conditionals

if x < y:
    print 'x is less than y'
elif x > y:
    print 'x is greater than y'
else:
    print 'x and y are equal'

if choice == 'a':
    print 'Bad guess'
elif choice == 'b':
    print 'Good guess'
elif choice == 'c':
    print 'Close, but not correct'

3.6  Nested conditionals

if x == y:
    print 'x and y are equal'
else:
    if x < y:
        print 'x is less than y'
    else:
        print 'x is greater than y'

if 0 < x:
    if x < 10:
        print 'x is a positive single-digit number.'

if 0 < x and x < 10:
    print 'x is a positive single-digit number.'

3.7  Catching exceptions using try and except

>>> speed = raw_input(prompt)
What...is the airspeed velocity of an unladen swallow?
What do you mean, an African or a European swallow?
>>> int(speed)
ValueError: invalid literal for int()
>>>

inp = raw_input('Enter Fahrenheit Temperature:')
fahr = float(inp)
cel = (fahr - 32.0) * 5.0 / 9.0
print cel

python fahren.py 
Enter Fahrenheit Temperature:72
22.2222222222

python fahren.py 
Enter Fahrenheit Temperature:fred
Traceback (most recent call last):
  File "fahren.py", line 2, in <module>
    fahr = float(inp)
ValueError: invalid literal for float(): fred

inp = raw_input('Enter Fahrenheit Temperature:')
try:
    fahr = float(inp)
    cel = (fahr - 32.0) * 5.0 / 9.0
    print cel
except:
    print 'Please enter a number'

python fahren2.py 
Enter Fahrenheit Temperature:72
22.2222222222

python fahren2.py 
Enter Fahrenheit Temperature:fred
Please enter a number

3.8  Short circuit evaluation of logical expressions

>>> x = 6 
>>> y = 2
>>> x >= 2 and (x/y) > 2
True
>>> x = 1 
>>> y = 0
>>> x >= 2 and (x/y) > 2
False
>>> x = 6
>>> y = 0
>>> x >= 2 and (x/y) > 2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: integer division or modulo by zero
>>> 

>>> x = 1
>>> y = 0
>>> x >= 2 and y != 0 and (x/y) > 2
False
>>> x = 6 
>>> y = 0
>>> x >= 2 and y != 0 and (x/y) > 2
False
>>> x >= 2 and (x/y) > 2 and y != 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: integer division or modulo by zero
>>>

3.9  Debugging

• What kind of error it was, and
  
  
• Where it occurred.

>>> x = 5
>>>  y = 6
  File "<stdin>", line 1
    y = 6
    ^
SyntaxError: invalid syntax

import math
signal_power = 9
noise_power = 10
ratio = signal_power / noise_power
decibels = 10 * math.log10(ratio)
print decibels

Traceback (most recent call last):
  File "snr.py", line 5, in ?
    decibels = 10 * math.log10(ratio)
OverflowError: math range error

3.10  Glossary

body:

The sequence of statements within a compound statement.

boolean expression:

An expression whose value is either True or False.

branch:

One of the alternative sequences of statements in a conditional statement.

chained conditional:

A conditional statement with a series of alternative branches.

comparison operator:

One of the operators that compares its operands: ==, !=, >, <, >=, and <=.

conditional statement:

A statement that controls the flow of execution depending on some condition.

condition:

The boolean expression in a conditional statement that determines which branch is executed.

compound statement:

A statement that consists of a header and a body. The header ends with a colon (:). The body is indented relative to the header.

guardian pattern:

Where we construct a logical expression with additional comparisons to take advantage of the short circuit behavior.

logical operator:

One of the operators that combines boolean expressions: and, or, and not.

nested conditional:

A conditional statement that appears in one of the branches of another conditional statement.

traceback:

A list of the functions that are executing, printed when an exception occurs.

short circuit:

When Python is part-way through evaluating a logical expression and stops the evaluation because Python knows the final value for the expression without needing to evaluate the rest of the expression.

3.11  Exercises

Enter Hours: 45
Enter Rate: 10
Pay: 475.0

Enter Hours: 20
Enter Rate: nine
Error, please enter numeric input

Enter Hours: forty
Error, please enter numeric input

Score   Grade
>= 0.9     A
>= 0.8     B
>= 0.7     C
>= 0.6     D
< 0.6    F

Enter score: 0.95
A

Enter score: perfect
Bad score

Enter score: 10.0
Bad score

Enter score: 0.75
C

Enter score: 0.5
F

1

In Python 3.0, you no longer get an error message; the division operator performs floating-point division even with integer operands.

Chapter 4  Functions

4.1  Function calls

>>> type(32)
<type 'int'>

4.2  Built-in functions

>>> max('Hello world')
'w'
>>> min('Hello world')
' '
>>>

>>> len('Hello world')
11
>>>

4.3  Type conversion functions

>>> int('32')
32
>>> int('Hello')
ValueError: invalid literal for int(): Hello

>>> int(3.99999)
3
>>> int(-2.3)
-2

>>> float(32)
32.0
>>> float('3.14159')
3.14159

>>> str(32)
'32'
>>> str(3.14159)
'3.14159'

4.4  Random numbers

import random

for i in range(10):
    x = random.random()
    print x

0.301927091705
0.513787075867
0.319470430881
0.285145917252
0.839069045123
0.322027080731
0.550722110248
0.366591677812
0.396981483964
0.838116437404

>>> random.randint(5, 10)
5
>>> random.randint(5, 10)
9

>>> t = [1, 2, 3]
>>> random.choice(t)
2
>>> random.choice(t)
3

4.5  Math functions

>>> import math

>>> print math
<module 'math' from '/usr/lib/python2.5/lib-dynload/math.so'>

>>> ratio = signal_power / noise_power
>>> decibels = 10 * math.log10(ratio)

>>> radians = 0.7
>>> height = math.sin(radians)

>>> degrees = 45
>>> radians = degrees / 360.0 * 2 * math.pi
>>> math.sin(radians)
0.707106781187

>>> math.sqrt(2) / 2.0
0.707106781187

4.6  Adding new functions

def print_lyrics():
    print "I'm a lumberjack, and I'm okay."
    print 'I sleep all night and I work all day.'

>>> def print_lyrics():
...     print "I'm a lumberjack, and I'm okay."
...     print 'I sleep all night and I work all day.'
...

>>> print print_lyrics
<function print_lyrics at 0xb7e99e9c>
>>> print type(print_lyrics)
<type 'function'>

>>> print_lyrics()
I'm a lumberjack, and I'm okay.
I sleep all night and I work all day.

def repeat_lyrics():
    print_lyrics()
    print_lyrics()

>>> repeat_lyrics()
I'm a lumberjack, and I'm okay.
I sleep all night and I work all day.
I'm a lumberjack, and I'm okay.
I sleep all night and I work all day.

4.7  Definitions and uses

def print_lyrics():
    print "I'm a lumberjack, and I'm okay."
    print 'I sleep all night and I work all day.'

def repeat_lyrics():
    print_lyrics()
    print_lyrics()

repeat_lyrics()

4.8  Flow of execution

4.9  Parameters and arguments

def print_twice(bruce):
    print bruce
    print bruce

>>> print_twice('Spam')
Spam
Spam
>>> print_twice(17)
17
17
>>> print_twice(math.pi)
3.14159265359
3.14159265359

>>> print_twice('Spam '*4)
Spam Spam Spam Spam
Spam Spam Spam Spam
>>> print_twice(math.cos(math.pi))
-1.0
-1.0

>>> michael = 'Eric, the half a bee.'
>>> print_twice(michael)
Eric, the half a bee.
Eric, the half a bee.

4.10  Fruitful functions and void functions

x = math.cos(radians)
golden = (math.sqrt(5) + 1) / 2

>>> math.sqrt(5)
2.2360679774997898

math.sqrt(5)

>>> result = print_twice('Bing')
Bing
Bing
>>> print result
None

>>> print type(None)
<type 'NoneType'>

def addtwo(a, b):
    added = a + b
    return added

x = addtwo(3, 5)
print x

4.11  Why functions?

• Creating a new function gives you an opportunity to name a group of statements, which makes your program easier to read, understand and debug.
  
  
• Functions can make a program smaller by eliminating repetitive code. Later, if you make a change, you only have to make it in one place.
  
  
• Dividing a long program into functions allows you to debug the parts one at a time and then assemble them into a working whole.
  
  
• Well-designed functions are often useful for many programs. Once you write and debug one, you can reuse it.

4.12  Debugging

4.13  Glossary

algorithm:

A general process for solving a category of problems.

argument:

A value provided to a function when the function is called. This value is assigned to the corresponding parameter in the function.

body:

The sequence of statements inside a function definition.

composition:

Using an expression as part of a larger expression, or a statement as part of a larger statement.

deterministic:

Pertaining to a program that does the same thing each time it runs, given the same inputs.

dot notation:

The syntax for calling a function in another module by specifying the module name followed by a dot (period) and the function name.

flow of execution:

The order in which statements are executed during a program run.

fruitful function:

A function that returns a value.

function:

A named sequence of statements that performs some useful operation. Functions may or may not take arguments and may or may not produce a result.

function call:

A statement that executes a function. It consists of the function name followed by an argument list.

function definition:

A statement that creates a new function, specifying its name, parameters, and the statements it executes.

function object:

A value created by a function definition. The name of the function is a variable that refers to a function object.

header:

The first line of a function definition.

import statement:

A statement that reads a module file and creates a module object.

module object:

A value created by an import statement that provides access to the data and code defined in a module.

parameter:

A name used inside a function to refer to the value passed as an argument.

pseudorandom:

Pertaining to a sequence of numbers that appear to be random, but are generated by a deterministic program.

return value:

The result of a function. If a function call is used as an expression, the return value is the value of the expression.

void function:

A function that doesn't return a value.

4.14  Exercises

def fred():
   print "Zap"

def jane():
   print "ABC"

jane()
fred()
jane()

Enter Hours: 45
Enter Rate: 10
Pay: 475.0

Score   Grade
> 0.9     A
> 0.8     B
> 0.7     C
> 0.6     D
<= 0.6    F

Program Execution:

Enter score: 0.95
A

Enter score: perfect
Bad score

Enter score: 10.0
Bad score

Enter score: 0.75
C

Enter score: 0.5
F

Chapter 5  Iteration

5.1  Updating variables

x = x+1

>>> x = x+1
NameError: name 'x' is not defined

>>> x = 0
>>> x = x+1

5.2  The while statement

n = 5
while n > 0:
    print n
    n = n-1
print 'Blastoff!'

1. Evaluate the condition, yielding True or False.
   
   
2. If the condition is false, exit the while statement and continue execution at the next statement.
   
   
3. If the condition is true, execute the body and then go back to step 1.

5.3  Infinite loops

5.4  ``Infinite loops'' and break

n = 10
while True:
    print n, 
    n = n - 1
print 'Done!'

while True:
    line = raw_input('> ')
    if line == 'done':
        break
    print line
print 'Done!'

> hello there
hello there
> finished
finished
> done
Done!

5.5  Finishing iterations with continue

while True:
    line = raw_input('> ')
    if line[0] == '#' :
        continue
    if line == 'done':
        break
    print line
print 'Done!'

> hello there
hello there
> # don't print this
> print this!
print this!
> done
Done!

5.6  Definite loops using for

friends = ['Joseph', 'Glenn', 'Sally']
for friend in friends:
    print 'Happy New Year:', friend
print 'Done!'

Happy New Year: Joseph
Happy New Year: Glenn
Happy New Year: Sally
Done!

5.7  Loop patterns

• Initializing one or more variables before the loop starts.
  
  
• Performing some computation on each item in the loop body, possibly changing the variables in the body of the loop.
  
  
• Looking at the resulting variables when the loop completes.

5.7.1  Counting and summing loops

count = 0
for itervar in [3, 41, 12, 9, 74, 15]:
    count = count + 1
print 'Count: ', count

total = 0
for itervar in [3, 41, 12, 9, 74, 15]:
    total = total + itervar
print 'Total: ', total

5.7.2  Maximum and minimum loops

largest = None
print 'Before:', largest
for itervar in [3, 41, 12, 9, 74, 15]:
    if largest is None or itervar > largest :
        largest = itervar
    print 'Loop:', itervar, largest
print 'Largest:', largest

Before: None
Loop: 3 3
Loop: 41 41
Loop: 12 41
Loop: 9 41
Loop: 74 74
Loop: 15 74
Largest: 74

smallest = None
print 'Before:', smallest
for itervar in [3, 41, 12, 9, 74, 15]:
    if smallest is None or itervar < smallest:
        smallest = itervar
    print 'Loop:', itervar, smallest
print 'Smallest:', smallest

def min(values):
    smallest = None
    for value in values:
        if smallest is None or value < smallest:
            smallest = value
    return smallest

5.8  Debugging

5.9  Glossary

accumulator:

A variable used in a loop to add up or accumulate a result.

counter:

A variable used in a loop to count the number of times something happened. We initialize a counter to zero and then increment the counter each time we want to ``count'' something.

decrement:

An update that decreases the value of a variable.

initialize:

An assignment that gives an initial value to a variable that will be updated.

increment:

An update that increases the value of a variable (often by one).

infinite loop:

A loop in which the terminating condition is never satisfied or for which there is no termination condition.

iteration:

Repeated execution of a set of statements using either a recursive function call or a loop.

5.10  Exercises

Enter a number: 4
Enter a number: 5
Enter a number: bad data
Invalid input
Enter a number: 7
Enter a number: done
16 3 5.33333333333

1

We will examine lists in more detail in a later chapter

Chapter 6  Strings

6.1  A string is a sequence

>>> fruit = 'banana'
>>> letter = fruit[1]

>>> print letter
a

>>> letter = fruit[0]
>>> print letter
b

>>> letter = fruit[1.5]
TypeError: string indices must be integers

6.2  Getting the length of a string using len

>>> fruit = 'banana'
>>> len(fruit)
6

>>> length = len(fruit)
>>> last = fruit[length]
IndexError: string index out of range

>>> last = fruit[length-1]
>>> print last
a

6.3  Traversal through a string with a loop

index = 0
while index < len(fruit):
    letter = fruit[index]
    print letter
    index = index + 1

for char in fruit:
    print char

6.4  String slices

>>> s = 'Monty Python'
>>> print s[0:5]
Monty
>>> print s[6:13]
Python

>>> fruit = 'banana'
>>> fruit[:3]
'ban'
>>> fruit[3:]
'ana'

>>> fruit = 'banana'
>>> fruit[3:3]
''

6.5  Strings are immutable

>>> greeting = 'Hello, world!'
>>> greeting[0] = 'J'
TypeError: object does not support item assignment

>>> greeting = 'Hello, world!'
>>> new_greeting = 'J' + greeting[1:]
>>> print new_greeting
Jello, world!

6.6  Looping and counting

word = 'banana'
count = 0
for letter in word:
    if letter == 'a':
        count = count + 1
print count

6.7  The in operator

>>> 'a' in 'banana'
True
>>> 'seed' in 'banana'
False

6.8  String comparison

if word == 'banana':
    print  'All right, bananas.'

if word < 'banana':
    print 'Your word,' + word + ', comes before banana.'
elif word > 'banana':
    print 'Your word,' + word + ', comes after banana.'
else:
    print 'All right, bananas.'

Your word, Pineapple, comes before banana.

6.9  string methods

>>> stuff = 'Hello world'
>>> type(stuff)
<type 'str'>
>>> dir(stuff)
['capitalize', 'center', 'count', 'decode', 'encode', 
'endswith', 'expandtabs', 'find', 'format', 'index', 
'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 
'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 
'partition', 'replace', 'rfind', 'rindex', 'rjust', 
'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 
'startswith', 'strip', 'swapcase', 'title', 'translate', 
'upper', 'zfill']
>>> help(str.capitalize)
Help on method_descriptor:

capitalize(...)
    S.capitalize() -> string
    
    Return a copy of the string S with only its first character
    capitalized.
>>>

>>> word = 'banana'
>>> new_word = word.upper()
>>> print new_word
BANANA

>>> word = 'banana'
>>> index = word.find('a')
>>> print index
1

>>> word.find('na')
2

>>> word.find('na', 3)
4

>>> line = '  Here we go  '
>>> line.strip()
'Here we go'

>>> line = 'Please have a nice day'
>>> line.startswith('Please')
True
>>> line.startswith('p')
False

>>> line = 'Please have a nice day'
>>> line.startswith('p')
False
>>> line.lower()
'please have a nice day'
>>> line.lower().startswith('p')
True

6.10  Parsing strings

From stephen.marquard@ uct.ac.za Sat Jan  5 09:14:16 2008

>>> data = 'From stephen.marquard@uct.ac.za Sat Jan  5 09:14:16 2008'
>>> atpos = data.find('@')
>>> print atpos
21
>>> sppos = data.find(' ',atpos)
>>> print sppos
31
>>> host = data[atpos+1:sppos]
>>> print host
uct.ac.za
>>> 

6.11  Format operator

>>> camels = 42
>>> '%d' % camels
'42'

>>> camels = 42
>>> 'I have spotted %d camels.' % camels
'I have spotted 42 camels.'

>>> 'In %d years I have spotted %g %s.' % (3, 0.1, 'camels')
'In 3 years I have spotted 0.1 camels.'

>>> '%d %d %d' % (1, 2)
TypeError: not enough arguments for format string
>>> '%d' % 'dollars'
TypeError: illegal argument type for built-in operation

6.12  Debugging

while True:
    line = raw_input('> ')
    if line[0] == '#' :
        continue
    if line == 'done':
        break
    print line

print 'Done!'

> hello there
hello there
> # don't print this
> print this!
print this!
> 
Traceback (most recent call last):
  File "copytildone.py", line 3, in <module>
    if line[0] == '#' :

    if line.startswith('#') :

    if len(line) > 0 and line[0] == '#' :

6.13  Glossary

counter:

A variable used to count something, usually initialized to zero and then incremented.

empty string:

A string with no characters and length 0, represented by two quotation marks.

format operator:

An operator, %, that takes a format string and a tuple and generates a string that includes the elements of the tuple formatted as specified by the format string.

format sequence:

A sequence of characters in a format string, like %d, that specifies how a value should be formatted.

format string:

A string, used with the format operator, that contains format sequences.

flag:

A boolean variable used to indicate whether a condition is true.

invocation:

A statement that calls a method.

immutable:

The property of a sequence whose items cannot be assigned.

index:

An integer value used to select an item in a sequence, such as a character in a string.

item:

One of the values in a sequence.

method:

A function that is associated with an object and called using dot notation.

object:

Something a variable can refer to. For now, you can use ``object'' and ``value'' interchangeably.

search:

A pattern of traversal that stops when it finds what it is looking for.

sequence:

An ordered set; that is, a set of values where each value is identified by an integer index.

slice:

A part of a string specified by a range of indices.

traverse:

To iterate through the items in a sequence, performing a similar operation on each.

6.14  Exercises

str = 'X-DSPAM-Confidence:  0.8475'

Chapter 7  Files

7.1  Persistence

7.2  Opening files

>>> fhand = open('mbox.txt')
>>> print fhand
<open file 'mbox.txt', mode 'r' at 0x1005088b0>

>>> fhand = open('stuff.txt')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory: 'stuff.txt'

7.3  Text files and lines

From stephen.marquard@uct.ac.za Sat Jan  5 09:14:16 2008
Return-Path: <postmaster@collab.sakaiproject.org>
Date: Sat, 5 Jan 2008 09:12:18 -0500
To: source@collab.sakaiproject.org
From: stephen.marquard@uct.ac.za
Subject: [sakai] svn commit: r39772 - content/branches/
Details: http://source.sakaiproject.org/viewsvn/?view=rev&rev=39772
...

>>> stuff = 'Hello\nWorld!'
>>> stuff
'Hello\nWorld!'
>>> print stuff
Hello
World!
>>> stuff = 'X\nY'
>>> print stuff
X
Y
>>> len(stuff)
3

7.4  Reading files

fhand = open('mbox.txt')
count = 0
for line in fhand:
    count = count + 1
print 'Line Count:', count

python open.py 
Line Count: 132045

>>> fhand = open('mbox-short.txt')
>>> inp = fhand.read()
>>> print len(inp)
94626
>>> print inp[:20]
From stephen.marquar

7.5  Searching through a file

fhand = open('mbox-short.txt')
for line in fhand:
    if line.startswith('From:') :
        print line

From: stephen.marquard@uct.ac.za

From: louis@media.berkeley.edu

From: zqian@umich.edu

From: rjlowe@iupui.edu
...

fhand = open('mbox-short.txt')
for line in fhand:
    line = line.rstrip()
    if line.startswith('From:') :
        print line

From: stephen.marquard@uct.ac.za
From: louis@media.berkeley.edu
From: zqian@umich.edu
From: rjlowe@iupui.edu
From: zqian@umich.edu
From: rjlowe@iupui.edu
From: cwen@iupui.edu
...

fhand = open('mbox-short.txt')
for line in fhand:
    line = line.rstrip()
    # Skip 'uninteresting lines'
    if not line.startswith('From:') :
        continue
    # Process our 'interesting' line
    print line

fhand = open('mbox-short.txt')
for line in fhand:
    line = line.rstrip()
    if line.find('@uct.ac.za') == -1 : 
        continue
    print line

From stephen.marquard@uct.ac.za Sat Jan  5 09:14:16 2008
X-Authentication-Warning: set sender to stephen.marquard@uct.ac.za using -f
From: stephen.marquard@uct.ac.za
Author: stephen.marquard@uct.ac.za
From david.horwitz@uct.ac.za Fri Jan  4 07:02:32 2008
X-Authentication-Warning: set sender to david.horwitz@uct.ac.za using -f
From: david.horwitz@uct.ac.za
Author: david.horwitz@uct.ac.za
...

7.6  Letting the user choose the file name

fname = raw_input('Enter the file name: ')
fhand = open(fname)
count = 0
for line in fhand:
    if line.startswith('Subject:') :
        count = count + 1
print 'There were', count, 'subject lines in', fname

python search6.py 
Enter the file name: mbox.txt
There were 1797 subject lines in mbox.txt

python search6.py 
Enter the file name: mbox-short.txt
There were 27 subject lines in mbox-short.txt

7.7  Using try, except, and open

python search6.py 
Enter the file name: missing.txt
Traceback (most recent call last):
  File "search6.py", line 2, in <module>
    fhand = open(fname)
IOError: [Errno 2] No such file or directory: 'missing.txt'

python search6.py 
Enter the file name: na na boo boo
Traceback (most recent call last):
  File "search6.py", line 2, in <module>
    fhand = open(fname)
IOError: [Errno 2] No such file or directory: 'na na boo boo'

fname = raw_input('Enter the file name: ')
try:
    fhand = open(fname)
except:
    print 'File cannot be opened:', fname
    exit()

count = 0
for line in fhand:
    if line.startswith('Subject:') : 
        count = count + 1
print 'There were', count, 'subject lines in', fname

python search7.py
Enter the file name: mbox.txt
There were 1797 subject lines in mbox.txt

python search7.py
Enter the file name: na na boo boo
File cannot be opened: na na boo boo

7.8  Writing files

>>> fout = open('output.txt', 'w')
>>> print fout
<open file 'output.txt', mode 'w' at 0xb7eb2410>

>>> line1 = 'This here's the wattle,\n'
>>> fout.write(line1)

>>> line2 = 'the emblem of our land.\n'
>>> fout.write(line2)

>>> fout.close()

7.9  Debugging

>>> s = '1 2\t 3\n 4'
>>> print s
1 2  3
 4

>>> print repr(s)
'1 2\t 3\n 4'

7.10  Glossary

catch:

To prevent an exception from terminating a program using the try and except statements.

newline:

A special character used in files and strings to indicate the end of a line.

Pythonic:

A technique that works elegantly in Python. ``Using try and except is the Pythonic way to recover from missing files.''.

Quality Assurance:

A person or team focused on insuring the overall quality of a software product. QA is often involved in testing a product and identifying problems before the product is released.

text file:

A sequence of characters stored in permanent storage like a hard drive.

7.11  Exercises

python shout.py
Enter a file name: mbox-short.txt
FROM STEPHEN.MARQUARD@UCT.AC.ZA SAT JAN  5 09:14:16 2008
RETURN-PATH: <POSTMASTER@COLLAB.SAKAIPROJECT.ORG>
RECEIVED: FROM MURDER (MAIL.UMICH.EDU [141.211.14.90])
  BY FRANKENSTEIN.MAIL.UMICH.EDU (CYRUS V2.3.8) WITH LMTPA;
  SAT, 05 JAN 2008 09:14:16 -0500

X-DSPAM-Confidence:  0.8475

Enter the file name: mbox.txt
Average spam confidence: 0.894128046745

Enter the file name: mbox-short.txt
Average spam confidence: 0.750718518519

python egg.py 
Enter the file name: mbox.txt
There were 1797 subject lines in mbox.txt

python egg.py 
Enter the file name: missing.tyxt
File cannot be opened: missing.tyxt

python egg.py 
Enter the file name: na na boo boo
NA NA BOO BOO TO YOU - You have been punk'd!

Chapter 8  Lists

8.1  A list is a sequence

[10, 20, 30, 40]
['crunchy frog', 'ram bladder', 'lark vomit']

['spam', 2.0, 5, [10, 20]]

>>> cheeses = ['Cheddar', 'Edam', 'Gouda']
>>> numbers = [17, 123]
>>> empty = []
>>> print cheeses, numbers, empty
['Cheddar', 'Edam', 'Gouda'] [17, 123] []

8.2  Lists are mutable

>>> print cheeses[0]
Cheddar

>>> numbers = [17, 123]
>>> numbers[1] = 5
>>> print numbers
[17, 5]

• Any integer expression can be used as an index.
  
  
• If you try to read or write an element that does not exist, you get an IndexError.
  
  
  
  
• If an index has a negative value, it counts backward from the end of the list.

>>> cheeses = ['Cheddar', 'Edam', 'Gouda']
>>> 'Edam' in cheeses
True
>>> 'Brie' in cheeses
False

8.3  Traversing a list

for cheese in cheeses:
    print cheese

for i in range(len(numbers)):
    numbers[i] = numbers[i] * 2

for x in empty:
    print 'This never happens.'

['spam', 1, ['Brie', 'Roquefort', 'Pol le Veq'], [1, 2, 3]]

8.4  List operations

>>> a = [1, 2, 3]
>>> b = [4, 5, 6]
>>> c = a + b
>>> print c
[1, 2, 3, 4, 5, 6]

>>> [0] * 4
[0, 0, 0, 0]
>>> [1, 2, 3] * 3
[1, 2, 3, 1, 2, 3, 1, 2, 3]

8.5  List slices

>>> t = ['a', 'b', 'c', 'd', 'e', 'f']
>>> t[1:3]
['b', 'c']
>>> t[:4]
['a', 'b', 'c', 'd']
>>> t[3:]
['d', 'e', 'f']

>>> t[:]
['a', 'b', 'c', 'd', 'e', 'f']

>>> t = ['a', 'b', 'c', 'd', 'e', 'f']
>>> t[1:3] = ['x', 'y']
>>> print t
['a', 'x', 'y', 'd', 'e', 'f']

8.6  List methods

>>> t = ['a', 'b', 'c']
>>> t.append('d')
>>> print t
['a', 'b', 'c', 'd']

>>> t1 = ['a', 'b', 'c']
>>> t2 = ['d', 'e']
>>> t1.extend(t2)
>>> print t1
['a', 'b', 'c', 'd', 'e']

>>> t = ['d', 'c', 'e', 'b', 'a']
>>> t.sort()
>>> print t
['a', 'b', 'c', 'd', 'e']

8.7  Deleting elements

>>> t = ['a', 'b', 'c']
>>> x = t.pop(1)
>>> print t
['a', 'c']
>>> print x
b

>>> t = ['a', 'b', 'c']
>>> del t[1]
>>> print t
['a', 'c']

>>> t = ['a', 'b', 'c']
>>> t.remove('b')
>>> print t
['a', 'c']

>>> t = ['a', 'b', 'c', 'd', 'e', 'f']
>>> del t[1:5]
>>> print t
['a', 'f']

8.8  Lists and functions

>>> nums = [3, 41, 12, 9, 74, 15]
>>> print len(nums)
6
>>> print max(nums)
74
>>> print min(nums)
3
>>> print sum(nums)
154
>>> print sum(nums)/len(nums)
25

total = 0
count = 0
while ( True ) :
    inp = raw_input('Enter a number: ')
    if inp == 'done' : break
    value = float(inp)
    total = total + value
    count = count + 1

average = total / count
print 'Average:', average

numlist = list()
while ( True ) :
    inp = raw_input('Enter a number: ')
    if inp == 'done' : break
    value = float(inp)
    numlist.append(value)

average = sum(numlist) / len(numlist)
print 'Average:', average

8.9  Lists and strings

>>> s = 'spam'
>>> t = list(s)
>>> print t
['s', 'p', 'a', 'm']

>>> s = 'pining for the fjords'
>>> t = s.split()
>>> print t
['pining', 'for', 'the', 'fjords']
>>> print t[2]
the

>>> s = 'spam-spam-spam'
>>> delimiter = '-'
>>> s.split(delimiter)
['spam', 'spam', 'spam']

>>> t = ['pining', 'for', 'the', 'fjords']
>>> delimiter = ' '
>>> delimiter.join(t)
'pining for the fjords'

8.10  Parsing lines

From stephen.marquard@uct.ac.za  Sat Jan  5 09:14:16 2008

fhand = open('mbox-short.txt')
for line in fhand:
    line = line.rstrip()
    if not line.startswith('From ') : continue
    words = line.split()
    print words[2]

Sat
Fri
Fri
Fri
    ...

8.11  Objects and values

a = 'banana'
b = 'banana'

>>> a = 'banana'
>>> b = 'banana'
>>> a is b
True

>>> a = [1, 2, 3]
>>> b = [1, 2, 3]
>>> a is b
False

8.12  Aliasing

>>> a = [1, 2, 3]
>>> b = a
>>> b is a
True

>>> b[0] = 17
>>> print a
[17, 2, 3]

a = 'banana'
b = 'banana'

8.13  List arguments

def delete_head(t):
    del t[0]

>>> letters = ['a', 'b', 'c']
>>> delete_head(letters)
>>> print letters
['b', 'c']

>>> t1 = [1, 2]
>>> t2 = t1.append(3)
>>> print t1
[1, 2, 3]
>>> print t2
None

>>> t3 = t1 + [3]
>>> print t3
[1, 2, 3]
>>> t2 is t3
False

def bad_delete_head(t):
    t = t[1:]              # WRONG!

def tail(t):
    return t[1:]

>>> letters = ['a', 'b', 'c']
>>> rest = tail(letters)
>>> print rest
['b', 'c']

8.14  Debugging

1. Don't forget that most list methods modify the argument and return None. This is the opposite of the string methods, which return a new string and leave the original alone.
   
   If you are used to writing string code like this:

   
   word = word.strip()
   

   It is tempting to write list code like this:

   
   t = t.sort()           # WRONG!
   

   Because sort returns None, the next operation you perform with t is likely to fail.
   
   Before using list methods and operators, you should read the documentation carefully and then test them in interactive mode. The methods and operators that lists share with other sequences (like strings) are documented at docs.python.org/lib/typesseq.html. The methods and operators that only apply to mutable sequences are documented at docs.python.org/lib/typesseq-mutable.html.
   
   
2. Pick an idiom and stick with it.
   
   Part of the problem with lists is that there are too many ways to do things. For example, to remove an element from a list, you can use pop, remove, del, or even a slice assignment.
   
   To add an element, you can use the append method or the + operator. But don't forget that these are right:

   
   t.append(x)
   t = t + [x]
   

   And these are wrong:

   
   t.append([x])          # WRONG!
   t = t.append(x)        # WRONG!
   t + [x]                # WRONG!
   t = t + x              # WRONG!
   

   Try out each of these examples in interactive mode to make sure you understand what they do. Notice that only the last one causes a runtime error; the other three are legal, but they do the wrong thing.
   
   
3. Make copies to avoid aliasing.
   
   
   
   If you want to use a method like sort that modifies the argument, but you need to keep the original list as well, you can make a copy.

   
   orig = t[:]
   t.sort()
   

   In this example you could also use the built-in function sorted, which returns a new, sorted list and leaves the original alone. But in that case you should avoid using sorted as a variable name!
   
   
4. Lists, split, and files
   
   When we read and parse files, there are many opportunities to encounter input that can crash our program so it is a good idea to revisit the guardian pattern when it comes writing programs that read through a file and look for a ``needle in the haystack''.
   
   Let's revisit our program that is looking for the day of the week on the from lines of our file:

   
   From stephen.marquard@uct.ac.za  Sat Jan  5 09:14:16 2008
   

   Since we are breaking this line into words, we could dispense with the use of startswith and simply look at the first word of the line to determine if we are interested in the line at all. We can use continue to skip lines that don't have ``From'' as the first word as follows:

   
   fhand = open('mbox-short.txt')
   for line in fhand:
       words = line.split()
       if words[0] != 'From' : continue
       print words[2]
   

   This looks much simpler and we don't even need to do the rstrip to remove the newline at the end of the file. But is it better?

   
   python search8.py 
   Sat
   Traceback (most recent call last):
     File "search8.py", line 5, in <module>
       if words[0] != 'From' : continue
   IndexError: list index out of range
   

   It kind of works and we see the day from the first line (Sat) but then the program fails with a traceback error. What went wrong? What messed-up data caused our elegant, clever and very Pythonic program to fail?
   
   You could stare at it for a long time and puzzle through it or ask someone for help, but the quicker and smarter approach is to add a print statement. The best place to add the print statement is right before the line where the program failed and print out the data that seems to be causing the failure.
   
   Now this approach may generate a lot of lines of output but at least you will immediately have some clue as to the problem at hand. So we add a print of the variable words right before line five. We even add a prefix ``Debug:'' to the line so we can keep our regular output separate from our debug output.

   
   for line in fhand:
       words = line.split()
       print 'Debug:', words
       if words[0] != 'From' : continue
       print words[2]
   

   When we run the program, a lot of output scrolls off the screen but at the end, we see our debug output and the traceback so we know what happened just before the traceback.

   
   Debug: ['X-DSPAM-Confidence:', '0.8475']
   Debug: ['X-DSPAM-Probability:', '0.0000']
   Debug: []
   Traceback (most recent call last):
     File "search9.py", line 6, in <module>
       if words[0] != 'From' : continue
   IndexError: list index out of range
   

   Each debug line is printing the list of words which we get when we split the line into words. When the program fails the list of words is empty []. If we open the file in a text editor and look at the file, at that point it looks as follows:

   
   X-DSPAM-Result: Innocent
   X-DSPAM-Processed: Sat Jan  5 09:14:16 2008
   X-DSPAM-Confidence: 0.8475
   X-DSPAM-Probability: 0.0000
   
   Details: http://source.sakaiproject.org/viewsvn/?view=rev&rev=39772
   

   The error occurs when our program encounters a blank line! Of course there are ``zero words'' on a blank line. Why didn't we think of that when we were writing the code. When the code looks for the first word (word[0]) to check to see if it matches ``From'', we get an ``index out of range'' error.
   
   This of course is the perfect place to add some guardian code to avoid checking the first word if the first word is not there. There are many ways to protect this code, we will choose to check the number of words we have before we look at the first word:

   
   fhand = open('mbox-short.txt')
   count = 0
   for line in fhand:
       words = line.split()
       # print 'Debug:', words
       if len(words) == 0 : continue
       if words[0] != 'From' : continue
       print words[2]
   

   First we commented out the debug print statement instead of removing it in case our modification fails and we need to debug again. Then we added a guardian statement that checks to see if we have zero words, and if so, we use continue to skip to the next line in the file.
   
   We can think of the two continue statements as helping us refine the set of lines which are ``interesting'' to us and which we want to process some more. A line which has no words is ``uninteresting'' to us so we skip to the next line. A line which does not have ``From'' as its first word is uninteresting to us so we skip it.
   
   The program as modified runs successfully so perhaps it is correct. Our guardian statement does make sure that the words[0] will never fail, but perhaps it is not enough. When we are programming, we must always be thinking, ``What might go wrong?''.
   
   Exercise 2   Figure out which line of the above program is still not properly guarded. See if you can construct a text file which causes the program to fail and then modify the program so that the line is properly guarded and test it to make sure it handles your new text file.
   
   
   Exercise 3   Rewrite the guardian code in the above example without two if statements. Instead use a compound logical expression using the and logical operator with a single if statement.

8.15  Glossary

aliasing:

A circumstance where two or more variables refer to the same object.

delimiter:

A character or string used to indicate where a string should be split.

element:

One of the values in a list (or other sequence), also called items.

equivalent:

Having the same value.

index:

An integer value that indicates an element in a list.

identical:

Being the same object (which implies equivalence).

list:

A sequence of values.

list traversal:

The sequential accessing of each element in a list.

nested list:

A list that is an element of another list.

object:

Something a variable can refer to. An object has a type and a value.

reference:

The association between a variable and its value.

8.16  Exercises

Enter file: romeo.txt
['Arise', 'But', 'It', 'Juliet', 'Who', 'already', 
'and', 'breaks', 'east', 'envious', 'fair', 'grief', 
'is', 'kill', 'light', 'moon', 'pale', 'sick', 'soft', 
'sun', 'the', 'through', 'what', 'window', 
'with', 'yonder']

python fromcount.py 
Enter a file name: mbox-short.txt
stephen.marquard@uct.ac.za
louis@media.berkeley.edu
zqian@umich.edu

[...some output removed...]

ray@media.berkeley.edu
cwen@iupui.edu
cwen@iupui.edu
cwen@iupui.edu
There were 27 lines in the file with From as the first word

Enter a number: 6
Enter a number: 2
Enter a number: 9
Enter a number: 3
Enter a number: 5
Enter a number: done
Maximum: 9.0
Minimum: 2.0

Chapter 9  Dictionaries

>>> eng2sp = dict()
>>> print eng2sp
{}

>>> eng2sp['one'] = 'uno'

>>> print eng2sp
{'one': 'uno'}

>>> eng2sp = {'one': 'uno', 'two': 'dos', 'three': 'tres'}

>>> print eng2sp
{'one': 'uno', 'three': 'tres', 'two': 'dos'}

>>> print eng2sp['two']
'dos'

>>> print eng2sp['four']
KeyError: 'four'

>>> len(eng2sp)
3

>>> 'one' in eng2sp
True
>>> 'uno' in eng2sp
False

>>> vals = eng2sp.values()
>>> 'uno' in vals
True

9.1  Dictionary as a set of counters

1. You could create 26 variables, one for each letter of the alphabet. Then you could traverse the string and, for each character, increment the corresponding counter, probably using a chained conditional.
   
   
2. You could create a list with 26 elements. Then you could convert each character to a number (using the built-in function ord), use the number as an index into the list, and increment the appropriate counter.
   
   
3. You could create a dictionary with characters as keys and counters as the corresponding values. The first time you see a character, you would add an item to the dictionary. After that you would increment the value of an existing item.

def histogram(s):
    d = dict()
    for c in s:
        if c not in d:
            d[c] = 1
        else:
            d[c] = d[c] + 1
    return d

>>> h = histogram('brontosaurus')
>>> print h
{'a': 1, 'b': 1, 'o': 2, 'n': 1, 's': 2, 'r': 2, 'u': 2, 't': 1}

>>> h = histogram('a')
>>> print h
{'a': 1}
>>> h.get('a', 0)
1
>>> h.get('b', 0)
0

9.2  Dictionaries and files

But soft what light through yonder window breaks
It is the east and Juliet is the sun
Arise fair sun and kill the envious moon
Who is already sick and pale with grief

fname = raw_input('Enter the file name: ')
try:
    fhand = open(fname)
except:
    print 'File cannot be opened:', fname
    exit()

counts = dict()
for line in fhand:
    words = line.split()
    for word in words:
        if word not in counts:
            counts[word] = 1
        else:
            counts[word] += 1

print counts

python count1.py 
Enter the file name: romeo.txt
{'and': 3, 'envious': 1, 'already': 1, 'fair': 1, 
'is': 3, 'through': 1, 'pale': 1, 'yonder': 1, 
'what': 1, 'sun': 2, 'Who': 1, 'But': 1, 'moon': 1, 
'window': 1, 'sick': 1, 'east': 1, 'breaks': 1, 
'grief': 1, 'with': 1, 'light': 1, 'It': 1, 'Arise': 1, 
'kill': 1, 'the': 3, 'soft': 1, 'Juliet': 1}

9.3  Looping and dictionaries

def print_hist(h):
    for c in h:
        print c, h[c]

>>> h = histogram('parrot')
>>> print_hist(h)
a 1
p 1
r 2
t 1
o 1

def print_sorted_hist(h):
    lst = h.keys()
    lst.sort()
    for c in lst:
        print c, h[c]

>>> h = histogram('parrot')
>>> print_sorted_hist(h)
a 1
o 1
p 1
r 2
t 1

9.4  Advanced text parsing

But, soft! what light through yonder window breaks?
It is the east, and Juliet is the sun.
Arise, fair sun, and kill the envious moon,
Who is already sick and pale with grief,

>>> import string
>>> string.punctuation
'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

import string                                          # New Code

fname = raw_input('Enter the file name: ')
try:
    fhand = open(fname)
except:
    print 'File cannot be opened:', fname
    exit()

counts = dict()
for line in fhand:
    line = line.translate(None, string.punctuation)    # New Code
    line = line.lower()                                # New Code
    words = line.split()
    for word in words:
        if word not in counts:
            counts[word] = 1
        else:
            counts[word] += 1

print counts

print a.translate(string.maketrans(' ',' '), string.punctuation

Enter the file name: romeo-full.txt
{'swearst': 1, 'all': 6, 'afeard': 1, 'leave': 2, 'these': 2, 
'kinsmen': 2, 'what': 11, 'thinkst': 1, 'love': 24, 'cloak': 1, 
a': 24, 'orchard': 2, 'light': 5, 'lovers': 2, 'romeo': 40, 
'maiden': 1, 'whiteupturned': 1, 'juliet': 32, 'gentleman': 1, 
'it': 22, 'leans': 1, 'canst': 1, 'having': 1, ...}

9.5  Debugging

Scale down the input:

If possible, reduce the size of the dataset. For example if the program reads a text file, start with just the first 10 lines, or with the smallest example you can find. You can either edit the files themselves, or (better) modify the program so it reads only the first n lines.

If there is an error, you can reduce n to the smallest value that manifests the error, and then increase it gradually as you find and correct errors.

Check summaries and types:

Instead of printing and checking the entire dataset, consider printing summaries of the data: for example, the number of items in a dictionary or the total of a list of numbers.

A common cause of runtime errors is a value that is not the right type. For debugging this kind of error, it is often enough to print the type of a value.

Write self-checks:

Sometimes you can write code to check for errors automatically. For example, if you are computing the average of a list of numbers, you could check that the result is not greater than the largest element in the list or less than the smallest. This is called a ``sanity check'' because it detects results that are ``insane.''



Another kind of check compares the results of two different computations to see if they are consistent. This is called a ``consistency check.''

Pretty print the output:

Formatting debugging output can make it easier to spot an error.

9.6  Glossary

dictionary:

A mapping from a set of keys to their corresponding values.

hashtable:

The algorithm used to implement Python dictionaries.

hash function:

A function used by a hashtable to compute the location for a key.

histogram:

A set of counters.

implementation:

A way of performing a computation.

item:

Another name for a key-value pair.

key:

An object that appears in a dictionary as the first part of a key-value pair.

key-value pair:

The representation of the mapping from a key to a value.

lookup:

A dictionary operation that takes a key and finds the corresponding value.

nested loops:

When there is one or more loops ``inside'' of another loop. The inner loop runs to completion each time the outer loop runs once.

value:

An object that appears in a dictionary as the second part of a key-value pair. This is more specific than our previous use of the word ``value.''

9.7  Exercises

Sample Line:
From stephen.marquard@uct.ac.za Sat Jan  5 09:14:16 2008

Sample Execution:
python dow.py
Enter a file name: mbox-short.txt
{'Fri': 20, 'Thu': 6, 'Sat': 1}

Enter a file name: mbox-short.txt
cwen@iupui.edu 5

Enter a file name: mbox.txt
zqian@umich.edu 195

python schoolcount.py
Enter a file name: mbox-short.txt
{'media.berkeley.edu': 4, 'uct.ac.za': 6, 'umich.edu': 7, 
'gmail.com': 1, 'caret.cam.ac.uk': 1, 'iupui.edu': 8}

Chapter 10  Tuples

10.1  Tuples are immutable

>>> t = 'a', 'b', 'c', 'd', 'e'

>>> t = ('a', 'b', 'c', 'd', 'e')

>>> t1 = ('a',)
>>> type(t1)
<type 'tuple'>

>>> t2 = ('a')
>>> type(t2)
<type 'str'>

>>> t = tuple()
>>> print t
()

>>> t = tuple('lupins')
>>> print t
('l', 'u', 'p', 'i', 'n', 's')

>>> t = ('a', 'b', 'c', 'd', 'e')
>>> print t[0]
'a'

>>> print t[1:3]
('b', 'c')

>>> t[0] = 'A'
TypeError: object doesn't support item assignment

>>> t = ('A',) + t[1:]
>>> print t
('A', 'b', 'c', 'd', 'e')

10.2  Comparing tuples

>>> (0, 1, 2) < (0, 3, 4)
True
>>> (0, 1, 2000000) < (0, 3, 4)
True

Decorate

a sequence by building a list of tuples with one or more sort keys preceding the elements from the sequence,

Sort

the list of tuples uring the Python built-in sort, and

Undecorate

by extracting the sorted elements of the sequence.

def sort_by_length(words):
    t = list()
    for word in words:
       t.append((len(word), word))

    t.sort(reverse=True)

    res = list()
    for length, word in t:
        res.append(word)
    return res

10.3  Tuple assignment

>>> m = [ 'have', 'fun' ]
>>> x, y = m
>>> x
'have'
>>> y
'fun'
>>> 

>>> m = [ 'have', 'fun' ]
>>> x = m[0]
>>> y = m[1]
>>> x
'have'
>>> y
'fun'
>>> 

>>> m = [ 'have', 'fun' ]
>>> (x, y) = m
>>> x
'have'
>>> y
'fun'
>>> 

>>> a, b = b, a

>>> a, b = 1, 2, 3
ValueError: too many values to unpack

>>> addr = 'monty@python.org'
>>> uname, domain = addr.split('@')

>>> print uname
monty
>>> print domain
python.org

10.4  Dictionaries and tuples

>>> d = {'a':10, 'b':1, 'c':22}
>>> t = d.items()
>>> print t
[('a', 10), ('c', 22), ('b', 1)]

>>> d = {'a':10, 'b':1, 'c':22}
>>> t = d.items()
>>> t
[('a', 10), ('c', 22), ('b', 1)]
>>> t.sort()
>>> t
[('a', 10), ('b', 1), ('c', 22)]

10.5  Multiple assignment with dictionaries

for key, val in d.items():
    print val, key

0 a
2 c
1 b

>>> d = {'a':10, 'b':1, 'c':22}
>>> l = list()
>>> for key, val in d.items() :
...     l.append( (val, key) )
... 
>>> l
[(10, 'a'), (22, 'c'), (1, 'b')]
>>> l.sort(reverse=True)
>>> l
[(22, 'c'), (10, 'a'), (1, 'b')]
>>> 

10.6  The most common words

import string
fhand = open('romeo-full.txt')
counts = dict()
for line in fhand:
    line = line.translate(None, string.punctuation)
    line = line.lower()
    words = line.split()
    for word in words:
        if word not in counts:
            counts[word] = 1
        else:
            counts[word] += 1

# Sort the dictionary by value
lst = list()
for key, val in counts.items():
    lst.append( (val, key) )

lst.sort(reverse=True)

for key, val in lst[:10] :
    print key, val

61 i
42 and
40 romeo
34 to
34 the
32 thou
32 juliet
30 that
29 my
24 thee

10.7  Using tuples as keys in dictionaries

directory[last,first] = number

for last, first in directory:
    print first, last, directory[last,first]

10.8  Sequences: strings, lists, and tuples--Oh My!

1. In some contexts, like a return statement, it is syntactically simpler to create a tuple than a list. In other contexts, you might prefer a list.
   
   
2. If you want to use a sequence as a dictionary key, you have to use an immutable type like a tuple or string.
   
   
3. If you are passing a sequence as an argument to a function, using tuples reduces the potential for unexpected behavior due to aliasing.

10.9  Debugging

reading:

Examine your code, read it back to yourself, and check that it says what you meant to say.

running:

Experiment by making changes and running different versions. Often if you display the right thing at the right place in the program, the problem becomes obvious, but sometimes you have to spend some time to build scaffolding.

ruminating:

Take some time to think! What kind of error is it: syntax, runtime, semantic? What information can you get from the error messages, or from the output of the program? What kind of error could cause the problem you're seeing? What did you change last, before the problem appeared?

retreating:

At some point, the best thing to do is back off, undoing recent changes, until you get back to a program that works and that you understand. Then you can start rebuilding.

10.10  Glossary

comparable:

A type where one value can be checked to see if it is greater than, less than or equal to another value of the same type. Types which are comparable can be put in a list and sorted.

data structure:

A collection of related values, often organized in lists, dictionaries, tuples, etc.

DSU:

Abbreviation of ``decorate-sort-undecorate,'' a pattern that involves building a list of tuples, sorting, and extracting part of the result.

gather:

The operation of assembling a variable-length argument tuple.

hashable:

A type that has a hash function. Immutable types like integers, floats and strings are hashable; mutable types like lists and dictionaries are not.

scatter:

The operation of treating a sequence as a list of arguments.

shape (of a data structure):

A summary of the type, size and composition of a data structure.

singleton:

A list (or other sequence) with a single element.

tuple:

An immutable sequence of elements.

tuple assignment:

An assignment with a sequence on the right side and a tuple of variables on the left. The right side is evaluated and then its elements are assigned to the variables on the left.

10.11  Exercises

Sample Line:
From stephen.marquard@uct.ac.za Sat Jan  5 09:14:16 2008

Enter a file name: mbox-short.txt
cwen@iupui.edu 5

Enter a file name: mbox.txt
zqian@umich.edu 195

Sample Execution:
python timeofday.py
Enter a file name: mbox-short.txt
04 3
06 1
07 1
09 2
10 3
11 6
14 1
15 2
16 4
17 2
18 1
19 1

1

Fun fact: The word ``tuple'' comes from the names given to seqences of numbers of varying lengths: single, double, triple, quadruple, quituple, sextuple, septuple, etc.

2

Python does not translate the syntax literally. For example if you try this with a dictionary it will not work as might expect.

3

This behavior is slightly different in Python 3.0.

Chapter 11  Regular expressions

import re
hand = open('mbox-short.txt')
for line in hand:
    line = line.rstrip()
    if re.search('From:', line) :
        print line

import re
hand = open('mbox-short.txt')
for line in hand:
    line = line.rstrip()
    if re.search('^From:', line) :
        print line

11.1  Character matching in regular expressions

import re
hand = open('mbox-short.txt')
for line in hand:
    line = line.rstrip()
    if re.search('^F..m:', line) :
        print line

import re
hand = open('mbox-short.txt')
for line in hand:
    line = line.rstrip()
    if re.search('^From:.+@', line) :
        print line

 From: stephen.marquard @uct.ac.za

 From:.+ @

 From: stephen.marquard@uct.ac.za, csev@umich.edu, and cwen @iupui.edu

11.2  Extracting data using regular expressions

From stephen.marquard@uct.ac.za Sat Jan  5 09:14:16 2008
Return-Path: <postmaster@collab.sakaiproject.org>
          for <source@collab.sakaiproject.org>;
Received: (from apache@localhost)
Author: stephen.marquard@uct.ac.za

import re
s = 'Hello from csev@umich.edu to cwen@iupui.edu about the meeting @2PM'
lst = re.findall('\S+@\S+', s)
print lst

['csev@umich.edu', 'cwen@iupui.edu']

import re
hand = open('mbox-short.txt')
for line in hand:
    line = line.rstrip()
    x = re.findall('\S+@\S+', line)
    if len(x) > 0 :
        print x

['wagnermr@iupui.edu']
['cwen@iupui.edu']
['<postmaster@collab.sakaiproject.org>']
['<200801032122.m03LMFo4005148@nakamura.uits.iupui.edu>']
['<source@collab.sakaiproject.org>;']
['<source@collab.sakaiproject.org>;']
['<source@collab.sakaiproject.org>;']
['apache@localhost)']
['source@collab.sakaiproject.org;']

[a-zA-Z0-9]\S*@\S*[a-zA-Z]

import re
hand = open('mbox-short.txt')
for line in hand:
    line = line.rstrip()
    x = re.findall('[a-zA-Z0-9]\S+@\S+[a-zA-Z]', line)
    if len(x) > 0 :
        print x

…
['wagnermr@iupui.edu']
['cwen@iupui.edu']
['postmaster@collab.sakaiproject.org']
['200801032122.m03LMFo4005148@nakamura.uits.iupui.edu']
['source@collab.sakaiproject.org']
['source@collab.sakaiproject.org']
['source@collab.sakaiproject.org']
['apache@localhost']

11.3  Combining searching and extracting

X-DSPAM-Confidence: 0.8475
X-DSPAM-Probability: 0.0000  

^X-.*: [0-9.]+

import re
hand = open('mbox-short.txt')
for line in hand:
    line = line.rstrip()
    if re.search('^X\S*: [0-9.]+', line) :
        print line

X-DSPAM-Confidence: 0.8475
X-DSPAM-Probability: 0.0000
X-DSPAM-Confidence: 0.6178
X-DSPAM-Probability: 0.0000

import re
hand = open('mbox-short.txt')
for line in hand:
    line = line.rstrip()
    x = re.findall('^X\S*: ([0-9.]+)', line)
    if len(x) > 0 :
        print x

['0.8475']
['0.0000']
['0.6178']
['0.0000']
['0.6961']
['0.0000']
..

Details: http://source.sakaiproject.org/viewsvn/?view=rev&rev=39772

import re
hand = open('mbox-short.txt')
for line in hand:
    line = line.rstrip()
    x = re.findall('^Details:.*rev=([0-9.]+)', line)
    if len(x) > 0:
        print x

['39772']
['39771']
['39770']
['39769']
...

From stephen.marquard@uct.ac.za Sat Jan  5 09:14:16 2008

^From .* [0-9][0-9]:

^From .* ([0-9][0-9]):

import re
hand = open('mbox-short.txt')
for line in hand:
    line = line.rstrip()
    x = re.findall('^From .* ([0-9][0-9]):', line)
    if len(x) > 0 : print x

['09']
['18']
['16']
['15']
…

11.4  Escape character

import re
x = 'We just received $10.00 for cookies.'
y = re.findall('\$[0-9.]+',x)

11.5  Summary

11.6  Bonus section for UNIX users

$ grep '^From:' mbox-short.txt
From: stephen.marquard@uct.ac.za
From: louis@media.berkeley.edu
From: zqian@umich.edu
From: rjlowe@iupui.edu

11.7  Debugging

>>> help()

Welcome to Python 2.6!  This is the online help utility.

If this is your first time using Python, you should definitely check out
the tutorial on the Internet at http://docs.python.org/tutorial/.

Enter the name of any module, keyword, or topic to get help on writing
Python programs and using Python modules.  To quit this help utility and
return to the interpreter, just type "quit".

To get a list of available modules, keywords, or topics, type "modules",
"keywords", or "topics".  Each module also comes with a one-line summary
of what it does; to list the modules whose summaries contain a given word
such as "spam", type "modules spam".

help> modules

>>> import re
>>> dir(re)
[.. 'compile', 'copy_reg', 'error', 'escape', 'findall', 
'finditer', 'match', 'purge', 'search', 'split', 'sre_compile', 
'sre_parse', 'sub', 'subn', 'sys', 'template']

>>> help (re.search)
Help on function search in module re:

search(pattern, string, flags=0)
    Scan through string looking for a match to the pattern, returning
    a match object, or None if no match was found.
>>> 

11.8  Glossary

brittle code:

Code that works when the input data is in a particular format but prone to breakage if there is some deviation from the correct format. We call this ``brittle code'' because it is easily broken.

greedy matching:

The notion that the ``+'' and ``*'' characters in a regular expression expand outward to match the largest possible string.

grep:

A command available in most UNIX systems that searches through text files looking for lines that match regular expressions. The command name stands for "Generalized Regular Expression Parser".

regular expression:

A language for expressing more complex search strings. A regular expression may contain special characters that indicate that a search only matches at the beginning or end of a line or many other similar capabilities.

wild card:

A special character that matches any character. In regular expressions the wild card character is the period character.

11.9  Exercises

$ python grep.py
Enter a regular expression: ^Author
mbox.txt had 1798 lines that matched ^Author

$ python grep.py
Enter a regular expression: ^X-
mbox.txt had 14368 lines that matched ^X-

$ python grep.py
Enter a regular expression: java$
mbox.txt had 4218 lines that matched java$

Enter file:mbox.txt 
38549.7949721

Enter file:mbox-short.txt
39756.9259259

Chapter 12  Networked programs

12.1  HyperText Transport Protocol - HTTP

12.2  The World's Simplest Web Browser

import socket

mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect(('www.py4inf.com', 80))
mysock.send('GET http://www.py4inf.com/code/romeo.txt HTTP/1.0\n\n')

while True:
    data = mysock.recv(512)
    if ( len(data) < 1 ) :
        break
    print data

mysock.close()

HTTP/1.1 200 OK
Date: Sun, 14 Mar 2010 23:52:41 GMT
Server: Apache
Last-Modified: Tue, 29 Dec 2009 01:31:22 GMT
ETag: "143c1b33-a7-4b395bea"
Accept-Ranges: bytes
Content-Length: 167
Connection: close
Content-Type: text/plain

But soft what light through yonder window breaks
It is the east and Juliet is the sun
Arise fair sun and kill the envious moon
Who is already sick and pale with grief

12.3  Retrieving web pages with urllib

import urllib

fhand = urllib.urlopen('http://www.py4inf.com/code/romeo.txt')
for line in fhand:
   print line.strip()

But soft what light through yonder window breaks
It is the east and Juliet is the sun
Arise fair sun and kill the envious moon
Who is already sick and pale with grief

import urllib

counts = dict()
fhand = urllib.urlopen('http://www.py4inf.com/code/romeo.txt')
for line in fhand:
    words = line.split()
    for word in words:
        counts[word] = counts.get(word,0) + 1   
print counts

12.4  Parsing HTML and scraping the web

12.5  Parsing HTML using Regular Expressions

<h1>The First Page</h1>
<p>
If you like, you can switch to the
<a href="http://www.dr-chuck.com/page2.htm">
Second Page</a>.
</p>

href="http://.+?"

import urllib
import re

url = raw_input('Enter - ')
html = urllib.urlopen(url).read()
links = re.findall('href="(http://.*?)"', html)
for link in links:
    print link

python urlregex.py 
Enter - http://www.dr-chuck.com/page1.htm
http://www.dr-chuck.com/page2.htm

python urlregex.py 
Enter - http://www.py4inf.com/book.htm
http://www.greenteapress.com/thinkpython/thinkpython.html
http://allendowney.com/
http://www.py4inf.com/code
http://www.lib.umich.edu/espresso-book-machine
http://www.py4inf.com/py4inf-slides.zip

12.6  Parsing HTML using BeautifulSoup

import urllib
from BeautifulSoup import *

url = raw_input('Enter - ')
html = urllib.urlopen(url).read()
soup = BeautifulSoup(html)

# Retrieve all of the anchor tags
tags = soup('a')
for tag in tags:
   print tag.get('href', None)

python urllinks.py 
Enter - http://www.dr-chuck.com/page1.htm
http://www.dr-chuck.com/page2.htm

python urllinks.py 
Enter - http://www.py4inf.com/book.htm
http://www.greenteapress.com/thinkpython/thinkpython.html
http://allendowney.com/
http://www.si502.com/
http://www.lib.umich.edu/espresso-book-machine
http://www.py4inf.com/code
http://www.pythonlearn.com/

import urllib
from BeautifulSoup import *

url = raw_input('Enter - ')
html = urllib.urlopen(url).read()
soup = BeautifulSoup(html)

# Retrieve all of the anchor tags
tags = soup('a')
for tag in tags:
   # Look at the parts of a tag
   print 'TAG:',tag
   print 'URL:',tag.get('href', None)
   print 'Content:',tag.contents[0]
   print 'Attrs:',tag.attrs

python urllink2.py 
Enter - http://www.dr-chuck.com/page1.htm
TAG: <a href="http://www.dr-chuck.com/page2.htm">
Second Page</a>
URL: http://www.dr-chuck.com/page2.htm
Content: [u'\nSecond Page']
Attrs: [(u'href', u'http://www.dr-chuck.com/page2.htm')]

12.7  Reading binary files using urllib

img = urllib.urlopen('http://www.py4inf.com/cover.jpg').read()
fhand = open('cover.jpg', 'w')
fhand.write(img)
fhand.close()

import urllib

img = urllib.urlopen('http://www.py4inf.com/cover.jpg')
fhand = open('cover.jpg', 'w')
size = 0
while True:
    info = img.read(100000)
    if len(info) < 1 : break
    size = size + len(info)
    fhand.write(info)

print size,'characters copied.'
fhand.close()

python curl2.py 
568248 characters copied.

curl -O http://www.py4inf.com/cover.jpg

12.8  Glossary

BeautifulSoup:

A Python library for parsing HTML documents and extracting data from HTML documents that compensates for most of the imperfections in the HTML that browsers generally ignore. You can download the BeautifulSoup code from www.crummy.com.

port:

A number that generally indicates which application you are contacting when you make a socket connection to a server. As an example, web traffic usually uses port 80 while e-mail traffic uses port 25.

scrape:

When a program pretends to be a web browser and retrieves a web page and then looks at the web page content. Often programs are following the links in one page to find the next page so they can traverse a network of pages or a social network.

socket:

A network connection between two applications where the applications can send and receive data in either direction.

spider:

The act of a web search engine retrieving a page and then all the pages linked from a page and so on until they have nearly all of the pages on the Internet which they use to build their search index.

12.9  Exercises

Chapter 13  Using Web Services

13.1  eXtensible Markup Language - XML

<person>
  <name>Chuck</name>
  <phone type="intl">
     +1 734 303 4456
   </phone>
   <email hide="yes"/>
</person>

13.2  Parsing XML

import xml.etree.ElementTree as ET

data = '''
<person>
  <name>Chuck</name>
  <phone type="intl">
     +1 734 303 4456
   </phone>
   <email hide="yes"/>
</person>'''

tree = ET.fromstring(data)
print 'Name:',tree.find('name').text
print 'Attr:',tree.find('email').get('hide')

Name: Chuck
Attr: yes

13.3  Looping through nodes

import xml.etree.ElementTree as ET

input = '''
<stuff>
    <users>
        <user x="2">
            <id>001</id>
            <name>Chuck</name>
        </user>
        <user x="7">
            <id>009</id>
            <name>Brent</name>
        </user>
    </users>
</stuff>'''

stuff = ET.fromstring(input)
lst = stuff.findall('users/user')
print 'User count:', len(lst)

for item in lst:
    print 'Name', item.find('name').text
    print 'Id', item.find('id').text
    print 'Attribute', item.get('x')

User count: 2
Name Chuck
Id 001
Attribute 2
Name Brent
Id 009
Attribute 7

13.4  Application Programming Interfaces (API)

13.5  Twitter web services

import urllib

TWITTER_URL = 'http://api.twitter.com/l/statuses/friends/ACCT.xml'

while True:
    print ''
    acct = raw_input('Enter Twitter Account:')
    if ( len(acct) < 1 ) : break
    url = TWITTER_URL.replace('ACCT', acct)
    print 'Retrieving', url
    document = urllib.urlopen (url).read()
    print document[:250]

python twitter1.py

Enter Twitter Account:drchuck
Retrieving http://api.twitter.com/l/statuses/friends/drchuck.xml
<?xml version="1.0" encoding="UTF-8"?>
<users type="array">
<user>
  <id>115636613</id>
  <name>Steve Coppin</name>
  <screen_name>steve_coppin</screen_name>
  <location>Kent, UK</location>
  <description>Software developing, best practicing, agile e

Enter Twitter Account:

<?xml version="1.0" encoding="UTF-8"?>
<users type="array">
  <user>
    <id>115636613</id>
    <name>Steve Coppin</name>
    <screen_name>steve_coppin</screen_name>
    <location>Kent, UK</location>
    <status>
      <id>10174607039</id>
      <source>web</source>
      </status>
  </user>
  <user>
    <id>17428929</id>
    <name>davidkocher</name>
    <screen_name>davidkocher</screen_name>
    <location>Bern</location>
    <status>
      <id>10306231257</id>
      <text>@MikeGrace If possible please post a detailed bug report </text>
    </status>
  </user>
  ...

13.6  Handling XML data from an API

import urllib
import xml.etree.ElementTree as ET

TWITTER_URL = 'http://api.twitter.com/l/statuses/friends/ACCT.xml'

while True:
    print ''
    acct = raw_input('Enter Twitter Account:')
    if ( len(acct) < 1 ) : break
    url = TWITTER_URL.replace('ACCT', acct)
    print 'Retrieving', url
    document = urllib.urlopen (url).read()
    print 'Retrieved', len(document), 'characters.' 
    tree = ET.fromstring(document)
    count = 0
    for user in tree.findall('user'):
        count = count + 1
        if count > 4 : break
        print user.find('screen_name').text
        status =  user.find('status')
        if status : 
            txt = status.find('text').text
            print '  ',txt[:50]

python twitter2.py 

Enter Twitter Account:drchuck
Retrieving http://api.twitter.com/l/statuses/friends/drchuck.xml
Retrieved 193310 characters.
steve_coppin
   Looking forward to some "oh no the markets closed,
davidkocher
   @MikeGrace If possible please post a detailed bug 
hrheingold
   From today's Columbia Journalism Review, on crap d
huge_idea
   @drchuck  #cnx2010 misses you, too.  Thanks for co

Enter Twitter Account:hrheingold
Retrieving http://api.twitter.com/l/statuses/friends/hrheingold.xml
Retrieved 208081 characters.
carr2n
   RT @tysone: Saturday's proclaimation by @carr2n pr
tiffanyshlain
   RT @ScottKirsner: Turning smartphones into a tool 
soniasimone
   @ACCompanyC Funny, smart, cute, and also nice! He 
JenStone7617
   Watching "Changing The Equation: High Tech Answers

Enter Twitter Account:

13.7  Glossary

API:

Application Program Interface - A contract between applications that defines the patterns of interaction between two application components.

ElementTree:

A built-in Python library used to parse XML data.

XML:

eXtensible Markup Language - A format that allows for the markup of structured data.

REST:

REpresentational State Transfer - A style of Web Services that provide access to resources within an application using the HTTP protocol.

SOA:

Service Oriented Architecture - when an application is made of components connected across a network.

13.8  Exercises

Enter Twitter Account:drchuck
Retrieving http://api.twitter.com/l/statuses/friends/drchuck.xml
Retrieved 194533 characters.
steve_coppin
   Kent, UK
   Looking forward to some "oh no the markets closed,
davidkocher
   Bern
   @MikeGrace If possible please post a detailed bug 
hrheingold
   San Francisco Bay Area
   RT @barrywellman: Lovely AmBerhSci Internet & Comm
huge_idea
   Boston, MA
   @drchuck  #cnx2010 misses you, too.  Thanks for co

Chapter 14  Using databases and Structured Query Language (SQL)

14.1  What is a database?

14.2  Database concepts

14.3  SQLite manager Firefox add-on

14.4  Creating a database table

import sqlite3

conn = sqlite3.connect('music.db')
cur = conn.cursor()

cur.execute('DROP TABLE IF EXISTS Tracks ')
cur.execute('CREATE TABLE Tracks (title TEXT, plays INTEGER)')

conn.close()

cur.execute('DROP TABLE IF EXISTS Tracks ')

cur.execute('CREATE TABLE Tracks (title TEXT, plays INTEGER)')

import sqlite3

conn = sqlite3.connect('music.db')
cur = conn.cursor()

cur.execute('INSERT INTO Tracks (title, plays) VALUES ( ?, ? )', 
    ( 'Thunderstruck', 20 ) )
cur.execute('INSERT INTO Tracks (title, plays) VALUES ( ?, ? )', 
    ( 'My Way', 15 ) )
conn.commit()

print 'Tracks:'
cur.execute('SELECT title, plays FROM Tracks')
for row in cur :
   print row

cur.execute('DELETE FROM Tracks WHERE plays < 100')
conn.commit()

cur.close()

Tracks:
(u'Thunderstruck', 20)
(u'My Way', 15)

14.5  Structured Query Language (SQL) summary

CREATE TABLE Tracks (title TEXT, plays INTEGER)

INSERT INTO Tracks (title, plays) VALUES ('My Way', 15)

SELECT * FROM Tracks WHERE title = 'My Way'

SELECT title,plays FROM Tracks ORDER BY title

DELETE FROM Tracks WHERE title = 'My Way'

UPDATE Tracks SET plays = 16 WHERE title = 'My Way'

14.6  Spidering Twitter using a database

import sqlite3
import urllib
import xml.etree.ElementTree as ET

TWITTER_URL = 'http://api.twitter.com/l/statuses/friends/ACCT.xml'

conn = sqlite3.connect('twdata.db')
cur = conn.cursor()

cur.execute('''
CREATE TABLE IF NOT EXISTS 
Twitter (name TEXT, retrieved INTEGER, friends INTEGER)''')

while True:
    acct = raw_input('Enter a Twitter account, or quit: ')
    if ( acct == 'quit' ) : break
    if ( len(acct) < 1 ) :
        cur.execute('SELECT name FROM Twitter WHERE retrieved = 0 LIMIT 1')
        try:
            acct = cur.fetchone()[0]
        except:
            print 'No unretrieved Twitter accounts found'
            continue

    url = TWITTER_URL.replace('ACCT', acct)
    print 'Retrieving', url
    document = urllib.urlopen (url).read()
    tree = ET.fromstring(document)

    cur.execute('UPDATE Twitter SET retrieved=1 WHERE name = ?', (acct, ) )

    countnew = 0
    countold = 0
    for user in tree.findall('user'):
        friend = user.find('screen_name').text
        cur.execute('SELECT friends FROM Twitter WHERE name = ? LIMIT 1', 
            (friend, ) )
        try:
            count = cur.fetchone()[0]
            cur.execute('UPDATE Twitter SET friends = ? WHERE name = ?', 
                (count+1, friend) )
            countold = countold + 1
        except:
            cur.execute('''INSERT INTO Twitter (name, retrieved, friends) 
                VALUES ( ?, 0, 1 )''', ( friend, ) )
            countnew = countnew + 1
    print 'New accounts=',countnew,' revisited=',countold
    conn.commit()

cur.close()

    countnew = 0
    countold = 0
    for user in tree.findall('user'):
        friend = user.find('screen_name').text
        cur.execute('SELECT friends FROM Twitter WHERE name = ? LIMIT 1', 
            (friend, ) )
        try:
            count = cur.fetchone()[0]
            cur.execute('UPDATE Twitter SET friends = ? WHERE name = ?', 
                (count+1, friend) )
            countold = countold + 1
        except:
            cur.execute('''INSERT INTO Twitter (name, retrieved, friends) 
                VALUES ( ?, 0, 1 )''', ( friend, ) )
            countnew = countnew + 1
    print 'New accounts=',countnew,' revisited=',countold
    conn.commit()

Enter a Twitter account, or quit: drchuck
Retrieving http://api.twitter.com/l/statuses/friends/drchuck.xml
New accounts= 100  revisited= 0
Enter a Twitter account, or quit: quit

import sqlite3

conn = sqlite3.connect('twdata.db')
cur = conn.cursor()
cur.execute('SELECT * FROM Twitter')
count = 0
for row in cur :
   print row
   count = count + 1
print count, 'rows.'
cur.close()

(u'opencontent', 0, 1)
(u'lhawthorn', 0, 1)
(u'steve_coppin', 0, 1)
(u'davidkocher', 0, 1)
(u'hrheingold', 0, 1)
...
100 rows.

Enter a Twitter account, or quit: 
Retrieving http://api.twitter.com/l/statuses/friends/opencontent.xml
New accounts= 98  revisited= 2
Enter a Twitter account, or quit: 
Retrieving http://api.twitter.com/l/statuses/friends/lhawthorn.xml
New accounts= 97  revisited= 3
Enter a Twitter account, or quit: quit

    if ( len(acct) < 1 ) :
        cur.execute('SELECT name FROM Twitter WHERE retrieved = 0 LIMIT 1')
        try:
            acct = cur.fetchone()[0]
        except:
            print 'No unretrieved twitter accounts found'
            continue

    url = TWITTER_URL.replace('ACCT', acct)
    print 'Retrieving', url
    document = urllib.urlopen (url).read()
    tree = ET.fromstring(document)

    cur.execute('UPDATE Twitter SET retrieved=1 WHERE name = ?', (acct, ) )

(u'opencontent', 1, 1)
(u'lhawthorn', 1, 1)
(u'steve_coppin', 0, 1)
(u'davidkocher', 0, 1)
(u'hrheingold', 0, 1)
...
(u'cnxorg', 0, 2)
(u'knoop', 0, 1)
(u'kthanos', 0, 2)
(u'LectureTools', 0, 1)
...
295 rows.

14.7  Basic data modeling

CREATE TABLE Pals (from_friend TEXT, to_friend TEXT)

INSERT INTO Pals (from_friend,to_friend) VALUES ('drchuck', 'lhawthorn')

CREATE TABLE People 
    (id INTEGER PRIMARY KEY, name TEXT UNIQUE, retrieved INTEGER)

CREATE TABLE Follows 
    (from_id INTEGER, to_id INTEGER, UNIQUE(from_id, to_id) )

14.8  Programming with multiple tables

import sqlite3
import urllib
import xml.etree.ElementTree as ET

TWITTER_URL = 'http://api.twitter.com/l/statuses/friends/ACCT.xml'

conn = sqlite3.connect('twdata.db')
cur = conn.cursor()

cur.execute('''CREATE TABLE IF NOT EXISTS People 
    (id INTEGER PRIMARY KEY, name TEXT UNIQUE, retrieved INTEGER)''')
cur.execute('''CREATE TABLE IF NOT EXISTS Follows 
    (from_id INTEGER, to_id INTEGER, UNIQUE(from_id, to_id))''')

while True:
    acct = raw_input('Enter a Twitter account, or quit: ')
    if ( acct == 'quit' ) : break
    if ( len(acct) < 1 ) :
        cur.execute('''SELECT id,name FROM People 
            WHERE retrieved = 0 LIMIT 1''')
        try:
            (id, acct) = cur.fetchone()
        except:
            print 'No unretrieved Twitter accounts found'
            continue
    else:
        cur.execute('SELECT id FROM People WHERE name = ? LIMIT 1', 
            (acct, ) )
        try:
            id = cur.fetchone()[0]
        except:
            cur.execute('''INSERT OR IGNORE INTO People 
                (name, retrieved) VALUES ( ?, 0)''', ( acct, ) )
            conn.commit()
            if cur.rowcount != 1 : 
                print 'Error inserting account:',acct
                continue
            id = cur.lastrowid

    url = TWITTER_URL.replace('ACCT', acct)
    print 'Retrieving', url
    document = urllib.urlopen (url).read()
    tree = ET.fromstring(document)

    cur.execute('UPDATE People SET retrieved=1 WHERE name = ?', (acct, ) )

    countnew = 0
    countold = 0
    for user in tree.findall('user'):
        friend = user.find('screen_name').text
        cur.execute('SELECT id FROM People WHERE name = ? LIMIT 1', 
            (friend, ) )
        try:
            friend_id = cur.fetchone()[0]
            countold = countold + 1
        except:
            cur.execute('''INSERT OR IGNORE INTO People (name, retrieved) 
                VALUES ( ?, 0)''', ( friend, ) )
            conn.commit()
            if cur.rowcount != 1 :
                print 'Error inserting account:',friend
                continue
            friend_id = cur.lastrowid
            countnew = countnew + 1
        cur.execute('''INSERT OR IGNORE INTO Follows 
            (from_id, to_id) VALUES (?, ?)''', (id, friend_id) )
    print 'New accounts=',countnew,' revisited=',countold
    conn.commit()

cur.close()

1. Creating tables with primary keys and constraints.
   
   
2. When we have a logical key for a person (i.e. account name) and we need the id value for the person. Depending on whether or not the person is already in the People table, we either need to: (1) look up the person in the People table and retrieve the id value for the person or (2) add the person the the People table and get the id value for the newly added row.
   
   
3. Insert the row that captures the ``follows'' relationship.

14.8.1  Constraints in database tables

cur.execute('''CREATE TABLE IF NOT EXISTS People 
    (id INTEGER PRIMARY KEY, name TEXT UNIQUE, retrieved INTEGER)''')
cur.execute('''CREATE TABLE IF NOT EXISTS Follows 
    (from_id INTEGER, to_id INTEGER, UNIQUE(from_id, to_id))''')

cur.execute('''INSERT OR IGNORE INTO People (name, retrieved) 
    VALUES ( ?, 0)''', ( friend, ) )

cur.execute('''INSERT OR IGNORE INTO Follows 
    (from_id, to_id) VALUES (?, ?)''', (id, friend_id) )

14.8.2  Retrieve and/or insert a record

        friend = user.find('screen_name').text
        cur.execute('SELECT id FROM People WHERE name = ? LIMIT 1',
            (friend, ) )
        try:
            friend_id = cur.fetchone()[0]
            countold = countold + 1
        except:
            cur.execute('''INSERT OR IGNORE INTO People (name, retrieved) 
                VALUES ( ?, 0)''', ( friend, ) )
            conn.commit()
            if cur.rowcount != 1 :
                print 'Error inserting account:',friend
                continue
            friend_id = cur.lastrowid
            countnew = countnew + 1

14.8.3  Storing the friend relationship

cur.execute('INSERT OR IGNORE INTO Follows (from_id, to_id) VALUES (?, ?)',
    (id, friend_id) )

Enter a Twitter account, or quit: 
No unretrieved Twitter accounts found
Enter a Twitter account, or quit: drchuck
Retrieving http://api.twitter.com/l/statuses/friends/drchuck.xml
New accounts= 100  revisited= 0
Enter a Twitter account, or quit: 
Retrieving http://api.twitter.com/l/statuses/friends/opencontent.xml
New accounts= 97  revisited= 3
Enter a Twitter account, or quit: 
Retrieving http://api.twitter.com/l/statuses/friends/lhawthorn.xml
New accounts= 97  revisited= 3
Enter a Twitter account, or quit: quit

People:
(1, u'drchuck', 1)
(2, u'opencontent', 1)
(3, u'lhawthorn', 1)
(4, u'steve_coppin', 0)
(5, u'davidkocher', 0)
295 rows.
Follows:
(1, 2)
(1, 3)
(1, 4)
(1, 5)
(1, 6)
300 rows.

14.9  Three kinds of keys

• A logical key is a key that the ``real world'' might use to look up a row. In our example data model, the name field is a logical key. It is the screen name for the user and we indeed look up a user's row several times in the program using the name field. You will often find that it makes sense to add a UNIQUE constraint to a logical key. Since the logical key is how we look up a row from the outside world, it makes little sense to allow multiple rows with the same value in the table.
  
  
• A primary key is usually a number that is assigned automatically by the database. It generally has no meaning outside the program and is only used to link rows from different tables together. When we want to look up a row in a table, usually searching for the row using the primary key is the fastest way to find a row. Since primary keys are integer numbers, they take up very little storage and can be compared or sorted very quickly. In our data model, the id field is an example of a primary key.
  
  
• A foreign key is usually a number that points to the primary key of an associated row in a different table. An example of a foreign key in our data model is the from_id.

14.10  Using JOIN to retrieve data

SELECT * FROM Follows JOIN People 
    ON Follows.to_id = People.id WHERE Follows.from_id = 2

import sqlite3

conn = sqlite3.connect('twdata.db')
cur = conn.cursor()

cur.execute('SELECT * FROM People')
count = 0
print 'People:'
for row in cur :
   if count < 5: print row
   count = count + 1
print count, 'rows.'

cur.execute('SELECT * FROM Follows')
count = 0
print 'Follows:'
for row in cur :
   if count < 5: print row
   count = count + 1
print count, 'rows.'

cur.execute('''SELECT * FROM Follows JOIN People 
    ON Follows.to_id = People.id WHERE Follows.from_id = 2''')
count = 0
print 'Connections for id=2:'
for row in cur :
   if count < 5: print row
   count = count + 1
print count, 'rows.'

cur.close()

python twjoin.py 
People:
(1, u'drchuck', 1)
(2, u'opencontent', 1)
(3, u'lhawthorn', 1)
(4, u'steve_coppin', 0)
(5, u'davidkocher', 0)
295 rows.
Follows:
(1, 2)
(1, 3)
(1, 4)
(1, 5)
(1, 6)
300 rows.
Connections for id=2:
(2, 1, 1, u'drchuck', 1)
(2, 28, 28, u'cnxorg', 0)
(2, 30, 30, u'kthanos', 0)
(2, 102, 102, u'SomethingGirl', 0)
(2, 103, 103, u'ja_Pac', 0)
100 rows.

14.11  Summary

14.12  Debugging

14.13  Glossary

attribute:

One of the values within a tuple. More commonly called a ``column'' or ``field''.

constraint:

When we tell the database to enforce a rule on a field or a row in a table. A common constraint is to insist that there can be no duplicate values in a particular field (i.e. all the values must be unique).

cursor:

A cursor allows you to execute SQL commands in a database and retrieve data from the database. A cursor is similar to a socket or file handle for network connections and files respectively.

database browser:

A piece of software that allows you to directly connect to a database and manipulate the database directly without writing a program.

foreign key:

A numeric key that points to the primary key of a row in another table. Foreign keys establish relationships between rows stored in different tables.

index:

Additional data that the database software maintains as rows are inserted into a table designed to make lookups very fast.

logical key:

A key that the ``outside world'' uses to look up a particular row. For example in a table of user accounts, a person's e-mail address might be a good candidate as the logical key for the user's data.

normalization:

Designing a data model so that no data is replicated. We store each item of data at one place in the database and reference it elsewhere using a foreign key.

primary key:

A numeric key assigned to each row that is used to refer to one row in a table from another table. Often the database is configured to automatically assign primary keys as rows are inserted.

relation:

An area within a database that contains tuples and attributes. More typically called a ``table''.

tuple:

A single entry in a database table that is a set of attributes. More typically called ``row''.

1

SQLite actually does allow some flexibility in the type of data stored in a column, but we will keep our data types strict in this chapter so the concepts apply equally to other database systems such as MySQL.

2

In general, when a sentence starts with ``if all goes well'' you will find that the code needs to use try/except.

Chapter 15  Automating common tasks on your computer

15.1  File names and paths

>>> import os
>>> cwd = os.getcwd()
>>> print cwd
/Users/csev

>>> os.path.abspath('memo.txt')
'/Users/csev/memo.txt'

>>> os.path.exists('memo.txt')
True

>>> os.path.isdir('memo.txt')
False
>>> os.path.isdir('music')
True

>>> os.listdir(cwd)
['music', 'photos', 'memo.txt']

15.2  Example: Cleaning up a photo directory

./2006/03/24-03-06_2018002.jpg
./2006/03/24-03-06_2018002.txt

import os
count = 0
for (dirname, dirs, files) in os.walk('.'):
   for filename in files:
       if filename.endswith('.txt') :
           count = count + 1
print 'Files:', count

python txtcount.py
Files: 1917

import os
from os.path import join
for (dirname, dirs, files) in os.walk('.'):
   for filename in files:
       if filename.endswith('.txt') :
           thefile = os.path.join(dirname,filename)
           print os.path.getsize(thefile), thefile

python txtsize.py
...
18 ./2006/03/24-03-06_2303002.txt
22 ./2006/03/25-03-06_1340001.txt
22 ./2006/03/25-03-06_2034001.txt
...
2565 ./2005/09/28-09-05_1043004.txt
2565 ./2005/09/28-09-05_1141002.txt
...
2578 ./2006/03/27-03-06_1618001.txt
2578 ./2006/03/28-03-06_2109001.txt
2578 ./2006/03/29-03-06_1355001.txt
...

<html>
        <head>
                <title>T-Mobile</title>
...

import os
from os.path import join
for (dirname, dirs, files) in os.walk('.'):
   for filename in files:
       if filename.endswith('.txt') :
           thefile = os.path.join(dirname,filename)
           size = os.path.getsize(thefile)
           if size == 2578 or size == 2565:
               continue
           fhand = open(thefile,'r')
           lines = list()
           for line in fhand:
               lines.append(line)
           fhand.close()
           if len(lines) > 1:
                print len(lines), thefile
                print lines[:4]

python txtcheck.py 
3 ./2004/03/22-03-04_2015.txt
['Little horse rider\r\n', '\r\n', '\r']
2 ./2004/11/30-11-04_1834001.txt
['Testing 123.\n', '\n']
3 ./2007/09/15-09-07_074202_03.txt
['\r\n', '\r\n', 'Sent from my iPhone\r\n']
3 ./2007/09/19-09-07_124857_01.txt
['\r\n', '\r\n', 'Sent from my iPhone\r\n']
3 ./2007/09/20-09-07_115617_01.txt
...

           lines = list()
           for line in fhand:
               lines.append(line)
           if len(lines) == 3 and lines[2].startswith('Sent from my iPhone'):
               continue
           if len(lines) > 1:
                print len(lines), thefile
                print lines[:4]

python txtcheck2.py 
3 ./2004/03/22-03-04_2015.txt
['Little horse rider\r\n', '\r\n', '\r']
2 ./2004/11/30-11-04_1834001.txt
['Testing 123.\n', '\n']
2 ./2006/03/17-03-06_1806001.txt
['On the road again...\r\n', '\r\n']
2 ./2006/03/24-03-06_1740001.txt
['On the road again...\r\n', '\r\n']

import os
from os.path import join
for (dirname, dirs, files) in os.walk('.'):
   for filename in files:
       if filename.endswith('.txt') :
           thefile = os.path.join(dirname,filename)
           size = os.path.getsize(thefile)
           if size == 2578 or size == 2565:
               print 'T-Mobile:',thefile
               continue
           fhand = open(thefile,'r')
           lines = list()
           for line in fhand:
               lines.append(line)
           fhand.close()
           if len(lines) == 3 and lines[2].startswith('Sent from my iPhone'):
               print 'iPhone:', thefile
               continue

python txtcheck3.py
...
T-Mobile: ./2006/05/31-05-06_1540001.txt
T-Mobile: ./2006/05/31-05-06_1648001.txt
iPhone: ./2007/09/15-09-07_074202_03.txt
iPhone: ./2007/09/15-09-07_144641_01.txt
iPhone: ./2007/09/19-09-07_124857_01.txt
...

           if size == 2578 or size == 2565:
               print 'T-Mobile:',thefile
               os.remove(thefile)
               continue
...
           if len(lines) == 3 and lines[2].startswith('Sent from my iPhone'):
               print 'iPhone:', thefile
               os.remove(thefile)
               continue

python txtdelete.py 
T-Mobile: ./2005/01/02-01-05_1356001.txt
T-Mobile: ./2005/01/02-01-05_1858001.txt
...

python txtcount.py 
Files: 1018

15.3  Command line arguments

name = raw_input('Enter file:')
handle = open(name, 'r')
text = handle.read()
...

python words.py
Enter file: mbox-short.txt
...

import sys
print 'Count:', len(sys.argv)
print 'Type:', type(sys.argv)
for arg in sys.argv:
   print 'Argument:', arg

python argtest.py hello there
Count: 3
Type: <type 'list'>
Argument: argtest.py
Argument: hello
Argument: there

import sys

name = sys.argv[1]
handle = open(name, 'r')
text = handle.read()
print name, 'is', len(text), 'bytes'

python argfile.py mbox-short.txt
mbox-short.txt is 94626 bytes

15.4  Pipes

>>> cmd = 'ls -l'
>>> fp = os.popen(cmd)