Python

Style

The Python style guide is in [PEP 8]

Python is very strict on indentation. Code blocks are kept together by their indent. Use either tabs or spaces (recommended) not both.

For readability long lines can be over more lines using () around a statement or \ to escape new-line's

(this = a(verylongline,
    so you can use parentheses)
)
you can also use \
    to escape the end-of-line character

We have our own template.

Modules

Modules need to be imported into your program by the import command.

To add the location of your own modules to the python search path put it in the PYTHONPATH see #sys.path below.

Import finds a file <modulename>.py or __init__.py in directory <modulename>. <modulename>.py or __init__.py are executed on import. Usually not many code is in modules to execute immediately, functions and classes are mostly in there.

import <module>

Import everything from the module, address components as <module>.<component>.

import <module> as <short>

Calls can have the short name. E.g. numpy is often imported as np

from <module> import *

Module components can be called without the module name. Beware of duplicates.

from <module> import <component>

Import a specific component from a modules, callable by just the component name.

We try to use modules that are available by default (on linux systems). If not it will be mentioned in this article. Only modules for which we use a very limited number of functions are listed here. More complex modules have there own article

See pip for module management.

os

Operating system things

os.environ

dict of environment variables e.g. os.environ['HOSTNAME']

sys

Provides a number of system variables

sys.argv

List of everything on the commandline. sys.argv[0] is the program itself. All elements of sys.argv are strings.

sys.version

The python version you run

sys.path

The directories python looks into when doing an import. The script location always in sys.path. Directories in the environment variable $PYTHONPATH are added to sys.path

sys.stdout.flush()

Print all output immediately

datetime

Date and time functions

from datetime import datetime
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")

shutil

Add shell things to python see Python:Files#Filehandling_and_metadata for examples for file handling.

time

Time functions

time.sleep(3)

Sleep for 3 seconds

from time import sleep 
sleep(3)

subprocess

Module to execute shell commands

In python2:

import subprocess
exitcode = subprocess.call("<any command>")
commandoutput = subprocess.check_output("<any command>")

Use ("command",shell=True) to have the call work like it would on the commandline

To catch error output too:

import subprocess
try:
    subprocess.check_output(shellcommmand,stderr=subprocess.STDOUT,shell=True)
except subprocess.CalledProcessError, e:
    print 'Output from {}: {}'.format(shellcommand,e.output))

In python3:

import subprocess
CompletedProcess = subprocess.run("<any command>")

The CompletedProcess returned has (args, returncode, stdout, stderr)

random

Generate random numbers.

random.random()

Return a floating point in the range from 0.0 to 1.0 (including both)

random.randint(start,stop)

Return an integer in the range from start to stop (including both)

Pick a random element from a list:

alist[random.randint(0,len(alist)-1)]

random.shuffle(alist)

Randomize the order of the list elements. The list itself is changed, nothing is returned.

threading

Enable parallel processing within 1 process. To be used for I/O bound functions.

t1=threading.Thread(target=<a function>)

Return a thread object to run <a function> in the background

t1.start()

Start the thread for the function targeted by t1

t1.join(<timeout>)

Wait until t1 is ready or until <timeout> has expired. Returns None always.

t1.is_alive()

Return True if t1 is still running (useful e.g. after join with timeout).

multiprocessing

Parallel processing by spawning sub-processes. Spread the load over different processors.

argparse

Module to parse the commandline arguments (sys.argv).

import sys 
import argparse

def main():    
    argparser = argparse.ArgumentParser()
    argparser.add_argument('--arg1',type=int,default=0)
    argparser.add_argument('--arg2',type=int,default=0)
    args = argparser.parse_args()
    print(args.arg1)
    if len(sys.argv) > 1:
        print(type(arg1))

main()

collections

from collections import defaultdict

WARNING; In a dictionary created with defaultdict a key will be added when you try to read a non-existing lower key. See Python:DataTypes#Dictionary or dict.

itertools

See pythondocs

Module to create various iterators. See the ranges section for the count function to create infinite ranges of integers.

traceback

See pythondocs

Module to show where things go wrong. Can be used in exception handling

bisect

Module to search lists.

To return the index of the first value higher than x.

from bisect import bisect
idx = bisect(alist, x)

Variables

Everything is an object in python. Objects can be variables and functions.

Variables are always pointers to objects.

a = 2
b = 2

Both a and b point to the same object (the immutable integer '2')

Beware making variables point to each other when it represents a mutable object.

a = [1,2,3]
b = a
a.append(4)

As both point to the same object (a list), b now also returns [1,2,3,4]

a = row[0] or "0"

Set a to 0 if row[0] has a value that evaluates to False (0, '' or None). Comes in handy for selections from databases where you expect a number but the field is empty.

Variables are local by default. If a routine has any assignment to a variable it is local. If you have defined a variable outside a routine and need assignments to it in the routine, you have to declare it global explicitly.

a = 'a string'

def main():
    global a
    print(a)
    a = "This would fail with 'local variable 'a' referenced before assignment' if 'a' was not declared as global"

main()

del <variable>

Remove a variable name. The garbage collector will release memory soon.

[Geeks for Geeks] has as good page about this.

Virtual environment

apt install python3-virtualenv

Install the virtualenv script, a nice wrapper for creating virtual environments

virtualenv --clear --always-copy -p <python binary> .venv

Create a fresh virtual environment in the current directory. Clear the existing virtual environment, copy the <pythonbinary> instead of creating a symbolic link.

This works too, can probably do the same as virtualenv but I haven't checked much of it.

apt install python3.11-venv

Install the virtual environment package

python -m venv .venv

Install the virtual environment in ./.venv with symbolic links to the python binary.

source .venv/bin/activate

Enter the virtual environment.

deactivate

Leave the virtual environment

In the virtual environment you can install and maintain python packages using pip

To update the python version itself you need to create a new virtual environment with the new python version