Python

Python https://www.python.org/ is a high level interpreter language, supporting object oriented programming, and having a lot high level libraries for all kind of applications. It is highly portable to all kinds of hardware devices and operating systems. Python scripts are converted to byte code that runs on a virtual machine. Additionally there is IronPython that runs on top of .NET (or Mono) or Microsofts python for .NET and Jython that runs on Java virtual machine. There are ten-thousands of python packages available at https://pypi.python.org/pypi

Extending python is connecting C routines to python. The other way around is also possible, embedding python is integrating the python interpreter in a c program. And have this run the python script.

Python scripts have usually the extension py but can also have no extension at all, since under Linux the magic first line

#!/usr/bin/python

tells exactly what to do when this file has to be executed, start the interpreter found under /usr/bin/python

The first thing to know when dealing with python scripts is that to group commands no {} brackets are used, everything is done with indention.

Warning

Be consequent and do not use tab characters but use space characters for the indent, so when using different editors no mess gets created.

Avoid long sequences of nested code segments, use functions and objects to encapsulate.

As general advice, if the code is getting large and nested, then its is time to learn a bit more about python since using the right methods (including, dictionaries, sets, classes, libraries) surprisingly compact code can be written.

Python2 and Python3

The file /usr/bin/python is probably a link to a command as /usr/bin/python2 or /usr/bin/python3 because different version of python might be installed on the computer. It selects therefore the default python to be used. To be more specific the magic line could be

#!/usr/bin/python3

to select python3

Note

Today still many python scripts are written for python2 that has incompatibilities with python3. So a python3 script will not run with a python2 interpreter and vice a versa.

Since python2 is no more actively developed, this book concentrates on python3.

If the computer is setup for python3 it fails when starting a python2 script. Therefore /usr/bin/python2 <script name> will force to use python2

A script 2to3 is installed that can automatically convert python2 scripts to python3 see: https://docs.python.org/2/library/2to3.html. Running 2to3<myscript>.py it will analyze the script and prints the changes to the screen. To rename the python2 script to <myscript>.py.bak and create a converted python3 script run it as 2to3 -w<myscript>.py. If python 2 is set as the active python interpreter run it as python3<myscript>.py. Note that python3 is a link to something as python3.1

Or set the magic line first line to choose the right version:

#!/usr/bin/python3

2to3 makes use of fixers that can be listed with 2to3 -l. Not all are active per default and need to be enabled with 2to3 -f<name of fixer> if desired.

The 2to3 works well on simple scripts, where the keyword print gets converted to the function call print(), but the troubles come with the libraries and file access.

To test python open a console and type python. Now you can type in commands and see what is going on.

On Getnoo Linux eselect python list will show what python version is the default and what are installed

How python finds its packages

Different python versions are probably installed on the same computer so the question arises what packages are available and taken. A way to see it is starting

python

Python 3.4.3 (default, Oct 30 2016, 21:08:25) [GCC 4.9.3] on linux Type "help", "copyright", "credits" or "license" for more information.

>>> from distutils.sysconfig import get_python_lib

>>> print(get_python_lib())

/usr/lib/python3.4/site-packages

User installed packages end up as ~/.local/lib64/python3.5/site-packages

Python documentation

Python documentation is available under https://docs.python.org/ or /usr/share/doc/python-docs-<xxx>/html/index.html http://www.onlamp.com/python/

There is a python doc server (pydoc server) for OpenRC add it to the start

/etc/init.d/pydoc-<version> start

The port is PYDOC_PORT=7464 and can be viewed:

http://localhost:7464/

The stuff comes from

/usr/share/doc/python-docs-2.5.1/html/index.htm

On trouble run it from a console /usr/bin/pydoc3.6 -b

Python IDEs

The following IDE's are available:

  1. The standard IDE (Integrated Development Environment) that comes with Python is idle:

    It requires tcl and tk support to run.

    Note

    For Gentoo set the tcl and tk useflags and then re-emerge it emerge -pv python

    idle is pretty basic but it might be the only IDE available (or working). To pass command line parameters idle -r<name of python script><command line parameter>

    For a specific python version start idle3.3

    Debugging is possible using its debug window, where the source can be viewed and all the variables. In the source, breakpoints can be set to lines.

    Important

    With the debug window open it debugs when run, with the debug window closed run runs.

    Click the Source checkbox to see single step and where the program is. For the default setting single step is a dark gray line and brake point is yellow. Not both colors can be seen at the same time so the dark gray line disappears when jumping to a yellow line. So if there is no expected dark gray line press step or out.

    It has individual windows, but when arranging them it looks like a real (but un-bloated) IDE

    Figure 15.1. idle

    Idle


  2. pycharm with its community edition gets installed under gentoo in /opt/pycharm-community/bin/pycharm.sh

  3. Microsoft Visual Studio Code is open source and runs also under Linux. It can use a already installed python interpreter. As regular used the tar.gz file can be downloaded and unpacked and ./code be run.

  4. eric7 is a nice complete IDE for python. It runs in a python virtual environment and might therefore have some issues with the none virtual environment.

  5. Eclipse has a plugin, add the link http://www.pydev.org/updates/to get the pydev plugin. Then go to window -> preferences and set the python interpreter as /usr/bin/python3

Embedded python debugging

Often python script are called by a external process, this might be a gui, and other python script or every thing else. The python script will then be started in a environment with additional environmental variable set, other command line parameters passed and maybe other processes or threads started in parallel. This environment is different from the one that would exist when the python script would have been started simply in a console.

Therefore you need a debugger with embedded debugging support as winpdb. Add the following line into the file that you would like to debug

import rpdb2; rpdb2.start_embedded_debugger_interactive_password() 

Then start the program using the python script as usual, however this time the python script will prompt for a password and wait for an external debugger. Start winpdb and go to file => password and then file => attach where the python script pop's up and can be selected. If you have problems with the interactive password you can use fix password as:

import rpdb2; 
rpdb2.start_embedded_debugger('hello')

Hint for winpdb: It shows at the current line capital letters that have the following meaning

C

CALL

A function is called (or some other code block entered)

L

LINE

The interpreter is about to execute a new line of code (sometimes multiple line events on one line exist)

R

RETURN

A function (or other code block) is about to return

E

EXCEPTION

An exception has occurred

*

RUNNING

The thread is still running (probably blocked in C code)

Python Programming

Variables are objects

Inside a python module there are global and local variables. However a child module has its own global name space and can not access global variables of the parent module. The parent module however can access the global variables of the child module by simply adding the child's module name as prefix (if imported this way)

Variables and almost everything in python are objects. Objects are instances of classes. Objects have an identifier, a name (that is optional) and a value.

Some objects as integers have immutable values and result in a not obvious behavior. This is best explained by the following:

>>> a=1
>>> b=1
>>> id(a)
134578352
>>> id(b)
134578352
>>> id(1)
134578352

To the object with the value 1 the name a gets assigned. After that the object with the name a and the value gets an additional name b. This can be verified by checking the id of the object using the names. The identifier can also be observed by passing the value to the id() command. Assigning the name b to an other value (=object) deletes the name b of the previous object.

To see that a is an object you can call the class add method of a:

>>> a.__add__(2)
3

This the same as a +2 that results in 3. In fact python converts the + sign to the class method __add__ , but luckily hides this to human.

Luckily there are objects with mutable values as lists, they behave as expected:

>>> a=[1]
>>> id(a)
3075342668L
>>> b=[1]
>>> id(b)
3074937644L

For mutable objects, each value assigned to a name creates a new object. To get the value of the first (and here only) value of the list you can either use the nice looking command using brackets or the class __getitem__ method:

>>> a[0]
1
>>> a.__getitem__(0)
1

However the following shows a common pitfall. The simple command b=a behaves differently, the list just gets a second name. To make a copy, a slice out of the list a needs to be done. The syntax for it is [<start index>:<end index>]. If the indexes are omitted as in [:] then a slice from the very beginning to the very end is done.

>>> a=[1]
>>> b=a
>>> id(a)
3074516172L
>>> id(b)
3074516172L
>>> c=a[:]
>>> id(c)
3074916588L

Python data types

Python data types or more precisely data classes can be grouped in mutable and immutable. Additionally there are sequences. Some sequences are mutable (as lists) and some immutable (as strings, tuples).

To use the wording of object oriented programming, if a data class gets used, it becomes an data object as instance of the data class.

Explaining in simple words, it is sometimes not obvious that the data is immutable:

>>> a="hello"
>>> id(a)
3073624256L
>>> a=a[1:3] 
>>> a
'el'
>>> id(a)
3073538048L

It seems that the immutable string a has been changed (even being immutable). However as the id() shows a new variable with the name a has been created.

With mutable data types the behavior is different:

>>> list=['h','e','l','l','o']
>>> id(list)
3074023660L
>>> del list[2]
>>> id(list)
3074023660L
>>> print list 
['h', 'e', 'l', 'o']

Using the python data types is a bit confusing since some commands are consistent among the different data types others not. To understand the logic behind, the different and many data types need to be understand in very detail.

There is a reason way different data types exist!

Strings have a lot of functions (or in term of object oriented programming) methods.

Table 15.2. String methods

s.strip() Removes blank (or character passed) characters from the beginning and end
s.split("/") Using the character passed, splits the string in substring and returns list of substrings
s.endwithsuffix("/") Returns true if the string ends with the character passed


List and tuples can have different type of variables inside. C has arrays and structures for that.To add a new element to a list do a+=[i].

Python strings and bytes

One big change between python2 and Python3 are the way strings are handled.

There are strings using unicode utf-8 and bytes (byte arrays).

To convert strings to bytes character encoding and decoding is required. Bytes can be declared with b'<here are bytes>' whereas a string is '<this is my unicode string>'.

Create bytes from strings is encoding strings and is done <stringofbytes>=<utf8string>.encode()

Creating strings from bytes is decoding bytes and is done <utf8string>=<stringofbytes>.decode()

Bytes arrays can contain any byte value or uft8 sequence. they can appear as follows:

x = b'El ni\xc3\xb1o'
s = x.decode()
print(s)

To convert a number into bytes

b=chr(128).encode()

This has also impact how files are handled. Files can be read as text (strings) or binary data (bytes).

Sequences

You can also get elements in a sequence counting from the back a[-1] gives the last element in a. Mathematically speaking the length len(a) is added to negative numbers.

>>> a="hello"
>>> a[0]
'h'
>>> a[-1]
'o'

The .count(<listelement>) method returns how many times the list element is in the list, so don't loop lists for that.

Mathematical expressions

2**8 is power 8 of 2 and so 256. In C, after including the math library, it would be pow(2,8)

>>> 2**8
256

Python can deal with complex numbers:

>>> a=2j
>>> a+(1+1j)
(1+3j)

Range checking gets more readable instead of C like (a>min)&&(a<max) it gets

>>> max=5
>>> min=2
>>> a=3
>>> min<a<max
True

Conditions

Every different from 0 is considered as True. this explains the following:

if a:
  pass
if len(a):
  pass

Instead of if: else: if: else: if: else: use if: elif: elif: else:, since no switch case as in C exists.

Important

As with every programming language, do not compare if two float variables are the same. Use math.isclose for that:

import math
a=0.1
b=0.2
print(a+b==0.3)              # gives false
print(math.isclose(a+b,0.3)) # gives true
format(a+b,".21g")           # shows the reason      
format(0.3,".21g")           # shows the reason

Loops

A count variable is not required in the for loop to access elements in a sequence (notice the indention):

>>> s=[0,2,4,6]
>>> for p in s:
...   p
... 
0
2
4
6

Inside the for loop do not modify s. If you want to do it use a while loop. As example:

>>> s=[0,2,4,6]
>>> while len(s)>0:
...   print s
...   s.pop(0)
... 
[0, 2, 4, 6]
0
[2, 4, 6]
2
[4, 6]
4
[6]
6

Without argument pop defaults to -1 and takes the last list member.element.

If you still want to use a count variable, you could use a while loop, but python stile uses the range command that creates a sequence that is passed to the for loop:

>>> for p in range(3):
...   p
... 
0
1
2

The range command can also do more as counting back and create more complex sequences.

Functions

Functions can return also more complex data as lists.

Functions can alternatively be called with <parametername>=<value> instead of just the values in correct sequence separated by commas. This looks like more writing but is safer and nicer code since it is independent of the sequence and also makes sure the parameter are passed in a consistent manner. To have good documented code it is also recommend to put every parameter to a new line and add a comment using the # character on every line.

*<parameter> means that the <parameter> can occur multiple times. It can be accessed using <parameter>[].

<parameter>=5 assigns a default value, this allows to call the function without passing any parameters.

Lambda

A lambda function just returns that it is a function

lambda x: x*x

it needs to be called with a (or multiple) parameter(s)

(lambda x: x*x)(2)

x is just the name of a place holder, any other name is ok.

A lambda function can also have a name and then used as a regular function.

lambda_add= lambda x,y: x+y
lambda_add(2,1)

A better use of lambda is to use it with iterables as lists.

An example is sorting a list containing lists. The second element in every list is used for the sorting.

element=1
two_dimensional_list = [
    [3, 7, 1],
    [2, 9, 4],
    [8, 5, 6]
]
two_dimensional_list.sort(key=lambda x: x[element])

The sort method has the parameter key that can hold a function. This function is applied to each element in the list. This creates a temporary list of results and sort is applied to this result list. The sorted result list is then used to sort the two_dimensional_list.

Without lambda a function would have to be written that passes the element back. The variable element must be globally to be known by the function, this is really not how clean programming should be.

two_dimensional_list = [
    [3, 7, 1],
    [2, 9, 4],
    [8, 5, 6]
]

def get_element(x):
  return x[element]

element=1
two_dimensional_list.sort(key=get_element)

Importing modules and packages

Modules are usually single py files. Packages contain one or many modules. Packages are py files inside a subdirectory and have an __init__.py file.

To use a module that is in the same directory. Note don't use - character in py file names.

import <module name> 

To import all modules from a package (This can create tons of unnecessary global variables and naming conflict)

from os import * 

So better

import os

And putting os in front of the module name to make it clear to where it belongs.

Python modules are usually put under /usr/lib/python<pythonversion>/site-packages but when developing a more flexible way is desired than creating and installing packages. The modules can be anywhere as long python finds it.

With the command export PYTHONPATH=/<here is my direcory>/ ; $PYTHONPATH the search paths for python can be expanded.

To not have the hassle with the PYTHONPATH

import sys

print(sys.path)

shows where python is looking for modules

Important

Modules in the sys.path will be found modules elsewhere not and cause an import error. The first path in sys.path is usually ' ' this means the current working dir

import os

print(os.getcwd())

Depending how the python script is started the cwd might be different and the python script fails (as when debugging in IDLE) to not have such effects add the missing path to sys.path using sys.path.append(<missing path>)

Note

__file__ holds the started script.

To test a module

python

import <module>

<module>.<function>

To test a package

python

import <package>

<package>.<module>.<function>

When writing a module, it is desired to run the module on its own, reasons are to debug the module or to have even a standalone utility program. Adding the following, makes starting the code just when calling the module directly with python on the command line. In this case the hidden variable __name__ contains __main__.

if __name__ == '__main__':
  <here comes my test code calling the stuff above> 

Instead of writing everything yourself look at https://pypi.python.org/pypi. There is also a tutorial about submitting and installing packages: https://wiki.python.org/moin/CheeseShopTutorial

Exceptions

A good python program should not crash. However sometimes it is not known in advance if a crash might happen. Examples are input from the user for a number, or parsing an xml file.

An other approach is to catch the exception and act afterwards. To initiate this add the python key word try: then the critical code. After that there is the choice for one of the two key words:

  1. except: This key word initiates the commands that are executed when the commands after try fail. Optionally the error can be added to except as except TypeError: to act just on type errors. This way multiple except's can follow a try:

  2. finally: This key word works as except:, except that this code is executed also when try: worked trouble free.

Print and formatting strings

In python 2 print was a keyword but in python 3 it is a build in function, therefore it has print() brackets.

To not end with new line characters tell print to no insert them

print("No new line after here", end='') 

There are different ways how to format what print is doing. If nothing print prints out the variables and adds a space in between. The variables can also be converted to a string using str() or repr().

Finally there are different format string methods (see https://pyformat.info/).

Don't care how it will be formatted:

print(a, b)   

Putting it together

print(str(a)+" "+str(b))

Old style (C like) using the % character

s = "Hello, %s! You are born %d" % ("Urs", 1962)
print(s)

using rep

print(repr(a).rjust(6),  repr(b).rjust(6))

using the .format method.

print('{0:6d} {1:6.2f}'.format(a, b))

The modern way is using {}

{} place holder for a string

{:d} decimal number

{:x} hex number

{:03x} hex number with 3 nibbles and printed preceding zeros

With Statement

When accessing external resources as a file runtime errors could happen since the file might not be there. A way out is considering this using try and programming the cases. As simpler way is using with:

with open("x.txt") as f:
    data = f.read()
    do something with data

open returns f and f can the be used on success.

For more details see https://effbot.org/zone/python-with-statement.htm).

Logger

It is common to print out information on the screen to know what is going on and avoid using a debugger. If the program gets stable it is desired to turn that off and maybe write log information in files somewhere under /var/log where having write permission.

#!/usr/bin/python3
import logging

logging.basicConfig(level=logging.INFO)

logging.debug("Debug")
logging.info("Info")
logging.warning("Warning")
logging.error("Error")
logging.critical("Critical")

allows this and brings many features as having different log levels and allow to adjust what will appear.

except ZeroDivisionError:
    logging.error("DonutCalculationError", exc_info=True)

will also log the execution information

logging.basicConfig allows to format and adjust the logging

logging.basicConfig(
      format="{asctime} - {name} - {levelname} - {message}",
      style="{",
      datefmt="%Y-%m-%d %H:%M",
      level=logging.DEBUG,
    )

Adds a time stamp, adds the name of the module.

logging.basicConfig(
   filename="app.log",
      encoding="utf-8",
      filemode="a",
      format="{asctime} - {name} - {levelname} - {message}",
      style="{",
      datefmt="%Y-%m-%d %H:%M",
      level=logging.DEBUG,
    )

Writes it into a file instead of printing to the screen.

Multiple Loggers

A bigger project might want to log differently and have loggers per module. This is done by creating a logger instance and defining and adding handlers.

import logging
logger = logging.getLogger(__name__)
stream_handler = logging.StreamHandler()
file_handler = logging.FileHandler("app.log", mode="a", encoding="utf-8") 
logger.addHandler(stream_handler)
logger.addHandler(file_handler)
logger.handlers

__name__ will give the logger the name of the module.

logger will show the logger instance and logger.parent the root logger and its log level. logging.basicConfig can just be applied to the root logger.

The StreamHandler will print to the screen and the FileHandler to a file. There are many other handlers available as ones that send emails.

To format a handler, a formatter needs to be defined and then added to the handler. This way different handlers can be formatted differently

formatter = logging.Formatter(
    "{asctime} - {levelname} - {message}",
    style="{",
    datefmt="%Y-%m-%d %H:%M",
)
stream_handler.setFormatter(formatter)
logger.setLevel("INFO") 

will set the log levels for all handlers of the logger

stream_handler.setLevel("DEBUG")

just for one handler

Threads

Threads are as processes and allow to run in parallel, with the exception that threads use the same memory space. As example it thread1 writes to a global variable thread2 can read it. A python script not creating threads is a single thread script.

In the simples case threads are started as follows

import threading

t =threading.Thread(target=<function name>)
t.start()

More advanced threads are started by defining a new subclass of the Thread class, then override the __init__(self [,args]) method to add additional arguments. Then, override the run(self [,args]) method to implement what the thread should do when started.

pass

The word pass is a python keyword and does not do anything. It is used to not cause a syntax error in the above code. Since a breakpoint can be set to pass it is quite helpful during program development. pass can be put as place holder for code that needs to be written.

Executing textural expressions

Since it is an interpreter language you can execute what is written in a string variable using the exec command.

>>> exec "pi=3.1415"
>>> pi
3.1415

Console

To read from the console raw_input() can be used that returns always a string. However input() could also be used that returns an object based on the input a string, and integer, ... . This looks like more coding work, but using the command type() can be used to check if the input was correct or try: can be used to just be optimistic and do and if it fails except TypeError: can be used to react.

Interrupting a running program can be done with a keyboard interrupt

while True:
      time.sleep(0.05) # be nice  
      try:  
        pass
        # some code here
      except KeyboardInterrupt: 
        pass
        # crtl+C brings you here

Handling command line parameters in python

Almost every program must deal with command line parameters. Therefore python offers modules for that. The module argparse should be used now. It replaced optparse that replaced the getopt module. Some python statements are required to let the program known with what command line parameter it has to deal. Those parameters will then be easily available in the options structure. Left overs that will not be parsed are called positional arguments and are arguments that usually are absolute necessary to run (as a filename).

Python logging

Instead of printing out information to track what the program is doing python offers a logger, see https://docs.python.org/3/howto/logging.html. After

import logging

a logging call looks similar as a print:

logging.info("This variable will be logged: "+i)

The logger can be configured. Levels can be set to control how much is printed. The levels are: DEBUG, INFO, WARNING, ERROR and CRITICAL and the outputs can be written in a file instead of the console:

logging.basicConfig(level=logging.INFO, filename='log.log')

Gui programming

Python programs with Graphical User Interfaces (GUI) make use of the following graphical libraries that give the applications different look:

  1. pyQT based on QT

  2. http://wxpython.org/ toolkit based on http://wxWidgets.org/ There is Boaconstructor a RAD (rapid development) GUI tool for it.

  3. http://pygtk.org/ that is the gtk=gimp tool kit. Glade is a tool to draw the GUI as xml file that can then be used by GtkBuilder to interface to different programming languages as python.

    anjuta is the standard ide for gnome and gnome is based on gtk. In anjuta a new project can be created for python pyGtk. It has dependencies to rope and glade to have support for it working. This creates a working hello world gui application using python. The gui itself is an xml file with the extension ui that is in the data subdirectory. The glade support in anjuta is responsible to edit this xml file using a gui and not a text editor and integrate it to the anjuta file. Glade itself would run as standalone program to edit this ui xml file, however a consistency mess with anjuta would then be probable.

  4. Tkinter is an interface to tk (tool kit).

    Tkinter comes with python and needs no installation, but looks a bit basic. Using ttk (Tk themed widgets) to look can be improved. An advantage is that it is stable, has not many dependencies and is portable.

Tkinter

Tk is not python, however python comes with Tkinter that connects python to Tk.

The result is that not all data that Tk has will be visible during debugging python.

Working with Tkinter is therefore not as working with pure python.

Links to documentation: https://tkdocs.com/tutorial/index.html

Nowadays tinker gets used together with ttk (Tk themed widgets).

ttk comes with a set of widgets allowing to set themes with the style= attribute.

matplotlib

Matplotlib for plotting and numpy for numerical calculation bring functionality as known from Matlab into python. Matplotlib uses a frontend/backend approach to output the plots. This allows to use different backends to put the results into files having different formats or embed in various GUI canvasses as tkinter, gtk, qt, qt4.

Matplotlib commonly uses numpy arrays but for simple things without a lot of computing as reading results from an instrument, python lists instead of numpy arrays can be used.

Note

For python3 matplotlib >=1.2 must be installed

Type Hints

Python is very open regarding data types, this might be ok for fast tests, but is a source of programming errors and lack of documentation. Type hint has been introduced to improve this.

For variable declaration:

age: int = 30 

For function and methods:

def hello_name(name: str) -> str:

or if nothing gets returned

def hello_name(name: str) -> None:

During debugging the debugger shows types that can be used for the type hints.

To have a static type checker install mypy and use it as mypy <script>.py

On most IDE's as pycharm mypy can be added as external tool. In pycharm add the variables $FilePath$ as argument and $ProjectFileDir$ for the working directory, so mypy gets called with the file opened in the editor. Add the argument --disallow-untyped-defs to get errors when type hints are missing.

See mypy -h for more options

Installing python packages

Pip

The standard way of installing python package is pip and using https://pypi.org/ and https://pip.pypa.io/en/stable/user_guide/

Important

However installing packages without the Linux package manager being involved is not advised.

Avoid using pip as root since it could impact python scripts that are used by the system (especially Gentoo Linux since all system program as emerge are written in python). Use it as user and install the python package under the users home directory. pip install --user <package> or pip install --user --break-system-packages <package> If installing as --user will not break the system so in this case do not worry about --break-system-packages

With --user and not running as root, the package goes in somewhere as ~/.local/lib or ~/.local/lib64 where the python interpreter will find it.

A way out of this is using a virtual environment

pip is also aware on what it did and what it can do. Commands as:

pip list and pip list --outdated

pip search <name>

pip uninstall <name>

pip install --upgrade <name>

pip install --user pymodbus==2.5.3

pip install --trusted-host files.pythonhosted.org <name> when getting error

pip requires connection to the Internet. If this can not be done download the *.whl or *.tar.gz package from https://pypi.org Some packages can have compiled binary code. For those packages multiple *.whl exist select the one matching your hardware.

then run

pip install --no-index --find-links=<local_folder><package filename name>

errors might appear form missing dependent packages

See https://thilinamad.medium.com/install-python-packages-via-pip-without-an-internet-connection-b3dee83b4c2d

pip uninstall <name>

Installing a python package

In Linux such packages should be installed as usual using the package manager (for Gentoo the emerge command) to no create a mess with the installed package database. The python files will be installed to /usr/lib64/python<m>.<n>/site-packages/<package name>

Additionally specially for Gentoo Linux there might be different python versions installed (see <m>.<n>) so it is good the package manager takes care about this. Otherwise python might be unable to find the package for the simple reason: Mismatch between python and package version numbers.

Python looks in the sys.path that can be seen by

python Python 3.5.4 (default, Jan 7 2018, 19:27:36) [GCC 6.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', '/usr/lib64/python35.zip', '/usr/lib64/python3.5', '/usr/lib64/python3.5/plat-linux', '/usr/lib64/python3.5/lib-dynload', '/usr/lib64/python3.5/site-packages'] >>>

Distutils creates further a <name>-<version>-<python version>.egg-info file (or directory) that contains information about the package. There is also a __pycache__ directory that holds the compiled and optimized bytecode files for the py file.

Using a python package

To use an installed python package it must be imported. To know how and what to import it must be known how it got installed.

Files as /usr/lib64/python<m>.<n>/site-packages/<name>.py can be imported as

import <name> 

It is common, that a package is installed into a directory under /usr/lib64/python<m>.<n>/site-packages/ and this directory holds different python modules (files). Such a module (file) /usr/lib64/python<m>.<n>/site-packages/<package name>/<name>.py can be imported as

from <package-name>  import <module-name>

Documenting Python

See https://realpython.com/documenting-python-code/

Doxygen supports python as well. The docstrings are considered directly by doxygen without the need to specially mark them.

Python has also a feature to attach a docstring to functions. After the function header a line starting by """ will open the docstring that can contain multiple lines and needs no special characters at the beginning of each line. The docstring is terminated by a second line starting with """. Such docstrings can be visible in the python interacting mode by the help() function.

The python module (file, main program) can hold a doxygen command as follows:

"""@package docstring
<Documentation>
"""

Alternatively for doxygen the # character used for comments can be takes as well.

Docstrings

objects have the __doc__ property that holds the docstring and can be made visible in the interpreter via help().

This means that the docstring must be placed after the object definition. Docstrings are placed within triple-double quotes (""")

Docstrings are therefore not comments (using #)

Docstrings can/should be formatted as reStructuredText

Testing Python code

Unittest

With unittest a program (test.py) can be made that calls the python program in various ways and compares the results against what is expected.

#!/usr/bin/python3
import unittest

class MyTest(unittest.TestCase):
      
    def test_default(self):
        <code>
        self.assertEqual(a,b)   
    
    def test_other(self):
        <code>
        self.assertEqual(a,b)      
      
if __name__ == '__main__':
    unittest.main()

Different test_<name> functions can be added that contain an assert<what> method.

The program test.py can then be called as

python -m unittest -v test.py

python test.py -v

test.py -v since it contains #!/usr/bin/python3

test.py MyTest -v to call the MyTest test cases

test.py MyTest.test_default_bin -v to just call a specific test case (for example the one causing troubles)

Code coverage

A good sign of code quality is if test run all the code. The program coverage can be used instead of the program python.

coverage run <program>.py

coverage report to get a text report

coverage html then open htmlcov/index.html to see a html report

coverage -h to get help

coverage does not check what get started and run in other processes

Python in virtual environments

All dependencies required to run a python script can be put into a virtual environment and therefore do not conflict with the systems python installation, dependencies, modules and versions. The virtual environment uses then a python interpreter with a specific version. Just packages installed in the virtual environment are available.

To create a virtual environment create a directory with a name. If used system wide and/or for different users then mkdir /srv/<my virtual environment> then change to it

cd /srv/<my virtual environment> and create the virtual environment

python -m venv .

or with a specific python version:

python3.10 -m venv .

-m calls the module venv that does the job of creating a virtual python environment

. means installation goes in current directory, add a path for an other directory I

Inside the directory run source bin/activate to enter

pip install <mypackage> to install inside the virtual environment

To exit the virtual environment, run: deactivate

Using package virtualenv

The package virtualenv https://virtualenv.pypa.io/en/latest/ needs to be installed, does the same thing as -m venv and more as example it supports also also python versions before python3.3.

For gentoo emerge virtualenv

https://www.youtube.com/watch?v=N5vscPTWKOk

Create a directory, preferably where all virtual environments go.

Then cd to this directory and create the virtual environment virtualenv <name of the virtual environment>

To enter into the virtual environment do source <path>/bin/activate

To verify being in the virtual environment:

which python

which pip

pip list

Installation is done as pip install <name of packages>

To get out of it deactivate

Python on Gentoo Linux

More than one python version is probably installed on your computer. In Gentoo Linux the command eselect python list shows what is installed and what is taken as default, first you must emerge eselect-python.

To switch to a particular version do eselect python set<n>

Which python version is active is set by links in /usr/bin

Since Gentoo allows to have more than one python version installed and usable at the same time (like version 2.7 and its new incompatible version 3.4), there needs to be some setup that this works. When installing the Gentoo-way a python scripts they end up under /usr/bin however a closer look shows there is just a link to a wrapper script ../lib/python-exec/python-exec2 inside ../lib/python-exec/ there are other directories as python3.3 that contains all scripts.

The wrapper python-exec hat its configuration file /etc/python-exec/python-exec.conf where global preferences are set. Obviously the preferred python version must be installed, otherwise gentoo tools as emerge will pop up errors.

The libraries are under /usr/lib/python<pythonversion> where additional installed packages are in /usr/lib/python<pythonversion>/site-packages. When updating and removing an old python version, it can happen that the old directory /usr/lib/python<pythonversion>/site-packages remains, due to unknown garbage inside, so take a look and delete it.

See https://wiki.gentoo.org/wiki/Project:Python

Python on Windows

Python is portable to many different CPU architectures and operating systems, however it runs only if a python interpreter is installed. On a Linux machines as Gentoo, python can be assumed to be installed but on a Windows machine not.

On Windows the python scripts the extension can be renamed from py to pyw to hide the black console window when graphical python scripts are executed. The interpreter pythonw.exe is then used instead of python.exe

As alternative there is Jython (the successor of JPython) http://www.jython.org/, this is python 2.x running on top of a Java virtual machine as java byte code. To use jython type jython. Jython with all its libraries can be installed as java -jar jython_installer-<version>.jar --console or if you like to risk a gui headache java -jar jython_installer-<version>.jar

Install Python on Windows

After installation, python can be started with

  1. IDLE console => python my.py

  2. console => my.py

  3. double click in file manager (but console applications disappear when done)

The best way to start is using Windows menu start and start IDLE. Then open a python script <name>.py and run it as module F5

It might be installed in a user directory and therefore just available for a single user as c:\Users\<username>\AppData\Local\Programs\Python\Python35\python.exe

Important

Make sure adding python to the path is selected otherwise python will not be found.

If the corresponding check box got not checked, then Python is not added to the windows path environmental variable so it will not be found in console window it need to be started with the complete path.

This can be fixed by adding this path to the windows environmental variables: Control Panel\System and Security\System\Advanced System Settings\Advanced\Environmental Variables

Figure 15.2. Windows Environmental Variables

Windows Environmental Variables


The shell might probably report that some modules (libraries) are not installed

The next problem is how to install those libraries, the standard way is using pip. Newer python version come with pip, on older versions pip needs to be installed first.

update pip (-m makes python know that pip is a module and python therefore knows where to look for it) is done with python -m pip install -U pip

python -m pip install --user bincopy==14.0.0 installs the missing module (bincopy) with a desired version (14.0.0)

A warning might comes that c:\Users\<username>\AppData\Roaming\Phyton\Phython35\Scripts should be set to the path this is recommended otherwise programs as pyinstaller are not found

If the installation fails due to certification errors install the certificates python -m pip install certifi

When debugging with IDLE then it can happen that imports of local modules fail. The reason is that sys.path does not contain the path of the script and the current working path is not the path where the script is. This can be fixed in the shell window of IDLE by sys.path.append('Z:\\<path of current script>'). In this example it is shown that Windows need to get two \\ characters since the first is a string escape character.

Install python packages under Windows

python -m pip list to show what is already there

python -m pip install --user <package> to do

Python with Microsoft Visual Studio Code

After installing Microsoft Visual Studio Code a python file can be opened and Visual Studio Code will ask for the python interpreter. A already installed interpreted can be choose or if not available Visual Studio Code will install one.

Visual Studio Code is a modern IDE and is much better looking as IDLE.

Microsoft Visual Studio Code is open source and runs also under Linux

Multi Platform programming

To know where the code is running use :

>>> import platform
>>> platform.system()
'Windows'

To get portable code avoid using commands as os.system("cd /home") that execute Linux commands. Use os.chdir(<path>) instead.

os.linesep instead of '\n' for Linux or '\r\n' for windows used in text files. However within python code it is recommended to use all the time '\n' on all platforms and have pythin deal with it.

Path incompatibilities

However the next problem arises the path syntax is not consistent between Windows and Linux.

Pyhon internally the '/' is used also for windows. Printing this out would cause confusion and or passing it to a windows program would cause an error.

The pathlib can be used to convert paths from String to an object. Then different methods are possible.

import pathlib     
# is an object maybe required to convert to string
s=pathlib.PureWindowsPath(<path>)
t=str(s)       # converts object to string
u=s.as_posix() # returns a string

Alternatively it can be dealt on string level using functions/constants/variables as

os.sep instead of the '\' for Windows and '/' for Linux characters. Don't be afraid seeing '\\' for Windows. '\' is used in Strings as escape character allowing to insert none printable characters. So a second '\' character needs to be inserted to indicate that it is a '\' character.

os.path

os.path.split() and os.path.join()

The os module (there are different implementations for the different os) puts an abstraction layer to it that can normalize path names, so it converts Windows forward slashes to backward slashes and have the drive name as directory name. Be aware that not all file systems understand utf-8 encoded file names.

Under Linux ~ is expanded to the users home directory /home/<username> on Windows this works as well but will be expanded to C:\Users\<username>

Note

Directories under C:\Users\<username> that start with a '.' character are an indication that they got ported from a non Windows system to Windows (as from Linux to Windows)

Lxml incompatibilities

lxml might be problematic so it is wise to include it as

# call lxml as ET so it is compatible with the python 
# built-in library xml but the library lxml has more
# features as pretty-print and xml declaration support
lxml=False
try:
   from lxml import etree as ET
   lxml=True
except:
   import xml.etree.ElementTree as ET

<some code>

if lxml==True:
   tree.write(self.sessionPath+os.sep+self.sessionFile,  
   encoding="utf-8", pretty_print=True,   
   xml_declaration=True) 
else:            
   tree.write(self.sessionPath+os.sep+self.sessionFile, 
   encoding="utf-8")    

<some code>

Pyinstaller for Windows and Linux

pyinstaller from https://pyinstaller.readthedocs.io/en/stable/ and https://readthedocs.org/projects/pyinstaller/downloads/pdf/latest/offers a nice way to create a Windows exe. To run pyinstaller Python needs to be installed under Windows.

Important

If the program has been developed under Linux first have it running under Windows to have all missing libraries and incompatibilities fixed.

pyinstaller allows converting python scripts into windows exe files not requiring to fiddle around with installing and setting up python on a windows computer. Those windows exe do not require administrator privilege to install them, since they contain usually all required code.

Yes, it is required to type into the Windows console commands as

pyinstaller <name>.spec creates the <name>.exe

The <name>.spec can be created automatically the first time by calling pyinstaller <name>.py

Important

Calling pyinstaller <name>.py will overwrites <name>.spec with default settings. Therefore rename <name>.spec with a useful name.

The <name>.exe can be found and started under dist\<name>\<name>.exe

Two ways are possible

  1. Everything packed in a single exe file, that when started extracts itself to a os temp directory as C:\Users\<username>\AppData\Local\Temp\<some name>\, then it runs from that and finally the directory gets removed. Since there is just one file it is easy to distribute it. Drawback is that it is slower to start and no files as readme.txt can be observed. Call pyinstaller --onefile <name>.py to get a default. The file <name>.spec will have a long exe section but no coll section.

  2. Directory structure that contains a small windows exe to be started and all other files as dll or text files as readme.txt.

    Having a directory structure is also better during testing and developing, since it can be easy observed what files are around and where. Regarding paths pyinstaller is a bit tricky since a boot loader starts the windows exe. Call pyinstaller <name>.py to get a default. The file <name>.spec will have a short exe section and a coll section.

It is best started in the Windows console so error messages indication missing components and file appear.

Files can be added in <name>.spec

 a = Analysis(
             datas=[('*.gif','.')], ...

The python tuple means copy it from source to destination.

There might be that pyinstaller does not detect all imports, missing imports can be added as

hiddenimports=['babel.numbers'],

There are also command line options when creating or overwriting a <name>.spec file.

--add-data '*.gif:.' copies *.gif to the bundle

--windowed hides the console window and shows just the gui window. Drawback no error and debug messages appear. Even when starting it in a console

--onefile puts everything in a <name>.exe file that when run gets first unzipped in temp folder of OS

--icon file.ico attaches a windows icon to replace the standard python icon seen in the windows file manager. it is not the icon in the window decoration of the application.

pyinstaller can also be installed on Linux (and Mac OS X, FreeBSD, Solaris and AIX) and can create single file executables using pyinstaller <name>.spec or if pyinstaller is installed locally ~/.local/bin/pyinstaller <name>.spec. This is a nice option to freeze all its dependencies to the python script and have it therefore archived and executable. The result will be different, it will a Linux binary (and not run under Windows).

Running Python scripts without installing Python

Under Windows, Python can also run from a memory stick without requiring an installation, as usual under Linux there are different ways to do this:

  1. http://portablepython.com/ comes with an exe that is an installer. There are 2.6 but also 3.2 versions available. The versions have different IDE and libraries included. The pyscripter is a full featured IDE with everything desired and more. SPE is an other ide that makes use of winpdb as debugger. py2exe and the necessary DLL's are also present. Once installed the portable python directory can be copied and moved anywhere.

  2. http://www.voidspace.org.uk/python/movpy/, unzip and go to the movpy directory and double click on movpyw to get the gui launcher. In the gui launcher the python script can be selected. There are also options to be clicked, on useful option is b that stops the console window after the script has run, this way it will be seen what the script has written to the console. IDLE is also available where the script can be debugged. Python version 2.5.1 comes with it.

  3. Convert the python script into a windows exe file using pyinstaller

Distribute python code

Python comes with distutils to have a standard way to distribute python code and therefore create a standard way to install it.

See: https://docs.python.org/3.5/distutils/

Creating a python package

To create distutils package a setup.py file has to be created that holds the information distutils requires.

For single py files this setup.py file can placed beside the py files and can look as follows:

from distutils.core import setup
setup(name='<name of the module>',
      version='<version>',
      scripts=['<name of the module, without py extension>'],
      )

For more complex program it is wise to create a package, a collection of files. So put them in a subdirectory and place a setup.py file above that directory that could look as:

from distutils.core import setup
setup(
      name="girttools",
      packages = ["girttools"],
      version="0",
      description='Tools for Gnu Infra Red Transceiver',
      author='Urs Lindegger',
      author_email='urs@linurs.org',
      url='http://www.linurs.org/girt.html',
      download_url = "http://www.linurs.org/girt/download/girttools-0.tgz",
      keywords = ["Infrared transceiver"],
      classifiers = [
        "Programming Language :: Python",
        "Programming Language :: Python :: 3",
        "Development Status :: 4 - Beta",
        "Intended Audience :: End Users/Desktop",
        "License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)",
        "Operating System :: POSIX :: Linux",
        "Topic :: Multimedia",
      ],
      long_description = """\
Gnu Infra Red Transceiver Tools
-------------------------------

Tools to setup and work with the Gnu Infra Red Transceiver that can replace all you IR remote controllers.
"""

Note

The setup function can be called with either packages this is a list of subdirectories containing the py files or py_modules a list of python files without adding the .py extension.

The long_description (or since it is python everything) can also be read out of a file:

long_description=open('README.md').read()

Valid classifiers can be found at https://pypi.org/pypi?%3Aaction=list_classifiers or https://pypi.org/classifiers/

Instead of py_modules that can list various py files packages is used containing the subdirectory that must contain a __init__.py file.

Finally when done packages can be register so other will find them https://pypi.python.org/pypi?:action=register_form. Python Package Index (“PyPI”) allows to classify your package so it can be found, therefore don't invent your own classifiers select them out of https://pypi.python.org/pypi?:action=list_classifiers

To include attentional files create a MANIFEST.in file as defined in https://docs.python.org/3.1/distutils/sourcedist.html. The MANIFEST.in file can also contain wild-carts and distutils specific commands as recursive directory functions.

Note

Adding files is not the same as having them installed. To install them to the same location as the package goes, use package_data in the setup.py file and not MANIFEST.in.

Finally MANIFET.in is read among other things defined in setup.py to create the MANIFEST file that contains all files

setup.py can also contain data_files that can be used to install files in any directory. Due to specifying the path the *.tar.gz package will no more be portable to different systems. An example would be installing a Linux init script to /etc/init.d, however on a Linux system using systemd instead of init or even on a Windows machine this makes not much sense to install it. It is better to do this on the system specific package as for Gentoo Linux in the ebuild. However you simply add them to the *.tar.gz using the MANIFEST.in file.

python setup.py check will check your setup.py

To get the *.tar.gz package in the subdirectory dist run python setup.py sdist (source distribution)

For windows an graphical windows installer can be created: python setup.py bdist_wininst

Note

setuptools is an enhanced alternative to distutils. Nice is that also setup tools uses a setup.py file. So Building the package is also done with python setup.py sdist

After having the tar.gz it could but should not be install it as : python setup.py install

Instead of that a package should be created that fits the Linux distribution and then be used with the package manager.

Note

__init__.py indicates that all files in the same directory belong to a package and the directory is the name of the package.

Alternatively distutils allows to install the package to an other directory with python setup.py install --home=~ any other directory with python setup.py install --home=<dir>

https://docs.python.org/3.1/distutils/builtdist.html helps to move packages to distributions and supports RPM red hat packages.

Gentoo python ebuild

For Gentoo Linux an ebuild needs to be developed that first installs the package in a sandbox and then copied (merged) to your system getting track what files will be added and check collisions.

Ebuilds can make usage of eclasses and there is a distutils eclass (https://wiki.gentoo.org/wiki/Project:Python/distutils-r1), so the ebuild for a python module can be very simple as:

 Copyright 2024 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=8

PYTHON_COMPAT=( python3_{10..13} )

DISTUTILS_USE_PEP517=setuptools
inherit distutils-r1

DESCRIPTION="Library for all sort of python code"
HOMEPAGE="https://www.linurs.org"
SRC_URI="http://www.linurs.org/download/${P}.tar.gz"

LICENSE="GPL-2"
SLOT="0"
KEYWORDS="~amd64 ~arm64 ~x86"

RDEPEND="dev-python/setuptools"  # Dependencies at runtime
DEPEND="${RDEPEND}"  # Dependencies at build time

src_install() {
distutils-r1_src_install
}

${P} is the filename of the ebuild including version number.

The inherit function loads distutils and this overwrites the standard install routines. So the ebuild contains no code just meta data about the python package and where to find it on the Internet remains.

The README.txt file gets zipped under /usr/share/doc/${P}/

Accessing C code

There is SWIG http://www.swig.org/ that creates wrapper code around the C or C++ code. This wrapper code can then be accessed by Python.

Or using cython: http://cython.org/

Python byte code

When a python code is started, then the source is converted into bytecode https://docs.python.org/release/2.5.2/lib/bytecodes.html and a virtual machine then executes this bytecode. When the same program is started again then the bytecode is started (except when the source has changed, then a new byte code is created). The files containing byte code have the extension pyc (or when optimized pyo). Be aware that the bytecode might be incompatible between different releases of python. Python has support for disassembling byte code:

>>> import dis
>>> def myfunc(alist):
>>>   return len(alist)
>>> dis.dis(myfunc)
  2           0 LOAD_GLOBAL              0 (len)
              3 LOAD_FAST                0 (alist)
              6 CALL_FUNCTION            1
              9 RETURN_VALUE        

Linurs startpage