Inside a python module there are global and local variables. However a child module has its own global name space and can not access global variables of the parent module. The parent module however can access the global variables of the child module by simply adding the child's module name as prefix (if imported this way)

Variables and almost everything in python are objects. Objects are instances of classes. Objects have an identifier, a name (that is optional) and a value.

Some objects as integers have immutable values and result in a not obvious behavior. This is best explained by the following:

>>> a=1
>>> b=1
>>> id(a)
134578352
>>> id(b)
134578352
>>> id(1)
134578352

To the object with the value 1 the name a gets assigned. After that the object with the name a and the value gets an additional name b. This can be verified by checking the id of the object using the names. The identifier can also be observed by passing the value to the id() command. Assigning the name b to an other value (=object) deletes the name b of the previous object.

To see that a is an object you can call the class add method of a:

>>> a.__add__(2)
3

This the same as a +2 that results in 3. In fact python converts the + sign to the class method __add__ , but luckily hides this to human.

Luckily there are objects with mutable values as lists, they behave as expected:

>>> a=[1]
>>> id(a)
3075342668L
>>> b=[1]
>>> id(b)
3074937644L

For mutable objects, each value assigned to a name creates a new object. To get the value of the first (and here only) value of the list you can either use the nice looking command using brackets or the class __getitem__ method:

>>> a[0]
1
>>> a.__getitem__(0)
1

However the following shows a common pitfall. The simple command b=a behaves differently, the list just gets a second name. To make a copy, a slice out of the list a needs to be done. The syntax for it is [<start index>:<end index>]. If the indexes are omitted as in [:] then a slice from the very beginning to the very end is done.

>>> a=[1]
>>> b=a
>>> id(a)
3074516172L
>>> id(b)
3074516172L
>>> c=a[:]
>>> id(c)
3074916588L

Python data types or more precisely data classes can be grouped in mutable and immutable. Additionally there are sequences. Some sequences are mutable (as lists) and some immutable (as strings, tuples).

To use the wording of object oriented programming, if a data class gets used, it becomes an data object as instance of the data class.

Explaining in simple words, it is sometimes not obvious that the data is immutable:

>>> a="hello"
>>> id(a)
3073624256L
>>> a=a[1:3] 
>>> a
'el'
>>> id(a)
3073538048L

It seems that the immutable string a has been changed (even being immutable). However as the id() shows a new variable with the name a has been created.

With mutable data types the behavior is different:

>>> list=['h','e','l','l','o']
>>> id(list)
3074023660L
>>> del list[2]
>>> id(list)
3074023660L
>>> print list 
['h', 'e', 'l', 'o']

Using the python data types is a bit confusing since some commands are consistent among the different data types others not. To understand the logic behind, the different and many data types need to be understand in very detail.

There is a reason way different data types exist!

Strings have a lot of functions (or in term of object oriented programming) methods.

List and tuples can have different type of variables inside. C has arrays and structures for that.To add a new element to a list do a+=[i].

One big change between python2 and Python3 are the way strings are handled.

There are strings using unicode utf-8 and bytes (byte arrays).

To convert strings to bytes character encoding and decoding is required. Bytes can be declared with b'<here are bytes>' whereas a string is '<this is my unicode string>'.

Create bytes from strings is encoding strings and is done <stringofbytes>=<utf8string>.encode()

Creating strings from bytes is decoding bytes and is done <utf8string>=<stringofbytes>.decode()

Bytes arrays can contain any byte value or uft8 sequence. they can appear as follows:

x = b'El ni\xc3\xb1o'
s = x.decode()
print(s)

To convert a number into bytes

b=chr(128).encode()

This has also impact how files are handled. Files can be read as text (strings) or binary data (bytes).

You can also get elements in a sequence counting from the back a[-1] gives the last element in a. Mathematically speaking the length len(a) is added to negative numbers.

>>> a="hello"
>>> a[0]
'h'
>>> a[-1]
'o'

The .count(<listelement>) method returns how many times the list element is in the list, so don't loop lists for that.

2**8 is power 8 of 2 and so 256. In C, after including the math library, it would be pow(2,8)

>>> 2**8
256

Python can deal with complex numbers:

>>> a=2j
>>> a+(1+1j)
(1+3j)

Range checking gets more readable instead of C like (a>min)&&(a<max) it gets

>>> max=5
>>> min=2
>>> a=3
>>> min<a<max
True

Every different from 0 is considered as True. this explains the following:

if a:
  pass
if len(a):
  pass

Instead of if: else: if: else: if: else: use if: elif: elif: else:, since no switch case as in C exists.

A count variable is not required in the for loop to access elements in a sequence (notice the indention):

>>> s=[0,2,4,6]
>>> for p in s:
...   p
... 
0
2
4
6

Inside the for loop do not modify s. If you want to do it use a while loop. As example:

>>> s=[0,2,4,6]
>>> while len(s)>0:
...   print s
...   s.pop(0)
... 
[0, 2, 4, 6]
0
[2, 4, 6]
2
[4, 6]
4
[6]
6

Without argument pop defaults to -1 and takes the last list member.element.

If you still want to use a count variable, you could use a while loop, but python stile uses the range command that creates a sequence that is passed to the for loop:

>>> for p in range(3):
...   p
... 
0
1
2

The range command can also do more as counting back and create more complex sequences.

Functions can return also more complex data as lists.

Functions can alternatively be called with <parametername>=<value> instead of just the values in correct sequence separated by commas. This looks like more writing but is safer and nicer code since it is independent of the sequence and also makes sure the parameter are passed in a consistent manner. To have good documented code it is also recommend to put every parameter to a new line and add a comment using the # character on every line.

*<parameter> means that the <parameter> can occur multiple times. It can be accessed using <parameter>[].

<parameter>=5 assigns a default value, this allows to call the function without passing any parameters.

Modules are usually single py files. Packages contain one or many modules. Packages are py files inside a subdirectory and have an __init__.py file.

To use a module that is in the same directory. Note don't use - character in py file names.

import <module name> 

To import all modules from a package (This can create tons of unnecessary global variables and naming conflict)

from os import * 

So better

import os

And putting os in front of the module name to make it clear to where it belongs.

Python modules are usually put under /usr/lib/python<pythonversion>/site-packages but when developing a more flexible way is desired than creating and installing packages. The modules can be anywhere as long python finds it.

With the command export PYTHONPATH=/<here is my direcory>/ ; $PYTHONPATH the search paths for python can be expanded.

To not have the hassle with the PYTHONPATH

import sys

print(sys.path)

shows where python is looking for modules, so links as

ln -s /home/<path>/<module name>.py /usr/lib/python3.2/<module name>.py

When writing a module, it is desired to run the module on its own, reasons are to debug the module or to have even a standalone utility program. Adding the following, makes starting the code just when calling the module directly with python on the command line. In this case the hidden variable __name__ contains __main__.

if __name__ == '__main__':
  <here comes my test code calling the stuff above> 

Instead of writing everything yourself look at https://pypi.python.org/pypi. There is also a tutorial about submitting and installing packages: https://wiki.python.org/moin/CheeseShopTutorial

A good python program should not crash. However sometimes it is not known in advance if a crash might happen. Examples are input from the user for a number, or parsing an xml file.

An other approach is to catch the exception and act afterwards. To initiate this add the python key word try: then the critical code. After that there is the choice for one of the two key words:

In python 2 print was a keyword but in python 3 it is a build in function, therefore it has print() brackets.

There are different ways how to format what print is doing. If nothing print prints out the variables and adds a space in between. The variables can also be converted to a string using str() or repr(). Finally there are two format string methods (see https://pyformat.info/). An old style (C like) using the % character and a new style using the .format method.

print(a, b)   
print(str(a)+" "+str(b))
print(repr(a).rjust(6),  repr(b).rjust(6))
print('{0:6d} {1:6.2f}'.format(a, b))

The modern way is using {}

{} place holder for a string

{:d} decimal number

{:x} hex number

{:03x} hex number with 3 nibbles and printed preceding zeros

To not end with new line characters tell print to no insert them

print("No new line after here", end='') 

Threads are as processes and allow to run in parallel, with the exception that threads use the same memory space. As example it thread1 writes to a global variable thread2 can read it. A python script not creating threads is a single thread script.

In the simples case threads are started as follows

import threading

t =threading.Thread(target=<function name>)
t.start()

More advanced threads are started by defining a new subclass of the Thread class, then override the __init__(self [,args]) method to add additional arguments. Then, override the run(self [,args]) method to implement what the thread should do when started.

The word pass is a python keyword and does not do anything. It is used to not cause a syntax error in the above code. Since a breakpoint can be set to pass it is quite helpful during program development. pass can be put as place holder for code that needs to be written.

Since it is an interpreter language you can execute what is written in a string variable using the exec command.

>>> exec "pi=3.1415"
>>> pi
3.1415

To read from the console raw_input() can be used that returns always a string. However input() could also be used that returns an object based on the input a string, and integer, ... . This looks like more coding work, but using the command type() can be used to check if the input was correct or try: can be used to just be optimistic and do and if it fails except TypeError: can be used to react.

Interrupting a running program can be done with a keyboard interrupt

while True:
      time.sleep(0.05) # be nice  
      try:  
        pass
        # some code here
      except KeyboardInterrupt: 
        pass
        # crtl+C brings you here

Almost every program must deal with command line parameters. Therefore python offers modules for that. The module argparse should be used now. It replaced optparse that replaced the getopt module. Some python statements are required to let the program known with what command line parameter it has to deal. Those parameters will then be easily available in the options structure. Left overs that will not be parsed are called positional arguments and are arguments that usually are absolute necessary to run (as a filename).

Instead of printing out information to track what the program is doing python offers a logger, see https://docs.python.org/3/howto/logging.html. After

import logging

a logging call looks similar as a print:

logging.info("This variable will be logged: "+i)

The logger can be configured. Levels can be set to control how much is printed. The levels are: DEBUG, INFO, WARNING, ERROR and CRITICAL and the outputs can be written in a file instead of the console:

logging.basicConfig(level=logging.INFO, filename='log.log')

Python programs with Graphical User Interfaces (GUI) make use of the following graphical libraries that give the applications different look:

Links to documentation: https://tkdocs.com/tutorial/index.html https://documentation.help/Python-3.6.3/tkinter.ttk.html

Nowadays tinker gets used together with ttk (Tk themed widgets).

ttk comes with a set of widgets allowing to set themes with the style= attribute.

Matplotlib for plotting and numpy for numerical calculation bring functionality as known from Matlab into python. Matplotlib uses a frontend/backend approach to output the plots. This allows to use different backends to put the results into files having different formats or embed in various GUI canvasses as tkinter, gtk, qt, qt4.

Matplotlib commonly uses numpy arrays but for simple things without a lot of computing as reading results from an instrument, python lists instead of numpy arrays can be used.

The standard way of installing python package is pip and using https://pypi.org/ and https://pip.pypa.io/en/stable/user_guide/ However installing packages without the Linux package manager being involved is not advised. pip needs to be first installed. For Gentoo emerge dev-python/pip

pip is also aware on what it did and what it can do. Commands as:

pip list and pip list --outdated

pip search <name>

pip uninstall <name>

pip install --upgrade <name>

pip install --trusted-host files.pythonhosted.org <name> when getting error

In Linux such packages should be installed as usual using the package manager (for Gentoo the emerge command) to no create a mess with the installed package database. The python files will be installed to /usr/lib64/python<m>.<n>/site-packages/<package name>

Additionally specially for Gentoo Linux there might be different python versions installed (see <m>.<n>) so it is good the package manager takes care about this. Otherwise python might be unable to find the package for the simple reason: Mismatch between python and package version numbers.

Python looks in the sys.path that can be seen by

python Python 3.5.4 (default, Jan 7 2018, 19:27:36) [GCC 6.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', '/usr/lib64/python35.zip', '/usr/lib64/python3.5', '/usr/lib64/python3.5/plat-linux', '/usr/lib64/python3.5/lib-dynload', '/usr/lib64/python3.5/site-packages'] >>>

Distutils creates further a <name>-<version>-<python version>.egg-info file (or directory) that contains information about the package. There is also a __pycache__ directory that holds the compiled and optimized bytecode files for the py file.

To use an installed python package it must be imported. To know how and what to import it must be known how it got installed.

Files as /usr/lib64/python<m>.<n>/site-packages/<name>.py can be imported as

import <name> 

It is common, that a package is installed into a directory under /usr/lib64/python<m>.<n>/site-packages/ and this directory holds different python files. Such a file /usr/lib64/python<m>.<n>/site-packages/<package name>/<name>.py can be imported as

from <package-name>  import <name>

With unittest a program (test.py) can be made that calls the python program in various ways and compares the results against what is expected.

#!/usr/bin/python3
import unittest

class MyTest(unittest.TestCase):
      
    def test_default(self):
        <code>
        self.assertEqual(a,b)   
    
    def test_other(self):
        <code>
        self.assertEqual(a,b)      
      
if __name__ == '__main__':
    unittest.main()

Different test_<name> functions can be added that contain an assert<what> method.

The program test.py can then be called as

python -m unittest -v test.py

python test.py -v

test.py -v since it contains #!/usr/bin/python3

test.py MyTest -v to call the MyTest test cases

test.py MyTest.test_default_bin -v to just call a specific test case (for example the one causing troubles)

A good sign of code quality is if test run all the code. The program coverage can be used instead of the program python.

coverage run <program>.py

coverage report to get a text report

coverage html then open htmlcov/index.html to see a html report

coverage -h to get help

coverage does not check what get started and run in other processes

After installation, python can be started with

The best way to start is using Windows menu start and start IDLE. Then open a python script <name>.py and run it as module F5

It might be installed in a user directory and therefore just available for a single user as c:\Users\<username>\AppData\Local\Programs\Python\Python35\python.exe

Python is not added to the windows path environmental variable so it will not be found in console window it need to be started with the complete path.

This can be fixed by adding this path to the windows environmental variables: Control Panel\System and Security\System\Advanced System Settings\Advanced\Environmental Variables

Figure 15.2. Windows Environmental Variables

The shell might probably report that some modules (libraries) are not installed

The next problem is how to install those libraries, the standard way is using pip. Newer python version come with pip, on older versions pip needs to be installed first.

update pip (-m makes python know that pip is a module and python therefore knows where to look for it) is done with python -m pip install -U pip

python -m pip install --user bincopy==14.0.0 installs the missing module (bincopy) with a desired version (14.0.0)

A warning might comes that c:\Users\<username>\AppData\Roaming\Phyton\Phython35\Scripts should be set to the path this is recommended otherwise programs as pyinstaller are not found

If the installation fails due to certification errors install the certificates python -m pip install certifi

When debugging with IDLE then it can happen that imports of local modules fail. The reason is that sys.path does not contain the path of the script and the current working path is not the path where the script is. This can be fixed in the shell window of IDLE by sys.path.append('Z:\\<path of current script>'). In this example it is shown that Windows need to get two \\ characters since the first is a string escape character.

To know where the code is running use :

>>> import platform
>>> platform.system()
'Windows'

To get portable code avoid using commands as os.system("cd /home") that execute Linux commands. Use os.chdir(<path>) instead.

os.linesep instead of '\n' for Linux or '\r\n' for windows used in text files. However within python code it is recommended to use all the time '\n' on all platforms and have pythin deal with it.

import pathlib     
# is an object maybe required to convert to string
s=pathlib.PureWindowsPath(<path>)
t=str(s)       # converts object to string
u=s.as_posix() # returns a string

# call lxml as ET so it is compatible with the python 
# built-in library xml but the library lxml has more
# features as pretty-print and xml declaration support
lxml=False
try:
   from lxml import etree as ET
   lxml=True
except:
   import xml.etree.ElementTree as ET

<some code>

if lxml==True:
   tree.write(self.sessionPath+os.sep+self.sessionFile,  
   encoding="utf-8", pretty_print=True,   
   xml_declaration=True) 
else:            
   tree.write(self.sessionPath+os.sep+self.sessionFile, 
   encoding="utf-8")    

<some code>

pyinstaller from https://pyinstaller.readthedocs.io/en/stable/ offers a nice way to create a Windows exe. To run pyinstaller Python needs to be installed under Windows.

pyinstaller allows converting python scripts into windows exe files not requiring to fiddle around with installing and setting up python on a windows computer. Those windows exe do not require to administrator privilege to install them, since they contain usually all required code. Two ways are possible

pyinstaller <name>.spec creates the <name>.exe.

The <name>.spec can be created automatically the first time by calling pyinstaller <name>.py

Files can be added in <name>.spec

 a = Analysis(
             datas=[('*.gif','.')], ...

The python tuple means copy it from source to destination.

There are also command line options when creating or overwriting a <name>.spec file.

--add-data '*.gif:.' copies *.gif to the bundle

--onefile creates single <name>.exe file <name>.exe

--windowed hides the console window and shows just the gui window. Drawback no error messages appear.

--onefile puts everything in a <name>.exe file that when run gets first unzipped in temp folder of OS

--icon file.ico attaches a windows icon to replace the standard python icon seen in the windows file manager. it is not the icon in the window decoration of the application.

The pyinstaller can also be run under Linux as well, since it is pure python. The same commands as above can be used. However the result will be different, it will a Linux binary.

Under Windows, Python can also run from a memory stick without requiring an installation, as usual under Linux there are different ways to do this:

To create distutils package a setup.py file has to be created that holds the information distutils requires.

For single py files this setup.py file can placed beside the py files and can look as follows:

from distutils.core import setup
setup(name='<name of the module>',
      version='<version>',
      scripts=['<name of the module, without py extension>'],
      )

For more complex program it is wise to create a package, a collection of files. So put them in a subdirectory and place a setup.py file above that directory that could look as:

from distutils.core import setup
setup(
      name="girttools",
      packages = ["girttools"],
      version="0",
      description='Tools for Gnu Infra Red Transceiver',
      author='Urs Lindegger',
      author_email='urs@linurs.org',
      url='http://www.linurs.org/girt.html',
      download_url = "http://www.linurs.org/girt/download/girttools-0.tgz",
      keywords = ["Infrared transceiver"],
      classifiers = [
        "Programming Language :: Python",
        "Programming Language :: Python :: 3",
        "Development Status :: 4 - Beta",
        "Intended Audience :: End Users/Desktop",
        "License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)",
        "Operating System :: POSIX :: Linux",
        "Topic :: Multimedia",
      ],
      long_description = """\
Gnu Infra Red Transceiver Tools
-------------------------------

Tools to setup and work with the Gnu Infra Red Transceiver that can replace all you IR remote controllers.
"""

The long_description (or since it is python everything) can also be read out of a file:

long_description=open('README.md').read()

Valid classifiers can be found at https://pypi.org/pypi?%3Aaction=list_classifiers or https://pypi.org/classifiers/

Instead of py_modules that can list various py files packages is used containing the subdirectory that must contain a __init__.py file.

Finally when done packages can be register so other will find them https://pypi.python.org/pypi?:action=register_form. Python Package Index (“PyPI”) allows to classify your package so it can be found, therefore don't invent your own classifiers select them out of https://pypi.python.org/pypi?:action=list_classifiers

To include attentional files create a MANIFEST.in file as defined in https://docs.python.org/3.1/distutils/sourcedist.html. The MANIFEST.in file can also contain wild-carts and distutils specific commands as recursive directory functions.

Finally MANIFET.in is read among other things defined in setup.py to create the MANIFEST file that contains all files

setup.py can also contain data_files that can be used to install files in any directory. Due to specifying the path the *.tar.gz package will no more be portable to different systems. An example would be installing a Linux init script to /etc/init.d, however on a Linux system using systemd instead of init or even on a Windows machine this makes not much sense to install it. It is better to do this on the system specific package as for Gentoo Linux in the ebuild. However you simply add them to the *.tar.gz using the MANIFEST.in file.

python setup.py check will check your setup.py

To get the *.tar.gz package in the subdirectory dist run python setup.py sdist (source distribution)

For windows an graphical windows installer can be created: python setup.py bdist_wininst

After having the tar.gz you could but should not install it as : python setup.py install Instead of that you should create a package that fits you Linux distribution and then use the package manager to have it installed. Under Linux python modules (py files) are copied to the directory /usr/local/lib/python<X>.<Y>/site-packages and packages (directories containing __init__.py) are copied into a subdirectory under /usr/local/lib/python<X>.<Y>/site-packages.

Alternatively distutils allows to install the package to an other directory as your home directory with python setup.py install --home=~ any other directory with python setup.py install --home=<dir> where no system conflicts should occur.

https://docs.python.org/3.1/distutils/builtdist.html helps you to move your package to distributions as creating a RPM red hat package.

For Gentoo Linux an ebuild needs to be developed that first installs the package in a sandbox and then copied (merged) to your system getting track what files will be added and check collisions.

Ebuilds can make usage of eclasses and there is a distutils eclass (https://wiki.gentoo.org/wiki/Project:Python/distutils-r1), so the ebuild for a python module can be very simple as:

EAPI=6

PYTHON_COMPAT=( python{3_4,3_5,3_6} pypy )

inherit distutils-r1

DESCRIPTION="Library for all sort of python code"
HOMEPAGE="https://www.linurs.org"
SRC_URI="http://www.linurs.org/download/${P}.tar.gz"
LICENSE="Unlicense"
SLOT="0"
KEYWORDS="~arm64 ~x86 ~amd64"
IUSE=""
DOCS=( README.txt )

${P} is the filename of the ebuild including version number.

The inherit function loads distutils and this overwrites the standard install routines. So the ebuild contains no code just meta data about the python package and where to find it on the Internet remains.

The README.txt file gets zipped under /usr/share/doc/${P}/