Python

Python https://www.python.org/ is a high level interpreter language, supporting object oriented programming, and having a lot high level libraries for all kind of applications. It is highly portable to all kinds of hardware devices and operating systems. Python scripts are converted to byte code that runs on a virtual machine. Additionally there is IronPython that runs on top of .NET (or Mono) or Microsofts python for .NET and Jython that runs on Java virtual machine. There are ten-thousands of python packages available at https://pypi.python.org/pypi

Extending python is connecting C routines to python. The other way around is also possible, embedding python is integrating the python interpreter in a c program. And have this run the python script.

Python scripts have usually the extension py but can also have no extension at all, since under Linux the magic first line

#!/usr/bin/python

tells exactly what to do when this file has to be executed, start the interpreter found under /usr/bin/python

The first thing to know when dealing with python scripts is that to group commands no {} brackets are used, everything is done with indention.

Warning

Be consequent and do not use tab characters but use space characters for the indent, so when using different editors no mess gets created.

Avoid long sequences of nested code segments, use functions and objects to encapsulate.

As general advice, if the code is getting large and nested, then its is time to learn a bit more about python since using the right methods (including, dictionaries, sets, classes, libraries) surprisingly compact code can be written.

Python2 and Python3

The file /usr/bin/python is probably a link to a command as /usr/bin/python2 or /usr/bin/python3 because different version of python might be installed on the computer. It selects therefore the default python to be used. To be more specific the magic line could be

#!/usr/bin/python3

To select python3

Note

Today still many python scripts are written for python2 that has incompatibilities with python3. So a python3 script will not run with a python2 interpreter and vice a versa.

Since python2 is no more actively developed, this book concentrates on python3.

A script 2to3 is installed that can automatically convert python2 scripts to python3 see: https://docs.python.org/2/library/2to3.html. Running 2to3<myscript>.py it will analyze the script and prints the changes to the screen. To rename the python2 script to <myscript>.py.bak and create a converted python3 script run it as 2to3 -w<myscript>.py. If python 2 is set as the active python interpreter run it as python3<myscript>.py. Note that python3 is a link to something as python3.1

Or set the magic line first line to choose the right version:

#!/usr/bin/python3

2to3 makes use of fixers that can be listed with 2to3 -l. Not all are active per default and need to be enabled with 2to3 -f<name of fixer> if desired.

The 2to3 works well on simple scripts, where the keyword print gets converted to the function call print(), but the troubles come with the libraries and file access.

To test python open a console and type python. Now you can type in commands and see what is going on.

On Getnoo Linux eselect python list will show what python version is the default and what are installed

How python finds its packages

Different python versions are probably installed on the same computer so the question arises what packages are available and taken. A way to see it is starting

python

Python 3.4.3 (default, Oct 30 2016, 21:08:25) [GCC 4.9.3] on linux Type "help", "copyright", "credits" or "license" for more information.

>>> from distutils.sysconfig import get_python_lib

>>> print(get_python_lib())

/usr/lib/python3.4/site-packages

User installed packages end up as ~/.local/lib64/python3.5/site-packages

Python documentation

Python documentation is available under https://docs.python.org/ or /usr/share/doc/python-docs-<xxx>/html/index.html http://www.onlamp.com/python/

There is a python doc server (pydoc server) for OpenRC add it to the start

/etc/init.d/pydoc-<version> start

The port is PYDOC_PORT=7464 and can be viewed:

http://localhost:7464/

The stuff comes from

/usr/share/doc/python-docs-2.5.1/html/index.htm

On trouble run it from a console /usr/bin/pydoc3.6 -b

Python IDEs

The following IDE's are available:

  1. The standard IDE (Integrated Development Environment) that comes with Python is idle:

    It requires tcl and tk support to run.

    Note

    For Gentoo set the tcl and tk useflags and then re-emerge it emerge -pv python

    idle is pretty basic but it might be the only IDE available (or working). To pass command line parameters idle -r<name of python script><command line parameter>

    For a specific python version start idle3.3

    Debugging is possible using its debug window, where the source can be viewed and all the variables. In the source, breakpoints can be set to lines.

    Important

    With the debug window open it debugs when run, with the debug window closed run runs.

    Click the Source checkbox to see single step and where the program is. For the default setting single step is a dark gray line and brake point is yellow. Not both colors can be seen at the same time so the dark gray line disappears when jumping to a yellow line. So if there is no expected dark gray line press step or out.

    It has individual windows, but when arranging them it looks like a real (but un-bloated) IDE

    Figure 15.1. idle

    Idle


  2. eric6 is a nice complete IDE for python.

    Python can make you crazy about the formatting, here eric draws vertical lines to help you horizontally positioning the text.

    To complete the python help insert in Settings > Preferences > Help Documentation > Python Documentation the link to the directory as /usr/share/doc/python-docs-<x>/html that holds the index.html file.

    It produces ~/.config/Eric6 and ~/.eric6

  3. spe is an other ide for python2

  4. drpython is an other ide for python2

  5. winpdb is a stand alone debugger with a GUI for python2 only (except some hacks are done). It is a highly featured python debugger especially when it comes to multi threading support. So use an context sensitive editor in one window and winpdb in a other window. The application uses os.fork() to create a child process. The function os.fork() returns two values, the value 0 to the child process and the PID of the child process to the parent process. Since the program forking is just one file (one program, one source), the two threads running the same source code, have to check the return value of os.fork() to know whether they run as child or as parent thread. The python debugger winpdb allows to debug either the parent process or whats probably more desirable to continue with the child process. To select what you desire the gui of winpdb has a console window where you can type in either fork child or fork parent.

  6. Eclipse has a plugin, add the link http://www.pydev.org/updates/to get the pydev plugin. Then go to window -> preferences and set the python interpreter as /usr/bin/python3

Embedded python debugging

Often python script are called by a external process, this might be a gui, and other python script or every thing else. The python script will then be started in a environment with additional environmental variable set, other command line parameters passed and maybe other processes or threads started in parallel. This environment is different from the one that would exist when the python script would have been started simply in a console.

Therefore you need a debugger with embedded debugging support as winpdb. Add the following line into the file that you would like to debug

      import rpdb2; rpdb2.start_embedded_debugger_interactive_password() 

Then start the program using the python script as usual, however this time the python script will prompt for a password and wait for an external debugger. Start winpdb and go to file => password and then file => attach where the python script pop's up and can be selected. If you have problems with the interactive password you can use fix password as:

import rpdb2; 
rpdb2.start_embedded_debugger('hello')

Hint for winpdb: It shows at the current line capital letters that have the following meaning

C

CALL

A function is called (or some other code block entered)

L

LINE

The interpreter is about to execute a new line of code (sometimes multiple line events on one line exist)

R

RETURN

A function (or other code block) is about to return

E

EXCEPTION

An exception has occurred

*

RUNNING

The thread is still running (probably blocked in C code)

Python Programming

Variables are objects

Inside a python module there are global and local variables. However a child module has its own global name space and can not access global variables of the parent module. The parent module however can access the global variables of the child module by simply adding the child's module name as prefix (if imported this way)

Variables and almost everything in python are objects. Objects are instances of classes. Objects have an identifier, a name (that is optional) and a value.

Some objects as integers have immutable values and result in a not obvious behavior. This is best explained by the following:

>>> a=1
>>> b=1
>>> id(a)
134578352
>>> id(b)
134578352
>>> id(1)
134578352

To the object with the value 1 the name a gets assigned. After that the object with the name a and the value gets an additional name b. This can be verified by checking the id of the object using the names. The identifier can also be observed by passing the value to the id() command. Assigning the name b to an other value (=object) deletes the name b of the previous object.

To see that a is an object you can call the class add method of a:

>>> a.__add__(2)
3

This the same as a +2 that results in 3. In fact python converts the + sign to the class method __add__ , but luckily hides this to human.

Luckily there are objects with mutable values as lists, they behave as expected:

>>> a=[1]
>>> id(a)
3075342668L
>>> b=[1]
>>> id(b)
3074937644L

For mutable objects, each value assigned to a name creates a new object. To get the value of the first (and here only) value of the list you can either use the nice looking command using brackets or the class __getitem__ method:

>>> a[0]
1
>>> a.__getitem__(0)
1

However the following shows a common pitfall. The simple command b=a behaves differently, the list just gets a second name. To make a copy, a slice out of the list a needs to be done. The syntax for it is [<start index>:<end index>]. If the indexes are omitted as in [:] then a slice from the very beginning to the very end is done.

>>> a=[1]
>>> b=a
>>> id(a)
3074516172L
>>> id(b)
3074516172L
>>> c=a[:]
>>> id(c)
3074916588L

Python data types

Python data types or more precisely data classes can be grouped in mutable and immutable. Additionally there are sequences. Some sequences are mutable (as lists) and some immutable (as strings, tuples).

To use the wording of object oriented programming, if a data class gets used, it becomes an data object as instance of the data class.

Explaining in simple words, it is sometimes not obvious that the data is immutable:

>>> a="hello"
>>> id(a)
3073624256L
>>> a=a[1:3] 
>>> a
'el'
>>> id(a)
3073538048L

It seems that the immutable string a has been changed (even being immutable). However as the id() shows a new variable with the name a has been created.

With mutable data types the behavior is different:

        >>> list=['h','e','l','l','o']
>>> id(list)
3074023660L
>>> del list[2]
>>> id(list)
3074023660L
>>> print list 
['h', 'e', 'l', 'o']

Using the python data types is a bit confusing since some commands are consistent among the different data types others not. To understand the logic behind, the different and many data types need to be understand in very detail.

There is a reason way different data types exist!

Strings have a lot of functions (or in term of object oriented programming) methods.

Table 15.2. String methods

s.strip() Removes blank (or character passed) characters from the beginning and end
s.split("/") Using the character passed, splits the string in substring and returns list of substrings
s.endwithsuffix("/") Returns true if the string ends with the character passed


List and tuples can have different type of variables inside. C has arrays and structures for that.To add a new element to a list do a+=[i].

Python Strings

One big change between python2 and Python3 are the way strings are handled, so the mess with character encoding in python2 seems to be solved (or has to reached a new dimension). Strings use unicode utf-8 make sure that locale prints out a language with utf-8 otherwise it happens that a python script crashes when it wants to print an utf-8 character in a console that does not support utf-8. Strings are strictly separated from bytes. To convert strings to bytes character encoding and decoding is required. Bytes can be declared with b'<here are bytes>' whereas a string is '<this is my unicode string>'.

Create bytes from strings is encoding strings and is done <stringofbytes>=<utf8string>.encode('utf-8')

Creating strings from bytes is decoding bytes and is done <utf8string>=<stringofbytes>.decode('utf-8').

This has also impact how files are handled. Then can be read as text (strings) or binary data (bytes).

Sequences

You can also get elements in a sequence counting from the back a[-1] gives the last element in a. Mathematically speaking the length len(a) is added to negative numbers.

>>> a="hello"
>>> a[0]
'h'
>>> a[-1]
'o'

The .count(<listelement>) method returns how many times the list element is in the list, so don't loop lists for that.

Mathematical expressions

2**8 is power 8 of 2 and so 256. In C, after including the math library, it would be pow(2,8)

>>> 2**8
256

Python can deal with complex numbers:

>>> a=2j
>>> a+(1+1j)
(1+3j)

Range checking gets more readable instead of C like (a>min)&&(a<max) it gets

>>> max=5
>>> min=2
>>> a=3
>>> min<a<max
True

Conditions

Every different from 0 is considered as True. this explains the following:

if a:
  pass
if len(a):
  pass

Instead of if: else: if: else: if: else: use if: elif: elif: else:, since no switch case as in C exists.

Loops

A count variable is not required in the for loop to access elements in a sequence (notice the indention):

>>> s=[0,2,4,6]
>>> for p in s:
...   p
... 
0
2
4
6

Inside the for loop do not modify s. If you want to do it use a while loop. As example:

>>> s=[0,2,4,6]
>>> while len(s)>0:
...   print s
...   s.pop(0)
... 
[0, 2, 4, 6]
0
[2, 4, 6]
2
[4, 6]
4
[6]
6

Without argument pop defaults to -1 and takes the last list member.element.

If you still want to use a count variable, you could use a while loop, but python stile uses the range command that creates a sequence that is passed to the for loop:

>>> for p in range(3):
...   p
... 
0
1
2

The range command can also do more as counting back and create more complex sequences.

Functions

Functions can return also more complex data as lists.

Functions can alternatively be called with <parametername>=<value> instead of just the values in correct sequence separated by commas. This looks like more writing but is safer and nicer code since it is independent of the sequence and also makes sure the parameter are passed in a consistent manner. To have good documented code it is also recommend to put every parameter to a new line and add a comment using the # character on every line.

*<parameter> means that the <parameter> can occur multiple times. It can be accessed using <parameter>[].

<parameter>=5 assigns a default value, this allows to call the function without passing any parameters.

Importing modules and packages

Modules are usually single py files. Packages contain one or many modules. Packages are py files inside a subdirectory and have an __init__.py file.

To use a module that is in the same directory. Note don't use - character in py file names.

import <module name> 

To import all modules from a package (This can create tons of unnecessary global variables and naming conflict)

from os import * 

So better

import os

And putting os in front of the module name to make it clear to where it belongs.

Python modules are usually put under /usr/lib/python&lt;pythonversion&gt;/site-packages but when developing a more flexible way is desired than creating and installing packages. The modules can be anywhere as long python finds it.

With the command export PYTHONPATH=/<here is my direcory>/ ; $PYTHONPATH the search paths for python can be expanded.

To not have the hassle with the PYTHONPATH

import sys

print(sys.path)

shows where python is looking for modules, so links as

ln -s /home/<path>/<module name>.py /usr/lib/python3.2/<module name>.py

Important

Modules in the sys.path will be found modules elsewhere not and cause an import error. The first path in sys.path is usually ' ' this means the current working dir (os.getcwd()). Depending how the python script is started the cwd might be different and the python script fails (as when debugging in IDLE) to not have such effects add the missing path to sys.path using sys.path.append(<missing path >)

Note

__file__ that holds the started script.

When writing a module, it is desired to run the module on its own, reasons are to debug the module or to have even a standalone utility program. Adding the following, makes starting the code just when calling the module directly with python on the command line. In this case the hidden variable __name__ contains __main__.

if __name__ == '__main__':
  <here comes my test code calling the stuff above> 

Instead of writing everything yourself look at https://pypi.python.org/pypi. There is also a tutorial about submitting and installing packages: https://wiki.python.org/moin/CheeseShopTutorial

Exceptions

A good python program should not crash. However sometimes it is not known in advance if a crash might happen. Examples are input from the user for a number, or parsing an xml file.

An other approach is to catch the exception and act afterwards. To initiate this add the python key word try: then the critical code. After that there is the choice for one of the two key words:

  1. except: This key word initiates the commands that are executed when the commands after try fail. Optionally the error can be added to except as except TypeError: to act just on type errors. This way multiple except's can follow a try:

  2. finally: This key word works as except:, except that this code is executed also when try: worked trouble free.

print and formatting strings

In python 2 print was a keyword but in python 3 it is a build in function, therefore it has print() brackets.

There are different ways how to format what print is doing. If nothing print prints out the variables and adds a space in between. The variables can also be converted to a string using str() or repr(). Finally there are two format string methods (see https://pyformat.info/). An old style (C like) using the % character and a new style using the .format method.

print(a, b)   
print(str(a)+" "+str(b))
print(repr(a).rjust(6),  repr(b).rjust(6))
print('{0:6d} {1:6.2f}'.format(a, b))

The modern way is using {}

{} place holder for a string

{:d} decimal number

{:x} hex number

{:03x} hex number with 3 nibbles and printed preceding zeros

pass

The word pass is a python keyword and does not do anything. It is used to not cause a syntax error in the above code. Since a breakpoint can be set to pass it is quite helpful during program development. pass can be put as place holder for code that needs to be written.

Executing textural expressions

Since it is an interpreter language you can execute what is written in a string variable using the exec command.

>>> exec "pi=3.1415"
>>> pi
3.1415

Console

To read from the console raw_input() can be used that returns always a string. However input() could also be used that returns an object based on the input a string, and integer, ... . This looks like more coding work, but using the command type() can be used to check if the input was correct or try: can be used to just be optimistic and do and if it fails except TypeError: can be used to react.

Interrupting a running program can be done with a keyboard interrupt

while True:
      time.sleep(0.05) # be nice  
      try:  
        pass
        # some code here
      except KeyboardInterrupt: 
        pass
        # crtl+C brings you here

Handling command line parameters in python

Almost every program must deal with command line parameters. Therefore python offers modules for that. The module argparse should be used now. It replaced optparse that replaced the getopt module. Some python statements are required to let the program known with what command line parameter it has to deal. Those parameters will then be easily available in the options structure. Left overs that will not be parsed are called positional arguments and are arguments that usually are absolute necessary to run (as a filename).

Python logging

Instead of printing out information to track what the program is doing python offers a logger, see https://docs.python.org/3/howto/logging.html. After

import logging

a logging call looks similar as a print:

        logging.info("This variable will be logged: "+i)

The logger can be configured. Levels can be set to control how much is printed. The levels are: DEBUG, INFO, WARNING, ERROR and CRITICAL and the outputs can be written in a file instead of the console:

        logging.basicConfig(level=logging.INFO, filename='log.log')

Gui programming

Python programs with Graphical User Interfaces (GUI) make use of the following graphical libraries that give the applications different look:

  1. pyQT based on QT

  2. http://wxpython.org/ toolkit based on http://wxWidgets.org/ There is Boaconstructor a RAD (rapid development) GUI tool for it.

  3. http://pygtk.org/ that is the gtk=gimp tool kit. Glade is a tool to draw the GUI as xml file that can then be used by GtkBuilder to interface to different programming languages as python.

    anjuta is the standard ide for gnome and gnome is based on gtk. In anjuta a new project can be created for python pyGtk. It has dependencies to rope and glade to have support for it working. This creates a working hello world gui application using python. The gui itself is an xml file with the extension ui that is in the data subdirectory. The glade support in anjuta is responsible to edit this xml file using a gui and not a text editor and integrate it to the anjuta file. Glade itself would run as standalone program to edit this ui xml file, however a consistency mess with anjuta would then be probable.

  4. Tkinter is an interface to tk (tool kit).

    Tkinter comes with python and needs no installation, but looks a bit basic. Using ttk (Tk themed widgets) to look can be improved. An advantage is that it is stable, has not many dependencies and is portable.

matplotlib

Matplotlib for plotting and numpy for numerical calculation bring functionality as known from Matlab into python. Matplotlib uses a frontend/backend approach to output the plots. This allows to use different backends to put the results into files having different formats or embed in various GUI canvasses as tkinter, gtk, qt, qt4.

Matplotlib commonly uses numpy arrays but for simple things without a lot of computing as reading results from an instrument, python lists instead of numpy arrays can be used.

Note

For python3 matplotlib >=1.2 must be installed

Installing python packages

Pip

The standard way of installing python package is pip and using https://pypi.org/ and https://pip.pypa.io/en/stable/user_guide/ However installing packages without the Linux package manager being involved is not advised. pip needs to be first installed. For Gentoo emerge dev-python/pip

pip is also aware on what it did and what it can do. Commands as:

pip list and pip list --outdated

pip search <name>

pip uninstall <name>

pip install --upgrade <name>

pip install --trusted-host files.pythonhosted.org <name> when getting error

Warning

Avoid using pip as root since it could impact python scripts that are used by the system (especially gentoo since all system program as emerge are written in python). Use it as user and install the python package under the users home directory. pip install --user bincopy==14.0.0 The package goes in something as ~/.local/lib64/python3.5/site-packages and the python interpreter will find it.

Installing a python package

In Linux such packages should be installed as usual using the package manager (for Gentoo the emerge command) to no create a mess with the installed package database. The python files will be installed to /usr/lib64/python<m>.<n>/site-packages/<package name>

Additionally specially for Gentoo Linux there might be different python versions installed (see <m>.<n>) so it is good the package manager takes care about this. Otherwise python might be unable to find the package for the simple reason: Mismatch between python and package version numbers.

Python looks in the sys.path that can be seen by

python Python 3.5.4 (default, Jan 7 2018, 19:27:36) [GCC 6.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', '/usr/lib64/python35.zip', '/usr/lib64/python3.5', '/usr/lib64/python3.5/plat-linux', '/usr/lib64/python3.5/lib-dynload', '/usr/lib64/python3.5/site-packages'] >>>

Distutils creates further a <name>-<version>-<python version>.egg-info file (or directory) that contains information about the package. There is also a __pycache__ directory that holds the compiled and optimized bytecode files for the py file.

Using a python package

To use an installed python package it must be imported. To know how and what to import it must be known how it got installed.

Files as /usr/lib64/python<m>.<n>/site-packages/<name>.py can be imported as

import <name> 

It is common, that a package is installed into a directory under /usr/lib64/python<m>.<n>/site-packages/ and this directory holds different python files. Such a file /usr/lib64/python<m>.<n>/site-packages/<package name>/<name>.py can be imported as

        from <package-name>  import <name>

Documenting Python

Doxygen supports python as well. The docstrings are considered directly by doxygen without the need to specially mark them.

Python has also a feature to attach a docstring to functions. After the function header a line starting by """ will open the docstring that can contain multiple lines and needs no special characters at the beginning of each line. The docstring is terminated by a second line starting with """. Such docstrings can be visible in the python interacting mode by the help() function.

The python module (file, main program) can hold a doxygen command as follows:

"""@package docstring
<Documentation>
"""

Alternatively for doxygen the # character used for comments can be takes as well.

Testing Python code

Unittest

With unittest a program (test.py) can be made that calls the python program in various ways and compares the results against what is expected.

#!/usr/bin/python3
import unittest

class MyTest(unittest.TestCase):
      
    def test_default(self):
        <code>
        self.assertEqual(a,b)   
    
    def test_other(self):
        <code>
        self.assertEqual(a,b)      
      
if __name__ == '__main__':
    unittest.main()

Different test_<name> functions can be added that contain an assert<what> method.

The program test.py can then be called as

python -m unittest -v test.py

python test.py -v

test.py -v since it contains #!/usr/bin/python3

test.py MyTest -v to call the MyTest test cases

test.py MyTest.test_default_bin -v to just call a specific test case (for example the one causing troubles)

Code coverage

A good sign of code quality is if test run all the code. The program coverage can be used instead of the program python.

coverage run <program>.py

coverage report to get a text report

coverage html then open htmlcov/index.html to see a html report

coverage -h to get help

coverage does not check what get started and run in other processes

Python on Gentoo Linux

More than one python version is probably installed on your computer. In Gentoo Linux the command eselect python list shows what is installed and what is taken as default, first you must emerge eselect-python.

To switch to a particular version do eselect python set<n>, then run the /usr/sbin/python-updater, that can update or down-date.

Which python version is active is set by links in /usr/bin

Since Gentoo allows to have more than one python version installed and usable at the same time (like version 2.7 and its new incompatible version 3.4), there needs to be some setup that this works. When installing the Gentoo-way a python scripts they end up under /usr/bin however a closer look shows there is just a link to a wrapper script ../lib/python-exec/python-exec2 inside ../lib/python-exec/ there are other directories as python3.3 that contains all scripts.

The libraries are under /usr/lib/python<pythonversion> where additional installed packages are in /usr/lib/python<pythonversion>/site-packages. When updating and removing an old python version, it can happen that the old directory /usr/lib/python<pythonversion>/site-packages remains, due to unknown garbage inside, so take a look and delete it.

See https://wiki.gentoo.org/wiki/Project:Python

Python on Windows

Python is portable to many different CPU architectures and operating systems, however it runs only if a python interpreter is installed. On a Linux machines as Gentoo, python can be assumed to be installed but on a Windows machine not.

On Windows the python scripts the extension can be renamed from py to pyw to hide the black console window when graphical python scripts are executed. The interpreter pythonw.exe is then used instead of python.exe

As alternative there is Jython (the successor of JPython) http://www.jython.org/, this is python 2.x running on top of a Java virtual machine as java byte code. To use jython type jython. Jython with all its libraries can be installed as java -jar jython_installer-<version>.jar --console or if you like to risk a gui headache java -jar jython_installer-<version>.jar

Install Python on Windows

After installation, python can be started with

  1. IDLE console => python my.py

  2. console => my.py

  3. double click in file manager (but console applications disappear when done)

The best way to start is using Windows menu start and start IDLE. Then open a python script <name>.py and run it as module F5

It might be installed in a user directory and therefore just available for a single user as c:\Users\<username>\AppData\Local\Programs\Python\Python35\python.exe

Python is not added to the windows path environmental variable so it will not be found in console window it need to be started with the complete path.

This can be fixed by adding this path to the windows environmental variables: Control Panel\System and Security\System\Advanced System Settings\Advanced\Environmental Variables

Figure 15.2. Windows Environmental Variables

Windows Environmental Variables


The shell might probably report that some modules (libraries) are not installed

The next problem is how to install those libraries, the standard way is using pip. Newer python version come with pip, on older versions pip needs to be installed first.

update pip (-m makes python know that pip is a module and python therefore knows where to look for it) is done with python -m pip install -U pip

python -m pip install --user bincopy==14.0.0 installs the missing module (bincopy) with a desired version (14.0.0)

A warning might comes that c:\Users\<username>\AppData\Roaming\Phyton\Phython35\Scripts should be set to the path this is recommended otherwise programs as pyinstaller are not found

If the installation fails due to certification errors install the certificates python -m pip install certifi

When debugging with IDLE then it can happen that imports of local modules fail. The reason is that sys.path does not contain the path of the script and the current working path is not the path where the script is. This can be fixed in the shell window of IDLE by sys.path.append('Z:\\<path of current script>'). In this example it is shown that Windows need to get two \\ characters since the first is a string escape character.

Multi Platform programming

To know where the code is running use :

>>> import platform
>>> platform.system()
'Windows'

To get portable code avoid using commands as os.system("cd /home") that execute Linux commands. Use os.chdir(<path>) instead.

os.linesep instead of '\n' for Linux or '\r\n' for windows used in text files. However within python code it is recommended to use all the time '\n' on all platforms and have pythin deal with it.

Path incompatibilities

However the next problem arises the path syntax is not consistent between Windows and Linux.

Pyhon internally the '/' is used also for windows. Printing this out would cause confusion and or passing it to a windows program would cause an error.

The pathlib can be used to convert paths from String to an object. Then different methods are possible.

import pathlib     
# is an object maybe required to convert to string
s=pathlib.PureWindowsPath(<path>)
t=str(s)       # converts object to string
u=s.as_posix() # returns a string

Alternatively it can be dealt on string level using functions/constants/variables as

os.sep instead of the '\' for Windows and '/' for Linux characters. Don't be afraid seeing '\\' for Windows. '\' is used in Strings as escape character allowing to insert none printable characters. So a second '\' character needs to be inserted to indicate that it is a '\' character.

os.path

os.path.split() and os.path.join()

The os module (there are different implementations for the different os) puts an abstraction layer to it that can normalize path names, so it converts Windows forward slashes to backward slashes and have the drive name as directory name. Be aware that not all file systems understand utf-8 encoded file names.

Under Linux ~ is expanded to the users home directory /home/<username> on Windows this works as well but will be expanded to C:\Users\<username>

Note

Directories under C:\Users\<username> that start with a '.' character are an indication that they got ported from a non Windows system to Windows (as from Linux to Windows)

Lxml

lxml might be problematic so it is wise to include it as

          # call lxml as ET so it is compatible with the python 
# built-in library xml but the library lxml has more
# features as pretty-print and xml declaration support
lxml=False
try:
   from lxml import etree as ET
   lxml=True
except:
   import xml.etree.ElementTree as ET

<some code>

if lxml==True:
   tree.write(self.sessionPath+os.sep+self.sessionFile,  
   encoding="utf-8", pretty_print=True,   
   xml_declaration=True) 
else:            
   tree.write(self.sessionPath+os.sep+self.sessionFile, 
   encoding="utf-8")    

<some code>

Converting Python scripts to a Windows exe

pyinstaller from http://www.pyinstaller.org/ offers a nice way to create a Windows exe. To run pyinstaller Python needs to be installed under Windows.

Important

Take the time to check what python version (as 3.5.2) runs well with pyinstaller and avoid using not supported versions

pyinstaller allows converting python scripts into windows exe files not requiring to fiddle around with installing and setting up python on a windows computer. Those windows exe do not require to administrator privilege to install them, since they contain usually all required code. Two ways are possible

  1. Everything packed in a single exe file. that when started extracts itself to a os temp directory as C:\Users\<username>\AppData\Local\Temp\<some name>\, then it runs from that and finally the directory gets removed. Since there is just one file it is easy to distribute it. Drawback is that it is slower to start and no files as readme.txt can be observed.

  2. Directory structure that contains a small windows exe to be started and all other files (as dll or text files as readme.txt.

    Having a directory structure is also better during testing and developing, since it can be easy observed what files are around and where. Regarding paths pyinstaller is a bit tricky since a boot loader starts the windows exe.

    Important

    Certain paths are different depending how python code is started.

pyinstaller <name>.spec creates the <name>.exe.

The <name>.spec can be created automatically the first time by calling pyinstaller <name>.py

Important

Do not call pyinstaller <name>.py since it overwrites <name>.spec with a default one unless a new default <name>.spec is required. Create a backup copy of <name>.spec with a useful name as <name>-onefile.spec and call it pyinstaller <name>-onefile.spec

Files can be added in <name>.spec

 a = Analysis(
             datas=[('*.gif','.')], ...

The python tuple means copy it from sorce to destination.

There are also command line options when creating or overwriting a <name>.spec file.

--add-data '*.gif:.' copies *.gif to the bundle

--onefile creates single <name>.exe file <name>.exe

--windowed hides the console window and shows just the gui window. Drawback no error messages appear.

--onefile puts everything in a <name>.exe file that when run gets first unzipped in temp folder of OS

--icon file.ico attaches a windows icon to replace the standard python icon seen in the windows file manager. it is not the icon in the window decoration of the application.

The pyinstaller can also be run under Linux as well, since it is pure python. The same commands as above can be used. However the result will be different, it will a Linux binary.

Note

pyinstaller can also be installed on Linux (and Mac OS X, FreeBSD, Solaris and AIX) and can create single file executables as on Windows using pyinstaller <name>.spec or if pyinstaller is installed locally ~/.local/bin/pyinstaller <name>.spec. This is a nice option to freeze all its dependencies to the python script and have it therefore archived and executable.

Running Python scripts without installing Python

Under Windows, Python can also run from a memory stick without requiring an installation, as usual under Linux there are different ways to do this:

  1. http://portablepython.com/ comes with an exe that is an installer. There are 2.6 but also 3.2 versions available. The versions have different IDE and libraries included. The pyscripter is a full featured IDE with everything desired and more. SPE is an other ide that makes use of winpdb as debugger. py2exe and the necessary DLL's are also present. Once installed the portable python directory can be copied and moved anywhere.

  2. http://www.voidspace.org.uk/python/movpy/, unzip and go to the movpy directory and double click on movpyw to get the gui launcher. In the gui launcher the python script can be selected. There are also options to be clicked, on useful option is b that stops the console window after the script has run, this way it will be seen what the script has written to the console. IDLE is also available where the script can be debugged. Python version 2.5.1 comes with it.

  3. Convert the python script into a windows exe file using pyinstaller

Distribute python code

Python comes with distutils to have a standard way to distribute python code and therefore create a standard way to install it.

See: https://docs.python.org/3.5/distutils/ http://www.diveintopython3.net/packaging.html

Creating a python package

To create distutils package a setup.py file has to be created that holds the information distutils requires.

For single py files this setup.py file can placed beside the py files and can look as follows:

        from distutils.core import setup
setup(name='<name of the module>',
      version='<version>',
      py_modules=['<name of the module, without py extension>'],
      )

For more complex program it is wise to create a package, a collection of files. So put them in a subdirectory and place a setup.py file above that directory that could look as:

        from distutils.core import setup
setup(
      name="girttools",
      packages = ["girttools"],
      version="0",
      description='Tools for Gnu Infra Red Transceiver',
      author='Urs Lindegger',
      author_email='urs@linurs.org',
      url='http://www.linurs.org/girt.html',
      download_url = "http://www.linurs.org/girt/download/girttools-0.tgz",
      keywords = ["Infrared transceiver"],
      classifiers = [
        "Programming Language :: Python",
        "Programming Language :: Python :: 3",
        "Development Status :: 4 - Beta",
        "Intended Audience :: End Users/Desktop",
        "License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)",
        "Operating System :: POSIX :: Linux",
        "Topic :: Multimedia",
      ],
      long_description = """\
Gnu Infra Red Transceiver Tools
-------------------------------

Tools to setup and work with the Gnu Infra Red Transceiver that can replace all you IR remote controllers.
"""

Note

The setup function can be called with either packages this is a list of subdirectories containing the py files or py_modules a list of python files without adding the .py extension.

The long_description (or since it is python everything) can also be read out of a file:

        long_description=open('README.txt').read()

Instead of py_modules that can list various py files packages is used containing the subdirectory that must contain a __init__.py file.

Finally when done packages can be register so other will find them https://pypi.python.org/pypi?:action=register_form. Python Package Index (“PyPI”) allows to classify your package so it can be found, therefore don't invent your own classifiers select them out of https://pypi.python.org/pypi?:action=list_classifiers

To include attentional files create a MANIFEST.in file as defined in https://docs.python.org/3.1/distutils/sourcedist.html. The MANIFEST.in file can also contain wild-carts and distutils specific commands as recursive directory functions.

Note

Adding files is not the same as having them installed. To install them to the same location as the package goes, use package_data in the setup.py file and not MANIFEST.in.

Finally MANIFET.in is read among other things defined in setup.py to create the MANIFEST file that contains all files

setup.py can also contain data_files that can be used to install files in any directory. Due to specifying the path the *.tar.gz package will no more be portable to different systems. An example would be installing a Linux init script to /etc/init.d, however on a Linux system using systemd instead of init or even on a Windows machine this makes not much sense to install it. It is better to do this on the system specific package as for Gentoo Linux in the ebuild. However you simply add them to the *.tar.gz using the MANIFEST.in file.

python setup.py check will check your setup.py

To get the *.tar.gz package in the subdirectory dist run python setup.py sdist (source distribution) or if you want to make it easy for windows and create an graphical windows installer, run : python setup.py bdist_wininst

Note

setuptools is an enhanced alternative to distutils. Nice is that also setup tools uses a setup.py file. So Building the package is also done with python setup.py sdist

After having the tar.gz you could but should not install it as : python setup.py install Instead of that you should create a package that fits you Linux distribution and then use the package manager to have it installed. Under Linux python modules (py files) are copied to the directory /usr/local/lib/python<X>.<Y>/site-packages and packages (directories containing __init__.py) are copied into a subdirectory under /usr/local/lib/python<X>.<Y>/site-packages.

Note

__init__.py indicates that all files in the same directory belong to a package and the directory is the name of the package.

Note

/usr/local/lib/python<X>.<Y>/site-packages is a location that allows to import those modules and packages into other python modules but it is not a location where the command line finds them. Python files to be started on the command line must be declared in the setup.py script as scripts. Under Linux those scripts will end up in the /bin directory where the command line finds them.

Alternatively distutils allows to install the package to an other directory as your home directory with python setup.py install --home=~ any other directory with python setup.py install --home=<dir> where no system conflicts should occur.

https://docs.python.org/3.1/distutils/builtdist.html helps you to move your package to distributions as creating a RPM red hat package.

Gentoo python ebuild

For Gentoo Linux an ebuild needs to be developed that first installs the package in a sandbox and then copied (merged) to your system getting track what files will be added and check collisions.

Ebuilds can make usage of eclasses and there is a distutils eclass (https://wiki.gentoo.org/wiki/Project:Python/distutils-r1), so the ebuild for a python module can be very simple as:

EAPI=6

PYTHON_COMPAT=( python{3_4,3_5,3_6} pypy )

inherit distutils-r1

DESCRIPTION="Library for all sort of python code"
HOMEPAGE="https://www.linurs.org"
SRC_URI="http://www.linurs.org/download/${P}.tar.gz"
LICENSE="Unlicense"
SLOT="0"
KEYWORDS="~arm64 ~x86 ~amd64"
IUSE=""
DOCS=( README.txt )

${P} is the filename of the ebuild including version number.

The inherit function loads distutils and this overwrites the standard install routines. So the ebuild contains no code just meta data about the python package and where to find it on the Internet remains.

The README.txt file gets zipped under /usr/share/doc/${P}/

Accessing C code

There is SWIG http://www.swig.org/ that creates wrapper code around the C or C++ code. This wrapper code can then be accessed by Python.

Or using cython: http://cython.org/

Python byte code

When a python code is started, then the source is converted into bytecode https://docs.python.org/release/2.5.2/lib/bytecodes.html and a virtual machine then executes this bytecode. When the same program is started again then the bytecode is started (except when the source has changed, then a new byte code is created). The files containing byte code have the extension pyc (or when optimized pyo). Be aware that the bytecode might be incompatible between different releases of python. Python has support for disassembling byte code:

>>> import dis
>>> def myfunc(alist):
>>>   return len(alist)
>>> dis.dis(myfunc)
  2           0 LOAD_GLOBAL              0 (len)
              3 LOAD_FAST                0 (alist)
              6 CALL_FUNCTION            1
              9 RETURN_VALUE        

Linurs Servernest startpage