Custom installation of Python
Installing Python 3
While CentOS 7 comes with Python 2.7 by default, Python 3 is the way to go moving forward. Note that Python 2 have reached EOL (end-of-life) in 2020 (see PEP 373)
Python 3 is an example of a non-trivial software package that can be installed across the cluster, which should not be in conflict with the system installation of Python.
There are three necessary tasks to accomplish this:
Install the necessary library dependencies for Python 3 to function on master and the compute nodes
Install the necessary library headers on
masterso Python 3 can be compiled with themCompile Python 3 on
masterand install it into/data/opt/tools/
Note
If GCC is not in your system, install the build environment
[root@master ~]# yum groupinstall "Development Tools"
Start by downloading the a more recent source code package of Python 3 (remember, work as install user):
[install@master ~]$ wget https://www.python.org/ftp/python/3.9.19/Python-3.9.19.tgz
Untar the Python source code, configure it with --prefix=/data/opt/tools/python-3.9.19 and compile it. This ensures that make install will copy all files into /data/opt/tools/python-3.9.19, which is owned by the install user and therefore doesn’t require sudo/root rights.
[install@master ~]$ tar xvzf Python-3.9.19.tgz
[install@master ~]$ mkdir build-python-3.9.19
[install@master ~]$ cd build-python-3.9.19
[install@master build-python-3.9.19]$ ../Python-3.9.19/configure --prefix=/data/opt/tools/python-3.9.19
[install@master build-python-3.9.19]$ make -j 12
In case of missing dependencies (headers, libraries) the Python build will output what optional modules were not compiled.
INFO: Could not locate ffi libs and/or headers
Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2 _curses _curses_panel
_dbm _gdbm _hashlib
_lzma _sqlite3 _ssl
_tkinter _uuid readline
zlib
To find the necessary bits, look in setup.py in detect_modules() for the module's name.
The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc atexit pwd
time
Failed to build these modules:
_ctypes
Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381
Try finding the missing dependencies, install the libraries and their headers via yum and then rebuild. Take notes on what packages you needed to install as RPM. You will need these notes later!
# example, install zlib library and headers
yum install zlib zlib-devel
Note
On RedHat based systems like CentOS library packages are usually split up into two parts: one package containing the libraries (e.g. zlib), and another package containing the necessary development headers (e.g. zlib-devel).
Finally, do a make install.
If you look at the first two levels of directories in /data/opt/tools/python-3.9.19, you will see the package provides python3, pip3 and other binaries in /data/opt/tools/python-3.9.19/bin.
[install@master python-3.9.19]$ tree -L 2 /data/opt/tools/python-3.9.19
/data/opt/tools/python-3.9.19
├── bin
│ ├── 2to3 -> 2to3-3.9
│ ├── 2to3-3.9
│ ├── easy_install-3.9
│ ├── idle3 -> idle3.9
│ ├── idle3.9
│ ├── pip3
│ ├── pip3.9
│ ├── pydoc3 -> pydoc3.9
│ ├── pydoc3.9
│ ├── python3 -> python3.9
│ ├── python3.9
│ ├── python3.9-config -> python3.9m-config
│ ├── python3.9m
│ ├── python3.9m-config
│ ├── python3-config -> python3.9-config
│ ├── pyvenv -> pyvenv-3.9
│ └── pyvenv-3.9
├── include
│ └── python3.9m
├── lib
│ ├── libpython3.9m.a
│ ├── pkgconfig
│ └── python3.9
└── share
└── man
8 directories, 18 files
To make this customized software installation usable, we will need to modify some of our environment variables:
[install@master ~]$ export PATH=/data/opt/tools/python-3.9.19/bin:$PATH
[install@master ~]$ export LD_LIBRARY_PATH=/data/opt/tools/python-3.9.19/lib:$LD_LIBRARY_PATH
Setting PATH and LD_LIBRARY_PATH variables will make your python installation usable.
[install@master ~]$ python3
Python 3.9.19
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
To enable man to find the new Python 3 man pages in /data/opt/tools/python-3.9.19/share/man update the MANPATH environment variable.
[install@master ~]$ export MANPATH=/data/opt/tools/python-3.9.19/share/man:$MANPATH
For software that need the Python include headers to be visible during compilation one can define CPATH or C_INCLUDE_PATH:
export CPATH=/data/opt/tools/python-3.9.19/include/python3.9:$CPATH
PATHdefines the search path for finding executables in your shell
LD_LIBRARY_PATHdefines the shared library search path the dynamic linker uses at startup
MANPATHdefines the search path for finding man pages
CPATHdefines a global include path for compilers such as
gcc
Installing Python 2
While Python 2 is installed by default by CentOS, let’s assume we have cluster users that require the latest Python 2 release. Or maybe you want to provide them with optimized builds for your particular cluster.
Follow the steps for Python 3 and install Python 2.7.16 in its own location at /data/opt/tools/python-2.7.16.
Verify your installation works by exporting the necessary environment variables and running python2.
[install@master ~]$ export PATH=/data/opt/tools/python-2.7.16/bin:$PATH
[install@master ~]$ export LD_LIBRARY_PATH=/data/opt/tools/python-2.7.16/lib:$LD_LIBRARY_PATH
[install@master ~]$ export MANPATH=/data/opt/tools/python-2.7.16/share/man:$MANPATH
[install@master ~]$ export CPATH=/data/opt/tools/python-2.7.2/include/python2.7:$CPATH
[install@master ~]$ python2
Python 2.7.16 (default, Mar 4 2019, 08:11:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>