Generic Python Library#

for pdms scoring pipeline and the refactoring of digital twin, to make the most of MLOps best practices, a generic python library is used resp. pdms-core-modules and digital-twin-core-modules.

Library Architecture#

The library structure contains two layers of the generic_core_module , this separates anything that needs to be contained in the library in the second layer of generic_core_module . Anything thats supports the creation of the python library is stored in the first layer of generic_core_module . 

generic_core_module
│   README.md
│   MANIFEST.ini
|   gitlab-ci.yml
|   requirements.txt
|   setup.py 
│
└───src
│   │
│   └───generic_core_module
│       │   main.py
|       |   __version__.py
│   └───tests
│       │   test_main.py
│   
└─── docs ..

supporting library creation#

There are five files supporting the creation of the generic_core_module : readme.md, manifest.ini, gitlab-ci.yml, requirements.txt and setup.py.

  • setup.py is used by the python package build, to build the python repostory taking in version.py, readme.md and the requirements.txt. Which modules can be used, by using the package is defined with packages = find_packages() .

  • readme.md contains the description of the generic_core_module and is used by the setup.py for a long_description of the python library. And the content of the long_description can be checked by running the following syntax in the root of the repository. pip install twine build | python -m build | twine check dist/*

  • manifest.ini determines which none .py files are included when building the python package e.g. requirements.txt

  • gitlab-ci.yml is a yml template which defines the ci/cd pipeline to build the python package and upload to the gitlab python repository. It currently contains two stages, testing the python package and building/uploading the package.

  • requirements.txt are the python packages used by the python library. When building the python package all dependencies in the requirements.txt are build into the python packages. And will be downloaded when you pip install the python package.

outstanding user stories#

  1. An outstanding improvement to be made is creating a gitlab template, that creates an empty python library following the structure as described below, and already containing the five supporting files, user story to create gitlab template