Some time ago I wrote about GitLab Continuous Integration and explained how code changes can be catched up to create or update software packages on remote systems. In order to communicate with the development systems, GitLab utilizes agents that are availble for plenty of platforms – including Microsoft Windows, we will focus on in this article.
In the following example, Python applications (*.py) are converted automatically into executables (*.exe) after code changes. But – why? I’m using this concept for generic Icinga monitoring plugins. Of course, Python scripts can also be executed on Windows after installing the runtime environment. Think about deploying this on a big amount of systems – it’s a huge administration effort. A single executable can be distributed easier and also offers less security risks than a complete runtime environment that consists of multiple sub-programs.
Python development guidelines SCSS style guide Shell scripting standards and style guidelines Sidekiq debugging Sidekiq style guide. GitLab Runner officially supported binaries are available for the following architectures: x86, AMD64, ARM64, ARM, s390x, ppc64le.
There are multiple applications for converting Python scripts into executables – I’m using pyInstaller. This tool combines all required Python modules and also the Microsoft Visual C++ runtime environment necessary for Windows into a single file. pyInstaller is also availble for other platforms and currently supports Python 2.7 and 3.
- Pytest is used to run unit tests in the Analytics project. The tests are executed from the root directory of the project with the pythonpytest CI pipeline job. The job produces a JUnit report of test results which is then processed by GitLab and displayed on merge requests. Writing New Tests.
- Java, PHP, Go, Python or even LaTeX, no limit here! In this blog post we review a few examples for the Python programming language. In GitLab, the different tests are called jobs. These jobs are executed by a gitlab-runner, that can be installed on the same server as you main GitLab.
My goal was to achieve that all code changes in the master branch forces the re-creation of the appropriate executables.
First of all, it is necessary to enable CI for a GitLab project. Proceed with the following steps:
- Select the project, click Settings and Project Settings.
- Click Builds and Save Changes.
- Move to the CI Settings pane to set advanced parameters such as timeouts and automatic builds.
- Click Save Changes.
- Select Runners, note/copy the CI-URL and the CI-Token – we will need this information later for registrating the runners.
To implement agent communication and automatic builds, additional preparations need to be fulfilled. The most important is to create a dedicated user account or service user. Depending on your infrastructure this step might vary (e.g. local account, Active Directory). It is important that this user has the permission to run services. Unfortunately, client systems like Windows 7 don’t offer a setting like this – in this case the user needs to have local administrative permissions. 🙁
Beyond that, the following software packages are required:
- Python: [click me!]
- Git for Windows: [click me!]
- PyInstaller: [click me!]
- GitLab Runner: [32bit] or [64bit]
The particular install process are self-explanatory. When installing Python, it is necessary to check “Add python.exe to path” – otherwise Python programs are not executable from the command line or PowerShell. For the Git installation a similar option “Use Git from the Windows Command Prompt” needs to be set, too.
PyInstaller is installed using a Python script – make sure to start it inside a command line with administrator privileges:
Python and PyInstaller
Let’s check whether Python and PyInstaller work as expected. The following source code should be catched up by PyInstaller without any issues:
The script prints a sentence and waits for an input before it closes itself. The following command creates an executable including the script, runtime environment and required Python modules:
The switch -F integrates the Python and Microsoft Visual C++ runtime environments, -y overwrites pre-existing files. If the utility finishes without any errors, new folders build and dist are created in the current directory. There is a folder with the same name like the script including the binary inside the dist folder. You can start it like this:
Gitlab Runner Python Download
By the way – for Python scripts that need to be executed under Windows, it is necessary to load the Python module os. Otherwise, the following error message is created when the executable is started:
Powershell and CI-Runner
For automating the packaging process, I’m using a PowerShell script. To ensure that custom and thus unsigned PowerShell scripts can be executed, I had to alter the execution policy:
Setting Up Gitlab Runner
The Microsoft Knowledge base offers good explanations on the particular settings.
The GitLab CI runner is moved into a directory on the hard drive, e.g. c:gitlab-ci-runner. To make manual debugging easier, it is also advisable to shorten the long file name. Afterwards the runner is registered to the GitLab system – for this task we need to enter the project token. After this, the runner is installed and started as Windows service:
To make GitLab push the code changes to the runners, we need to create the .gitlab-ci.yml file. As I already mentioned in the last post, this file controls the behavior of runners. The following script automatically executes a PowerShell script after every commit to the master branch.
Gitlab Runner Python Example
The PowerShell script looks like this:
The script is kinda simple, it moves into every sub-folder and converts a Python script with the same name.
Conclusion
There are still a couple of things in the scenario that could be extended – for example:
Gitlab Python Ci
- Compiling only on particular nodes (e.g. depending on additional dependencies or hardware ressources)
- Functional tests after compiling – e.g. using unittest
- Automatic upload of converted files using artefacts
I’m really interested in deep-diving into artefacts, so stay tuned. 🙂