Pipeline Templates

Process

The abstract pipeline template describes the process which is implemented by the pipelines in the different pipeline tools. The process is divided in to 2 main steps which is then split into tasks.

• Build
• Deploy

The number of tasks are mainly defined by the selected component type of component creation and the technology within the provisioning process.

Build

The build step has the function to build your application and create an executable Docker container from source code. This ensures a permanent releasability of your application and covers the aspects of quality and security. It fulfills the common practice of continuous integration and continuous delivery.

Process source code

This task covers all source code based actions like compiling, packaging and execute unit test.

Security tasks

This task covers security scan actions. This could include static code analysis, linting, credential scanning, AST, OSS governance and SCA.

Create container

This task covers all steps to build a container and upload it to an image repository.

Scan container

This task cover the SCA scan of the container that was build previously.

Push container

This task will push the container to the container registry.

Push API spec

This task will push the API spec to ODJ. It is related to the container and can be different per stage.

Sign container

Signs the container depending on the selected runtime environment to ensure that the container can be executed.

Deploy

The deploy step can be executed multiple times depending on the stages that have been configured in the pipeline. The "productive stage" (by default PROD) have some special task here. It fulfills the common practice of continuous delivery.

Custom preparations (optional)

Additional tasks that can be executed before a deployment is triggered.

Deploy

This task deploys the container to the defined stage.

Scan container (productive stage only)

Running the final SCA scan of the container that was deployed to production. This scan will then be constantly tracked and will report new vulnerabilities depending on the SBOM stored in the SCA tool.

Custom tests (optional)

Custom tests may contain different additional tests that you may want to execute after your deployment. This may include feature or integration tests, security tests or performance tests. It's up to you to add them.

Component types and pipelines

The different component types have different effects to the pipeline. Due to this definition, the pipeline will add or remove steps automatically.

Deployable

This type has the goal to produce a docker container from source code. This is a typical software package that you want to deploy an run on your runtime environment.

All selected tools from the categories will be used (e.g. dependency scanner, static code analysis,...)

Example:

A component of the online shop which handles the backend logic of a shopping cart

Library

This type is for producing a shared library which has not the goal to run it on a runtime environment. The creation of a docker container is skipped. The pipeline steps for deploying are skipped.

All selected tools from the categories will be used (e.g. dependency scanner, static code analysis,...)

Example:

Shared library that contains code that is shared between multiple components and products

Supportive

This type is for producing a supportive docker container that is started manually in your runtime. The pipeline steps for deploying are skipped.

All selected tools from the categories will be used (e.g. dependency scanner, static code analysis,...).

Example:

A container that is added to the k8s cluster to analyses network, migrate databases or access cluster internal resources for support cases

---

Pipeline templates in Azure DevOps

Documentation about the ODJ pipeline templates implemented with Azure DevOps Pipelines.

Template extension hierarchy

The ODJ pipeline templates are based on 3 levels of extension hierarchy.

Technical description

A detailed technical description, how to use it and FAQs of the whole pipeline templates in Azure DevOps can be found in the ODJ Pipeline templates repository.

Generic pipeline

The generic pipeline defines the default process. This template also covers the security dependant step topics like static code analysis and SCA (software composition analysis).

This file is part of the ODJ pipeline templates GIT repository.

A detailed description of the generic template can be found here.

Technology pipeline

The technology pipeline file handles the build specific parts of the pipeline. It coverts the tasks for the build tool (like Gradle, NPM...) and the interaction with the required scanning tools like SonarQube or Snyk. It also takes care about things like showing the unit test coverages and unit test executions in the pipeline.

This file is part of the ODJ pipeline templates GIT repository.

A detailed description of the technology template and how they are used can be found here.

Component pipeline

The component pipeline file is part of the customers GIT repository that ODJ creates within the provisioning process. The file is named "odj-azure-pipeline.yml" and stored in the root folder (for multi-repo) or in the subfolder of the component (for mono-repos).

This file extends a technology template or directly the generic template. This can be chosen within the provisioning of the component.

Pipeline extension points

The pipeline templates can be extended via extension points. With them you can add your own required tasks to add specific features depending to your build.

Extension points are added with the generic pipeline template. The technology pipeline templates also offer the option to add additional extension points. They are documented in the generic pipeline templates or in the technology specific documentation files.

Deploy tool

The deploy tool is used in the ODJ pipeline templates to trigger a deployment of an artifact via the ODJ. It will trigger the deployment in ODJ Core and checking the status of the deployment. The logs in Azure Devops will be grouped and you can fold / unfold the logs with the little triangle at the left of the group start.

Looking for latest stage run

At this step we are looking for the latest stage run ID before we start with the deployment so that we know which ID we have to ignore at looking for the new run ID.

Modify artifacturl for the component/s

At this step we are modify the components configuration of the core product to add the new artifact url (Docker) for the main component and eventually for added additional components.

Waiting for deployment to start

At this step we are looking for a new stage run ID and this will be considered as true if it is not matching the ID from step "looking for latest stage run".

Checking component/s status in zones

At this we fetch all zones for component/s that start with "stage-[stage]-component-deployment-[component name]" and then looking for the status of all nodes until all done, one is in error or it runs in the timeout.

Checking stage status

At this step we checking the overall status of stage run when the components check was done otherwise it will skipped. This check will be done until we got the status done, error or it runs in the timeout.

Status Calculation

The end status will be calculated depending on the result of the component and stage status. The status will be reflected through the pipeline status as well.

Component status	Stage status	End status (pipeline)
Error / Timeout	Done / Error / Timeout	Error
Done	Error / Timeout	Warning
Done	Done	Done

A warning now indicates that your current component was deployed to your runtime, but there is a general error in the stage run (like an issue with another component or other infrastructure components).

Beta support

The pipeline template contains some beta support for special environments that cannot be provisioned via the ODJ.

AppCloud (Cloud Foundry)

Deploy to STACKIT external appcloud instance (bring your own appcloud)

info

Only external AppCloud is supported.

OCP is End-of-life

This solution will be replaced with full ODJ advanced product deployments support when the STACKIT advanced product integration for AppCloud is finished. There will be no migration scenario!

Prerequisite

• odj_runenv_provider: 'stackit' (must be changed by hand)
• odj_runenv_runtime: 'appcloud' (must be changed by hand)
• component type is "deployable" or "supportive"

The following actions will be done:

• Download deployable from pipeline artifact storage
• Use cf cli to push the application to appcloud

Actions

Please follow the documentation included in the pipeline templates repo.

OCP4

This support will build a container with our default pipelines and push it to Artifactory so that it can be used in on-prem OCP4.

OCP is End-of-life

Migrate your applications away from OCP as it is end-of-life! This support will the be dropped from ODJ.

Prerequisite

• odj_runenv_provider: 'onprem' (must be changed by hand)
• odj_runenv_runtime: 'ocp4' (must be changed by hand)
• Artifactory is provisioned
• component type is "deployable" or "supportive"

The following actions will be done:

• build docker container
• use dependency scanner
• push to Artifactory (must be activated or it fails the build!)

Actions

This steps describe how to replace the original ODJ deployment with your custom deployment.

Define variable "docker_repository_host"

Variable "docker_repository_host" must be defined by hand with the correct target docker registry in Artifactory (e.g. schwarzit-xx-sit-odj-soul-docker-local.jfrog.io) in the product specific variable group.

If it does not exist in the product specific variable group, then just add it.

Example:

Define variable "docker_image_prefix"

This variable must be defined with '' (an empty string). If it does not exist in the product specific variable group, then just add it.

Disable ODJ deployment

As ODJ cannot deploy out-of-the-box to on-prem OCP, you have to disable it manually with the pipeline parameter "odj_deployment: false".

Example:

Custom deploy jobs

You have to add your own deploy job to the pipeline. Don't forget to add the required agent-pool to your project.

Example:

---

Permission (SIT)

Describes the permission concept of users to the pipeline templates repository to maintain them. This is not related to the component pipeline files in the customers repositories or pipelines.

Main Azure DevOps organization (SIT)

This permission overview is for the main repository in https://dev.azure.com/schwarzit/schwarzit.odj-pipeline-templates.

ADO group	containing users/groups	Description
[schwarzit.odj-pipeline-templates]\Owner	- [TEAM FOUNDATION]\azuredevops-xx-schwarzit-odj-pipeline-templates-owner - ado-odj-deliver@schwarzit.onmicrosoft.com - External developers from ODJ	Administrators for pipeline template repository (ODJ Deliver & Tooling team)
[schwarzit.odj-pipeline-templates]\Member	- [TEAM FOUNDATION]\azuredevops-xx-schwarzit-odj-pipeline-templates-member	Contributors for other scripts
[schwarzit.odj-pipeline-templates]\Reader	- [TEAM FOUNDATION]\azuredevops-xx-schwarzit-odj-pipeline-templates-readers - External developers from any team	Read access for all ODJ users. This role will be ordered automatically for all ODJ users
[schwarzit.odj-pipeline-templates]\Approvers	- Dominik Doll - Thomas Pawlassek	Team to approve PR's for changes in the pipeline templates
[schwarzit.odj-pipeline-templates]\Approvers - Mendix	- Dominik Doll - Thomas Pawlassek - Björn Sammet	Team to approve PR's for changes in the pipeline templates for Mendix

SIAM Roles (SIT)

SIAM role	Additional approver
azuredevops-xx-schwarzit-odj-pipeline-templates-owner	- Dominik Doll - Thomas Pawlassek
azuredevops-xx-schwarzit-odj-pipeline-templates-member	-
azuredevops-xx-schwarzit-odj-pipeline-templates-readers	-

Synced Azure DevOps organization (SIT)

This permission overview is for the all synced organizations.

ADO group	containing users/groups	Description
[ORGA.odj-pipeline-templates]\Owner	- Dominik Doll - Thomas Pawlassek - ado-odj-deliver@schwarzit.onmicrosoft.com	Limited access to synced repos. Changes in this orgas should only be done in case of emergency!
[ORGA.odj-pipeline-templates]\Member	-	-
[ORGA.odj-pipeline-templates]\Reader	- [TEAM FOUNDATION]\azuredevops-xx-schwarzit-odj-pipeline-templates-readers - External developers from any team	Read access for all ODJ users. This role will be ordered automatically for all ODJ users.