Stream of Consciousness

Mark Eschbach's random writings on various topics.

k8s: Pipelines in Pipelines

Categories: programming

Tags: kubernetes continuous-delivery tekton

A few searches later turns out Tekton has an experimental set of CRDs which might be able to help! Installation is fairly straight forward: kubectl apply --filename https://storage.googleapis.com/tekton-releases-nightly/pipelines-in-pipelines/latest/release.yaml.

With a pipeline like:

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: go-trusted-service-container
spec:
  description: |
    Clones, tests, and builds Golang artiactes    
  params:
    - name: repo-url
      type: string
      description: The git repo URL to clone from.
    - name: docker-image
      type: string
      description: Docker image to store product as
    - name: TARGETARCH
      type: string
  workspaces:
    - name: scm
      description: |
        Workspace cloned from the scource code management system        
    - name: git-credentials
      description: My ssh credentials
  tasks:
    - name: fetch-source
      taskRef:
        name: git-clone
      workspaces:
        - name: output
          workspace: scm
        - name: ssh-directory
          workspace: git-credentials
      params:
        - name: url
          value: $(params.repo-url)
    - name: build
      runAfter: ["fetch-source"]
      taskRef:
        name: golang-build
      workspaces:
        - name: source
          workspace: scm
      params:
        - name: TARGETOS
          value: linux
        - name: TARGETARCH
          value: $(params.TARGETARCH)
        - name: package
          value: "./cmd/service"
        - name: docker-image
          value: "$(params.docker-image)"

Running with both architectures as:

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: go-trusted-services-containers-all
spec:
   tasks:
   - name: container-amd64
     taskRef:
        apiVersion: tekton.dev/v1beta1
        kind: Pipeline
        name:  go-trusted-service-container
        podTemplateSpec:
           nodeSelector:
              "kubernetes.io/arch": "amd64"
     params:
        - name: TARGETARCH
          value: amd64
   - name: container-amd64
     taskRef:
        apiVersion: tekton.dev/v1beta1
        kind: Pipeline
        name:  go-trusted-service-container
        podTemplateSpec:
           nodeSelector:
              "kubernetes.io/arch": "amd64"
     params:
        - name: TARGETARCH
          value: arm64

Well that failed spectacularly! podTemplateSpec fails validation which makes sense when I think through. taskRef does not define additional fields.

Maybe buildah ?

While researching how to solve this issue looks like Buildah does support multi-architecture builds which would simplify this whole mess. Many of the toolmakers seem pretty annoyed at the requests for multi-arch oddly.

Turns out buildah requires a privileged container. I would rather avoid that.

Maybe img ?

Well that looks promising. Let’s give this another whirl this time.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: golang-build-img
  labels:
    app.kubernetes.io/version: "0.1"
  annotations:
    tekton.dev/pipelines.minVersion: "0.12.1"
    tekton.dev/categories: Build Tools
    tekton.dev/tags: build-tool
    tekton.dev/displayName: "golang build"
    tekton.dev/platforms: "linux/amd64,linux/arm64"
spec:
  description: >-
    Tests then builds a Golang service    
  params:
    - name: package
      description: base package to build in
    - name: docker-image
      description: Docker image URL in repository:tag format
    - name: TARGETOS
      description: target operating system.  generally linux
    - name: TARGETARCH
      description: target architecture.  generally AMD64 or ARM64
  workspaces:
    - name: source
  steps:
    - name: build
      image: docker.workshop.meschbach.org/mee/platform/golang-builder:1.18.3
      workingDir: $(workspaces.source.path)
      script: |
        go test ./pkg/...
        CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -ldflags='-w -s -extldflags "-static"' -o service $(params.package)        
    - name: package
      image: r.j3ss.co/img
      workingDir: $(workspaces.source.path)
      args: ["build", "--tag","$(params.docker-image)",  "--file $(params.package)/Dockerfile", "$(workspaces.source.path)"]

After some experimentation this resulted in a lot of extra priveleges.