Serving assets from Minio through k8s ingress

Categories: tech

I was thinking: I have a Minio instance and a bucket holding static content on my homelab cluster. What would it take to expose the content via my public ingress? Specifically I run haproxy-ingress.

C4Context
  title Static Minio assets via haproxy-ingress
  Boundary(inet,"Internet"){
    Person(rando,"Rando")
  }
  System_Boundary(k8s,"k8s"){
    System(haproxy,"haproxy-ingress")
    System(minio,"Minio")
    ContainerDb(bucket, "Static Assets")
    Rel(haproxy, minio, "Exposes")
    Rel_R(minio, bucket, "Contains")  
  }
  Rel(rando, haproxy, "HTTP(S)")

To make this work there are several key elements:

Path needs to be prefixed with the target bucket name.
If we are exploring a directory then the index.html should be appended to the end.
Ignore ACME verification for provisioning a certificate but require all other requests to be over HTTPS.
Host should be updated to an allowed name for the API.

For haproxy-ingress we will use the annotation haproxy-ingress.github.io/config-backend on our ingress. This allows us to inject HAproxy configuration stanzas into the actual ingress configuration. The value of this annotation injected into a shared backend for all ingresses targeting the specified service. Likely this service is used to expose the API and Console.

acl is used to match conditions for the request and set a flag. By default https-request is already set by haproxy-ingress when HTTPS is in use. Two new matches will also be detected. A scope to the target host to expose the bucket as and one for the ACME challenge. haproxy-ingress sets the variable req.host with the host request which can be matched on. ACME verification starts with /.well-known/acme-challenge. I believe these are scoped to the backend so a unique name within that scope should be used.

      acl soc_integ var(req.host) -i soc.example.com
      acl wellknown_cert_check path -i -m beg /.well-known/acme-challenge

Actions may have conditional suffix which can test flags set with acl using a stanza. The following will redirect only when the target is the site, not already an HTTPS request, nor an ACME setup.

      http-request redirect scheme https if soc_integ !https-request  !wellknown_cert_check

Requests may be rewritten via set-header and set-path. These need to be scoped to the target site. First line should be an allowed host for the S3 API. Second is the bucket name prefixed with /.

      http-request set-header Host s3.example.com if soc_integ
      http-request set-path /com-meschbach-soc-integ%[path] if soc_integ

Last is to handle the default file. When the file with a trailing slash we append index.html to ensure we can get the index file. nginx can be internally set to try a number of resources however no easy similar find to try_files. This uses a different version of if allowing one to avoid the acl expression. Great for one offs.

      http-request set-path %[path]index.html if soc_integ { path_end / }

Putting it all together it looks something like the following:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: website-soc-meschbach-integ
  namespace: minio-prod
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    haproxy-ingress.github.io/config-backend: |
      acl soc_integ var(req.host) -i soc.example.com
      acl wellknown_cert_check path -i -m beg /.well-known/acme-challenge
      http-request redirect scheme https if soc_integ !https-request  !wellknown_cert_check
      http-request set-header Host s3.example.com if soc_integ
      http-request set-path /com-meschbach-soc-integ%[path] if soc_integ
      http-request set-path %[path]index.html if soc_integ { path_end / }      
spec:
  ingressClassName: public
  tls:
    - hosts:
        - soc.example.com
      secretName: website-soc-meschbach-integ
  rules:
    - host: soc.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: private-r0-minio
                port:
                  number: 9000

Benefits

Really easy to spin up a bucket and get a static site with TLS.
Direct path through the ingress controller. No additional hops through additional applications. No additional configuration, failure points, etc.
Highly configurable for ingresses.

Drawbacks

Specific to a given ingress controller and possibly version.
Directly exposes the minio instance. Even with proper path rewriting, having an additional proxy to expose the S3 setup directly would provide an additional layer of protection. Either through various requests, CPU + memory DoS attacks, etc.
Errors or missing files will result in direct responses from Minio in all it’s XML glory.

Stream of Consciousness

Mark Eschbach's random writings on various topics.