Skip to main content

Running mabl Link on Kubernetes

Eva Ferreira
  • Updated

Running mabl Link on a Kubernetes cluster uses the mabl Link Docker image, which includes everything the Link Agent needs. This article explains how to run the Link Agent as a Kubernetes deployment, how many resources to give it, and how to stop it without interrupting tests.

Before you start

Gather the following:

  • A "Link Agent" API key. API key access is limited to workspace owners. For a tunnel shared by every workspace in your company, use a company key. See Company-scoped Link tunnels.
  • A tunnel name, between 1 and 24 characters long, using lowercase letters, numbers, and dashes.
  • The name of your Kubernetes namespace.

Use the mablhq/link-agent:latest-ubuntu image on arm64 nodes, such as AWS Graviton. The default mablhq/link-agent:latest image is the Alpine variant, which is published for x86-64 only, so it can't run on arm64 nodes. See Link with Docker for the variants.

Store the API key in a secret

Keep the API key in a Kubernetes secret rather than in the pod spec, where anyone who can read the deployment would see it:

kubectl create secret generic mabl-link-agent \
  --namespace={your-namespace} \
  --from-literal=api-key={your-api-key}

A deployment keeps the number of Link Agents you ask for running and replaces any that stop. Set replicas to 2 or more for high availability: every replica serves the same tunnel, and mabl spreads connections across them.

The following example runs two Link Agents on the tunnel qa-env-01. Replace {your-namespace} with your namespace:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mabl-link-agent
  namespace: {your-namespace}
  labels:
    app: mabl-link-agent
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mabl-link-agent
  template:
    metadata:
      labels:
        app: mabl-link-agent
    spec:
      # Longer than the Link Agent's own drain, so that running tests can finish
      terminationGracePeriodSeconds: 90
      containers:
      - name: mabl-link-agent
        image: mablhq/link-agent:latest
        args:
          - "--api-key"
          - "$(MABL_LINK_API_KEY)"
          - "--name"
          - "qa-env-01"
        env:
          - name: MABL_LINK_API_KEY
            valueFrom:
              secretKeyRef:
                name: mabl-link-agent
                key: api-key
        resources:
          requests:
            cpu: "2"
            memory: "4Gi"
          limits:
            memory: "4Gi"
            # No CPU limit: throttling the agent adds latency to every connection through it.
        readinessProbe:
          exec:
            command:
            - grep
            - "^ready$"
            - /opt/mabl/link-agent/run/status
          initialDelaySeconds: 10
          periodSeconds: 10
        volumeMounts:
        # The Link Agent's working files, which must stay writable on a read-only root filesystem
        - name: link-agent-run
          mountPath: /opt/mabl/link-agent/run
      volumes:
      - name: link-agent-run
        emptyDir: {}

To create the deployment, save the spec to a file and apply it:

kubectl apply -f mabl-link-agent-deployment.yaml

# Optionally follow the logs of one of the agents
kubectl logs -f deployment/mabl-link-agent -n {your-namespace}

If the deployment doesn't come up, check the pod events with kubectl describe pod, and confirm the secret and namespace names match.

Choosing resources

The requests in the example match the minimum host tier, which covers one tunnel with light concurrent use. Size the pod the way you would size a VM:

  • Memory: set the memory limit to cover the tunnels the pod serves. The Link Agent needs about 400 MB, plus about 500 MB for each tunnel, plus one more tunnel's worth while an update installs. During a migration from legacy Link, allow up to 2 GB more.
  • CPU: plan for about one core for every 150 to 200 Mb/s of traffic through the pod, and avoid CPU limits below two cores. A tunnel that is throttled on CPU adds latency to every connection through it.
  • Scaling: to handle more traffic, add replicas rather than enlarging one pod, so that the tunnel keeps working if one pod is replaced.

See Sizing and scaling Link Agents for the full guidance.

Network requirements

The pods need outgoing access to api.mabl.com and to one of the Link tunnel endpoints. If your cluster's egress allows outgoing UDP on port 443, the Link Agent uses QUIC, which is faster. Otherwise it uses a secure WebSocket over TCP port 443. See Link Agent requirements.

mabl recommends raising the socket-buffer limits on each node that runs a Link Agent pod, which increases tunnel throughput over long network paths. Those limits are kernel settings on the node, so apply them on each node rather than in the pod. See Raise the host's socket-buffer limits.

Clusters with a read-only root filesystem

Many clusters require pods to set readOnlyRootFilesystem: true, often through an admission policy that applies to every workload. On those clusters, keep the emptyDir volume from the example mounted at /opt/mabl/link-agent/run. Each Link 3.0 tunnel runs as a separate program, called a tunnel process, which the Link Agent downloads to that directory and runs from there. The directory must be writable and allow running programs. A default emptyDir allows both.

If the Link Agent connects over legacy Link but not over Link 3.0, and its log shows no writable, executable directory for the Link 3.0 worker, the volume is missing or doesn't allow running programs. Check that the volume is mounted at /opt/mabl/link-agent/run.

A default emptyDir takes its mount options from the node, so on nodes that mount it with noexec, the Link Agent can't run the tunnel process from it. Link 3.0 needs a directory it can run programs from, so choose one of the following:

  • Ask your cluster administrator for an exception. Exempt the Link Agent's volume or pod from the noexec rule.
  • Use a memory-backed volume. Kubernetes mounts emptyDir with medium: Memory in a way that allows running programs. The volume's contents count against the pod's memory limit, and the volume can hold up to three versions of the tunnel process, about 175 MB each, for every tunnel. Raise the memory limit by about 600 MB for each tunnel, and set the same amount as the volume's sizeLimit.
  • Run the Link Agent outside the cluster, on a VM or a Docker host that allows running programs from a writable directory.

For example, a memory-backed volume for a pod that serves one tunnel:

      volumes:
      - name: link-agent-run
        emptyDir:
          medium: Memory
          sizeLimit: 600Mi

With a read-only root filesystem, the Link Agent also can't update itself or write log files inside the container:

  • Updates - To move to a new Link Agent version, pull the newest image and restart the deployment with kubectl rollout restart deployment/mabl-link-agent -n {your-namespace}. If you pin an image tag, change the tag instead.
  • Logs - The Link Agent's output still reaches kubectl logs.

For the directories the Link Agent writes to on other installations, see Writable directories.

A Link Agent pod reports Ready once its tunnel is set up. The readiness probe in the example reads the Link Agent's status file, /opt/mabl/link-agent/run/status, which holds the agent's current status and contains ready once the agent has registered with mabl and set up its tunnel.

While an agent is in maintenance mode, its status file shows that it is draining, so the pod reports not ready until you take it out of maintenance.

When the tunnel is active, the Link Agents also appear with a Connected status on the networking page in the mabl app: Settings > Networking.

When Kubernetes stops a Link Agent pod, the agent lets in-flight connections finish before it exits. Keep terminationGracePeriodSeconds at 90 seconds or more so that Kubernetes doesn't stop the pod before the agent has finished. The default of 30 seconds can cut off tests that are running through the tunnel.

For planned work, such as draining a node, you can also put each Link Agent into maintenance mode first, so that new tests go to the other replicas while it drains.

If you shut down a Kubernetes-based Link Agent from the mabl app, the deployment starts a replacement within seconds, because it keeps the number of replicas you asked for. To stop Link Agents for good, delete the deployment:

kubectl delete deployment mabl-link-agent -n {your-namespace}

To stop routing tests to the tunnel, see Removing mabl Link tunnels.

Next steps

When the Link Agent is installed and running, configure your tests to run over mabl Link.

Related articles