Skip to content

Getting started

A hub and two cells, one command per cluster, then a placement.

What you need

  • Two or more Kubernetes clusters. One is the hub, which holds the registry and mints; the others are cells, which receive deploys. The hub cluster can also be a cell.
  • helm and kubectl.
  • A CI platform that issues OIDC tokens (GitHub Actions, GitLab, Buildkite), or any other OIDC issuer under your control.

Each cell needs its own service account issuer

The hub tells one cell's agent from another by the iss claim of its token. Managed providers give every cluster a distinct issuer URL already. A stock kubeadm or kind cluster issues as https://kubernetes.default.svc.cluster.local, and every one of them collides, so those clusters need --service-account-issuer set before they can be told apart. See ADR-009.

1. Install the hub

helm install cellcast oci://ghcr.io/ethan-kane-ops/charts/cellcast \
  --namespace cellcast-system --create-namespace \
  --set hub.oidc.issuers[0].url=https://token.actions.githubusercontent.com \
  --set hub.oidc.issuers[0].provider=github

The issuer is not optional in practice. A hub with none authenticates nobody and refuses every request. That is the correct state for a broker that cannot tell who is asking, and it is not a working install.

A self-hosted issuer needs its CA

The hub fetches each issuer's discovery document and keys over HTTPS and verifies that certificate against the system roots. GitHub Actions, GitLab SaaS and Buildkite chain to a public root and need nothing extra. A self-hosted GitLab, a GitHub Enterprise Server or an internal Keycloak usually presents a certificate from the organisation's own CA, and the hub cannot reach it until that CA is trusted:

kubectl -n cellcast-system create configmap issuer-roots --from-file=ca.crt=./internal-ca.crt
helm upgrade cellcast ... --set hub.oidc.caConfigMap=issuer-roots

The bundle is added to the system roots rather than replacing them, so a public issuer configured alongside an internal one keeps working.

A complete worked fleet, four cells and three policies, is in examples/. The rest of this page builds one of those objects at a time.

2. Enrol each cell

One command per cell, given a kubeconfig for that cell:

$ cellcast cell add --kubeconfig ./prod-euw1.yaml --name prod-euw1 \
    --labels env=prod,region=euw1 \
    --mint-service-account deployer --mint-namespace apps \
    | kubectl apply -f -

It reads the cell's service account issuer from the cell's own discovery document and prints a TrustConfig and a Cluster. The issuer is the value that is easy to get wrong by hand, and wrong it fails silently: the agent's capacity reports are refused and the cell never wins a placement. Nothing is applied until the output is piped.

--mint-service-account and --mint-namespace decide what every deploy credential for this cell can do, which is why neither has a default.

The command leaves two things to you, and says so on stderr.

The Secret holding the hub's credential for the cell. cellcast does not put credential material on stdout, so the Secret goes straight from a file to the API server. Use a kubeconfig that can create tokens for deployer in apps and nothing more, not an administrative one:

kubectl -n cellcast-system create secret generic prod-euw1-kubeconfig \
  --from-file=kubeconfig=./prod-euw1-minter.yaml

The hub trusting the cell's issuer. stderr prints the --oidc-issuer value for this cell. Add it to the hub's hub.oidc.issuers; a cell whose issuer the hub does not verify never reports capacity.

A kubeconfig that authenticates through an exec plugin, such as aws eks get-token, works exactly as it does for kubectl. If the cell refuses its discovery document to that identity, pass the issuer yourself with --issuer. On EKS it is the cluster's OIDC provider URL, which aws eks describe-cluster reports.

What it generated

A TrustConfig says how the hub authenticates to the cell. It references a credential and never contains one:

apiVersion: cellcast.io/v1beta1
kind: TrustConfig
metadata:
  name: prod-euw1
  namespace: cellcast-system
spec:
  provider: kubernetes
  credentialSource:
    # The Secret created above. When the cell is the hub's own cluster,
    # `inCluster: true` replaces this and stores nothing at all.
    secretRef:
      name: prod-euw1-kubeconfig
      key: kubeconfig
  kubernetes:
    # What gets minted: a token for this service account, in this namespace,
    # in that cell. Nothing broader is reachable through cellcast.
    serviceAccountName: deployer
    namespace: apps

A Cluster registers the cell and says who may report its capacity:

apiVersion: cellcast.io/v1beta1
kind: Cluster
metadata:
  name: prod-euw1
  namespace: cellcast-system
  labels:
    env: prod
    region: euw1
spec:
  endpoint: https://ABCDEF1234.gr7.eu-west-1.eks.amazonaws.com
  provider: eks
  state: LIVE
  trustConfigRef:
    name: prod-euw1
  # Who may publish capacity for this cell. Unset means nobody, not anybody.
  reporter:
    issuer: https://oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLE
    subject: system:serviceaccount:cellcast-system:cellcast-agent

Check both were accepted:

$ kubectl -n cellcast-system get trustconfig,cluster
NAME                                PROVIDER     READY   DETAIL
trustconfig.cellcast.io/prod-euw1   kubernetes   True

NAME                            PROVIDER   STATE   ACCEPTING   ENDPOINT
cluster.cellcast.io/prod-euw1   eks        LIVE    True        https://prod-euw1.example.com

READY: False on the TrustConfig means the hub could not find the Secret or the service account it names. Fixing it here costs less than finding it during a deploy.

3. Install the agent in each cell

helm install cellcast-agent oci://ghcr.io/ethan-kane-ops/charts/cellcast-agent \
  --namespace cellcast-system --create-namespace \
  --set cellName=prod-euw1 \
  --set hub.endpoint=https://cellcast.example.com

cellName must match the Cluster above. Within one heartbeat interval the hub is scoring the cell.

Until a cell reports, it has no capacity, so it is excluded from scoring rather than treated as empty. A naive least-loaded ranking reads a missing entry as zero load, which is the best possible score, so the first cluster to break badly enough that its agent stopped reporting would become the target for every deploy in the estate.

4. Say who may deploy where

Nothing is permitted until a policy says so.

apiVersion: cellcast.io/v1beta1
kind: PlacementPolicy
metadata:
  name: app-prod
  namespace: cellcast-system
spec:
  # Which callers this policy is about. Matched on the token's claims.
  subjects:
    - issuer: https://token.actions.githubusercontent.com
      subject: repo:acme/checkout:ref:refs/heads/main
  # Which cells they may reach. A label selector over registered Clusters.
  permittedCells:
    matchLabels:
      env: prod
  strategy: LeastLoaded
  tokenTTL:
    default: 15m
    max: 30m

5. Place something

From the pipeline, with its identity token in CELLCAST_TOKEN:

$ cellcast place --hub https://cellcast.example.com \
    --workload checkout-api --dry-run --explain
would place checkout-api on prod-euw1 (policy app-prod, LeastLoaded, confidence high)

CELL        ADMITTED  STAGE       REASON                                       UTILISATION
prod-euw1   yes                                                                0.42
prod-euw2   yes                                                                0.77
dev-euw1    no        permission  cell is not permitted by this caller's policy 0.01

--dry-run runs the whole decision and stops before minting, and --explain shows every cell and why it was or was not chosen. Running it before the first real deploy is how a policy that permits more than intended gets caught while it still costs nothing.

Drop both flags and cellcast writes a kubeconfig for the existing deploy step:

$ cellcast place --hub https://cellcast.example.com --workload checkout-api --ttl 15m
placed checkout-api on prod-euw1 (policy app-prod, LeastLoaded, confidence high)
credential valid for 15m
kubeconfig written to cellcast.kubeconfig (0600, as apps/deployer)

$ kubectl --kubeconfig cellcast.kubeconfig -n apps apply -f deploy.yaml

Next