Crossplane's mental model and moving parts

It is long overdue to continue writing this series on building platforms with Crossplane. Where the previous post mentioned why we’re throwing out a bunch of Terraform code it also just namedrops Crossplane, mic drops, and leaves everyone confused.

So it’s time to do some explaining

The very basic introduction

In short, and to play an association game, Crossplane is a piece of software you can run in a Kubernetes cluster. This piece of software act as a control plane to manage your infrastructure resources. Or as the official website puts it in their more polished tagline:

Crossplane Is the Cloud-Native Framework for Platform Engineering

Your first reaction might be “so a bit like opentofu/terraform/pulumi/ansible/<foo>, but with more YAML?” But (thank the great Spaghetti monster) the answer to that question is “no. not at all”. Most toolchains for Infrastructure as Code-tools differ in syntax, but all follow the same loop of plan → apply → destroy either by installing something on your local machine or through CI/CD. No one runs crossplane apply to apply changes to infrastructure, there is no plan to inspect to verify changes before applying and there is no refreshing status of a pipeline and read through the job output to see what happened.

The mental model

You tell the cluster what you want to exist, and a controller in the cluster creates it, and spends the rest of the lifecycle for that resource making sure the description matches what actually exists.

This loop has a name: reconciliation. In Kubernetes this happens continuously, not just when you run terraform apply or destroy, but for the entire lifecycle of a resource the controller wakes up, compares what you have asked for against what actually exists, corrects the difference, and goes back to sleep.

What you as a Platform Engineer get out of it is a set of building blocks to build user friendly APIs that abstracts away (or at least reduces) the need for your developers to know or care about the underlying infrastructure. No matter if it’s Azure or a different cloud vendor, a bunch of bare metal with a wildly expensive VMWare, or something else entirely.

What the users get is a familiar interface. Just like their container deployments, infrastructure components are now also kubernetes resources. They can use kubectl, all the GitOps tools like Flux or ArgoCD, and every other tool a developer usually have very strong opinions on.

An example

1. A developer creates a claim that describes some infrastructure they need
2. A GitOps tool picks up the change
3. The control plane is made aware of the changes
4. The claim is resolved as managed resources
5. Resources are created, updated or deleted
6. The control plane monitors changes in actual infrastructure and corrects drift

Hybrid

This also makes the concept of a “hybrid platform” interesting. If the developers don’t care, or need to care, about finer details on the infrastructure, we can build interfaces where the user does not have to deal with different resource models, permission models, workflows or authentication. The control plane takes care of it.*

(* yes. you could say that modules in OpenTofu/Terraform can solve the same thing. I partly disagree. It can solve most of the problem with enough duct tape, but not as well as a control plane approach is able to)

Explaining the building blocks

Basicly. Crossplane has two halves:

1. The bottom half talks to the underlying infrastructure layers through providers and managed resources.
2. The top half is where the platform team and developers interact. The land of XRDs, Compositions, and Functions.

You might not need both. “Just” doing the first half can still result in something useful for your IT-team, but it’s in the second half the “platform engineering” (and imho, fun) happens.

So, to follow along in a bottom up approach, there are ~4 concepts that are important to understand:

1. Providers
   A package you install in your cluster to bring controllers and custom resource definitions to communicate with external APIs. provider-azure-family knows how to talk with Azure, provider-upjet-github talks with GitHub etc. The name and the concept is very similar to people familiar with OpenTofu/Terraform, and often, Crossplane use crossplane/upjet to generate the provider from existing OpenTofu/Terraform code.
   
2. Managed Resources
   A 1:1 mapping with a real resource. A LinuxVirtualMachine managed resource (MR) represents a Virtual Machine running Linux in Azure. The schema is generated from the vendor’s own API schema and usually feels familiar if you’re migrating from other tools to deploy resources.
   
   I find using the anatomy of the MRs to be a nice mental note when working with compositions: Everything under spec.forProvider is what I want to deploy, and everything under status.atProvider contains what actually exists. The controller’s whole job is to make sure these two match.
   
3. Compositions and XRDs
   In short, this what your users care about. The API contract between a developer and the platform.
   
   A Composite Resource Definition (XRD) defines the API. What kind of resource it is, which group it belongs to, the input fields with optional defaults and validation rules to make sure inputs are sanitized.
   
   A Composition is the implementation of the API and provides the coveted abstraction of infrastructure-concept I keep mentioning =). It might deploy a single resource, or ten different resources that gives the user a complete runtime environment.
   
   (ps: as a mental model, the comparison with a OpenTofu/Terraform module is close enough on the most important bits: variables.tf is your XRD, and main.tf is your composition).
   
4. Functions
   Finally we have functions. A quick history lesson is needed to explain why these are great:
   
   In the beginning, Compositions used patch-and-transform to map input fields from the XRD. This YAML-dialect works fairly well and is easy to understand, until you need conditionals, looping over a list of unknown length, or more creative logic, and it turns ugly or straight up impossible.
   
   Thus, functions was introduced to provide more ways to describe resources in a Composition. function-go-templating let you run wild with (sometimes unreadable) template tags, function-kcl gives you the option KCL, and with function-kro you can have the same YAML+CEL experience as in Kro. And if none of them fits your purpose, you can (just) create your own bespoke functions.

All wired together, these four components are what allow you to create nice interfaces your users will actually love using!

No tool is perfect for everything

You might be sold already and think this is better than a posh hotel breakfast, but every technology comes with a set of caveats. Nothing is perfect for everything. We move away from Terraform to Crossplane because we see that the amount of duct tape and plumbing needed to make the plan → apply → destroy tools are not a good fit for our requirements to build a true hybrid platform, with a great self-service experience, and continuous reconciliation. Your shop is probably different, and you should take that into account before you climb up to the diving board and make a big decision.

• The initial setup
  Crossplane runs on Kubernetes. That means you now have a cluster it’s really important to take good care of. If you’re already running a bunch of container workloads you should already have all the processes in place for upgrades, monitoring, backup etc. in place. If not, you have big work coming before you’re ready.
  
• There is no dry run
  No terraform plan, no -whatif, no approval gate in your CI/CD. Tools like crossplane render help, and while the tools are good they are not amazing and like everytime you want to test something against a Cloud API they have gaps. You need to design a process for how you do dev→test→release to build the confidence that you’re not breaking something important.
  
• Provider coverage
  The ecosystem is not as mature as others. Though many providers are generated by using crossplane/upjet and this strategy is good for attaining broad coverage fast, it also means providers are trailing upstream and the further away from the most mainstream resources you get, the gaps becomes larger and more annoying. (If you’re willing and allowed, you can contribute to make this situation better though!)
  
• Migrations are time-consuming
  You can adopt current infrastructure into Crossplane by using management-policies to observe resources that already exist, but it’s time-consuming, careful, and (imho) boring work.
  
• Learning tax
  Getting started with tools like OpenTofu/Terraform is easy. You download a binary, create a small bit of code, run apply, and resources are created. Getting everything together to run Crossplane requires more investment, and because the mental model is different, the knowledge transfer for experiences infrastructure engineers are not as clean as you might think. Especially debugging errors is less transparent in the beginning and there are definitively well camouflaged pit-falls that separates working code from good code.