FuncOps: Orchestration and The Mothership Connection

It’s going to take some time to understand and articulate the full awesomeness of Crowbar 2’s approach to DevOps. One of the ways is to envision “Functional Operations,” similar to “Functional Programming”… this is early, but it’s a peek at our thinking. Paraphrasing Parliament Funcadelic: “Make my Func the FuncOps, I want to get FuncOps.”

What Erlang teaches us about orchestration.

Erlang’s design allows for intergalactic scalability and concurrency. Let’s riff off of a great design!

Erlang’s basic principles:

  • Functional programming through pattern matching
  • Variables immutable
  • VM manages resources and processes
  • Concurrent/parallel processing
  • Light weight functional processes
  • No shared data between processes
  • Async message passing between processes
  • No garbage collection.  Dedicated heaps.
  • Sane failure management
  • Live code update

How they’re being made manifest in Crowbar:

Pattern matching data for function assignment:

  • incoming data is assigned to the right NodeRole through a Crowbar function.

Immutable variables:

  • messages passed to jigs for nodes are immutable
  • a receiver node cannot transform a crowbar instruction and think anyone else will know

Crowbar (the “brain” or VM) manages resources and processes:

  • system purpose language (switch/lb/web/app/db are resources interacting)
  • data (that is processed by crowbar is the state of the system)
    • consensus
    • resource use
    • resource availability
    • code (the crowbar language)
    • the rules by which this data is evaluated
    • calls to the API to change nodes

Nodes are lightweight because they are:

  • fast to spin up
  • crowbar (will) manage resource allocation
  • crowbar will manage resource loss/exhaustion

No Shared Data:

  • Nodes are independent actors
  • Processes, too, are independent actors
  • Both are able to message in their domain to establish/confirm consensus

Concurrent processing:

  • domains of deployments are handled in parallel
  • within a deployment, nodes are run in defined order

Async Message passing:

  • crowbar processes communicate through messages
  • messages are usually the outcome of node state transformation
  • messages are sometimes “monitoring data” which is sort of a state transformation

Garbage collection:

  • disused nodes are a rarity
  • they are very quick to clean up when physical limits are met
  • they should have been out of the path of operations for a while now

Sane failure management:

  • well defined and recorded state
  • caller function/node will retry

Live update:

  • annealer exmanines impact of local change to entire deployment
  • changes to crowbar code affects only new calls to the namespace
  • looks like a phased migration


Three tiers of system orchestration:

  1. CMDB Territory: the code managing the resources
  2. chef search/attribute injection: Managing nodes as objects – and their interrelationships
  3. Managing the deployments

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s