SATURN 2014 DevOps and Delivery Session (notes)

Notes by Scott Shipp, edited by Tamara Marshall-Keim

Impact of Architecture on Continuous Delivery
Russell Miller, SunView Software, Inc.

First, context: This was a greenfield, from-scratch project for a nontrivial social-monitoring tool. It was also their first attempt at the native cloud. It was a pilot for a truly agile project. Go to to see a beta version.

Miller uses the term “continuous delivery” (CD) as defined in Jez Humble’s book Continuous Delivery. It leverages continuous integration, automated testing, and automated deployment. Releases are frequent, small, and predictable.

For example, take Amazon drone delivery. It eliminates waste, and customers do not have time to cancel the order. It also provides quicker feedback from the customer. So CD vs. the traditional release model is similar to drone delivery vs. freight train delivery. “This is a good metaphor for lean vs. legacy.”

Another example is Formula 1 racing. CD is like a pit crew in a Formula 1 race. The product is continuously usable and made incrementally better like a race car going through a pit stop. The car is always running.

CD provides the team continuous learning. Use the Act > Observe > Orient > Decide cycle to continuously learn and innovate.

A key component of CD is the Build Pipeline, which is similar to a car assembly line or to a John Deere tractor assembly line. They are also following lean principles. The Build Pipeline allows you to talk about and measure releases by the “flow” through the pipe.

How to start? SunView Software made a goal to deliver every week. They found that they had a bottleneck at deploying from testing to staging and production. Following lean principles, they stopped the line to do root-cause analysis about why things were not going well. But they stopped the line too often.

Measuring done:

  • Lean allows you to have a framework to talk about what is truly done.
  • Is it “done” or is it done, Done, DONE?
  • The team must finish all the way through.

SunView also used a Kanban Board to visualize flow. For a good reference about flow, see the Harvard Business Review article “Six Myths of Product Development.” A quotation (see slides) from this article discusses how the “batch size” is invisible to the product owner in traditional cases.

SunView thought they were doing small batches, but in fact the size was much greater than they knew. Once they read Donald Reinertson’s book The Principles of Product Development Flow: Second Generation Lean Product Development and learned about the “fluidity principle,” they understood. Reinertson says, “loose coupling between product subsystems enables small batches…”

So … they granularized functionality to microservices, leveraged the cloud, and aimed for nothing manual in the deployment to the cloud.

One thing to watch out for: features that cut across different components of the app. To aid in this, they used “feature switches.” For an example of this, see the Twitter Decider Framework, in which feature switches help to manage risk.

Next, they used “Branch by Abstraction.” Source control branches are a waste to lean thinking.

Did they meet their goal? Yes. For 34 of 36 weeks, they delivered new functionality.

Lessons learned:

  • Track velocity. They should have but didn’t early on, so they don’t have the necessary metrics and comparison points.
  • Architecture impacts batch size.
  • Decouple deployments.
  • Virtualize the pipeline.

Russell Miller also runs a podcast,

Presentation link:

The New Era of Integrated Software Delivery with DevOps
Sujatha Perepa, IBM

A key topic for Perepa’s customers, who are customers from the federal government, is DevOps.

What is DevOps?

  • Collaborative development
  • Integrated processes and tools between Dev and Ops
  • Continuous testing – No wait/lag time; virtual test environments
  • Continuous release and deployment, all automated
  • Continuous monitoring – Not overkill and badgering developers; more proactive because metrics exist to inform what to do next time

Why do you need DevOps? We already have a good traditional method, but the CLOUD, MOBILE, BIG DATA, and other emerging technologies are driving the necessity of DevOps. In a survey by the IBM Institute for Business Value, customers said that they want technology to do more with less.

Traditional release challenges:

  • Cost
  • Bottlenecks at the point of waiting for build engineers to deploy
  • Risk of instability, especially when there is integration of multiple vendors’ products

DevOps ecosystem and standards: The Open Services for Lifecycle Collaboration (OSLC) is a standards body for integration technologies. It includes Saas, PaaS, and IaaS. See

Promoting DevOps to customers:

  • Adoption roadmap
  • Maturity models – assess what you already have; identify and then automate manual processes; create a feedback loop with customers; be reasonable with monitoring metrics.

Adoption by the Maturity Model Approach

  1. Document key paint points.
  2. Assess current capabilities.
  3. Create a “heat map” by looking at all requirements and overlaying colors for priorities
  4. Establish the roadmap.

Perepa described a case study as an example and emphasized that the process is not linear but a journey.

How DevOps influences software engineers and architects:

  • In DevOps, software architects influence integration.
  • In DevOps, architects are no longer responsible only to the boundaries of their systems.
  • Google “DevOps,” and you will find that there is an emerging role called a “DevOps Architect.”
  • Both engineers and architects must collaborate and have visibility end to end.

DevOps for the Cloud

  • Cloud is the classic DevOps playground.
  • VMs create a build-once-and-reuse configuration for environments.

Perepa also discussed DevOps for mobile devices.

Presentation link:

Is Your Team Instrument Rated (Or: Deploying 89,000 Times a Day
J. Paul Reed, Release Engineering Approaches

By show of hands:

  • How many people have DevOps at their organization? (1)
  • How many are starting a DevOps initiative at their organization? (a few)
  • How many are new to DevOps? (many)

DevOps is about culture. Specifically,

  • Incentives
  • Human factors (methods for facilitating and fostering the incentives)

Why compare DevOps to aviation? Most industries’ evolutions fit a predictable pattern:

  1. Craft
  2. Trade
  3. Science
  4. Industry

Where is DevOps on this spectrum? First, some considerations: There are many stories of failed DevOps initiatives. In addition, many DevOps communities compare DevOps to incident-response teams. DevOps conferences feature some training from firefighters, for example. So DevOps is integrating/acquiring knowledge from other areas.

Aviation is a metaphor for DevOps. Think of Devs as pilots. Think of Ops as air traffic controllers.

Also, aviation has scaling issues similar to software. For example, there are 86 landings or takeoffs per minute.           So there are many parallels from a complexity standpoint. This may be a possible educational model for DevOps.

What is “instrument-rating”? It is a rating given to pilots who have reached a certain level of experience. First, there is “visual rating” based on visual flight rules:

  • “See and avoid” technique
  • For pilots who fly based on what they can see
  • These pilots cannot fly (because they cannot see) in the clouds.

Second, there is an “instrument rating” based on instrument flight rules:

  • Able to fly without seeing
  • Visual landmarks are not provided on maps for instrument-rated pilots.
  • A holistically different approach to piloting
  • Sometimes described as “flying in the system”

A video demo was given to show the difference.

What is “flying in the system”?

1. Standardization

  • Op primitives for creating a process
  • “TERPS” is the primitives of instrument rating.
  • TERPS are incentives in the incentives + human-factors framework.

2. Definition of operating procedures

  • Derived in a specific context
  • Since you have “TERPS,” you do not have to “come up from first principles” every time.

3. Communication or, more precisely, the precision of communication

  • Example: 1 paragraph provides 30 people the flight plan from the SFO to JFK airport and encodes as 5 lines.
  • This “increases the bandwidth of communication.”
  • It is the first step toward automating it.

Several risks exist, including hesitancy to use appropriate terms to communicate about a situation and misuse of defined terminology matters. Avoiding risk sets expectations: Who has responsibility? What are reasonable outcomes?

The best example of remediation in aviation is “approach procedures.” Remediation processes can be integrated into processes.

What “flying in the system” is not:

  • Not static: The Capital-P process still has deviations, but you can now call them out.
  • Not blind reliance on automation: You must grapple with misreading or looking at the wrong metrics.
  • Not fun-forbidden: You know when you can accept slack and fun in the system.

How to get there:

  • Define the current situation.
  • Seek optimizations on current practices.
  • Create an organizational vocabulary.
  • Make sure there’s ownership.
  • Formalize roles and responsibilities.

Do not give someone a pager and say “we’re doing DevOps now.” Current limitations need to be internalized.

Holding patterns:

  • Know when to hold ’em.
  • Related to LEAN W-I-P.

Finally, investigate outcomes:

  • Do true “postmortems,” not retrospectives.
  • Separate responsibilities and have someone to help with this.
  • Have “no blame” postmortems.
  • Postmortems should be a little bit uncomfortable and produce actionable items.

Summary: “Have an operational model that accounts for incentives + human factors.”

Presentation link:

Further resources:

Paul Reed also runs the podcast The Ship Show.


One response to “SATURN 2014 DevOps and Delivery Session (notes)

  1. Pingback: SATURN 2014 SESSION NOTES, Pt. 2 | Stack Trace

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s