SATURN 2015 Keynote: Rethinking Architecture in the Context of DevOps (Session Notes)

Mark Schwartz, U.S. Citizenship and Immigration Services

Schwartz discussed some projects that he has led and lessons learned from the experiences in building systems for the government. He is CIO of one of three agencies that deal with immigration. USCIS processes 7 million applications per year for green cards, refugee status, citizenship, and other cases. USCIS is part of the Department of Homeland Security, which is important because the agency is under two tiers of enterprise architecture, and everybody wants to tell everybody else what to do. USCIS manages about 70 legacy IT systems, and Schwartz discussed three new projects.

Three years ago, USCIS became 100% agile in a “magic-by-policy” process. Schwartz issued a memo, and it was easy as that! Except… then he had to define what agile means and which practices to follow, provide references in case people wanted to actually learn about them, and other difficult practicalities. The union also had to approve the policy, so Schwartz needed to explain what agile meant to the union as well. Now USCIS builds in AWS, a public cloud, and the agency is beginning to adopt DevOps practices.

Schwartz referred to the first project that he described as the “transformation project.” It’s been in development for nine years, with four years to go. Fifteen development teams of ten or so people work on this project. Its lifecycle cost estimate is $2.9 billion, but this estimate includes development costs plus a projection of maintenance costs. The original estimate was for 10 years of operation, but then they learned that they needed to estimate for 20 years of operation. This gives you a sense of the scope of the commitment.

The second project is myUSCIS, which helps people with immigration processes; users include applicants and their legal representation. This project is much smaller and includes only two teams. The third project is called DID(IT), for Digital Innovation and Development (Information Technology). USCIS has distributed locations all across the country, and they all build “rogue” applications. Schwartz sees a need for providing a way to build applications quickly and a platform to put them on.

The three projects range from a huge transformation project to a small user-facing project, and each will exercise DevOps practices in different ways. The first goal is responsiveness, which means reducing cycle time dramatically. The second goal is quality, which means gathering more feedback throughout the process.

What does architecture look like in the government? The model is all about governance and compliance. Architecture make sure that every project is done the right way. Schwartz calls this the “across the street” aspect. Enterprise architects and quality assurance are literally across the street from each other. They have a technical reference model for USCIS and for DHS, which specifies what components architects can use. For example, IE is the browser, and Java is the development platform. Development teams must stay within the bounds of what has been approved or gain approval to go outside those bounds.

There is a conflict here. The goals of DevOps and the cycle time required are not compatible. It’s a given that standardization is a good thing, but Schwartz sees costs involved in standardizing too. Anytime you constrain a problem, you reduce the size of the solution space. Standards constrain the solution space for development teams. The biggest factor has been bringing in people who are used to working in other types of environments; the government has an initiative to hire people from big technology companies, and these hires want to use different technologies than the approved ones. Knee-jerk standardization interferes with weighing all the benefits. Standards also impose a cost in checking to see that everyone is using the standards; of course, if they aren’t, there are costs associated with that too.

For the transformation project, 15 teams will work on the same code base at the same time. How should they handle architecture for this system to support that? Another challenge for the architecture is that the system will have to interface with 30 other systems. Originally, one team would pilot the proposed architecture, then the architecture team worked with the developers, provided input into the processes, and published standards for the system. After a few months, feedback was poor. Development teams were nervous about evolving their own architecture without getting approval from the architecture design team, building a waterfall into the project. Also, it seemed like the architecture design team owned the architecture, so the developers didn’t feel as responsible for it as for developing the business functions. They had to change the dynamic.

Schwartz tasked the architecture team with serving the developers, facilitating meetings where developers made decisions about design, and helping by doing research and reference implementations. After a few months, the feedback showed that developers were making decisions but had trouble turning them into execution. They still didn’t feel like they owned the implementation.

For the third experiment, the results are not yet in. They’ve brought in teams from leading technology companies to work with the developers. To solve the execution problem, the architecture team will try to create reusable platforms that the rest of the teams will want to use because it will accelerate their processes. The architecture team is essentially entering a marketplace and selling to the development team.

In contrast, myUSCIS and DID(IT) are trying to change the ways that the government interacts with the public. For myUSCIS, they began by building reusable components that they knew would push things forward and make it easy for developers to execute the architects’ vision. The architects found that they could rig up prototypes quickly. For example, all applications are in English, but not all applicants speak English. Legal reasons make providing translations of the applications difficult. The new system will provide some components in Spanish, but the answers must be in English. So they are looking for ways that the system can recognize that the answer is in Spanish and warn the user. A test web page was built quickly, and they let the lawyers try it out.

The DID(IT) project is a single code base that looks like multiple applications. They created a standard tool set and wrapper on which developers could build applications quickly. In this case, they were willing to make the tradeoff between standardization for the speed of development within that homogeneous space. It optimizes for speed of delivery.

These components make up the USCIS IT “laboratory,” and it has provided these lessons: First, the goal of traditional architecture was enforcing standards, but in the DevOps environment, architecture must be seen as an enabler instead of a constraint. Second, architecture actually will emerge; teams are effective in creating it, and you want teams to be able to innovate. Third, they became conscious of the danger of removing responsibility for architecture from the development team. It’s less about whether you are going to “impose” architecture and more about whether the developers “own” the decisions. Fourth, there must be a decentralized approach to architecture, and any solution they consider has to take this into account. Fifth, the developers need to get used to working without an architecture runway. Last, loose coupling is the right approach; decouple the work of teams so they can proceed at their own pace and not have wait times, but understand the tradeoff is that boundaries become less clear. It becomes harder to test interfaces, so you need to build defensively by validating all your assumptions.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s