Lessons Learned from Architecture Reviews
Thursday, May 6, 2009, 9:00 am
Introduced by Linda Northrop of the SEI, who called Wirfs-Brock “the champion of the practitioner.” “She writes and teaches so that we can practice.”
Lessons learned are from two perspectives:
- An outside expert called in to review a system architecture or product line whose success is critical. Alone or with small team of trusted colleagues. Look at strengths and weakness of products and enterprise systems.
- A presenter and defender of the architecture for review. How to present ideas that sell concepts, give reviewers focused things for them to address.
You need to explain to a healthy skeptic
- what your design is and why it’s a good solution
- rationale – why you made a key decision
- your thought process
When I come into a review, sometimes people are fearful. I may be viewed as a threat. I have to develop a relationship with the people for whom I do the review.
Notion of collaboration is important: to work together on a joint intellectual effort. Architecture reviewers should be collegial, should trust and share information. If review is in collaborative environment, there’s little friction, there’s mutual respect, and there’s give and take.
Those you work with may not share your views, values, motives. Root cause is the illusion of transparency: tendency of people to overestimate the degree that their mental state is known by others. Tendency to skip over important background information. People overestimate how well people understand each other’s mental states.
In-group bias: preferential treatment for members of same tribe or group. When building a review process as an outsider coming in, must be aware of this. In defensive posture, there can be different way of rationalizing architecture decisions.
Eoin Woods, Software Systems Architecture: Working With Stakeholders Using Viewpoints and Perspectives suggested format for presenting an architecture idea:
- Alternatives: options considered and reasons for ruling them out
- Effects: What does the decision imply
- Evidence: Confirmation that decision is good
My format as an insider presenting to collegial people, friendly colleagues:
- Design idea
- Design notes
- Issues, uncertainties
If you have a collegial environment in which constructive feedback is given, you can use this sort of format.
When I give advice, a triage mentality helps me as a reviewer focus my energy and efforts. If I see fatal flaws, I don’t need to spend much time other than to say it is really bad. I focus on constructive observations where my review can be constructive.
Sort comments into
- Recommendations (feel strongly about)
- Observations (positive or not)
It’s easy as a reviewer to focus on problems, but don’t ignore the good stuff. Just as important to note good practices, as well as risks.
I found it hard to get people to focus on scenarios for qualities or failures. As reviewers, we want to find potential weaknesses and vulnerabilities of architecture, and realistic scenarios are hard to get agreement on. Concrete failure scenarios are important but can be hard to agree on. Lesson: rainy day scenarios are hard. They’re uncomfortable for people, but they yield good insights.
People want to discuss normal operation of the system. You have to ask, “Fine, but what happens when it breaks?”
Scale review process to the system at hand. Can characterize whether a review is gold, silver, or bronze depending on criticality.
Architecture review board in an organization leads to balanced discussions—less in-fighting, parochialism. Levels playing field, makes managing easier. Forum for airing differences of opinion. Makes work of reviewing more pleasant and professional.
Agile architecture reviews: Clients want us to assess whether application or architecture is agile-ready: can we deliver incrementally and well tested, and integrate complex systems? Developed proposals for achieving automation of testing, which is especially difficult for a SOA that integrate COTS components. Problem is how to test component to deliver functionality incrementally in an automated way. Key isn’t low or no documentation—that’s a red herring—it’s to enable a steady pace of development and deliver software incrementally. Test automation is critical, and acceptance test. Easy configuration and reconfiguration are important. Incremental development and delivery place greater requirements on the architecture.
Lesson: Beware of the technical stack. Architects focus on low-level technical details. Architects talk about the tools they are using. It’s a red flag to me when architects can speak only of technical elegance or complexity. Needs to be driven with real practical experience.
Lesson: Merging existing systems is really hard. Reviewers must point out the obvious: things like hidden requirements in peoples’ heads or buried in custom code.
Lesson: Risks compound. A bunch of medium-risk things joined together become major risks.
Lesson: Get the right people involved, e.g., bring in VP of marketing and other key stakeholders. Be a catalyst for communication in an organization. Critical to get to bottom of requirements by structuring questions for reviewers.
Lesson: Ask the right questions, a la Columbo—ask leading questions with air of innocence, leading people to incriminate themselves. “I like to disarm you with questions.” Architects love to talk about their decisions. I keep asking questions.
- Evaluation – how good do you think it will be?
- Accuracy – how did you come up with those numbers?
- Completeness – is that all?
- Relevance – does this apply here?
- Purpose – why did you suggest that?
- Extension – tell me more.
Instructor Excellence, Bob Powers – count to 10, then ask again.
Ask clarifying questions
- Why do you say that?
- What exactly do you mean?
- How does this relate to what we discussed earlier?
- Can you give me an example?
- Are you saying…or…?
- Can you restate your concern?
Learn to accept criticism
- Acknowledge critic’s viewpoint
- Be sure you understand
- Take appropriate action
- Don’t become defensive
Paul Graham defined a disagreement hierarchy:
- Name calling
- Ad hominem
- Responding to tone
- Refuting the central point
Lesson: Mood affects judgment. When in a good mood, people judge things more favorably. When grumpy, we judge more harshly.
Lesson: Recognize cognitive biases, distortions in how people naturally tend to process and interpret information. Not everyone shares the same biases. Examples:
- Contrast effect: people can’t avoid comparing items against each other rather than against a fixed standard. Lesson: Increase information availability – people decide based on what they remember. To increase information availability, make it recent, vivid, easy to imagine.
- Ambiguity effect – people favor a choice where there is a known probability to a choice rife with ambiguity.
Q: What’s the common theme here?
A: Architects don’t know how to communicate well to others. I work with clients to help them communicate design ideas. Problems stem from underestimation of technical complexity, from excessive optimism about things working and scaling, and poor documentation (it’s in the heads of the architects).
-Bill Pollak, SEI