Notes by Scott Shipp
Past, Present, and Future of APIs for Mobile and Web Apps
Ole Lensmar, SmartBear Software
Ole Lensmar is from Sweden and has been in the API space since the late ’90s. He has created one of the most popular API testing tools in the world SoapAPI. He also is the CTO of SmartBear solutions.
Once upon a time, people tried to connect distributed systems with
- COM / DCOM
- J2EE / RMI
They all had their challenges. They were complex, proprietary, limited, etc. To get the scale you wanted, it was very difficult.
… the internet came along.
They realized it wasn’t just for hypertext but could be used to connect distributed systems together.
Examples: HTTP, XML, POX, HTTP+XML, SOAP (Microsoft).
SOAP tried to take the notion of RPC further and provide a messaging infrastructure that did not necessarily bind you to HTTP.
Thesis from Roy Fielding in 2000. REST is now by far the most popular way to build web APIs, but took a long time to take off.
SOAP 1.1 (2003)
Everyone did SOAP their own way.
WS-I Basic Profile (2004)
Created a rule-set for how to apply SOAP standards.
People who do SOAP today adhere to this in order to prevent “everyone doing SOAP their own way.”
Doc/Literal replaces RPC.
Simultaneously people were talking about a top-down (design APIs first) vs. bottom-up design.
Web APIs Emerge
- Simultaneously Salesforce and eBay released XML API (2000). Helped them build an ecosystem around their SaaS offerings.
- Others followed: Del.icio.us (2003), Flickr (2004); Facebook, Twitter, Google Maps, etc. (2006)
Also…Web applications became much more dynamic.
- Consumed APIs.
- Web 2.0 Technology Stack.
- “Programmable Web” launches in 2005.
…web starts turning into a platform.
SOAP became “enterprisy”
- “WS Death Star” Joke emerges.
- WS-* standards became lengthy.
- Too many standards led to too much confusion, and people searched for alternatives.
(That being said, SOAP is still huge in the enterprise. SOAP users are still growing every day.)
SOA architectures became “mainstream.”
Public APIs did not utilize SOAP. Very difficult to apply SOAP to that space.
The client landscape continues to evolve as well.
- Mobile takes the lead.
- Device proliferation. Aided by single backend interfaced by APIs.
APIs fuel cloud and infrastructure.
- Amazon S3 (2006) and EC2 (206). Both are REST APIs that lacked web interfaces for several years.
- Twilio (2007)
APIs are at the heart of applications.
Provide basis for composite architectures.
APIs just continue to grow.
- Chart on slide – ProgrammableWeb API Growth 2005-2013
- API directories: programmableweb.com, apihub.com, publicapis.com
- apicommons.org, a collection of shared API definitions
- QoS standards for REST growing – OAuth, OpenID Connect, Tokens, etc.
Question: will proliferation of QoS standards kill REST as it did for SOAP?
API growth recap
Slides – diagrams of example enterprise architectures shown from 10 years ago until now. From monolithic to distributed.
What Ole calls this: Picking components (APIs) off the shelf as the key ingredients in a distributed application architecture.
What is REST?
Not a technology. An architectural style. (SOAP is not an arch style…it is a technology).
Components of REST
- Resources identified with URIs
- HTTP verbs for actions
- Representations and content-types control semantics. (Each REST API has its own).
- Big component of REST
- HATEOAS (“Hypermedia As The Engine Of Application State”)
- – a big divider (religious war) in the REST community
Pros: Allows server to change URIs if it needs to.
Cons: Human concept. Someone must read this and adapt to the changing list of resources.
Examples provided for HATEOAS RES requests and responses.
REST API descriptions
- A competing (to HATEAOS) circle.
- Provide metadata, schema, etc.
- Is bottom-up. Exactly same (history repeating itself) as SOAP evolution.
- Example: Swagger definition and Swagger UI.
LOOKING AHEAD—REST CHALLENGES:
- REST is RPC-centric in its way of thinking
- Netflix has spoken of this the most. REST Too chatty and didn’t scale for them. Created “experience APIs.” Exactly same as Doc/Literal 10 years ago
- Binary Protocols – CORBA-like alternatives from Facebook (Thrif) Google (ProtoBuf) Apache/Hadoop (Avro)
- Async / Real-Time APIs – pushes data to the clients when needed. WebSockets good example
- SDKs vs. APIs – Developers do not want to write HTTP calls. Give them an SDK in their native language. This is easier for them. SDKs pose great challenges too, however. API provider must keep them all up to date and make version-changes work, dependency conflicts, etc.
- Internet of Things (IoT) – IoT devices have limited power and bandwidth. Have pub-sub model. Must minimize on the wire formats. IoT Brokers who connect and integrate many
- APIs are at the heart of so many strategies: mobile, web, partner/integration, developer
- Must treat APIs first not as afterthought
- Avoid (REST) religion
- Choose what’s best for you and your users
Parting thought: Love your APIs and they will love you back.
Presentation link: http://resources.sei.cmu.edu/asset_files/Presentation/2014_017_101_89993.pdf
Making a Language Switch at Google
Patrick Riley, Google
Patrick Riley has led Google’s efforts around search logs. Made significant changes to log contents and way the data is analyzed.
They record each searche’s query, timestamp, user-agent, etc.
Then they collect them all together.
Logs are used for “Google Zeitgeist,” identifying Chrome vs. IE user-base behaviors (quantitatively)
In other words: Search logs very useful for providing good things to users.
Google has a core infrastructure for log processing. Two groups use it: 1. content experts / library maintainers 2. Engineers and analysts who write small bits of code to get interesting things from the logs.
- for a long time was the main language they used for log analysis
- A domain-specific language. Had its pros and cons. Pro and con: language was limiting. Provided security. Limited creativity.
- Other pros/cons listed.
Why switch from Sawzall now?
- Google Trends example. Sawzall was great at say, comparing “baseball” and “football” searches on Google.
- Things became more complicated in the world. Sawzall could not readily analyze log records split up over multiple AJAX requests that comprised one user interaction. Could not deal with information fragmented over multiple logs.
- They outgrew Sawzall. What language should replace?
….this talk focuses on how Google went through the decision-making process.
Needed everyone at Google to buy in and contribute to the change.
First, identified stakeholders (and concerns).
Second, found partnerships. Partnership between team lead of infrastructure team and Patrick Riley with foot in camp of engineers/analysts and camp of log content experts.
Third, identify possibilities that are better than Sawzall.
- Collect champions. (Champions of other options.)
- Have a small expert evaluation of each … with EMPHASIS on: security and performance. They were the “immovable constraints.”
- Evaluate options by all the stakeholders. Looked at other qualities: productivity, maintainability. Very diverse group. Lots of people who had small slices of responsibility and therefore small investment in the decision outcome. Required true buy-in and to do that they needed to be seen as making the right decision (not just making the right decision).
- Collected volunteers from different stakeholders. They wrote small “Hello World” example log analysis programs for evaluation. Important to note: Sawzall was still kept as an evaluation option.
- Collected surveys based on experiences using these other languages for their usual day-to-day activities. Received many free-form responses.
- Simultaneously, created evaluation criteria via partnership with core team and champions.
Fourth, Created a summary doc.
- doc summarized all that happened over the evaluation
- intended to be a *description* of what happened during the evaluation to aid objectivity.
- iterated publically over the document to allow greater feedback, earlier problem-catching and champion-resolution
Fifth, Based on summary doc, wrote a small (two page) decision doc.
- Explained why.
- Acknowledged tradeoffs.
- Frequently referenced summary doc.
Sixth, a series of steps.
Reflecting–was it the right choice?
- It’s been a year.
- Search log analysis has been still growing in complexity and Go is handling it much better than they could with Sawzall.
- Anecdotally, engineers and analysts have given positive feedback.
- There have been a few rough edges and surprises. Ex: knowing exactly which part of the log is being accessed in a given time.
Morals of the story:
- Find representatives from groups of stakeholders.
- Find champions for every option.
- Find people willing to commit their time (very valuable resource)
- Seprate description from evaluation
- Make data-gathering and summarization as public as possible
- Single owner for a decision and document it well
A web developer who works in open source on Asp.net and Azure Cloud for Microsoft. Has been a developer for more than 20 years, working to create large systems.
Check out The HanselMinutes Podcast “Fresh Air for Developers”
This Developer’s Life
Ratchet & The Geek
Also, http://getinvolvedintech.com. Feature-length documentary about becoming a social developer.
Anecodote: gave a talk on the Web and an 80-year-old engineer asked him about becoming a web developer. Had designed the fundamentals of the Intel processor but had a “rocketmail” account. Missed all the bad stuff: 1-pixel gifs, tables, etc.
How do you teach a CS expert about today’s internet? (This presentation)
Think about the cloud in terms of the basic characteristics of the things that make up an OS.
(memory management, graphics subsystems, threading, events, etc.)
Virtual machines are the change because they provide a level of portability…take a VM on a USB key somewhere else.
Can run Linux in Azure.
VMDepot is an example of how there are VMs you can go acquire uploaded from the community.
A VM is something you must “water” like a houseplant…must keep it alive. You manage OS+Apps.
- VM = your first house.
- IaaS – Renting a room.
- PaaS – Hotel
The old way of scaling out was like “yak shaving” – all the stuff you have to do that has nothing to do with the stuff you have to do but you still have to do it. Flying out a consultant from F5 Networks to configure the load balancer. Weeks to scale out.
Now…can adjust a slider or execute at one command.
The cloud doesn’t care about language choice.
“Delightful parallelism” required for the cloud.
See solution: Azure WebJobs
Do something N times…and abstract away all the underlying parallel batching and so forth.
This is what people aren’t talking about enough. The “other virtual machines living in the browser.”
The web was initially kinda similar to a dumb terminal. The “fridge” (server) does the work and the browser displays it.
When Berners-Lee made the web he thought of documents not applications in the browser.
Then Java applets happened. Then Silverlight. Flash.
And people proved some amazing capabilities:
SenchaUI did an implementation of Facebook that was as fast as Facebook implemented in mobile native.
“The avalanche has already started. It is too late for the pebbles to vote.” -Kosh – Babylon 5
Could have applied to HTML5 and JS
“JS is the x86 of the web.” – Brendan Eich
Remember to keep it simple.
JQuery, TypeScript, CoffeeScript, etc. provide possibly too many layers over simple things.
People need to remember there are more servers available to you than you think. All the clients are now servers, so to speak. Maybe do more processing there in the browser where you now have “very powerful servers” available to you.