Description Discovery and Profiles - The Next Level in Web APIs

Citation preview

FACILITATING THE SPREAD OF KNOWLEDGE AND INNOVATION IN PROFESSIONAL SOFTWARE DEVELOPMENT

Description, Discovery, and Profiles: The Next Level in Web APIs eMag Issue 30 - August 2015

ARTICLE

ARTICLE

ARTICLE

Description, Discovery, and Proiles: A Primer

The Power of RAML

From Doodles to Delivery: An API Design Process

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

1

Description, Discovery, and Proiles: A Primer While the process of implementing Web APIs has become common, the tooling for describing, discovering, and understanding the meaning of the tens of thousands of API-based services has yet to settle into a widely-accepted set of standards. here is still quite a bit of opportunity when it comes to deining and implementing tools around the “meta-level” of APIs.

From Doodles to Delivery: An API Design Process Having a basic understanding of web based applications is a good foundation for designing a working Web API. But, if you want to create a good API you need a lot more than that. Designing a good API is hard work and it’s easy to feel overwhelmed when it’s your job to make one.

The Power of RAML RAML, or the RESTful API Modeling Language, is a relatively new spec based on the YAML format- making it easily read by both humans and machines. But beyond creating a more easily understood spec, Uri Sarid, the creator of RAML, wanted to push beyond our current understandings and create a way to model our APIs before even writing one line of code.

APIs with Swagger: An Interview with Reverb’s Tony Tam Ater a lurry of activity from their open working group, Swagger 2.0 was oicially released in September 2014. Our interview took place in March 2015, less than one year from the start of the 2.0 process and right ater Reverb announced that the responsibility for leading the future of the Swagger speciication would be handed over to SmartBear, the Massachusetts-based sotware tools company.

The APIs.json Discovery Format: Potential Engine in the API Economy

Programming with Semantic Proiles: In the Land of Magic Strings, the Proile-Aware is King As this article’s author, Mark Foster, puts it, “Absent proiles, the API space will be relegated to blindly passing around ‘magic strings, fooling ourselves into thinking we are passing reliable semantic information.” Here, Foster — one of the editors of the ALPS speciication — explains what semantic proiles are and how they can transform the way Web APIs are designed and implemented.

FOLLOW US

In the fast growing world of APIs and microservices, inding just the right API when you are developing a web, or mobile application, or possibly integrating between existing systems, is always a tedious task.

Proiles on the Web: An Interview with Erik Wilde Erik Wilde talks to Mike Amundsen about Proiles, Description, Documentation, Discovery, his Sedola project and the future of Web-level metadata for APIs.

CONTACT US GENERAL FEEDBACK [email protected] ADVERTISING [email protected] EDITORIAL [email protected]

facebook.com /InfoQ

@InfoQ

google.com /+InfoQ

linkedin.com company/infoq

internationally MIKE An known author AMUNDSEN and lecturer, Mike

A LETTER FROM THE EDITOR

Amundsen travels throughout the world consulting and speaking on a wide range of topics including distributed network architecture, Web application development, and other subjects. In his role of Director of Architecture for the API Academy, Amundsen heads up the API Architecture and Design Practice in North America. He has authored numerous books and papers on programming over the last 15 years. His most recent book is a collaboration with Leonard Richardson titled “RESTful Web APIs” published in 2013. Mike’s 2011 book, “Building Hypermedia APIs with HTML5 and Node”, is an oft-cited reference on building adaptable Web applications.

This eMag focuses on three key areas of “meta-language” for Web APIs: API Description, API Discovery, and API Proiles. You’ll see articles covering all three of these important trends as well as interviews with some of the key personalities in this fast-moving space. API description languages and utilities have been around since the Interface Description Language (IDL) era of the 1980s and 90s. Now we have formats like Swagger, RAML, and Apiary Blueprint as the most popular in the ield of Web APIs.

4

API discovery work — inding an API to meet your needs — is primarily a human-driven search, select, and on-boarding process today. However, enterprise-level tooling such as ZooKeeper, Consul, and etcd are pointing the way toward automating much of that process. Finally, API proiles have emerged from the quiet land of library and information science into the world of APIs for the Web. This is a very new ield but a few vendors are already taking a look at the possibility of supporting API Proiles today.

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

READ ONLINE ON InfoQ

Description, Discovery, and Profiles: A Primer

by Mike Amundsen

While the implementation of Web APIs has become common, the tooling for describing, discovering, and understanding the meaning of the tens of thousands of API-based services has yet to settle into a widely accepted set of standards. There is still quite a bit of opportunity when it comes to deining and implementing tools around the meta-level of APIs.

For now, three areas continue to see quite a bit of interest and activity. They are:

Description This is the ability to easily describe APIs including implementation details such as resources and URLs, representation formats (HTML, XML, JSON, etc.), status codes, and input arguments in both human-readable and machine-readable form. There are a few key players setting the pace here.

Discovery This is searching for and selecting Web APIs that provide the de-

sired service (e.g. shopping, user management, etc.) within speciied criteria (e.g. uptime, licensing, pricing, and performance limits). Right now, this is primarily a human-driven process but there are new players attempting to automate selected parts of the operation.

Profiles Long a focus of librarians and information scientists, proiles that deine the meaning and use of vocabulary terms carried  within  API requests and responses are gaining renewed interest for Web APIs. Although it is still an experimental idea, there is some

evidence that vendors and designers are starting to implement support for Web API proiles. Each of these three categories of Web API metadata has key players and trends.

Describing API Implementations Currently, most of the focus on API design and implementation is on  description  formats. The most commonly mentioned formats are Swagger, RAML, and API Blueprint, but the list of available formats is quite long. Each takes a slightly diferent approach but essentially ofers the same basic

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

5

Example of an API Blueprint description 001 FORMAT: 1A 002 003 # Resource and Actions API 004 This API example demonstrates how to deine a resource with multiple actions. 005 006 # /message 007 This is our resource 008 009 ## GET 010 Here we deine an action using the `GET` HTTP request method. 011 012 As with every good action it should return a response 013 014 + Response 200 (text/plain) 015 016 Hello World! 017 018 ## PUT 019 OK, let’s add an update action and send a response back conirming the posting was a success 020 021 + Request (text/plain) 022 023 All your base are belong to us. 024 025 + Response 204 features: a way to describe a Web API at varying levels of detail. API-irst Most approaches support the  API-irst  concept. You describe your API using a meta-language based on XML, JSON, or YAML, and the resulting document (or set of documents) is used to auto-generate implementation assets such as server-side code, human-readable documentation, test harnesses, SDKs, or even fully functional API clients. An example of the API-irst approach is  Apiary’s  API Blueprint format. It’s based on Markdown and aims for human-readable descriptions of APIs that are also machine-readable. In the example above, you can see there is a single resource (/message) that supports both  GET and  PUT. You can also see there is support

6

for human-readable text to describe the way the API operates.

RAML,  Swagger, and similar formats work essentially the same way. With the API-irst approach, you need tooling to convert the meta-language created at design time into sometime useful at run time. For example, the Swagger  Code Generator  tool parses description documents and generates compliant client-side code, while the  RAML for JAXRS  project provides two-way transformation between RAML descriptions and JAX-RS-annotated Java code. Code-irst Very few description models support the  code-irst  approach, by which you generate the service description from the source code. However, the most famous

of these - the Web Service Deinition Language  (WSDL) - is still popular with the enterprise community, and there is a great deal of tooling and support for WSDL built into common editing platforms such as Microsoft Visual Studio and Eclipse. Code 1 is an example of a simple Web API described using WSDL. When using the code-irst approach, you need tools to turn your source code into usable API-description meta-data Eclipse and Visual Studio have  a one-click process for creating WSDL iles from code. There are also several tools that can  consume the WSDL ile and generate implementation assets. For example,  SmartBear’s  SoapUI  utility can generate code, create human-readable documentation, and even build and run test suites based on WSDL iles. Documenting APIs Most API description formats also support the creation of human-readable documentation. This is true for RAML, Apiary, and Swagger. In fact, the open source  Swagger UI  tool is so well known as a documentation generator (see the igures 1&2) that some mistakenly think that Swagger is only used to generate human-readable API documentation. There are also formats that are designed with a focus on generating human-readable documentation. A well-known example of this is Mashery’s  I/O Docs  (see Code 2), which also provides testing support. Whether you are focused on generating code from meta-languages or documentation from code (or any of the other possibilities), API description gets you only partway through the process of creating and deploying Web APIs. Another important part of the process is

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

HelloService WSDL example 001 007 008

009

010

011 012

013

014

015 016

017

018

019

020

021

022 023

024

026

027

028

029

033

034 035

036

040

041

042

043 044

045 WSDL File for HelloService 046

047

049

050

051 Code 1

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

7

Figure 1. SoapUI consuming WSDL.

Figure 2. Human-readable documentation generated by the Swagger UI utility.

8

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

Mashery’s I/O Docs example 001 { 002 “name”: “Lower Case API”, 003 “description”: “An example api.”, 004 “protocol”: “rest”, 005 “basePath”: “http://api.lowercase.sample.com”, 006 “publicPath”: “/v1”, 007 “auth”: { 008 “key”: { 009 “param”: “key” 010 } 011 }, 012 “headers”: { 013 “Accept”: “application/json”, 014 “Foo”: “bar” 015 }, 016 “resources”: { 017 “Resource Group A”: { 018 “methods”: { 019 “MethodA1”: { 020 “name”: “Method A1”, 021 “path”: “/a1/grab”, 022 “httpMethod”: “GET”, 023 “description”: “Grabs information from the A1 data set.”, 024 “parameters”: { 025 “param1”: { 026 “type”: “string”, 027 “required”: true, 028 “default”: “”, 029 “description”: “Description of the irst parameter.” 030 } 031 } 032 }, 033 “MethodA1User”: { 034 “name”: “Method A1 User”, 035 “path”: “/a1/grab/{userId}”, 036 “httpMethod”: “GET”, 037 “description”: “Grabs information from the A1 data set for a speciic user”, 038 “parameters”: { 039 “param1”: { 040 “type”: “string”, 041 “required”: true, 042 “default”: “”, 043 “description”: “Description of the irst parameter.” 044 }, 045 “userId”: { 046 “type”: “string”, 047 “required”: true, 048 “default”: “”, 049 “description”: “The userId parameter that is in the URI.” 050 } 051 } 052 } 053 } 054 } 055 } 056 } Code 2

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

9

Figure 3. Search interface for ProgrammableWeb.

iguring out what APIs and services are out there and what it takes to consume them. For that, you need to discover the APIs.

Discovering APIs in the real world API discovery is the ability to locate the Web API needed for a particular job. For example, you may be looking for a Web API that allows you to support online shopping, or manage user accounts, or process helpdesk requests, etc. In an ideal world, you should be able to initiate a search, ind the API that its your

needs, gain access to that API, implement connecting code, and start using it with a minimum of efort. However, reality is diferent. Discussion about API discovery often conlates the  application programming interface (API) with a live, up-andrunning service. In the irst case, we’re just talking about the interface - one that you might use to design, implement, and deploy your  own  service. In the second case, we’re referring to an existing instance of the service itself - something you can connect to

remotely and start using immediately. Since the hurdles and beneits to each are diferent, it’s worth some time to review some examples. API Commons When you are looking for a published set of API speciications that you can implement yourself, the API Commons can be a good source. The goal of API Commons is to “Provide a simple and transparent mechanism for the copyright free sharing and collaborative design of API speciications, interfaces and data models.” For

10 Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

Figure 4. The Mashery API Network.

example, if you have already designed an API that you would like to share with others to so that they can implement their own service  using your model, you can publish your model at API Commons and encourage others to use it. If, on the other hand, you are wondering if someone else has already dealt with the same problem, you can search API Commons for an existing design that will work for you. This publish-and-subscribe pattern makes it possible for similar services to use the same interface without the need for detailed coordination between parties. In the best case, an API consumer built to work with an API design registered in the API Commons will be able to work with any other service that uses the same design. Creative Commons inspired the operating model of API Commons, which is currently main-

tained by  Kin Lane and Steve Willmott. Searchable service listings When people talk about API discovery, they mostly mean discovering an available running instance: a usable service. A handful of services keep track of usable services and most of them operate as human-readable search engines. One of the bestknown examples  is ProgrammableWeb’s  API Directory  (see igure 3). This kind of solution works when you want to: • search for services that that it your criteria, • evaluate the ones that seem to meet your needs, • engage in the on-boarding process for that service, and inally • write your API consumer code to match the API of your selected service.

It is possible that the service you select will support one or more of the API description languages we covered above, and that can ease the work of creating an API consumer for your selected service. A potential downside of this kind of discovery service is that not all directories are carefully vetted (some rely on self-registration). You may need to wade through several APIs that don’t support the protocol or format you need, are no longer active, or do not meet your performance or licensing requirements, etc. Also, the process of going through the search, evaluation, on-board, and interface-generation loop can get tedious if you plan on using several third-party APIs. Aggregation services There is another type of API discovery approach: the API aggregator. The aggregator acts as proxy for one or more existing

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015 11

Figure 5. Apache Zookeeper and HashiCorp’s Consul ofer run-time discovery within an enterprise.

Web services and ofers a single, uniied API for you to code against. An example of this is Intel’s  Mashery API Network. (Figure 4) If you plan on consuming several third-party Web APIs, an aggregator ofers the chance to greatly reduce your on-boarding and integration eforts. Aggregator services take on the work of consuming and normalizing the back-end API and managing your access keys, and some even ofer a custom API for you - one that makes is easier to share related data between each third-party API. Conigurable run-time discovery Some people think about API discovery services as a conigurable  local  service-discovery engine. These discovery engines

exists within the boundaries of a single company and handle  inding and connecting to one or more running instances of a service such as a data store or business component. This approach has been growing over the last few years and examples include Apache  ZooKeeper, HashiCorp’s  Consul, and CoreOS’s etcd. (Figure 5) The advantage of this discovery approach is that it provides an additional level of indirection when connecting to running services within your own organization. You can remove the actual address and connection parameters from your running code and place this information in coniguration iles. Some services will even allow you to set limits on latency and responsiveness in ways that will automatically ignore or route

around slow-running or unavailable service instances and connect to the next available healthy instance. Of course, this kind of abstraction has downsides. First, the added complexity only pays of in large installations. Second, coniguration models for these services are currently not standardized, which means you end up with a strong dependency on a single vendor. Finally, this kind of run-time discovery service is not yet available for use with publicly available third-party API services outside your organization. Web-scale discovery with APIs. io and APIs.json An interesting alternative to a single-source collection of APIs is the  distributed search  approach of  APIs.io. A joint efort

12 Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

An XMDP example 001 003 004 sample HTML proile 005 006 007 author 008 A person who wrote (at least part of) the document. 009 keywords 010 A comma and/or space separated list of the 011 keywords or keyphrases of the document. 012 copyright 013 The name (or names) of the copyright holder(s) 014 for this document, and/or a complete statement of copyright. 015 date 016 The last updated date of the document, in ISO8601 date format. 017 identiier 018 The normative URI for the document. 019 rel 020

021

022 script 023 A reference to a client-side script. When used with the 024 LINK element, the script is evaluated as the document loads and 025 may modify the contents of the document dynamically. 026

027

028

029 030 Code 3 between  3scale  and Kin Lane of API Evangelist, this is a classic search engine that doesn’t actually  host  API documents but tracks where the discovery iles are found on the Web and ofers a search interface for the data in those iles. All the discovery iles are in a format known as APIs.json. Unlike API description documents, APIs.json iles don’t contain details of all the available URLs, representations, and response code. Instead, these iles contain pointers  to those description documents along with pointers to terms of service, licensing, contact info, and other related data. This format is quite new but may ofer some glue to help connect API descriptions and other data all in a single place. Since the format is machine readable, it ofers the potential to automate

some of the search and possibly even the on-boarding details for Web APIs. Still at the detail level Both API description and API discovery aim to make it easier to build and locate APIs for the Web. However, many eforts focus on low-level implementation details such as describing protocol methods, return codes, and the shape of payloads. These are essential when it comes to writing code, but sometimes can get in the way of  designing  good APIs. Because most approaches focus on implementation details, they usually describe the speciics of a  single instance  (or mirrored cluster) of a running service. If you want to focus on higher-level design aspects (use cases, inputs, and outputs) free of the details of protocols, rep-

resentations, and resources, you need something else.

Leveraging profiles for shared understanding A more recent development in API metadata services is the idea of creating and sharing API  proile information. Unlike description documents, proile documents ofer a high-level view of what the API supports and, in some cases, how clients and servers can expose features in a machine-readable way. Proiles on the Web: A short history Web proiles have been around for quite some time. The  HTML 4.01  speciication introduced the proile  attribute in 1999. The  meta data proile  was deined as either a globally unique name (URI) or a link (URL) to an

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015 13

An example of a DCAP document

001 Description template: Person id=person 002 minimum = 0; maximum = unlimited 003 Statement template: givenName 004 Property: http://xmlns.com/foaf/0.1/givenname 005 minimum = 0; maximum = 1 006 Type of Value = “literal” 007 Statement template: familyName 008 Property: http://xmlns.com/foaf/0.1/family_name 009 minimum = 0; maximum = 1 010 Type of Value = “literal” 011 Statement template: email 012 Property: http://xmlns.com/foaf/0.1/mbox 013 minimum = 0; maximum = unlimited 014 Type of Value = “non-literal” 015 value URI = mandatory

Code 4

A search API described by an ALPS document 001 { 002 “alps” : { 003 “version” : “1.0”, 004 “doc” : { 005 “href” : “http://example.org/samples/full/doc. html” 006 }, 007 “descriptor” : [ 008 { 009 “id” : “ind-user”, 010 “type” : “safe”, 011 “doc” : {“value” : 012 “User search form” 013 }, 014 “descriptor” : [ 015 { 016 “id” : “userName”, 017 “type” : “descriptor”, 018 “doc” : { “value” : “input for search” } 019 }, 020 { “href” : “#userStatus” } 021 ] 022 }, 023 { 024 “id” : “userStatus”, 025 “type” : “descriptor”, 026 “description” : {“value” : “results ilter”}, 027 “ext” : [ 028 { 029 “href” : “http://alps.io/ext/range”, 030 “value” : “active,inactive” 031 } 032 ] 033 } 034 ] 035 } 036 } Code 5

actual document. It was designed to allow document authors to provide additional descriptive information about the contents of the response (e.g. useful indexing properties of the document, terms of use, etc.). In 2003,  Tantek Çelik  deined the XHTML meta data proile (XMDP). XMDP is a format for deining document proiles that are both human-readable and machine-readable. (Sound familiar?) An XMDP document resembles a modern API description. (Code 3) The proile property never gained widespread use and so was dropped from HTML5, although there is an efort to  deine the proile property  separately. Removing the proile property from HTML prompted  Erik Wilde  to pen  RFC 6906  for deining the proile  link-relation value  and registering it with the  Internet Assigned Numbers Authority  (IANA). Wilde’s idea was to standardize the URI-style proile — an opaque identiier that “allows resource representations to indicate they are following one or more proiles.” The ability to describe the details of payload via proiles has prompted increased interest in applying proiles to only not human-readable documents but also to API responses. In the last few years, several URL-style proile implementations have emerged, too.  For this article, we’ll look at two of them: DCAP and ALPS. Dublin Core Application Proiles (DCAP) In 2009, the  Dublin Core Metadata Initiative  (DCMI) released their DCAP format for describing proile metadata. Focused on supporting Resource Description Framework (RDF) documents, DCAP “deines metadata records which meet speciic application

14 Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

Figure 6. Managing proile vocabulary with the Rapido API designer.

needs while providing semantic interoperability with other applications on the basis of globally deined vocabularies and models.” (Code 4) DCAP was created to improve the ability to share data semantics in a world where the representation format (RDF) is highly constrained and generic (e.g. triples). A key advantage of DCAP is that it does not dictate which terms are used in a response (e.g. givenName, familyName, customer, user, etc.) but dictates how those terms are communicated. This paves the way for shareable vocabularies for online use. There is some tooling for DCAP including online editors, validators, and HTML generators but the use of DCAP has been limited to libraries, information science, and the academic community. It is not common to ind DCAP used for business-related APIs on the Web.

DCAP, ALPS also borrows ideas from XDMP. ALPS includes metadata for both data elements (e.g.  userName, userStatus, etc.) and use-case elements (e.g. ind-user, etc.). (Code 5) ALPS documents focus on the interface-level interactions — what Eric Evans calls the “bounded context”. It does this without addressing implementation details like protocol (HTTP, XMPP, etc.), format (HTML, JSON, etc.) or even resource URLs. This free-

dom from implementation details means ALPS documents can be used as source material for API design tools, for generating human-readable docs, and even as part of a discovery process to help in the selection of APIs for a desired use. Currently, ALPS is best described as an “unstable” speciication and there are few examples of its use on the Web. Ronnie Mitra’s experimental Rapido API designer (see igure 6) can use ALPS

Application-Level Proile Semantics (ALPS) In 2013, Leonard Richardson, Mark Foster, and I released the irst draft of  ALPS. Similar to

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015 15

documents as inputs and  Pivotal’s  Spring Data tools produce ALPS documents as part of their API build process. While there is renewed interest in using proiles for Web APIs, it is too early to know whether this is just an experiment that will fade or it is the start of a trend focusing on independently deining the data and action semantics of a Web API.

The challenge ahead The key areas of Web API metadata include description, discovery, and proiles. Initiatives from Swagger, RAML, and Apiary currently dominate the description space among a handful of other players in this healthy ecosystem. There is also support for using description formats to automate the generation of both code and documentation and a strong set of utilities has grown around a few key formats. The API discovery space continues to be dominated by human-driven search and selection while a few key API aggregators like Mashery continue to provide an aggregator approach for subscribing to remote APIs. There is a growing world of automated, coniguration-based location services aimed at supporting connection to enterprise-level service instances and some of that approach may start to show up in eforts to provide automated service discovery to Web-based APIs. API proiles — commonly used in library and information sciences — have been getting renewed interest for business-related Web APIs. Current initiatives are either experimental or have limited reach but there are vendors showing support for API proiles. The Web is a dynamic and fast-moving space and it should be interesting to keep an eye on this meta-level of the API ecosystem for some time to come.

In an ideal world, you should be able to initiate a search, ind the API that its your needs, gain access to that API, implement connecting code, and start using it with a minimum of efort.

16 Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

READ ONLINE ON InfoQ

From Doodles to Delivery: An API Design Process

Ronnie Mitra is director of API design for CA’s API Academy. He helps organizations around the world design and successfully implement APIs. Ronnie has over thirty years of experience failing (and learning) iteratively.

Understanding Web-based applications is a good foundation for designing a Web API, but if you want to create a good API, you need a lot more than that. Designing a good API is hard work and it’s easy to feel overwhelmed when it’s your job to make one. Good design is deinitely achievable and an iterative process can help us reach it through three important phases: sketching, prototyping, and implementation.

Good API design happens in iterations Before we can begin designing an API, we must understand its purpose. You should know why you are building an API before you begin and there are many resources  available  to those seeking  help. Deining your motiva-

tion is only the irst step. The real trick is making good design decisions all the way to implementation. Succeeding with an API design means designing an interface whose  usage  fosters its purpose. As API designers, we make decisions that have an impact on the success of the product. There are big decisions to be made, such as the transport protocol that the API will use or the message format that it will support, but there are many

smaller decisions related to the controls, names, relationships, and sequences of an interface. When you put them together, all these decisions drive a pattern of usage. If you’ve made only good decisions, that pattern will support and foster the purpose of the API perfectly. If you want to make a correct design decision, you’ll probably irst need to make a wrong one and learn from that experience. In fact, you’ll probably need to make many mistakes before

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015 17

you can get close to being correct. This is the crux of iteration: no one gets it right the irst time but given enough opportunities, you can get closer to perfection. It’s easy to say we should be iterating our API designs but it’s hard to do that in the real world. A common challenge is the dificulty in changing an API after releasing it. Changing an API that is in use can be expensive and risky. As Joshua Bloch says, “Public APIs, like diamonds, are forever.” One way to address this problem is to stop breaking the interface every time we change it. This is an excellent habit and is a cardinal rule for good API design. But sometimes we can’t avoid changes that break the system and fundamental design changes are especially invasive. We instead need to make these changes well before we release the interface. We ideally should iron out all usability and design issues before change becomes expensive. Of course, inding those problems and ixing them is only possible if we can iterate early and often before the irst release.

An iterative design process Each iteration gives us a chance to measure our design candidate according to its usage. For example, will developers be able to accomplish their goals using what we have built? Can this interface actually be implemented? How will people feel if we ask them to use this API? We should be able to achieve the best API design by designing and implementing many interfaces without releasing them. Reviewing and testing each interface would provide powerful insight into how we could improve the end product. In practice, this type of grand iterative design is impossible to achieve. Few of us have

the time, money, or inclination to design and implement a working API over and over. For any non-trivial interface, this type of iterative design would just take too much time. A more reasonable approach is to perform iterations very early in the design process. These early designs should have just enough detail to expose the biggest opportunities for improvement, but not so much that they are hard to build. We can gradually increase the level of detail (or idelity) until we arrive at a fully realized implementation. This graduated process is popular in the design world, which typically breaks it down into three important phases: 1 Sketching 2 Prototyping 3 Implementing The Power of sketching Sketching is a universal act of design. Architect  Frank Gehry’s sketches are famous enough to have been the subject of a  ilm. Many of his building projects started life as a series of sketches drawn on paper. Gehry draws hundreds of these sketches, each time moving closer to an ideal design. Interaction designer  Bill Verplank  describes sketching as an essential irst act in the design process.  Bill Buxton  has written an entire book on the value of sketching for user-experience design, calling out its key characteristics of disposability and low cost of exploration amongst others. Incorporating sketching in the early phases of an API design process allows us to experiment with the conceptual model of the interface. This is a chance to not only deine the ideas that we have in our head but also to explore new avenues and ask the

“what if?” questions that can lead to real innovation. Good sketches should be easy to make and easy to destroy. If they take too long or are too diicult to create, they become harder to walk away from. Disposability is important because it allows us to take risks. We can experiment with diferent types of interface styles and capture the abstract concepts that are loating around our minds. Once these ideas are material, we can review and discuss them. We get to decide what we like and dislike about a particular notion and start the process again with new sketch that applies that knowledge. Sketches are rarely evaluated by users who are not embedded in the design team, not because there isn’t value in testing assumptions early but because there is a inite amount of user testing that can be done in a real project. Testing every sketch with real users would be an expensive endeavour with limited return. Sketching with proiles It’s useful to use a proile or meta language  when sketching an API. Proiles provide a set of constructs from which a sketch can develop. A good proile can be analogous to the boxes and lines that form many sketches of systems diagrams. These are elements that both the designer and evaluators understand and make it easier to develop sketches quickly. In fact, a good way to start the sketching process is to deine the list of words that will feature prominently in the API interface. Which words must a user know? Which best capture the goals and tasks of your target audience? Answering these questions and creating an interface vocabulary will help you as you form an early sketch of the interface.

18 Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

Figure 1 The wonderful thing about proiles is that they provide a formalised way to share and reuse this type of information. For example, we might start our design by picking words from an XML-schema document, a vocabulary retrieved from Schema.org, or perhaps an ALPS or RDF document, depending on our needs. In the design phase, the goal should not be to create a comprehensive vocabulary. We want an early vocabulary as a starting point from which we can draw out other types of detail. We might ind that a list of twenty words captures the essence of our interface and leave it at that. The vocabulary provides a base from which we can sketch the resources and relationships of the API. This may include the URIs, relationships between resources, link text, and other structural and navigational elements. Again, there is no need to sketch every detail; the aim is to express the most important parts of the API. Above all, it’s important not to go too far with the initial sketches. For example, try

to avoid diving into error-low modelling or response messages at this stage. We can add these ideas later or may even sketch them on their own. A single sketch does not have to relect the entire interface. In fact, it is useful to sketch certain details separately. For example, we might sketch a basic error low that will be relevant for the entire API or a message format that can be applied to all responses. We can apply these ideas to a working model during the prototyping phase later. Prototyping The prototyping stage is an opportunity to build a higher-idelity model of the interface and test some of the assumptions we have made during sketching. A good API prototype should be invokable. It should handle real request messages and provide responses when required. Developers should even be able to create a simple application with the prototype. Prototypes should always be less expensive to create than a full-on implementation. One way

to keep the cost down is to simulate response messages rather than serving real responses from your back-end systems. This is sometimes referred to as “mocking  the interface” and is a great way to build a prototype quickly. However you build the prototype, it is important to ind a level of investment that makes it possible to build new iterations. We should be able to build two or three diferent prototypes based on sketches, learning as we go along. We may even go back to sketching to experiment with new directions based on what we learn from the prototyping phase. Prototypes let your team gather early user feedback on your design and observe real usage. If the interface has the right level of idelity, you can ask potential users to build an application and note the challenges they face. Implementers should also participate in the prototype evaluation. Well-designed APIs should not only be easy to use but also sustainable and reliable. While the interface shouldn’t

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015 19

Figure 2

expose the inner details of data models or server architecture, implementers can advise the design team on environment constraints and implementation costs for some of the decisions that have been made. The design cycle is like a scientiic process, and the prototype phase is your chance to test any hypotheses before it is too late to make changes. Implementing The implementers’ job is to turn a prototype interface into something that can be relied upon for serious use. The delivery of a safe, reliable, and scalable implementation is challenging and will usually require a design process of its own. The inal prototype and supporting sketches form a description of what the interface should look like. They relect the collective design decisions on what needs to be built. It’s useful to use a formal interface description language (or  IDL) to transition from the prototype to the implementation phase. For example, when you are happy with your prototyped API you might choose to describe

it in an API Blueprint document (or  Swagger,  RAML,  WADL, or whichever format best its your working environment). Although implementation is the goal, design shouldn’t stop at this stage. This is an opportunity to further test the assumptions you’ve made throughout the design process with real usage data. Just as prototype APIs allow us to observe usage, implemented APIs allow us to analyse usage at a macro level. For example, you might want to validate the design assumptions you’ve made. Are application developers really using that convenience operation you created for them? Are you getting the type of users you expected? Are new users having trouble with speciic parts of the interface? This analysis may lead you to starting the sketching process again to further improve your interface.

Automating the process with tools Tooling and technology can radically improve the design process. Tools that lower the cost of sketching and prototype creation will allow design teams

to produce more designs in less time, leading to improved design decisions. Incorporating tooling and automation is an essential part of most design processes. In the world of building design,  SHoP  (a consortium of architects based in New York) is succeeding through innovation, collaboration and tool based design. Their architects’ process includes prototyping tools that allow designers to incorporate the physical characteristics of the materials they want used. This method allows them to create thousands of design iterations, each incorporating implementation details that can easily be evaluated. There is ample opportunity for this type of tool-based optimization in the API design space. Indeed, some excellent Web-API design tools already exist in the service-description domain. It is now common to ind that IDLs provide supporting tools to ease the task of writing a description. These editors shorten the creation time for IDLbased design, making it easier to create more descriptions in a shorter time. Swagger, RAML, and API Blueprint all ofer ex-

20 Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

Figure 3 cellent editing tools to support their languages. Even IDLs like WADL that are released only as speciications beneit from tools like SoapUI. Apiary  (Figure 2) provides an editor for the API Blueprint language that is particularly compelling because it ofers a complete worklow of tools to support the design process. By simply writing an API description in API Blueprint, the designer can evaluate the documentation, invoke a prototype, and even analyse invocations. Most tooling is constrained to the particulars of the description language it supports. The designer must understand the syntax of the IDL and design the interface using this language. While this style of design may

appeal to users who are familiar with programming languages, it can also limit the abstract and experimental thinking that is valuable in the early sketching phases. These characteristics make IDLs and their associated tools an ideal it for the prototyping process, but less helpful for the early experimental sketching. Sketching with visual tools Interest in the use of visual tools for API design is rising. Rather than supporting the editing of an IDL directly, these tools allow designers to play with a visual representation of the interface. Visual-modelling tools provide an IDL based mostly on pictures or diagrams. While this perspective constrains the idelity

of the interface representation, it also allows the designer to see the entirety of the interface from a high level. This presents opportunities for improvement that may not be apparent in written form. Visual editors that are easy to use make good candidates for automated sketching. The mix of low idelity, abstract representation, and disposability is exactly the right it. I’ve been working on a visual-modelling tool called  Rápido  (Figure 3) to support API sketching. Rápido constrains the user to lower-idelity detail not as a side efect of its form factor but by design. Users can model proiles, navigational elements, and response data but not logical lows or dynamic responses.

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015 21

Figure 4

When designers start a new Rápido project, they are asked to create a vocabulary for the API. After coming up with an initial list of words (or importing an ALPS vocabulary from an external source), the designer can start to form the conceptual model for the API, creating resources, experimenting with URI names, and even linking states together in a hypermedia canvas. Designers can increase the idelity by implementing static response messages for each resource or state they are sketching. When the sketching phase is complete, the design can be exported in IDL format ready to be plugged into higher-idelity tools for the prototyping phase. The goal of the Rápido experiment is to deliver the experience of a quickly drawn “cocktail napkin” sketch to the Web-API design space. If the idelity is correct and the constraints are not too tight, it can form an important part of an API-design tool chain.

Succeed with an iterative process Follow the iterative style of design described in this article and you will give your team a chance to create an efective API. Create and evaluate many low-idelity designs at the beginning of the process to foster experimentation and ideation. Build higher-idelity prototypes and mock implementations to evaluate early design ideas. Finally, implement the design for real users and gather data for analysis of real world usage. The speciic details of an iterative design process will depend on your unique environment and project. The exact amount of detail, number of iterations, and evaluation techniques are left to the designer. As a general rule of thumb, the number of iterations should decrease as you move through the design process, while the evaluation efort will probably increase. A good API design process allows you to produce the best

interface possible. The secret to building a great API is not expert guidance or secret knowledge. Instead, it is the application of an iterative process that is optimized by great tools, languages, and proiles. Use this formula to deliver an API that will help your organization succeed.

22 Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

READ ONLINE ON InfoQ

The Power of RAML

Michael Stowe, a RAML enthusiast, has over 10 years experience building applications for diferent use cases including law enforcement, the medical ield, non-proits, and industrial companies, and has contributed to several open-source projects. He now works as a developer relations manager at MuleSoft. He has spoken on RAML and API design at multiple conferences including API Con, API Strategy and Design, and API World. Look for Michael’s new book on REST API design soon, and follow @mikegstowe on Twitter to stay up to date with his latest posts. You can view his talks and read his thoughts at: http://www.mikestowe.com

RAML, or RESTful API is a relatively new spec based on the  YAML  format, making it easily read by both humans and machines. Beyond creating a more easily understood spec (one that could be handed to the documentation team without worry),  Uri Sarid, the author of RAML, wanted to push beyond our current understanding and build a way to model our APIs before writing even one line of code. On his Untangled blog, Roy Fielding, the creator of REST, points out that we tend to design for the short term, with focus on what is known and what we are currently thinking about. “Unfortunately, people are fairly good at short-term design, and usually awful at long-term design,” he writes. The challenge with APIs is that they are usually intended

to last, hopefully for years. After all, an API requires a substantial investment on the part of the developer, but also a substantial investment on part of the consumer — who relies on it and has to implement it. Sarid started thinking about how to solve the challenge of long-term design with APIs in 2009. He wanted a tool that could model (or design) your API in just a few lines and

then quickly generate a prototype, which developers around the world could try out and give you feedback. In 2013, he released RAML 0.8. Interest in RAML has grown as companies large and small have realized the beneit of being able to design APIs in a human-readable format, to see exactly what their API will look like as they design it, and to create

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015 23

Figure 1 a live, functional example of the API against which developers can make real calls with just the click of a button. (Figure 1) All of the RAML tools are open source and freely available at  http://RAML.org/projects. My employer, MuleSoft, ofers freely hosted versions of these tools that you can use within its Anypoint Platform for APIs.

The API Design Cycle However, simply creating a prototype is not enough. Working under the premise that an API is a contract not just between machines but between a provider and a user, the MuleSoft created the API Design Cycle. (Figure 2) After designing and creating a prototype, a company needs a way to share its API with others and solicit feedback. This valuable feedback would let the API team identify design laws such as data or structural inconsistencies as well as confusing aspects of the API. The identiication of design issues in this stage is critical as once you release an API, you usually cannot ix it without breaking backwards compatibility, which can hinder its use.

This cycle is also the basis behind  spec-driven development. API Console and API Notebook To accomplish this, MuleSoft developed two more tools: API Console and API Notebook. API Console, similar to other API consoles such as  Mashery I/O Docs  and  Swagger, provides both documentation and an interactive environment in which developers can enter data and make calls. This means that developers can quickly see what

resources and methods are available while prototyping the API and test them out for instant validation. Once you launch your API, you can provide interactive documentation (on top of your static documentation) to developers who want to try out and debug calls. API Notebook, on the other hand, takes interactivity and exploration a step further by letting developers use JavaScript to call your (and other) APIs. With API Notebook, they are also able to manipulate the data to see how

Figure 2

24 Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

it would work in a real-world use case. This means developers can test your prototype for their use case without having to write anything more than a few lines of JavaScript. In the screenshot above, users are able to connect to the  Instagram API, authenticate using OAuth, and search for im-

ages with the tag “kitten”, being walked through each step along the way. But what truly makes API Notebook a powerful tool for RAML is that you can create use cases for your consumers to work through. Not only can you create these scenarios both for your prototype and for your API once you

launch it into production, your consumers can also create their own scenarios using markdown. This lets your consumers share bugs or use cases for feedback and support purposes without having to share any proprietary code — or forcing you to work through their entire application.

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015 25

This simple feature greatly reduces the efort your consumers have to put into getting their questions answered, while giving your support team reproducible errors that they can quickly identify without having to guess whether or not they are caused by the client’s code. Once you have collected this feedback, you are able to ine-tune your design and decide whether or not it is ready for production. The other advantage, as mentioned, is that when your API is ready to go live, you already have these two amazing tools for use in production! Many other tools provided by the community can help you take your API live, including tools for generating the code for your API and for testing your API, such as Abao. There are also numerous tools for documentation beyond API Console and API Notebook, including  RAML to HTML, which generates a single HTML ile (similar to API Console) for documentation, and  RAML 2 HTML for PHP, a script that creates a full, multipage documentation site: This script is completely customizable, letting you modify the overall template and even content blocks to match your site exactly. There are also numerous parsers for RAML that let you easily integrate your spec into your own custom applications, whether they are written in Ruby, PHP, JavaScript, .NET, Python, or Java.

Human readable Because RAML is designed to be human readable, designing a new API or deining a pre-existing API is easy. While you can create RAML in any editor, you’ll ind that using the online API Designer or an IDE plugin such as the ones for Sublime or Visual Studio  (both available at  RAML. org/projects) make this process

even easier by providing tooltips, autocomplete, and real-time validation as you go. To get started, just begin your RAML ile like so: 001 #%RAML 0.8 002 title: This is My API 003 baseUri: http://api. domain.com 004 version: 1

In the above, we are irst declaring that this is a RAML spec, 001 002 003 004 005 006 007 008

using RAML version 0.8 (version 1 is set to be released soon), declaring the title of our API, the base URI for our API, and what version our API is (in this case, version 1). Declaring resources in RAML is as simple as /resourceName, and methods are likewise as quick and simple to add just by referencing the HTTP verb: RAML has numerous other features that let you fully deine

#%RAML 0.8 title: This is My API baseUri: http://api.domain.com version: 1

/resource1: get: description: This gets the collection of resource1 009 post: 010 description: This adds a new item to the collection

RAML ofers you the ability to host multiple responses, returning diferent status codes, headers, and bodies: 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025

#%RAML 0.8 title: This is My API baseUri: http://api.domain.com version: 1 /resource1: get: responses: 200: headers: cache-control: example: | public, no-cache, no-store body: application/json: example: | {“name”:”Michael Stowe”} application/xml: example: | Michael Stowe 400: #... 401: #...

26 Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015

your API with schemas and parameters. It also allows for resource nesting (similar to how SaaS lets you nest CSS), ile includes (letting you pull in multiple iles to keep your spec easy to read and organized), and even the setting of variables or properties for consistent use throughout the spec. You could take advantage of these features in your spec like so: 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039

#%RAML 0.8 title: This is My API baseUri: http://api.domain.com version: 1 /resource1: get: responses: 200: body: application/json: schema: | { “type”: “object”, “$schema”: “http://json-schema.org/draft-03/schema”, “id”: “http://jsonschema.net”, “required”: true, “properties”: { “irstName”: { “type”: “string”, “required”: true }, “lastName”: { “type”: “string”, “required”: true, “minLength”: 3, “maxLength”: 36 } } } /sub-resource: get: queryParameters: irstName: description: “the user’s irst name” example: John required: true type: string

Developer friendly Beyond its simple format and available tooling, RAML lets developers incorporate coding best practices such as patterns and code reuse. This not only greatly reduces the work but also helps keep the API uniform across resources and methods. While you can always  extract schemas to keep your spec orga-

nized or pull in schemas, examples, and other RAML snippets using the “!include” command, RAML ofers two additional useful and unique features for templates:  traits  and  resourceTypes. Deining common attributes with traits RAML’s  traits  allow you to deine common attributes (or traits)

for methods (GET, PUT, POST, PATCH, DELETE, etc.) such as whether or not they can be iltered, searched, or paged. In creating a trait, you are actually creating a template that accepts parameters to provide the attributes to the method in a couple lines, while providing as much lexibility and customization to that trait as you need:

Description, Discovery, and Profiles: The Next Level in Web APIs // eMag Issue 30 - Aug 2015 27

001 traits: 002 -searchable: 003 queryParameters: 004 query: 005 description: | 006 JSON array [{“ield1”,”value1”,”operator1”},…]