Reasoning on REST versioning, with example code

Versioning and ease of use

The way to “properly” version REST services, according to most texts I’ve read and people I’ve talked to, is to use custom media types. I think this is a good solution implementation wise. Usability wise I think it sucks because it makes it very hard to test the service using your browser since you need to specify a custom accept header.

Example

Instead of rambling about pros and cons here, like many have already done. I made an example using Java (Jersey, Guice, Tomcat) and nginx to illustrate how to both implement a media type versioned service as well as publishing it in a way that also makes it easy to explore using a browser.

tl;dr

It’s all here https://github.com/chids/java-rest-versioning/.

Visualizing cost and improvement areas for software maintenance

In chapter 4 “Motivation” of his book Implementation Patterns (Addison-Wesley Professional, 2008) Kent Beck cites Structured Design: Fundamentals of a Discipline of Computer Program and Systems Design (Prentice Hall, 1979) with the formula for calculating the cost of software as the sum of the cost of development plus the cost of maintaining it:

COST_total = COST_develop + COST_maintain

The cost of maintenance is then broken down to:

COST_maintain = COST_understand + COST_change + COST_test + COST_deploy

I like this formula and when I recently revisited Implementation Patterns I started thinking about the cost components of software maintenance and how we work with them.

I’ve put together a short illustrated text on my reasoning and published it here.

Confluence: Render XML response from HTTP service

tl;dr;

Here’s the macro, example of it’s usage and the XSLT used to pretty print.

Background

We document our HTTP based services in Confluence and I wanted to…

• embed service invocations in wiki pages to show the result of calling a certain URL
• pretty print the output
• not have to write a Java plugin in Confluence

Steps

Making a HTTP request from a macro

I started of with the HTML macro and verified that it worked.

Formatting the response

Our services produces as compact XML as possible to save space, they don’t pretty print. Hence I needed to format the service response and after finding these XSLT’s I saved one of them as an attachment to a page within Confluence and paired it with the XSLT macro.

Alternating the URL displayed (and/or used to make the request)

As I’ve written about earlier we document all of our services in one space and selectively “publish” a subset of those to another space from which we allow export of PDF’s as the deliverable documentation. In order for this to work we need the service examples to show, and use, different domain names for accessing services based on wether the documentation is viewed in our internal space or in the space used to export PDF documentation to our clients. The logic for this is the first block in the macro in which it looks at which space the macro is rendered and selects the base URL based on that.

Caveats

Query parameters

Due to what appears to be a limitation or bug in the handling of user macros Confluence won’t properly render the macro if the URL to the service contains an equal (=) sign. So in order to support query parameters the macro replaces any occurrences of two colons (::) with an equal sign. It’s all illustrated in the Gist showing the macro.

What about JSON?

We use this solution for JSON as well but without the XSLT formatting stage which renders our JSON responses butt ugly, if you have a solution on how to format JSON as well – feel free to post a comment.

Nginx: set the accept header based on file extension

tl;dr

https://gist.github.com/874201

Background

Most of our services responds to HTTP GET requests and produce an XML or JSON response. The format of the response is determined by the HTTP Accept header. This is fine when our users program their clients but since our APIs are exposed over HTTP most of the initial sampling and testing can be done simply by pointing a browser to the endpoint and experimenting with query params etc. So how to do play with your browser and still be able to choose wether to get the result as JSON or XML? Well set the accept header by using Poster for Firefox or Simple REST client, REST Console, or Advanced REST Client for Chrome. There’s certainly more alternatives available for any and all browser that you may use. Or you can fire up a console and run cURL.

All clients and command lines are fine, and usually invaluable to have when your getting deep with the API. But for those initial play around tests it’d be nice just to be able to surf to the resource and say: I want this resource to respond with XML or JSON.

Keep it simple…

…therefore we support the use of file extensions .xml and .json respectively. So a service that we expose as:
http://some.api.domain/some-service/v1/fetch?foo=bar
might also be invoked as:
http://some.api.domain/some-service/v1/fetch.json?foo=bar
or
http://some.api.domain/some-service/v1/fetch.xml?foo=bar

Accept header manipulation in Nginx

Now, where not really interested in introducing this handling in all our services. They all have automagic response format serialization through the Accept header. Hence we’d simply like to alter the accept header for requests that provide a file extension and we already use Nginx as a reverse proxy in front of our application servers – see this gist for an example nginx configuration.

Atlassian Confluence: Context aware includes

tl;dr

https://gist.github.com/870528

Background

I’ve written earlier about how we utilize Atlassian Confluence to document our APIs in general and my user macro hack to actually perform service calls and display results on page rendering (read the section “User macro to render service response” for more details).

Our space structure

We mainly classify services into three groups; public, partner and internal. Public being services where we don’t really care about their usage – we might not encourage broad public use but we’re not trying to limit it. Partner services are offered within some form of commercial agreement and requires a API key. Internal services are not accessible from outside our internal networks.

This has led me to do the following space structure in our Confluence wiki:

• API, this is the space that contains all of our services
• PARTNER, this space contains the subset of the API contents that are available to partners

The partner space uses the include macro to include those pages from the API space that are available to partners. The bread and butter of all our service docs are the various macros that call our services and display the result to provide concrete examples. These will always use the internal address of a service to circumvent the need for any API key handling. However the internal domain should not be used in the examples displayed to our partners.

Context aware includes

The partner space uses includes from the api space and the page in the api space is the one using the macro to perform and display the service call. Hence I needed to have the macro display a different base URL depending on the space key. This wasn’t bloody obvious but proved very simple once I grasped the Veolcity context and the objects it provide. The magic happens with:
$renderContext.getOriginalContext().getSpaceKey()

See the gist for a complete example.

Giving tech talks in upper secondary school

I’ve been a member of Transfer for some time now and thought it might be worth a mention. Specifically I’d like to hear about similar initiatives elsewhere, if you know any – please drop me a line. Transfer is a non-profit knowledge transfer network whose aim is to bring professionals in all sorts of diverse areas to Sweden’s upper secondary schools. Teachers can use the Transfer website to request a lecture in an area and Transfer then matches those requests with the profiles of the affiliated professionals, which can then either accept or decline the request. Sort of like a dating service.

To be fair, I’ve only done two lectures and the amount of requests within my area of expertise are somewhat sparse. Nonetheless, giving a lecture to a young audience that’s still in school and usually very curious is challenging and thus very rewarding. I sincerely recommend participating, regardless of what area you do work in.

The slides from my latest lecture (given in November 2010) on databases in general are now up on Slideshare.

Stockholm #killdashnine 1

Bckgrnd

Basho posted “Data Durability Is Not An After-market Add-on; Announcing KillDashNine” and since I’m a fan of durable data in general, Riak and Dry Martini in particular I decided to do the Stockholm version of #killdashnine. The venue was Little Quarter (the inner bar at restaurant Marie Laveau), home to my favorite bar staff.

Killing-dash-nine-shit

Me, Jesper, Sven and Fredrik made only four people in attendance so we certainly have some room for improvement. Also we didn’t actually kill any databases on location. Although me and Jesper did spend some time last week beating the crap out of ActiveMQ using kill -9 on the broker while producing and consuming persistent messages. Using our very non-scientific, nor exhaustive, approach it all looked well although some people experience problems with KahaDB.

I’m aiming to keep this going on the 9th of the month and hopefully I’ll muster the energy to prep some proper kill-demos for the next gathering.

Goals for 2011

In no particular order:

• Travel – I love the new and unknown, therefore: take a solid vacation. Solid meaning minimum six weeks spent in another country. Hot candidates are India, South America, Africa, and Indonesia.
• Work – I see myself as very diverse individual with a multidisciplinary palette which I often find hard to convey, therefore: Revamp my resumé/CV by taking the time to explore the various ideas I’ve had over the years on how to break free from the traditional CV/resumé format in order to communicate my diverse skill set, experience and strengths.
• Work – I have had very positive experiences of mentor/mentee relations in the past but I haven’t had a mentor in almost 10 years, therefore: Try to find a mentor which is in someway senior to me but not within IT but rather in finance, human resources or type of management.
• Travel – I’ve enjoyed all my tips to the US (New York, North Carolina & Las Vegas) and therefore: visit the US west coast and see at least three of: San Francisco, Los Angeles, Seattle, Portland, Sacramento, Fresno, Santa Cruz, Tijuana, Oakland. Also, this is separate from the solid vacation goal.
• Work – I originate from programming and after a few years in exile I’m back into doing it almost full time, therefore: become self-sufficient in one new programming language. Obvious candidates are Scala, Erlang, and Clojure.
• Home – I love watching movies, therefore: realize a slim, no fuzz, home media setup: A thin display (TFT/LED) that supports as much DLNA/UPnP stuff as possible in combination with either a media player such as Sony PS3, Mac Mini or, preferably, just a capable enough NAS.

Two years late: Comments on old WMB best practices

In September 2008 Shravan K Kudikala posted an article on IBM DeveloperWorks entitled “WebSphere Message Broker development and deployment best practices” consisted of various best practice bullet points. This product being my key competence at the time I had a few comments on his article but for some reason I never got around to finish and publish it. While cleaning out various drafted blog posts I found this and even though this is somewhat embarrassingly old I thought I might as well post since I obviously took the time to write it.

DISCLAIMER: Read the above – this is old material. Always refer to and double check agains the latest product documentation.

• Often, to separate configuration information from business logic, customers externalize configuration information to a file or database. This technique can reduce performance, because reading a configuration or parameters file is a one-time activity at the time of the first instance of a node is created or at the time the first message is processed, instead of a loop checkup for each message. Since Message Broker is more CPU-oriented than I/O-oriented, it is usually best to avoiding I/O operations involving files or databases when possible.
  • While far from a simple decision or guideline to provide I’d opt for building flows highly configurable at first. Should stress testing reveal that the externalization of configuration data poses too much of a performance increase, inline it. But generally I’d say that the flexibility that an external configuration solution provides (be it database, file, or a registry) enables rapid change and drives a more module solution.
• Do not use trace nodes on production environments. Using ${Root} expression is expensive operation as this causes the complete message tree parsing. This happens even if the destination is not an active one.
  • Clarification; do use trace nodes at key points in your flows but be sure that your production environment is configured with trace nodes disabled. As of version 6.1.0.3 of WebSphere Message Broker; consider using the new flow monitoring solution instead of trace nodes for a more flexible monitoring solution.
• Wherever possible use user-exits and redirect the audit / logging information appropriately. User exit feature gives the flexibility to activate & deactivate them dynamically during message processing.
  • This also holds true for trace nodes and the event monitoring introduced in version 6.1.0.3.
• Save intermediate results in the message tree in order to avoid recalculating them in subsequent nodes. If the message contains user maintainable data folders in the headers like MQRFH2 usr folder then store the intermediate results in it for subsequent nodes.
  • For temporary results that are only needed during the execution of a flow, consider using the global or local environment.
• Using destination list is more recommended rather than using more nodes when message has to be written to multiple destinations.
  • Destinaion lists are a good way to provide dynamic destinations. Multiple nodes are more explicit and easily understood when the destinations are fixed. Route nodes in combination with MQOutput-nodes provide a way to specify conditions to determine which destinations should receive a message. Be sure to weigh ease-of-understanding versus dynamicity (and whether that dynamicity is really needed!).
• Always have exceptional handling mechanism for the message flows rather than relying on the default broker exception handler. The default exceptional handler can block the message consumption when a single poisoned message processing is failed.
  • For MQ-driven flows the default exception handling should be fine as long as you’ve defined a backout queue on the flow input queue.
• Use subflows for code reuse across multiple message flows, especially if you are using or considering multiple protocols.
  • Do note that a subflow is the equivalent of an include statement. A subflow is merely a IDE construct and what the runtime receives is a message flow that is the result of an “merge” done by the tooling in which it simply inlines all subflows in the main flow. Therefore changing a subflow will never change any deployed flow using that subflow. Nor is it possible to deploy a subflow on it’s own. Sometimes this is a good thing, however for a “pure” service re-use it is not, for those scenarios; consider using a separate flow.
• Revisit all the java nodes and ensure that there is a clearMessage() called on every MbMessage object especially in the finally block. MbMessage object is used to create output message tree, environment tree, local environment tree, exception list tree, etc. So where ever the message trees are created, clear them out in the try – finally block.
  • The clearMessage() call is not necessary. You can safely exclude try/finally blocks that only exist to call clearMessage().
• It is strongly recommended to have fewer, more complex nodes rather than having large number of nodes with the processing logic spread across.
  • Again, performance versus maintainability. I approach this as any kind of software optimization and prefer to build flows that are easily understood – this often means a larger number of nodes that only have a specific responsibility, and are named to explain that responsibility. In my view optimization should be done when actual bottle necks are identified.

Adventures in GIS

I spent a fair amount of time during September and October 2010 with cleaning up and refactoring our generic coordinate library. GIS not being a domain I’ve had much, if any, to do with since before joining Hitta.se it’s been a deep dive into this pretty tricky domain. Some of the key insights gained during the work was that there are different types of coordinate systems. WGS84 not being a projected coordinate system but rather a geodetic while Swedish systems RT90 and SWEREF99 are projected. Understanding transformations becomes much easier when you know that ;-)

The actual transformation math is not something I’ll even try to understand, nor actually care for, but I now understand how transformations are done and why. We’re lucky that we have Tom to help us out in GIS-crunch times.

In the process of structuring and simplifying the code base I’ve played around with some of the less and more known GIS-libraries written in Java. Most notably there was a short period of time when it looked like GeoTools would simply plug in and free us from all transformation headaches. It was a nice illusion to live in for a week or so. Turned out it was, at the time, more or less impossible to get a clear idea of which libraries (out of way too many) that we actually needed. I ended up simply removing them one at the time from the classpath and running tests. When I’d slimmed down the amount of required libraries to a bare minimum (still too many) it turned out that the code was leaking memory all over. It soon become to cumbersome to justify. Hence we’re now rolling our own bare minimum transformation code which is slim, fast and does not leak.