Archive for April 2014
One of my principal goals with mountebank is to make it cross-platform and as easy to install as possible. Nikos Baxevanis showed the cross-platform nature of mountebank by using F# to verify SMTP calls from the system under test. I like this a lot; I think the .NET community has been under-represented by such tools in the past, and mountebank definitely aims to correct that.
I’ve started the process of creating self-contained packages for mountebank to make the installation as transparent as possible. Due to the nature of being a traveling consultant, I often bounce between technology stacks. Nonetheless, I find the process of having to install a JDK to use a testing tool when the rest of my stack is Microsoft-based to be a hassle. I love CSS transpilers, but I also find it taxing to set up ruby just to manage CSS when I’m developing Java. I don’t want mountebank to be a similar hassle, requiring node.js just to use. My goal is to make it as usable out of the box as possible.
The latest version continues to ship as an npm package. However, it also ships with several other options that allow you to simply download and run from a local directory, even without node.js installed, or to use a Linux package manager to install (again without node.js installed). There’s still quite a bit of work yet to do – getting the packages in the primary package repositories, and adding additional packages like homebrew.
What about Windows?
mountebank runs on Windows, but requires
npm install -g --production. Without the
--production flag, it tries to install some testing dependencies that don’t run on Windows. I definitely aim to follow suit with Windows (or get some help from friendly developers) to get an msi and self-contained zip file for Windows. Please do reach out if you’re interested in helping.
API Docs: The Traditionalist View
I’ve written a lot of APIs in my time (think RESTful-type APIs, not library APIs). Often times, clients don’t have access to the code or don’t care to pore through it, so documentation is important. I’ve generally held the following to be somewhat of a maturity model for API documentation:
- No docs – call me and I’ll show you how to use it!
- Go to the wiki
- The docs are on a special page of the web application hosting the service. This is nice because the docs
are versioned with the app; in fact, they are part of the app.
- The docs are auto-generated from the comments, annotations, attributes, or source code
Recent experiences have made me question this maturity model. Indeed, I’m beginning to wonder if deriving the documentation from the code is exactly backwards from the natural order of things. In short, I’ve begun to wonder: Is it better to generate the docs from the code, or test the code through the docs? I’m starting to think that the way we write API documentation based on the code may be a bit of the tail wagging the dog.
Experience 1 – Cracks in the Traditionalist View
On my current project, we’re auto-generating docs using Swagger plugins for Java and using Swagger UI to present them. Swagger – a wire format for exposing docs – is nice. The Java plugins are nice, reading many built-in annotations of the framework we’re using for exposing services. The UI is nice. None of it fits our needs perfectly, but that’s the nature of borrowing code, and, at first, the win of having documentation generated from our code seemed completely worth it.
The belated observation is that, in our noble effort to keep the documentation complete, we’ve made some subtle and insidious design compromises to satisfy the metadata Swagger requires. This isn’t a knock on the Swagger community – the plugins really are very nice. Our Java Swagger plugins use reflection to, for example, describe the structure of the return type of the resource (controller) method representing the response body. Any such technology would have to do the same.
There are a few examples of the design smells we’re noticing, but the big one is that we have an anemic domain model since our service is essentially just a pass-through to a downstream system that doesn’t have the common courtesy to expose itself over HTTP. Using lists and maps would be actually quite a bit simpler for our scenario. All we originally created the domain objects for was for serialization and deserialization, but that’s a relatively simple problem to solve with lists and maps. However, we piggy-backed Swagger on top of those domain objects, and now we’re stuck with them.
If we suddenly switched to lists and maps, any use of reflection would fail to adequately describe the shape of the response object, since simple reflection depends on data known at compile time. To dynamically generate documentation through a tool, we would actually have to call our service at runtime and inspect the lists and maps being returned for the keys and values. This isn’t feasible, so we’re stuck (for now) with an anemic domain model.
Experience 2 – An Alternative Presents Itself
When I was writing mountebank (the tool we’re using for stubbing TCP and HTTP dependencies), I did what I imagine most open source developers do when they think they’re close to the first release. The sequence goes like this:
- Write a bunch of crappy documentation by hand (following the third option on the maturity model – inline with the app)
- Take one last look at the code to “polish” it
- In a fit of passion, write a bit more code, including complete rewrites of core functionality and substantial enhancements
- (Roughly one month later) notice how out-of-date the docs are. Write more code to ease the pain.
Eventually I realized I could ignore the docs no more, but in a desperate effort to continue writing code, I rationalized that I should test the docs to prevent them going stale again. So I added markup in the HTML to give me hooks to test, wrote the necessary code to interpret that markup and assert on the entire JSON, post-processed out the ephemeral bits like timestamps, and verified my docs. Simple enough. So then I wrote the next bit of documentation and re-ran the test. That was when something magical happened.
Turns out, I had had this bug in the code for quite some time, and never noticed it. That was the case despite a fair bit of automated functional testing which examined substantial bits of the (parsed) JSON structure. I fixed it and carried on writing the docs. In the process, I discovered several more bugs, as well as a few “UX” fixes (e.g. the ordering of fields to make developer comprehension easier when testing over curl). I had not up to this point faithfully tested against the full JSON structure, nor had I previously tested the JSON structure as text instead of some internal representation. I caught more bugs in the app writing documentation than in all previous tests I had written added together.
I realize some would call this BDD, but it’s a constrained version of BDD, where the docs are really my production API docs, and my experience makes me wonder if this isn’t a better way of writing docs. The docs were, without question, much harder to write, as I painstakingly wrote out example narratives by hand. However, they now serve as the bests tests I have in my entire codebase.
I’ve been coding for a long time, and I can’t remember the last time I experienced such a welcome surprise. What I intended to be a routine regression test turned out to completely reshape the way I think about the documentation, at least in mountebank. Based on just one experience, I’m not confident enough to suggest that the traditionalist view of API documentation is wrong. I am, however, hedging my bets.
How to test your docs, or how your docs test your API
Here’s an example request; the hooks are bolded:
<code data-test-id="predicate injection example" data-test-step="3" data-test-type="http"> POST /test HTTP/1.1 Host: localhost:4545 Content-Type: application/json <?xml> <customer> <name>Test</name> </customer> </code>
and an example response
<code data-test-id="predicate injection example" data-test-verify-step="3" data-test-ignore-lines="["^Date"]"> HTTP/1.1 400 Bad Request Connection: close Date: Thu, 09 Jan 2014 02:30:31 GMT Transfer-Encoding: chunked </code>
I won't say that writing code to parse these
data- attributes and execute them correctly was easy, but you can see how it works in the code. I will say that it was definitely worth it.