Documenting RESTful APIs with RESTdoc

[danjemiolo]

When we were designing Zile and its many components, we created wiki pages documenting their design and use cases; included in those wiki pages were REST APIs (tables) that described the public interface clients and other components would use to interact with our implementation. Everyone here knows what these tables look like, so I'm not going to spend any time describing them. This post is about a tool I created to automate the REST documentation process and eliminate redundancy between HTML/wiki docs and code comments. The tool is called RESTdoc, and here's the summary from RESTdoc's own docs:

RestDoc is a utility for generating REST API documentation from RESTful resource scripts in Zero applications. Given a set of Zero-style scripts, it will generate an HTML page with a REST table ("Gregorio table") for each resource type; the tables include such information as the HTTP methods, URIs, data formats, applicable status codes, and descriptions of each type of operation.

To use RESTdoc, you must first include RESTdoc-style comments in your Zero scripts. RESTdoc comments are just like JavaDoc comments, with the following new tags: @success, @error, and @format. The new tags are used to list the HTTP status codes for success and errors, as well as the expected data format in the request or response body for each method. RESTdoc will ignore any method that is not a Zero REST method (onList(), etc.) or that does not have RESTdoc-style documentation preceding its definition. Like Zero itself, RESTdoc only supports Groovy and PHP scripts, but it is possible to add other languages in the future.

To run RESTdoc from your Zero application's directory, just type:

Code: Select all

$ restdoc


By default, RESTdoc will look in ./app/resources for Zero REST scripts and it will save the generated HTML page in ./docs/rest. If you want to run RESTdoc from a script that is not in your application's directory, you can use the input and output flags to override the defaults:

Code: Select all

$ restdoc -input /my-app/app/resources -output /my-docs


Finally, RESTdoc does not produce any output if all goes well. You can turn on console logging using the verbose flag if you're having trouble debugging a problem:

Code: Select all

$ restdoc -verbose


As an example, comments like this:

Code: Select all

/**
*
* Returns all contact lists from a user's profile. The response content is a
* JSON object with one field for each list; the keys are the list names, the
* values are lists of user IDs.
*
* @format JSON
* @success 200
* @error 403, 404
*
*/


create tables like the one in the attached screenshot.

The code currently works with Groovy or PHP resources and produces HTML that can standalone or be copied into a wiki page. Any feedback would be appreciated - the code and sample applications can be found in SVN under /branches/p_dj_restdoc. The tool can be run by executing RestDoc's main() or by using the RESTdoc Ant task; the latter is demonstrated in the build.xml files of each sample application - just run 'ant docs'. I will try to incorporate any feedback before submitting an enhancement request and patch for zero.tools.

TODOs: incorporate .bnd files when generating a resource's URI patterns.

[danjemiolo]

Update: I've added the support for .bnd files, so now the URI patterns shown will include the proper prefixes (e.g., /profiles/{profilesId}/contacts).

[pmuellr]

Have you considered trying to use language-specific annotations, so that you can more closely tie this meta-information into the implementation? I believe Groovy supports Java annotations directly, and PHP can do annotation-like stuff (if you write some code), by reflecting over the docComments available in the code. (our PHP interpreter may not support this yet)

The advantage to having it 'closer to the code', or more precisely, available to the runtime code, is that you can actually validate yourself against it. If you ever try to return a return code that you claim you don't return, RuntimeException. etc.

The disadvantage to having it available (perhaps only) to the runtime code is that you need to have your server running to get the information. I don't see this as a big deal. In fact, it's an advantage to some extent, in that you can imagine a client ASKING for this information so that they can 'auto-discover' stuff about your services. Remember back in the WSDL days, AXIS allowed you to append "?help" or something to the endpoint url of a service, and it gave you back some informatino about your service? Man, that was sweet. Except that I think it gave you back WSDL. :-)

[brandon]

Groovy 1.0 doesn't support Java annotations. 1.1 does (will, it's in beta).

Am wondering if you made use of Groovy's GRAM project?

[danjemiolo]

I didn't use GRAM - one of my main goals was to avoid adding dependencies to zero.tools for each language supported[1]. I might need to reverse that decision if people ask for more interesting features, but at present it's not really rocket science (or compiler science).

As for annotations, I can see where they'd be beneficial, but I'd like to get the zero.core team's input on whether they would actually take advantage of it first (since that's where startup and per-request validation would occur). At the moment, Groovy and PHP support similar JavaDoc-style docs, and I wouldn't want to have them split (and start the legal approval process for various annotation processing libraries) and then find out that annotations are undesirable to zero.core. Steve? Michael? Thoughts?


[1] http://codeunleashed.com/2007/07/restdoc.htm

[brettk]

Nice work, Dan! This kind of tooling has actually been on the to-do list for awhile. It was even called RestDOC at one point, but I had been using the term ZeroDoc more recently. Since you're running with it, let me share some of the directions we were looking at for it:

- Publishing the RestDoc at a well-known URI when the service was deployed (similar to what Pat described)
- Adding microformats to the HTML so that an application (like a mashup maker) could consume the service
- Another alternative - publishing the REST interface using an ATOM feed
- Put the RestDoc in a well-known location in the package so that the Catalog can extract it and present the REST API in the Catalog entry
- Include a generated form to allow actual interaction with a deployed version of the service
- We also need to think about how this fits or could fit with webtools.

Anyway, I'll give you some more feedback once I play with what you've done so far.

[danjemiolo]

Great ideas. I can handle #'s 1 and 4 easily (meaning, by the end of the day). Generating clients and UIs is also possible, but it won't be done by the end of the day. :) I think I will complete 1 & 4 and then run it by Aaron for inclusion in zero.tools. Once it's in place, I'll start on the client stuff. I will need more input from you and Pat (and whoever else is interested) in determining the best way to record the metadata needed for such code/markup generation.

From a webtools perspective: I think we'll want to provide the doc and form-based UI in a production setting - these artifacts benefit the client programmer, not the developer who wrote them. In other words, I don't want them to go away when the webtools dependency is removed. We can always add a flag to the package task to ignore doc-related files if it becomes necessary.

[ajtarter]

I think this tool may be better served in a runtime library (not zero.tools) that can execute on the request of a well-known uri, e.g. /restdoc. This way when the client requested it, they could be sure that they were getting documentation that reflects the current state. I agree that webtools is not a good location, since this functionality should stick with the application into production. So, I would suggest putting it in a new library of its own or adding it to zero.core.

The current zero.tools commands cannot be run through a web interface, and I don't think we ever want them to be. Furthermore, as a zero.tools inclusion, the command would need to be re-run each time there was a change to the APIs. It could be an implied step before each run, but it still would require a restart of the app to regen the doc. If you are using a scripting language you may not want to restart.

[danjemiolo]

That works for me - I think I'd like to keep a command line option so that people can create HTML pages without running Zero, but that's not a problem. The RestDoc class already has the interface needed to be invoked programmatically (and with non-FileWriter Writers), so I will tweak things a bit and write a handler that will generate docs at runtime. This will definitely make client/UI generation more seamless.

I'll keep it a separate library for now, but it can always be added to zero.core if desired.

[ajtarter]

I think the mainstream usage should be dynamic generation at runtime, but I agree that there may be some cases where you just want to generate as part of building your app without running it. So, I agree that it would also be useful as a zero.tools command.... let me know if you need help adding it.