A personal django based information manager

For quite some time I was looking for the “optimal” software to manage all kinds of information:

• To Do Lists
•  Reminders
• Writetups of ideas or concepts for some new application or article
• Code snippets worth remembering

Ideally, there is one central place for all this stuff - not one for to do lists, another for article drafts, and a third for storing code snippets. The “place” needs to be somewhere on the web, of course, to make it accessible from everywhere.

I tried many of the applications to be found on the web. There’s backpack, the excellent minimal information management app from 37signals; there’s google notebook with its seamless integration in firefox; there are countless more examples. Needless to say, I didn’t evaluate them all. There might well be the perfect application for me somewhere out there. But none of the solutions I looked at really was the kind of structured information dump supporting all types of content I care about, and none really met all of my requirements:

• editing of content in ASCII, to allow easy exchange from and to other systems
• support for some simplified markup language to structure content, preferably markdown
• code highlighting
• simple priorization of items
• tagging and tag-based retrieval of items
• free of charge (I already pay for web hosting)

This list of requirements had been growing for some time. When it reached the state listed above I realized two things: first, the time had come to start coding the application myself. Second, developing this would enable me to add another feature that I had been thinking about for some time: automatic documentation of potentially reusable code snippets.

The need for code snippet documentation

In my day job I do a lot of user interface prototyping for a large web application. For any single feature I create code in ASPX/HTML, CSS, JavasScript and C#. If I identify a component that I think or know will be reused, I try to implement it as a reusable component - an ASP.NET user or custom control, a method in a JavaScript library, and so on.

But quite often I do not want to reuse a specific control or method, but rather some technique or bit of knowledge I know I appied some time ago an a similar but somewhat different problem. What I would like to retrieve in this situation is some code or other documentation that shows how the technique is applied.

But there’s a problem: in order to find the code, I need to remember the context in which I used it. And that’s something I easily forget, especially if half a year has passed since I wrote the code I’m looking for now. What’s even worse: sometimes I remember that a used the sought-after technique on a certain page, only to realize way later that it’s an outdated version.

Now if I had the personal information manage outlined above, I could easily create an entry describing the technique I want to remember. But then the technique would be documented in two different places, which is bad. A good solution would propagate any changes in the code directly to the documentation.

So a new feature for my personal information manager was born: it should also contain a collection of documentation entries taken directly from the source code. An inspiration for this is of course JavaDoc (link) or the builtin documentation features from Visual Studio, but I have my own list of requirements for this feature:

• The documentation system must work with all kinds of source code: ASPX/HTML, CSS, JavaScript, C#, Python are currently on my list.
• The markup for entering the documentation info should be easy to parse (I’m not interested in writing parsers. I’m interested in up-to-date documentation!)
• The resulting documentation entries should contain the following information:
  • a title for the snippet or technique used
  • an optional description
  • a list of arbitrary tags to be used for retrieval
  • the source code itself
• Syntax highlighting

The solution

The obvious solution for marking up documentation entries in any kind of code is to embed them in comments. I came up with a very simple scheme that looks like this (if applied to HTML):

<!-- @doc.title { Title of this documentation entry }
      @doc.desc { Longer description of the code that follows }
      @doc.tags { documentation tags retrieval memorize find }
      @doc.code  -->
<div class="ingenious_layout_trick">
    <span>More HTML...</span>
</div>
<!-- @doc.end -->

Obviously this type of markup can be applied to any kind of code. With this idea in place I wrote a simple, regex-based parser that walks through a configurable list of directory trees, scans all the files for embedded documentation and saves the resulting entries via Django’s ORM to the database. For the sake of a simple implementation, I decided to ensure the up-to-dateness of the documentation by deleting all existing documentation entries prior to the parsing process. This has the obvious drawback that identical entries get a different URL in the application after each parsing pass, but this didn’t bother me so far.

The final product

I actual implemented a django application with the above features and call it my “dev log”. I am constantly tweaking and improving it, but I’ve been using it for some time now and I’m very pleased with the result. Here are some screen shots.

Overview with two lists: log entries and documentation entries

Tag search is available through the search box on top of the page and through the tag cloud in the sidebar

Editing an entry

• Markdown enabled
• clicking on tag in tag cloud adds to tag field
• pressing TAB adds four spaces
• ‘Save’ button posts the content via ajax, so no disruption by a postback while editing

Viewing a log entry

A documentation entry - contents retrieved from code

Concluding remarks

This application is only as good as the information contained in it - and it certainly calls for quite some discipline to really enter all the information one may want to retrieve at a later date. When the documentation repository gets bigger and bigger, the application will probably also need full text search in order to remain usable. But that shouldn’t be a problem with Django, I think. And until then, I enjoy my personal information and knowledge management tool.