A Day In The Lyf

There's a discussion on the altdotnet Yahoo group about managing configuration files. How do you manage updating multiple configuration files to change the appropriate values when deploying to a different environment?

The solution I hit on was to create a custom MSBuild task (yeah, I screwed up and used MSBuild instead of NAnt). When called from our build script, it looks something like this:

Notice that each call to UpdateConfig takes the list of config files that will be changed and a config mapping file. That mapping file is what is read to update the config files given the environment. Here's an example of what the mapping file looks like:

Each config file is scanned looking for each XPath expression in the mapping file. On each match, the entire node (and all its child nodes) of the original config file are replaced with the node under the appropriate environment tag in the mapping file. It's a bit verbose, but simple enough, and it supports as many environments as you want to have.

The MSBuild task itself is fairly simple, delegating most of its work to a separate object called XmlMerger:

XmlMerger just finds the nodes that need updating and replaces them from the mapping file. Notice that it also accepts namespace information (see the NHibernate example in the build script snippet above), which is occasionally needed:

That's it. Now the whole process is hands-free, so long as you remember to update the mapping file when needed. The config files we put into subversion are set to work in the development environment (everything is localhost), so anybody can checkout our code and start working without having to tweak a bunch of settings first. The deployment process calls our build script, which ensures that the appropriate config values get changed.

While it seems to be fairly common knowledge in the functional programming world, I don’t think most object-oriented developers realize that closures and objects can be used to implement each other. Ken Dickey showed how it can be done rather easily in Scheme, complete with multiple inheritance and dynamic dispatch.

That’s not to say, of course, that all OO programmers should drop their object hats and run over to the world of functional programming. There is room for multiple paradigms.

Take the well-known Command pattern, often advertised as having two advantages over a more traditional API:

1. Commands can be easily decorated, giving you some measure of aspect-oriented programming. CruiseControl.NET uses a Command-pattern dispatch for the web interface, and decorates each command with error-handling, etc, providing a nice separation of concerns.
2. Commands can give you easy undo functionality. Rails migrations are a good example.

Recently, I had to retrofit Undo onto an existing legacy (and ugly) codebase, and I was able to do it quite elegantly with closures instead of commands.

What are closures?

Briefly (since better descriptions lie elsewhere), a closure is a procedure that “remembers” its bindings to free variables, where free variables are those variables that lie outside the procedure itself. The name come from LISP, where the procedure (or “lambda”, as LISPers call them) was said to “close over” its lexical environment. In C# terms, a closure is simply an anonymous delegate with a reference to a free variable, as in:

Notice that the anonymous delegate references the variable mark. When the delegate is actually called, it will be within a lexical scope that does not include mark. To make that work, the compiler wraps the closure in a class that remembers both the code to execute and any variable bindings (remember – objects and closures can be interchanged).

As always, Wikipedia has a nice write-up. A C#-specific description can be found here.

What does a closure-based Undo look like?

The legacy code I needed to update maintained the entire object state serialized in XML. This was terrible for a number of reasons, but it did have the advantage of making undo easy in principle; just swap out the new XML with the XML before making the previous API call. I wanted something like this:

In actual practice, the undo part of that could be wrapped up in some boilerplate code:

Notice that the undo procedure is referencing originalXml. That variable will be saved with the closure, making for a rather lightweight syntax, even with the static typing.

Getting Started

Implementing a single undo is really quite easy. Here’s a simple test fixture for it:

...and the code to make it work:

I won’t go into how CircularBuffer works, but it’s such a simple data structure that you can figure it out.

Naturally, with undo, we’ll want redo:

Conceptually, this should be fairly easy:

However, we’re not actually adding anything to the redo buffer yet. What we need to do is rather interesting—we don’t want to add to the redo buffer until Undo is called. Closures to the rescue:

But let’s say I undo, redo, and then want to undo and redo again. That won’t work as written, and making it work is starting to get pretty ugly:

It’s becoming apparent that what we really want is infinite recursion, lazily-evaluated. How ‘bout a closure?

Now we see how easy it is to decorate closures—remember that the ability to decorate commands is an oft-quoted advantage of them. However, closures provide a more lightweight approach to programming than commands.

The elegance of this approach is hard to deny. All it takes is getting over the conceptual hump that functions are just data. Think about it—we just added a function that took two functions as arguments and returned another function.

What also was apparent to me is how much TDD helped me get to this point. It may not be obvious from the few snippets I’ve shown here, but building up to the DecoratedAction abstraction was a very satisfying experience.

For reference, here’s the full CommandProcessor class. The bit I haven’t shown, CanUndo and CanRedo, along with an event that fires when either one change, is there so that we know when to enable or disable a menu option in a UI.

Ayende recently asked on the ALT.NET mailing list about the various methods developers use to provide lookup values, with the question framed as one between lookup tables and enums. My own preference is to use both, but keep it DRY with code generation.

To demonstrate the idea, I wrote a Ruby script that generates a C# enum file from some metadata. I much prefer Ruby to pure .NET solutions like CodeSmith—I find it easier and more powerful (I do think CodeSmith is excellent if there is no Ruby expertise on the team, however). The full source for this example can be grabbed here.

The idea is simple. I want a straightforward and extensible way to provide metadata for lookup values, following the Ruby Way of convention over configuration. XML is very popular in the .NET world, but the Ruby world views it as overly verbose, and prefers lighter markup languages like YAML. For my purposes, I decided not to mess with markup at all (although I’m still considering switching to YAML—the hash of hashes approach describes what I want well). Here’s some example metadata:

enums = {
  'OrderType' => {},
  'MethodOfPayment' => {:table => 'PaymentMethod',},
  'StateProvince' => {:table => 'StateProvinces',
                      :name_column => 'Abbreviation',
                      :id_column => 'StateProvinceId',
                      :transformer => lambda {|value| value.upcase},
                      :filter => lambda {|value| !value.empty?}}
}

That list, which is valid Ruby code, describes three enums, which will be named ‘OrderType’, ‘MethodOfPayment, and ‘StateProvince.’ The intention is that, where you followed your database standards, you should usually be able to get by without adding any extra metadata, as shown in the ‘OrderType’ example. The code generator will get the ids and enum names from the OrderType table (expecting the columns to be named OrderTypeId and Description) and create the enum from those values. As ‘StateProvince’ shows, the table name and two column names can be overridden.

More interestingly, you can both transform and filter the enum names by passing lambdas (which are like anonymous delegates in C#). The ‘StateProvince’ example above will filter out any states that, after cleaning up any illegal characters, equal an empty string, and then it will upper case the name.

We use a pre-build event in our project to build the enum file. However, if you simply overwrite the file every time you build, you may slow down the build process considerably. MSBuild (used by Visual Studio) evidently sees that the timestamp has been updated, so it rebuilds the project, forcing a rebuild of all downstream dependent projects. A better solution is to only overwrite the file if there are changes:

require File.dirname(__FILE__) + '/enum_generator'

gen = EnumGenerator.new('localhost', ‘database-name’)
source = gen.generate_all(‘Namespace', enums)

filename = File.join(File.dirname(__FILE__), 'Enums.cs')
if Dir[filename].empty? || source != IO.read(filename)
  File.open(filename, 'w') {|file| file << source}
end

I define the basic templates straight in the EnumGenerator class, but allow them to be swapped out. In theory, the default name column and the default lambda for generating the id column name given the table name (or enum name) could be handled the same way. Below is the EnumGenerator code:

class EnumGenerator
  FILE_TEMPLATE = <<EOT
//------------------------------------------------------------------------------
// <auto-generated>
//     This code was generated by a tool from <%= catalog %> on <%= server %>.
//
//     Changes to this file may cause incorrect behavior and will be lost if
//     the code is regenerated.
// </auto-generated>
//------------------------------------------------------------------------------

namespace <%= namespace %>
{
    <%= enums %>
}
EOT

  ENUM_TEMPLATE = <<EOT
public enum <%= enum_name %>
{
<% values.keys.sort.each_with_index do |id, i| -%>
    <%= values[id] %> = <%= id %><%= ',' unless i == values.length - 1 %>
<% end -%>
}

EOT

  # Change the templates by calling these setters
  attr_accessor :enum_template, :file_template

  attr_reader :server, :catalog

  def initialize(server, catalog)
    @server, @catalog = server, catalog
    @enum_template, @file_template = ENUM_TEMPLATE, FILE_TEMPLATE
  end
end

The code generation uses erb, the standard Ruby templating language:

  def transform(template, template_binding)
    erb = ERB.new(template, nil, '-')
    erb.result template_binding
  end

template_binding describes the variables available to use in the template in much the same way that Castle Monorail’s PropertyBag describes the variables available to the views. The difference is that, because Ruby is dynamic, you don’t have to explictly add values to the binding. The rest of the code is shown below:

  def generate(enum_name, attributes)
    table = attributes[:table] || enum_name
    filter = attributes[:filter] || lambda {|value| true}
    values = enum_values(table, attributes)
    values.delete_if {|key, value| !filter.call(value)}
    transform enum_template, binding
  end

  def generate_all(namespace, metadata)
    enums = ''
    metadata.keys.sort.each {|enum_name| enums << generate(enum_name, metadata[enum_name])}
    enums = enums.gsub(/\n/m, "\n\t").strip
    transform file_template, binding
  end

  private
    def enum_values(table, attributes)
      sql = get_sql table, attributes
      @dbh ||= DBI.connect("DBI:ADO:Provider=SQLNCLI;server=#{server};database=#{catalog};Integrated Security=SSPI")
      sth = @dbh.execute sql
      values = {}
      sth.each {|row| values[row['Id']] = clean(row['Name'], attributes[:transformer])}
      sth.finish

      values
    end

    def get_sql(table, attributes)
      id_column = attributes[:id_column] || "#{table}Id"
      name_column = attributes[:name_column] || "Description"
      "SELECT #{id_column} AS Id, #{name_column} AS Name FROM #{table} ORDER BY Id"
    end

    def clean(enum_value, transformer=nil)
      enum_value = '_' + enum_value if enum_value =~ /^\d/
      enum_value = enum_value.gsub /[^\w]/, ''
      transformer ||= lambda {|value| value}
      transformer.call enum_value
    end

Caveat Emptor: I wrote this code from scratch today; it is not the same code we currently use in production. I think it’s better, but if you find a problem with it please let me know.

We use log4net as our production logger, which has proven to be tremendously flexible. However, one problem I ran into was configuring the AdoNetAppender that logs to the database. It expects the connection string to be defined in the configuration file, which I didn’t want to do since it was already defined in our NHibernate config file.

This proved to be a relatively easy fix (found here):

However, the problem is that log4net whined to standard error about not having the connection string defined. The result was that any console application had its output garbled (including our tests, since some of them used the production logger).

The solution turned out to be going ahead and putting a connection string in the config file, but making it obviously invalid (e.g., “<ignore>”). Then, when the logger is configured, temporarily redirect standard error:

Voila.

My wife is in the middle of a research project comparing diet to the age of reproduction in African house snakes. She has to collect quite a bit of data, and when I finally looked at the spreadsheets she was maintaining, I was ashamed that I had not written something for her earlier.

This was really the first Windows Forms application that I’ve had the opportunity to do in years (my UI’s aren’t very inspiring). However, I have to maintain a couple at work that were primarily written by former colleagues, and I’ve always been a bit dismayed at the enormous amount of duplication that the standard event-driven application generates.

Despite the fact that the application I wrote for my wife was nothing more than a one-off application, one which you don’t expect to have to maintain, I focused on eliminating the duplication I see in the Windows applications at work. The result isn’t something that I would even begin to consider done for a corporate application, but I found the duplication removal techniques worth writing about. The code can be found here.

The biggest gains in removing duplication, and the ones most readers are likely to be least familiar with, are the use of higher order functions. My impression is that most C# developers aren’t very comfortable with higher order functions. Actually, I think that’s probably true for most developers working within mainstream commercially developed (Microsoft, Borland, Sun) languages. They’re simply not emphasized enough.

For example, all the forms had a ListView to display the data. All of them had to define the column header names and the data that goes in each row. It looked something like this:

Having the subclass define the column header names and the data that goes in each row didn’t bother me. What did bother me was having to specify the order that the headers and data needed to be shown in two different place. However, while the header names were static, the data would be different for each invocation. The result was to specify the order only once, in an associative array (I used .NET 2.0’s generic Dictionary, which seemed to maintain the order I entered the items). The key would be the column name, and the value would be a function to retrieve the data value.

The important things to note are that the subclass is passed, in a template method, a collecting parameter, associations, each entry of which represents a column name along with a way of retrieving the value for a row in that column. The delegate used to retrieve the value can be passed a single state parameter, which is needed by the report forms that need to pass in the source object for each row. Given that information, the superclass can manage most of the work. (AddListViewAssociations would have been abstract, except for the fact that Visual Studio’s designer doesn’t much care for abstract classes.)

For example, here is the information for the measurement form that was first given to show the problem:

One of the benefits of removing the ordering duplication is that the column names now sit beside the functions for retrieving the values, making it easier to understand. Notice that the GetterDelegate definition actually accepts an object parameter. C#’s anonymous delegate syntax lets you ignore unused parameters, making for a somewhat more readable line.

One of the forms shows the information about feedings per snake, and needed that parameter. Below is the entire implementation of the form (aside from the designer-generated code).

In case you’re wondering what this form does, it allows you to select a snake, or all snakes, and see the feeding information in the ListView. It also lets you export all the data to a CSV file. Not bad for 30 lines of code.

Another thing that bothered me about all the event handlers was how similar they looked. The workflow was abstracted in the superclass into a HandleEvent method:

HandleEvent takes a function that handles the meat of the event handler and wraps it within the code that’s common to all event handlers. Here’s a couple examples:

Managing the ListView proved to be fertile territory for removing duplication through higher order functions. For example, I used the first row’s data to set the column alignments automatically—if it looked like a number or date, right-align the data; otherwise left-align it.

Look how alike IsNumber and IsDate look. We can simplify:

I used a similar trick to auto-size the column widths in the ListView based on the width of the largest item. Here’s the refactored code:

If this were a more long-lived application, I really should have bit the bullet and created my own ListView subclass. The methods above reek of Feature Envy.

Being able to treat functions as first-class objects is extremely useful. For some reason, it doesn’t get the attention it deserves in most development books. And it’s often somewhat obscured by intimidating sounding names like “lambda expressions” thanks to its roots in lambda calculus. However, much of what I was able to do in this application was possible only because I was able treat functions as data and pass them as parameters. And it was helped by the fact that I didn’t have to explicitly define each function as a method; I could create them anonymously like any other data object (although C#’s anonymous delegate syntax is somewhat obscured by the static typing).

Kent Beck called one of the patterns in Smalltalk Best Practice Patterns “Execute Around Method.” It’s a useful pattern for removing duplication in code that requires boilerplate code to be run both before and after the code you really want to write. It’s a much lighter weight method than template methods (no subclassing), which can accomplish the same goal.

As an example, I’ve written the following boilerplate ADO.NET code countless times:

Notice that the connection and parameter management overwhelms the actual code that each method is trying to get to. And the duplication means I have multiple places to change when I decide to do something differently. However, since the using block encloses the relevant code, a simple Extract Method refactoring is not as easy to see.

Here’s the result of applying an Execute Around Method pattern to it.

Much nicer, no?

Pramod Sadalage and Scott Ambler have suggested using a series of numbered change scripts to version your database. Start with a base schema, and every subsequent change gets its own change script, grabbing the next number. That version number is stored in a table in the database, which makes it easy to update—you just run all change scripts, in order, greater than the version stored in your database.

The Ruby on Rails team implemented this technique in their migrations code. It’s quite elegant. This blog uses a Rails application called Typo; here’s one of its migrations:

  class AddArticleUserId < ActiveRecord::Migration
    def self.up
      add_column :articles, :user_id, :integer

      puts "Linking article authors to users"
      Article.find(:all).each do |a|
        u=User.find_by_name(a.author)
        if(u)
          a.user=u
          a.save
        end
      end
    end

    def self.down
      remove_column :articles, :user_id
    end
  end

That migration is called 3_add_article_user_id.rb, where 3 is the version number. Notice that it’s written in Ruby, not in SQL. It adds a column called user_id to the articles table and updates the data. The data update is particularly interesting—we get to use the ActiveRecord O/RM code instead of having to do it in SQL (although you can use SQL if you need to). The Rails migration code can also rollback changes; that’s what the down method is for.

The problem I’ve always had with this scheme is that we have many database objects that I’d like to version in their own files in our source control system. For example, here’s our directory structure:

We have several files in each directory, and it’s convenient to keep them that way so we can easily check a subversion log and see the history of changes for the database object. For us to use the migrations scheme above, we’d have to create a stored procedure in a migration, and later alter it in a separate migration. Since the two migrations will be in separate files, our source control wouldn’t give us a version history of that stored procedure.

We came up with a hybrid solution. Schema changes to the tables use a migration scheme like Rails. Database objects are versioned in separate files. Both the schema changes and the peripheral database object changes are updated when we update the database.

For this to work, we have to be a little careful with how we create the database objects. We want them to work regardless of whether we’re creating them for the first time or updating them, which means ALTER statements won’t work. The solution is simply to drop the object if it exists, and then create it. This is a fairly common pattern.

I wrote an NAnt and MSBuild task to do the dirty work. It runs both the schema migrations and the database object updates. Both are optional, so if migrations are all you want, that’s all you need to use. It expects all migrations to be in the same directory, and match the pattern 1.comment.sql, where 1 is the version number. It will be stored in a database table whose default name is SchemaVersion, with the following structure:

CREATE TABLE SchemaVersion (
  Version int, 
  MigrationDate datetime, 
  Comment varchar(255)
)

I’ve only tested it on SQL Server, but I think the task should work for other DBMS’s as well (it uses OLEDB). Migrations can contain batches (using the SQL Server GO command) and are run transactionally. Unlike the Rails example, the .NET migrations use SQL, and I don’t yet have any rollback functionality.

You can include any extra SQL files you want in the DatabaseObjects property. Both NAnt and MSBuild have convenient ways to recursively add all files matching an extension.

Here’s an NAnt example:

    <target name="migrate" description="Update the database">
         <loadtasks assembly="Migrations.dll" />
         <migrateDatabase
             connectionString="Provider=SQLNCLI;Data Source=localhost;Integrated Security=SSPI;Initial Catalog=Northwind"
             migrationsDirectory="db/migrations"
             commandTimeout="600"
             batchSeparator="go"
         >
            <fileset>
                <include name="db/functions/**/*.sql"/>
                <include name="db/procedures/**/*.sql"/>
                <include name="db/triggers/**/*.sql"/>
                <include name="db/views/**/*.sql"/>
            </fileset>
         </migrateDatabase>
     </target>

And here it is using MSBuild:

  <PropertyGroup>
      <ConnectionString Condition="$(ConnectionString)==''">
Provider=SQLNCLI;Data Source=localhost;Integrated Security=SSPI;Initial Catalog=Northwind
      </ConnectionString>
  </PropertyGroup>

  <ItemGroup>
       <DatabaseObjects Include="db/functions/**/*.sql"/>
       <DatabaseObjects Include="db/procedures/**/*.sql"/>
       <DatabaseObjects Include="db/triggers/**/*.sql"/>
       <DatabaseObjects Include="db/views/**/*.sql"/>
  </ItemGroup>

  <Target Name="dbMigrate">
      <MigrateDatabase 
          ConnectionString="$(ConnectionString)"
          MigrationsDirectory="db/migrations"
          DatabaseObjects="@(DatabaseObjects)"
          CommandTimeout="600"
          TableName="version_info"
      />
  </Target>

The source code and binaries can be found here.