Daveblog5000

In "Offense" of Scala's Option class; a guarded defense of Cédric's rant

2010-07-30T00:00:00-04:00

In a recent blog entry, Cédric Beust calls out scala’s Option class as nothing more significant than null checks. Commenters rightly set him straight that the thesis of his blog post was marred by an ignorance of idiomatic use of the class.

But, it’s hard to really blame Cédric, when you look at what he had to go on. Odersky’s book states that one should use pattern matching with Option, and the scaladoc for Option is just abysmal:

This class represents optional values. Instances of Option are either instances of case class Some or it is case object None.

That is the entire description of the class, no examples, nothing. Worse, the method that commenters called out as idiomatic, flatMap, has the following description:

If the option is nonempty, return a function applied to its value. Otherwise return None.

This is not what Option#flatMap actually appears to do (nor is documented to do; it’s documented to return an Option[B]!):

scala> val s:Option[String] = Some("foo")
scala> val n:Option[String] = None
scala> val f = { (x:String) => Some(x + "bar") }
scala> s.flatMap(f)
Some(foobar)
scala> n.flatMap(f)
None
scala> val result = s.flatMap(f)
r: Option[java.lang.String] = Some(foobar)
scala> r.getClass
res6: java.lang.Class[_] = class scala.Some

It certainly doesn’t call itself out as “the way” to use Option. A simple example in the scaladoc could have gone a long way.

So, angry post + helpful commenters == problem solved, right?

Wrong.

The Option class is, really, an implementation of the NullObject pattern, and a more elegant way to handle optional values. In scala, we might have this method signature:

/**
 * Updates the full name
 * @param lastName the last name
 * @param firstName the first name
 */
def updateName(lastName:String, firstName:Option[String])

This means “update my name; lastName is required and firstName is optional”. In java, this method might look like this:

/**
 * Updates the full name
 * @param lastName the last name, may not be null
 * @param firstName the first name, may be null
 */
public void updateName(String lastName, String firstName) {
    if (lastName == null) {
        throw new IllegalArgumentException("lastName required");
    }
    StringBuffer b = new StringBuffer(lastName);
    if (firstName != null) {
        b.append(", ");
        b.append(firstName);
    }
    this.fullName = b.toString();
}

So, what’s the right way to do it in Scala? According to the commentors:

def updateName(lastName:String, firstName:Option[String]):Unit = {
  val b = new StringBuffer(lastName)
  firstName.foreach( (name) => b.append(", "); b.append(firstName) )
}

Yech. Does anyone else think that calling a method called “foreach” on our “optional value” is just nonsensical? Or that the idiomatic way to treat an optional value is as a collection, e.g. by using the for comprehension? This just feels hacky. Naming is one of the most important (and challenging) things in software engineering, and Option’s API is an utter failure (even its name is wrong; when one has an option, one typicaly has many choices, not just one or nothing. Optional is really what is meant here, so why are we afraid of adding a few more letters? Especially given how “precise” some of the documentation is, mathematically speaking, why are we not being precise with English?). If Option is just shorthand for a “list of zero or one elements”, and we get no better methods than what comes with List, then what’s even the point of the class?

I’m not saying we remove all the collection methods from Option, but how about a throwing us a bone to make our code readable and learnable without running to the scaladoc (or REPL) to see what’s going on? I mean, there’s a method on Option called withFilter whose documented purpose (I’m not making this up) is: “Necessary to keep Option from being implicitly converted to Iterable in for comprehensions”. Am I expected to believe that it’s ok to have this hacky pile of cruft, but we can’t get a readable method for “do something to the contents if they are there”?

class Option[A] { 
  def ifValue[U]( f: (A) => U ):Unit = foreach(f)
  def unlessValue[U]( f: () => U):Unit = if (self.isEmpty) f
}

def updateName(lastName:String, firstName:Option[String]):Unit = {
  val b = new StringBuffer(lastName)
  firstName.ifValue( (name) => b.append(", "); b.append(firstName) )
}

Which would be less surprising? Couple this with some better scaladoc:

/** This class represents an optional value.  
 *
 * To use as a null object:
 * val optional = getSomePossiblyOptionalValue
 * <pre>
 * option.ifValue { (actualValue) => 
 *   // do things with the value, if it was there
 * }
 * </pre>
 * or
 * <pre>
 * optional.unlessValue { log.debug("missing optional value") }
 * </pre>
 * 
 * To use as a Monad or collection:
 * <pre>
 * val first8upper = option.flatMap( (y) => Some(y.toUpperCase) ).
 *                          flatMap( (y) => Some(y.substring(0,8)) )
 * </pre>
 */
class Option[T] {
  // etc.
}

With these examples, you cover the two main uses of this class, show newcomers how to use it, and demonstrate its superiority over null checks.

It’s frustrating to see this, because Scala has so much potential to be a lucid, accessible, readable language, but API usability and learnability are just not prioties. Scala needs to take some lessons from Ruby in terms of API design (and Java in terms of API documentation).

Of course, none of this does save you from null, because Scala is perfectly happy to assign null to anything. It kinda makes the whole thing seem a bit pointless.

My first iPhone App - Part 5

2010-07-18T00:00:00-04:00

See Part 1, Part 2, Part 3, and Part 4 first

I’ve been working on the app more than my recent lack of blog entries indicates. At this point, I have what could roughly be called a beta version; almost all the features are there, and things seem to be generally working pretty well.

User Experience

The biggest change in the UX is the ability to add tasting notes, date tasted, and location tasted. This is a new screen accessible from the main entry screen. The most obvious way to do this in my mind, was a big button at the bottom:

In designing the new screen, the “where tasted” and “when tasted” were straightforward; I used stock controls. For the tasting notes, I needed a UITextView, which is akin to an HTML TEXTAREA. The visual appearance of this control is pretty lacking compared to the UITextField; there is no nice beveled edge, no rounded corners, and no placeholder text. I really just wanted a multi-line field much like the UITextField, but there is nothing available to create that.

So, I hacked something together.

An option for the UITextField’s appearance is to have a beveled edge with square corners. In this configuration, you can adjust the height of the text field. So, I placed such a field on the screen and sized it about the size of my tasting notes field and made the background color white. I then put the tasting notes field on top of it, with a clear background color, and, well, it looked pretty good:

I then implemented some UITextViewDelegate methods to give the apperance of placeholder text:

- (BOOL)textViewShouldBeginEditing:(UITextView *)textView {
    if ([textView.text isEqualToString:DEFAULT_TASTING_NOTES_TEXT]) {
        textView.text = @"";
    }
    return YES;
}

- (void)textViewDidEndEditing:(UITextView *)textView {
    if ([textView.text length] == 0) {
        textView.text = DEFAULT_TASTING_NOTES_TEXT;
    }
}

I considered using some third-party controls that mimic this behavior, but didn’t want to get side-tracked adding new frameworks to my app at this point.

User Testing

Once I had this, I handed my phone to Amy for some more user testing; She brought up a few obvious things that I had completely internalized and begun ignoring:

Clicking “Save” on the details screen brought you back to the “new wine” screen, instead of just saving and bringing you back to the top. A minor but obvious annoyance.
She kept tapping on the “Choose Varietal” text field, thinking that would bring up the varietal chooser, instead of clicking the much smaller blue “disclosure” button
She was a bit sad that the wines we had entered in the app would not be available on our shared Google Spreadsheet

To smooth the navigation after saving, I used a stock feature of the UINavigationController to “pop” more than once up the chain. Since my design of the details screen used Apple’s delegate pattern (essentially, the “add new wine” view controller was the delegate to the details view’s lifecycle; when you click “Save” on the detail view, it triggers a callback in the “add new wine” view controller; the perfect place to save before the detail view controller popped back two screens).

The problem with the “Choose Varietal” control had bugged me, too, but I got used to it and didn’t think about it. The solution was very simple, though hacky. I placed a clear button on top of the field the exact size of the field and had it trigger the same action as the blue disclosure button. Problem solved.

As to maintaining the list up on Google Docs, I think I may need to implement this sooner rather than later; I think it’s important to be able to get your data out of an application, and Google Docs is a reasonably user-friendly way to do it (as opposed to emailing some CSV file).

Other Random Bits

I still didn’t get around to setting up iCuke for testing; I really should because I don’t know what missing retain calls might be lurking. I also finally created an icon, using a picture I took in Napa. Not sure I like it, but it beats the white blob:

(Taken from my original).

Finally, the app no longer starts up on the actual device. A seemingly serious problem that I assume would be remedied by a re-install from scratch, however I have a few wines that I’ve added and don’t particularly wanted to lose them. Not sure how I could gain access to the SQL database to get them out, but I’m currently downloading the 4.0.1 update for my phone and the 2+ GB SDK update (!).

As a followup, I had to re-install the application from scratch, though I was able to access the SQLite database from an iTunes backup. I really need to implement a quicker backup/export mechanism…

My first iPhone App - Part 4

2010-07-08T00:00:00-04:00

See Part 1, Part 2, and Part 3 first

The past week was spent trying to understand the best way to expand my application’s features without have a ton of duplicated code or UI. It was also a learning experience on Core Data. Thankfully, Stackoverflow and its amazing contributers were very helpful.

Core Data Blunders

It started as I made my first foray into implementing the search screen (i.e. the home screen of my app):

Three of the searches (most recent wines, wines by rating, and all wines) end up doing the same thing more or less: Query for some data, sort it a certain way, and show it in a UITableView. I ended up creating a custom table cell view and I wanted all three to use it:

Access to Core Data-provided entities is done through objects of the class NSFetchedResultsController, which takes what amounts to a database query and provides many ways of accessing the results, including caching, callbacks, and iteration. When using a table view, you typically have your view controller handle callbacks for this class, which allows the table to update itself when items are added, removes, or changed. All of this boilerplate is given to you at the start of the project. So far so good.

My plan was to create additional NSFetchedResultsController instances inside my table view controller class and then switch between them. A fine idea that lead that numerous random difficult-to-reproduce crashes.

It turns out the basic idea of what I was doing was good, but my implementation exposed a sore lack of understanding of how all this stuff fit together. I’m still not sure I fully grasp it (and wish the book I bought had a bit of a deeper dive into Core Data), but after some more reasoning, I got around it. Essentially, leaving the unused NSFetchedResultsController instances connected to my table view and having caching turned on and not properly reloading data when switching searches created a situation where they were all pretty confused about the underlying state of the database.

With some judicious management of these instances, as well as disabling caching, I now have a fully-functional “Recent Wines”, “Wines By Rating” and fancy indexed “All Wines” view (index meaning I can jump to wines by letter, a la the Contacts application).

Unfortunately, the result of turning off caching is that it takes a noticable blip of time to summon any of these views. I may just come full circle and end up with three different UITableViewController instances and NIBs just so I can leave Caching on.

User Experience

Once I had gotten back to a stable app, I loaded it up on my phone and headed into the field. A few nights ago, Amy and I ended up at one of our favorite restaurants/wine bars in DC, Proof. We’ve been there several times, and many of their wines-by-the-glass are in the Wine Brain. While they rotate their selections, I was curious as to what I’d had there previously. Unfortunately, I had yet to implement the search-by-location feature :) Combing through the “All Wines” view was a bit frustrating. This made obvious several features that are now on the top of my list (some planned previously, some not):

Find wines by location
Actually entering the location at which I had a wine
Location-aware assistance of location (e.g. “Are you at Proof? Here’s what you’ve had there…”)
Ability to text-search the “all wines” list

The good news is that my cursory reading of my book and the API docs indicates that these things are going to be very easy to implement.

Objective-C

There’s a lot to like about Objective-C, and a lot to dislike. Even though the handling of multiple arguments is a bit strange, I find it actually results in fairly readable code. It feels very Apple; different, but usable. Even though it’s a lot to type out something like:

NSString *stripped = [canonicalName stringByTrimmingCharactersInSet:
                       [NSCharacterSet whitespaceAndNewlineCharacterSet]];

It is pretty readable, regardless of what programming language you are coming from. That being said, stack traces and error messages on crashes are nigh-useless, and the overall testability of iPhone apps is pretty behind the times. This months issue of Prag Prog’s magazine has an interesting article on iCuke, which I look forward to trying out. I’ve resulted to keeping a text file of test cases that I have to manually run through to make sure I haven’t broken anything, and it feels, well, 1997. The rumours of a Apple’s switch to Ruby are too good to be true, but one can always dream.

Continued in Part 5

My first iPhone App - Part 3

2010-06-29T00:00:00-04:00

See Part 1 and Part 2 first

Just a quick update for tonight. I’m starting to get the hang of how the user interface elements I create in Interface Builder get hooked up and become available in code. It’s a bit odd and not terribly intuitive, but it’s starting to make a bit more sense. Under the covers there’s some sort of dependency injection going on (even if Apple doesn’t call it that) that smells of Spring-style magic; objects get created for me and populated if I do the right thing.

At any rate, I decided that, before making any more changes, I needed to get a reasonable data set into the app. I spent two painful hours exporting my Google Doc Wine Brain into CSV and parsing it with Objective-C, mapping it to my Core Data model objects. In the end, I was successful, but, WOW, what a pain compared to Ruby or Perl.

Seeing this large set of data on the home screen was very motivational to start working on a drill-down/search interface, as you can see here:

Before delving into my new homescreen, I decided to work up a better color scheme that was more “wine-like”. I’m not a big fan of apps that have fancy graphics and obscure the idiomatic iPhone UI, but a bit of a color splash would go a long way (and, not to mention, be well within my graphic design skill level). In addition, more user testing (via Amy once again) revealed that the rating control needed a much starker contrast between the selected and unselected choices. There isn’t much I have control over via the API for this, so I chose an alternate style of UISegmentedControl that used the “iPhone blue” for the selected state. I think it works OK with the color scheme, though I’d prefer the “merlot red” that I chose for the nav bar and wine type:

I’m starting to think that my desire to have wine entry confined to one screen might not pan out. The “slot-machine” control will become more and more unusable as more varietals are added, and there’s really no way to even add a new varietal here. I think I may have to go with the drill-down method where by I push another table view onto the stack to select (and possibly add) a varietal. For another day.

Lastly, I took a stab at a better home screen:

This pretty-well represents the ways I think I’d want to navigate the data. Time will tell once I implement these searches, but this seems like a reasonable start.

I also have to say that, despite Objective-C’s overall wackiness, it was pretty easy to put this together and have the “Add New Wine” button bring up the “Add Wine” screen and to have the “All Wines” button navigate to the old home screen. Apple has done a reasonable job of making it easy to make an idiomatic, normal-looking iPhone app.

Continued in Part 4, and Part 5

My first iPhone App - Part 2

2010-06-27T00:00:00-04:00

See Part 1 first

For the past several days I’ve had the app on my phone and used it to enter in wines to see how the UI was working. Yesterday, I integrated Core Data (Apple’s Object/Relational Mapper (ORM), for lack of a better description) so that I could actually save some data. I discovered some nuances to using Core Data, as well as a refinement in the UI.

UI

Last night, I had both my girlfriend Amy and my good friend Clay (who is also reasonably into wine) play around with the “new wine” screen. Neither have an iPhone, and there were a few obvious problems:

They were very unaccustomed to the “Done” button being in the upper-right corner, as well as the “slot-machine” control
Both had trouble with the “Wine Name” concept
Both had trouble dismissing the keyboard
Clay thought he was entering search terms and not a new Wine

Later that evening, I had a glass of wine at a bar and entered it into my app. Up until this point, I was just typing in wines I knew about, so this was the first “real use” of the app in the field. I quickly realized that the “Wine Name” concept, while important, was too vague and confusing to be the first field you enter. Most wines are known by the vineyard and some don’t even have proper names or appellations. So, I swapped the fields and added some examples in the placeholder text:

I also added a title to indicate that this screen is for adding a new wine. Finally, I used the background button trick along with some events on the rating and type controls to dismiss the keyboard if you tap anywhere outside of a text field.

This feels much cleaner and flows a bit better. While varietal is probably the third most important thing (at least for New World wines), the slot-machine control would be awkwardly placed right below the vineyard/vintage text fields, so I left it where it was.

As to Clay and Amy’s confusion over some idiomatic iPhone design choices, I’m not worrying about them a whole lot; a typical iPhone user will understand what’s going on (I hope :).

Core Data

Backing all this is Apple’s Core Data, which grew out of the Enterprise Objects Framework (and was, incidentally, my first exposure to an ORM way back in 1998), is the simplest and most idiomatic way to store and manage data on the iPhone. As with Interface Builder, you create the model visually, using a simplistic data modelling tool. I set up the bare bones of my data model and generated the classes.

The first problem I ran into was in changing the model after the database had been created on the device. Core Data doesn’t migrate your model for you by default. I ended up starting over with a “versioned” model and implementing their default migration. This will probably work for simple changes, but for more complex changes, it looks like it’s somewhat difficult to migrate the model. An unfortunate reason to do some Big Design Up Front.

My next problem was initializing reference data. In my case, I wanted the list of varietals as well as the list of wine types (e.g. “red”, “white”) to be in the database and managed by Core Data. I ended up creating an array of the values inside the app and then checking the database for their existence, adding them if they were missing. It’s a bit cheesy, but I can extract the lists to plist files later. It did give me a chance to play with Objective-C’s blocks feature. I needed to map my list of Varietal objects to a list of strings, so I could compare that list against my default list. While NSArray doesn’t provide a map function, it does provide the equivalent of Ruby’s each or Scala’s foreach. In true Apple/Objective-C style, it’s called enumerateObjectsUsingBlock :)

NSMutableArray *strings = [NSMutableArray arrayWithCapacity:[array count]];
[array enumerateObjectsUsingBlock:^(id obj, NSUInteger i, BOOL *stop) {
    [strings addObject:[obj valueForKey:@"name"]];
}];

The stop pointer is an out-only variable that you set to YES to allow for an early exit from the loop. I’m glad they’ve adopted this functional style; I forsee explicit loops becoming just as quaint as GOTOs in the near future.

One thing bugged me about this process, however. I had these generated classes and it wasn’t clear how to add my own methods to them without losing the code every time I re-generated my model. Enter mogenerator. This provides a more sophisticated bit of code generation, givng you two classes for each model: one for machines (i.e. for Core Data to manage) and a subclass for people (to which I can add custom methods). Perfect! I wasn’t able to incorporate it into XCode, but I’m happy running in on the command line as needed for now.

Random Bits

I spent more time than I would’ve liked tracking down a nasty bug where every attempt to convert a string to a number (for the wine vintage year) resulted in a nil number. It turns out I was confused about when some object lifecycle methods were being called and my pointer to an NSNumberFormatter was, in fact, pointing to nil. Unlike Java, where you’d get a NullPointerException, or C, where you’d get a core dump, it seems Objective-C just returns nil and doesn’t even issue a warning. Very strange. Once I realized that, the solution was obvious.

I’m still not 100% sure I’m structuring my code properly, but Stack Overflow’s stalwart iPhone SDK answerers have been incredibly helpful in pointing me in the right direction.

Next Steps

Now that I’ve got the ability to save and retrieve data, the next step is to start looking to browsing, viewing, and editing the database. I think the domain is restricted enough that I can use the iPhone’s idiomatic drill-down table navigation to filter the database.

Continued in Part 3, Part 4, and Part 5

My first iPhone App - Part 1

2010-06-23T00:00:00-04:00

Background

Last year, I spent most of my free time becoming familiar with Ruby, Rails, and Scala. I’m by no means an expert, but I’m trying to follow the advice of the Pragmatic Programmers and learn a new language/system every year. This year, it’s going to be the iPhone API and Objective-C. I coded some WebObjects applications in Objective-C many years ago, so the language isn’t totally unfamiliar, but the iPhone SDK, obviously, is. After working my way through the first several chapters of the Dudney and Adamson’s “iPhone SDK Development” book, I decided I was ready to try my first real app.

First App

My girlfriend and I enjoy wine, but are largely novices. On a trip to Napa we started writing down the wines we’d tried and keeping personal ratings of them. The idea being twofold: try to remember good wines/avoid bad ones, and try to hone in on what our tastes are. There are so many wines and so many flavors and smells, it’s just impossible to keep in your head. So, I created a Google Spreadsheet and a form to allow us to enter wines on the go, via our smart phones. The Goolge Docs form is less than usable, and this seems like the perfect job for a native iPhone app.

User Experience

Aside from learning the ins and outs of the SDK, another main goal is focus on making a highly-usable application (at least for myself and my girlfriend). So, before diving into Core Data and modelling the domain, my first version of the applicaton simply presents the “add a new wine” screen. This will be the screen I spend the most time in, and it’s important to make it fast and easy.

The Google Spreadsheet currently collects:

Wine Name
Vineyard
Vintage (year)
Varietal
Type (e.g. red/white)
Where we had
When we had
My rating
Her rating
Tasting Notes

That’s obviously too much for one iphone screen, and, if I’m entering something in a hurry, I might not have time to go through all that. Further, I think adding multiple ratings makes things a bit confusing, so I focussed my screen on the identifying characteristics (name/vineyard/vintage), the rating, and the type:

As you can see, I tried to fit the “where” on the bottom, as well as a “enter more information” button. After loading this on the phone, it was just not working out, plus I was missing the varietal, which is fairly important (at least to me).

My next attempt simply ignores the “add more” concept, and, by using the segmented control, I was able to fit the type into a more compact area, while including the varietal as a “slot-machine”/picker.

My plan is to load this onto the phone and try using it the next few times I’m out. I figure I can do without the tasting notes or location if it keeps the UI simple

The Code

Objective-C and XCode are a whole different world from Java or Ruby. First thing that bit me is the memory management. I was familiar with the rules (or so I thought). Basically, if you call a method like “copyXXX” or “initXXX” to create a new object, you must release it when you are done with it. If you get access to an object through any other means, you should not release it.

// my class has an NSArray called varietals that I'm initializing here
varietals = [NSArray arrayWithObjects:
                  @"Barbera",
                  @"Cabernet Franc",
                  // snip
                  @"Zinfandel",
                  @"Other",
                  nil];

Since I’m not calling an “init” style method, I figured I didn’t need to release it in dealloc. For some reason, this made me decide that I didn’t need to retain it. This caused my first crash and trip through the debugger to figure out what in the heck is going on.

// my class has an NSArray called varietals that I'm initializing here
varietals = [[NSArray arrayWithObjects:
                  @"Barbera",
                  @"Cabernet Franc",
                  // snip
                  @"Zinfandel",
                  @"Other",
                  nil] retain];

Since I needed varietals to survive the method where I was initializing it, I had to retain it.

Other struggles

Interestingly, laying out the UI and getting that hooked up isn’t too bad; the thing I’m having the most trouble with is figuring out what is the “unsurprising” way of doing things; tutorial code from books is rarely very well written; it’s usually done in a certain way to make concepts clear, but I haven’t yet found a definitive “here’s how you organize your code” or “iPhone best practices” that seems relatively comprehensive.

At any rate, I figure if I can get the UX to be reasonably good, the rest should sort itself out.

I’ll continue posting about my progress here on my blog. If I can get the app polished and working, my girlfriend might demand and Android version. Would be interesting to compare the two paradigms (especially since Android is all Java, which has been my bread-and-butter for many years).

Continued in Part 2, Part 3, Part 4, and Part 5

In defense of vi

2010-06-14T00:00:00-04:00

Sadly, I was unable to attend RailsConf this year. It was in Baltimore, and it would’ve been a lot of fun, but it just wasn’t in the cards. One of the great things about RailsConf is that the videos are posted online very quickly and are always of high quality. While I always like hearing DHH speak, I try to never miss one of Uncle Bob Martin, author of the must-go-read-this-right-now Clean Code.

This year, he gave an awesome talk on the (lack of) innovation in software development. He pointed out that we were still writing the same type of code now as we were 40 years ago. In one part, he asks the crowd what editor they use (knowing full well most will say vim). He then proceeds to make light of the audience for using a 19-year-old editor based on a 34-year-old editor. While his criticism is brief, I think it speaks more to the sorry state of text editing than to the developers (like me) who are “still” using vim.

I’m sure that unclebob is one of the more advanced IDE users, but it just pains me watching my co-workers meander through their code, selecting things with the mouse, and deleting long swaths of text by just hitting the delete key a lot.

While an IDE is great for learning a new environment (especially one as pedantic and verbose as Java or Objective-C), the editing capabilities are counter-intuitive and degenerate. vim may be hard on the newbie, but it’s intuitive and logical. When you understand the “zen” of vim, you can fly through your code, playing it like a musical instrument. It affords muscle memory and allows sophisticated chaining of movement, intent, and action. You can see what you want to accomplish and your hands make it so, all without reaching for a mouse, cursoring, using the delete key, or contorting your hand with some awful combination of key modifiers.

Think of it as a piano; it’s a very simple base of simple sounds and actions that, when chained together, can be used to make beautiful music. An IDE is more like a fancy Casio keyboard with built-in songs and beats. Shiny, but hollow.

Are private methods code smells?

2010-05-26T00:00:00-04:00

Having had discussions with co-workers on the utility of private methods and recently on the scala mailing list, there’s been the notion that private methods (or even protected methods) are code smells – indicators that there is something wrong with the design of your code. I believe in many cases they can be, but that private (and protected) methods still have their uses, especially in maintaining a clean design under constraints of getting things done. First, we need to know what we are talking about.

Definitions

While the technical meaning of access modifiers varies with the language, the intent these concepts communicate is what we’re talking about here. I assert that the intent of public/protected/private is (and/or should be) as follows:

public - this method is part of the published API and will not change within major versions of the class
protected - this method is a hook for modifying the behavior of this class using subclasses. It, too, will not change within major versions of the class (of course, it also might exist for code-reuse internal to the class hierarchy).
private - this method was refactored out of a well tested public or protected method for reasons of clarity or internal re-use. This method may absolutely change, even in patch releases, and should not be relied upon to even exist
package private (bonus for Java) - this method was written by someone lazy or ingorant, or by someone who acknowledges that this code should be pulled out into its own class, but hasn’t done so, yet still wants to test it seperately, thus breaking encapsulation.

Why would private methods be code smells?

The most concise argument is that private methods could indicate that the class that contains them is doing too many things. Consider this code from shorty, my URL shortener:

override def contextInitialized(event:ServletContextEvent) = {
  event.getServletContext.log("Initializing our hasher/DB")
  val dirName = event.getServletContext.getInitParameter(
    ShortyServlet.DB_DIR_PARAM)
  if (dirName != null) {
    val dir = new File(dirName)
    if (dir.exists()) {
      if (dir.isDirectory()) {
        val hasher = URIHasher(DB(dir))
        hasher.start
        uriHasher = hasher
        event.getServletContext.setAttribute(
          ShortyServlet.URI_HASHER_ATTRIBUTE,uriHasher)
      }
      else {
        throw new ServletException(dir.getAbsolutePath 
          + " is not a directory")
      }
    }
    else {
      throw new ServletException(dir.getAbsolutePath + " doesn't exist")
    }
  }
  else {
    throw new ServletException("You must supply a value for " 
      + ShortyServlet.DB_DIR_PARAM)
  }
}

Is it totally clear what this method does? If not, it basically checks that the directory configured for our database exists and is a directory, giving us a specific error message if not. It’s a bit long and full of error checking, so the meat where it creates our URI hasher and gives it to the servlet is somewhat obscured. Here’s a cleaned up version:

override def contextInitialized(event:ServletContextEvent) = {
  event.getServletContext.log("Initializing our hasher/DB")
  getDBDir(event) match {
    case Right(dir) => {
      val hasher = URIHasher(DB(dir))
      hasher.start
      uriHasher = hasher
      event.getServletContext.setAttribute(
        ShortyServlet.URI_HASHER_ATTRIBUTE,uriHasher)
    }
    case Left(errorMessage) => 
      throw new ServletException(errorMessage)
  }
}

private def getDBDir(event:ServletContextEvent) = {
  val dirName = event.getServletContext.getInitParameter(
    ShortyServlet.DB_DIR_PARAM)
  if (dirName != null) {
    val dir = new File(dirName)
    if (dir.exists()) {
      if (dir.isDirectory()) {
        Right(dir)
      }
      else {
        Left(dir.getAbsolutePath + " is not a directory")
      }
    }
    else {
      Left(dir.getAbsolutePath + " doesn't exist")
    }
  }
  else {
    Left("You must supply a value for " + ShortyServlet.DB_DIR_PARAM)
  }
}

We’ve added lines of code, but our public method is a lot clearer: we get the dir for our DB; if we get a “right”, we have a usable dir, and if we get a “left” (the Scala convention for an error), we have the error message to use for our exception.

So, is getDBDir now a code smell? This is a private method and, this means that contextInitialized was tested and we extracted the private method as step three of the TDD “Test/Fix/Refactor” cycle.

Of course, getDBDir has nothing to do with the ServletContextListener and is really all about checking for a valid path. So, we should extract it to a new class, right?

Well, if we did do that, we’ve now actually added a public API to our codebase. Unless we make the new class completely private to the enclosing class, we’ve now introduced a new public class that must be tested and managed as part of our system’s public API.

I don’t see the benefit to doing that. I don’t particularly want this method to be in the public API (at least not now). So, what about making it a private inner class? That doesn’t seem to be much different than what we have now, though it might save a few minutes on a future extraction.

When IS this a code smell?

This was a simple example. Consider a class like SignedUrl in my Gliffy Ruby client. This class has a fair number of private methods and, overall, is pretty large. The private methods are vague things like handle_params and get_signing_key. The private methods aren’t really the problem, however; They are a symptom of the fact that this class is just doing too many things. A better design would be to split this class up into something like a UrlParams, UrlSigner and UrlAssembler (just off the top of my head). The result would be more smaller classes, each with fewer public and private methods.

What about protected methods?

In Ruby, you don’t see a lot of protected methods. In well-designed Java frameworks (such as Spring), you’ll see them as I’ve described above: hooks into a helper class that allow you to customize how that class behaves via subclassing. In non-library code, protected methods tend to get used for code-reuse in narrowly-focused class hierarchies. For example, the project I’m working on has a BaseController that provides common methods to the actual web controllers. Having used protected methods for both of these purposes, I think that, honestly, both can be code smells.

Protected methods as hooks

In the case of providing “hooks”, I think it’s clear that by providing them at all, you are acknowledging that your class might be poorly designed (or that your language lacks the expressiveness to better design it). Consider the venerable (and now deprecated) SimpleFormController, a Spring MVC class that provides many protected-method hooks, one of which is protected Map referenceData(HttpServletRequest request). This is called during the lifecycle of the controller to get any data needed by the form that should be available to the view (for example, a list of U.S. states from the database). Your subclass of SimpleFormController overrides this to provide the data as a map.

Why not create an interface called ServletRequestToReferenceData that contains this method, and inject an instance into SimpleFormController? This would be very Spring-like. My guess as to why this isn’t the case is that there is simply too much overhead in making yet another one-method interface and the designer of SimpleFormController just felt this was a reasonable tradeoff.

In Scala, however, this could simply take a function:

  def referenceData[A]((HttpServletRequest) => Map[String,A]);

  // ...


  controller.referenceData{ request => 
    // create our map based on the request
  }

In Ruby, the overhead of creating a new class isn’t nearly as onerous, and we could still inject a block as we do in Scala.

Ultimately, I would say that you have to make a tradeoff here, and it has to take the language and frameworks into account; a class with many, many hooks might be a code smell, but using it as a means to avoid language ceremony in the name of OO purity is probably a reasonable design tradeoff.

Protected methods as code sharing

This use of protected methods is harder to justify, but incredibly handy. Still, this could be another case of Java (e.g.) not providing the necessary language features to make class extraction more straightforward. As mentioned, in my current application, I have a BaseController. It has a helper method as follows:

protected Person getAndValidateLoggedInPerson() {
  Person p = this.personService.getPerson(
    this.authenticationService.loggedInUserId());
  if (p == null) {
    throw new NotFoundException;
  }
  return p;
}

Because of Spring MVC, we have specialized exceptions for HTTP errors, such as “NOT_FOUND”. Here we contain the logic to identify the id of the logged-in person as well as the check for existence. Having this available to all controllers in the entire system is very handy.

But, is this a code smell?

We could make a class that does this, turning this code:

Person p = getAndValidateLoggedInPerson();

into this:

Person p = this.personValidatorAndGetter.getPerson();

creating a one method class that has many lines of injected dependeicies and other boilerpalte with a few actual lines of code.

I think this is yet another case of pragmatically dealing with language ceremony. In Scala or Ruby, the overhead of creating re-usable bits of code like this is far lower than for Java; In Scala one could envision a simple function, possibly with some implicit parameters. Ultimateley, “too much” of this sort of thing should be a code smell, but small bits of this, in the name of simplicity, clarity, and simply keeping the codebase smaller isn’t an automatic red flag.

Conclusions

So, are non-public methods code smells? If we take the wikipedia definition, which is clear that a code smell indicates merely the possibility of deeper problems in your code, then, yes, private and protected methods are code smells.

But, should they be avoided at all costs? Absolutely not. There’s many valid reasons to use them, and they do not deserve the blame for an ill-concieved class. Coding in the real world is about tradeoffs; we have to do the best thing we can with the time, resources, and tools at our disposal. These tradeoffs may not result in an According-to-Hoyle OO (or functional) design, but we’re not writing code to provide examples of programming paradigms or design patterns; we’re writing it to accomplish something and provide value. And, as professionals, we need to do it at a predictable rate that doesn’t incur too much technical debt.

Video of My Talk at ScalaDays 2010

2010-04-30T00:00:00-04:00

Video of my talk at ScalaDays is online. Check it out. Feedback very much appreciated!

Scala Days Impressions

2010-04-22T00:00:00-04:00

I had the good fortune to not only attend ScalaDays 2010, but also to speak (generously supported by my employer). The experience overall was great. I talked to some really enthusiastic and interesting people (including two high school kids from Austria who were using Lift because, after three years of Java, they were tired of it already!)

Overall Impressions

Scala 2.8 has some really powerful new features; I'm almost glad I don't have a significant codebase using 2.7, because I would hate to not be able to use 2.8
The conference had a wide variety of talks, at varying skill levels; this was not a "beginners-only" conference, though there were plenty of introductory talks (e.g. the Akka talk).
The Scala team seems very interested in moving Scala into industry and changing the perception of it as an "academic" language (though I'm not convinced they know what to do, nor should this be their responsibility).
Most people I talked to who weren't using Scala at their jobs were looking at the strategy I had employed (using it for better testing), or were just frustrated with not being able to take advantage of Scala's productivity.
Scala's flexible syntax and powerful constructs are being used for a very wide variety of things; there were talks on using it for Processing, parallel collections, and cross-compilation to non-standard architectures (i.e. chips with no JVM available).
The term "monad" was mentioned during exactly one talk and not at all for the rest of the conference. This made me feel very good, because the impression I get on the Scala mailing list sometimes is that if you don't understand mathematical type theory, you are a moron who can never understand Scala. This impression has been proved false.

Talks I Attended

It was very interesting to compare this conference to a No Fluff Just Stuff conference. At those, the speakers are professional, both in that they are getting paid and that they are just, really, really good at speaking and presenting. Their talks, however, tend to be introductory and not terribly advanced. I think that's fine, these are the type of speakers you want to have when you need to "get the message out" on some new and interesting technologies.

The speakers at ScalaDays, however, were just regular people using Scala and wanting to talk about it. As such, things like presentation style and slides were highly varied in terms of quality. That being said, even the speakers who had the most trouble, still had some interesting things to talk about. Following are my quick thoughts on each of the talks I attended:

Opening Keynote

by Martin Odersky

His talk was on the new features of Scala 2.8 which, as I said above, are quite exciting. Package objects, new collections library, and default/named arguments are just some of the highlights. All-in-all, some very exciting features.

Runing Scala in the Browser

by Wolfgang Kuehn

He had created a way to write Scala code and turn it into JavaScript, much like how GWT does this with Java. His approach wasn't geared around UI widgets, however, but in creating AJAX applications using WebSockets. I've always felt JavaScript was assembly language (which is why I think coffeescript is so interesting), and this was another way to leverage JavaScript without having to deal with it.

Scala Parallel Collections

by Aleksander Prokopec

This was a preview of a Scala 2.8.1 (or later) feature, where a subpackage of the new 2.8 collections library was created that could perform the collection operations in parallel. Consider a filter on a collection; this is highly parallelizable. This library is being created to basically implement that in a transparent fashion. Really interesting stuff. They had even considered load balancing, via some interesting work-stealing mechanisms. In only 30 minutes, there wasn't a huge amount of detail, but this seems to be another way in which parallel computation could be done very easily.

Developing the Scala Bindings to the Fly Object Space

by Channing Walton, Nigel Warren, Simon Kent

The concept of "object spaces" never really sunk in with me, but it wasn't actually that relevant to the talk. This was basically a comparison of a Java library and it's straight-port to Scala. As expected, the Scala implementations were all much more succinct and (to me) clear. The speakers hadn't used a lot of Crazy Scala Magic™ to achieve the real code reduction that was shown. This talk made it very clear that even if you write "Java in Scala", you still get a lot of productivity gains.

CCSTM: Library-based STM for Scala

by Nathan Bronson, Hassan Chafi, Kunle Olukotun

Software Transactional Memory is one of the go-to features of Clojure. Clojure supports it natively, while Scala does not. This talk was on an implementation of STM as a library for Scala. It seemed relatively simple to use and understand. While it's not as "clean" as the Clojure way of doing things, the speaker made a solid case for a library implementation being a better tradeoff, given the difficulty in adding it to the language at this point in Scala's life (it's easy to forget how mature the Scala language and compiler are, despite how "new" it seems).

Lightweight language support for type-based, concurrent event processing

by Philipp Haller

This talk proposed a function type called "Translucent Functions" that were better-suited to concurrency than the partial functions used in the actor library. Honestly, it was a bit over my head, and I wasn't able to catch up with the speaker. Perhaps the paper would be more elucidating for me.

Data Parallel Programming in Scala

by Bruce Lester

This talk was very similar to the one on Scala Parallel Collections. Here, the speaker had created a collection library separate from the Scala one, designed specifically to allow parallel processing. He had a very different approach, using control structures to indicate areas where parallelism could occure. It was pretty interesting, but the Scala 2.8 Collections Library implementation seemed a lot cleaner to me than this.

Type Specialization in Scala

by Iulian Dragoș

One of the new features of 2.8 is the ability to "specialize" your code to work with primitive types and avoid boxing/unboxing. This is something Java cannot really do. This talk was an overview of how that worked and what specializations were done to the Scala library. I guess it's nice that this kind of optimization if available, but I think it would have negligible gains for most real-world applications that weren't doing constant number crunching. Even in that case, you don't necessarily need this.

Leaky Monads

by Josh Suereth

How could I not attend the only session (and mention!) of Monads? The speaker gave a good overview of using Monads to implement automatic resource management (e.g. File.open { |file| file.readlines.whatever }). The idea was that since Monads are designed to hold onto the objects they manage, it's a bit inconvienient when dealing with the results of operating on managed resources. Thus, his monads "leaked" their contents to create more concise code.

Type-safe SQL embedded in Scala

by Christoph Wulf

Akin to LINQ, this talk showed how to write SQL directly inside Scala code, but maintain type safety. It uses a compiler plugin to achieve this, and he talked a lot about how to tell the difference between embedded SQL and regular Scala code. I guess the plugin introspects your database to get the types and generates a bunch of code using some Scala classes that get substituted in during compilation. Pretty interesting. I'm not sure I'm really missing SQL right in my code, but it is certainly cleaner than JDBC.

Scala at LinkedIn: Distribute Computing with Norbert

by Chris Conrad

This was a talk about how LinkedIn is using Scala to glue together some enabling technologies to create their social network graph. It seemed really clean and simple, especially since the underlying technologies are all Java-based. This was mostly an overview of the API and some usage examples. It's open source, and looks pretty interesting. I'm not sure it would be useful for me at my job, but definitely seems like a cool project.

Akka

by Jonas Boner

I was looking forward to this one quite a bit, as akka gets a lot of hype on Twitter. I think 30 minutes was just too short, as it was pretty light on details. There was a lot of assertions about Akka that the speakers just didn't have time to elucidate or substantiate. I would love to see a comparison of Scala Actor vs. Akka Actor code and discuss why the Akka way is better. Similarly, I'd love to see the basis for all the assertions on Akka's performance; it certainly gets a lot of hype. Jonas is currently working on publicizing the production deployments of Akka (which would go a long way) as well as establishing commercial support for it (which also goes a long way :).

Scala IDE for Eclipse

by Miles Sabin

I went to this purely for my co-workers. I hate IDEs in general, and Eclipse does some pretty evil things to my co-workers, yet many of them still swear by it. At any rate, Miles was almost impossible for me to understand (despite being English!) and most of what I got out of the presentation was that Eclipse was very much designed to work with only Java, requiring some nasty hacks to get Scala working with it. It seems Miles' Herculean efforts have been used to get around this and there now seems to be a nice development environment for extending and enhancing the Scala plugin.

sbt Design and Implementation

by Mark Harrah

While sbt isn't my ideal build tool, but it's a billion times better than Maven and Mark's talk actually made me appreciate it a lot more. He is unabashedly in favor of terse symbols over long method names, and he spent some time explaining the rational behind them (thus making things make a bit more sense). His assertion is that you will eventually internalize the symbols and achieve productivity gains at a cost of a slight bump at the start. I tend to disagree with this, because how often is one modifying their build? This is why learning and retaining Ant and Maven is so difficult (I can't write a copy in Ant without a trip to the docs). At any rate, this was a good talk and great overview of sbt, and, despite me not being into the crazy symbols, I will take it over Maven or Ant any day. At least it's based in a real programming language and not some XML nonsense.

Processing with Spde

by Nathan Hamblen

This was a brief over of Processing, for those unfamiliar with this (horribly‐named) visualization tool and then a review of the Scala bindingns the presenter had created. Interestingly, he created SPDE as an SBT plugin so that users could gain the benefit of SBT's automatic re-compilation and running features. His plugin + a text editor is a makeshift IDE for Processing, which is pretty cool. It was very high-level and didn't get into too much detail, but the Scala code was, as expected, much more concise than the Java code.

Sneaking Scala Into your Organization

by Me!

I was pretty nervous, as the many of the previous speakers' topics were all quite heady, but I had practiced my ass off, and delivered the talk as well as I could. I think it was pretty well received and got some good questions. I had talked with a few people previously in the conference who were using testing as a way to get Scala into their jobs and a few other people who were interested in what I had to say. Several people were curious as to how the other developers at OPOWER liked Scala. I felt pretty good about it.

Future of Scala

by The Scala Team

For this, we broke into small groups and basically unloaded on a poor member of the Scala team as to how anything related to Scala could be better. Then, the Scala team assembled all of this info while Martin gave a very long term roadmap for Scala. I do worry that the universe will never catch up to Martin and Scala team as they add more and more advanced and innovative features.

After this, a member of the Scala team did a very fast tour of the feedback. I wish the entire list could be posted; there was a lot of very heartening complaints in there that would not be obvious from the Scala mailing list (e.g. hate on the underscore, desire to be less academic in terminology) This fed my (arguably self-biased) impressions that people want to get Scala into industry and that there is a large hole in this part of the Scala universe.

There were also some updates on IntelliJ's support for Scala. They sound way ahead of the curve compared to Eclipse and NetBeans, but I was a bit frightened by IDEA's guy saying that they way people learn languages is to ask for auto-completion and see what's available. I see this trend all to often and I think it's harmful. More on that later.

Overall

Despite still being stuck in Geneva, I'm glad I went. Giving my talk was fun, but it was also great to hear about all the many ways in which people are using Scala. I went in expecting massive academia overload, with a lot of theoretical type system and functional programming talk and got, instead, a wide variety of topics and just the right amount of academia (i.e. not that much). Really encouraging. Also encouraging is that ScalaDays 2011 will likely be in the Bay Area, which is a much nicer place to get stuck that stupid Geneva.

The only four types of classes in your OO system

2010-03-08T00:00:00-05:00

Object-Oriented design is hard, especially in a large application. It's not always clear where logic should go, and there's often no "right place" to put a piece of code. I've found that there are four distinct types of classes that, if you stick to them, can make your code a lot more understandable, and can provide clear direction as to the age-old question of "where does this code go?"

0. Background

The J2EE way is to have model objects be stupid structs, and have all business logic in a service layer (this is actually very close to a classic "functional programming" way of doing things; ironic that many Java devs eschew FP). Spring lets you do whatever you want, but more or less follows this pattern. While the "Rails Way" is to put business logic on the model objects, I think the "service layer" concept is eventually going to be common practice. So, I've found that I very rarely make a pure "by-the-book according-to-Hoyle" OO-compliant class; I've settled on four patterns that seem to cover pretty much everything. I've also noticed that when these patterns get mixed together, you get trouble.

1. The Record

The record is a dumb struct that you usually need to appease your object-relational mapper. You may need them elsewhere to just name and type some set of data that you either can't model as a tuple because of your language, or don't want to model as a tuple because of some complexity. A record typically has methods that merely expose it's contents and often need to be mutable for the reasons stated. You might have derived fields that are convieniences and not based on your core business logic. An easy example is a person. They have a name and a birthdate, and you might derive their age from that:

2. The Immutable Object

This is the closest to a pure "object-oriented" design. Classes of this type are immutable and should hold data you will use a lot in your system. They may also probably have some business-logic attached as methods; this business logic should be entirely focused on the object and its contents. Typical methods will give you more complex information about the data the object contains, or will vend new objects of the same type, based on the method and parameters called.

This is the most clear distinction (in my mind) between functional programming and object-oriented programming. In an FP world, the data being operated on would be loosely defined (if at all) and you'd have functions that transform it. In an OO world, your object's data is clearly defined (by the class fields/accessors), and the operations available are the methods of the class. When you require that the objects of the class be immutable, you have a very nice encapsulated package of data and operations. This, to me, seems a lot easier to deal with than a "module" of functions and some tuples (or lists of tuples) that the functions operate on. Scala makes it very easy to create classes like this. It's probably one of the few languages that does so (Java certainly is no help, but it can be done).

The benefits here are huge; immutability allows your codebase to be much more comprehensible, and allows you to use these objects in concurrent situations without worry. Since they are immutable, their methods are immediate targets for caching if you discover you need to do this to improve performance.

3. The Builder

While you can certainly use methods (or create methods) on Immutable Object classes to "build up" the object you want, this is often cumbersome, and results in a lot of object creation for no real reason. The "builder" can be used to make this a bit simpler. The Builder is a throwaway class whose sole purpose is to create Immutable Objects. This obviously creates a very tight coupling between the two classes, but this can be worth it. This is very preferable to a mutable class and, depending on your operating environment, is preferable to making many intermediate objects you will need to create the Immutable Object.

4. The Service

The analog of The Record, the service has no data and all logic. EJBs are services; they have no internal state, operating on their arguments and returning a result. Methods of services can be very functional in nature (operating solely on structs or immutable objects), or they may provide functionality that implements complex business logic not logically part of an immutable object's class. In a vanilla n-tier application, you use services to get data in and out of your database (you might call these DAOs and you might distinguish different types of services for partitioning, but these are all the same sort of class).

Like records, Services are not OO at all; these are the functions to your C programs structs. But, there is a good reason for this design; you separate concerns, don't need to worry about concurrency (services have no state), and can even horizontally partition where serivces actually run.

Reading a File in Scala vs. Java vs Ruby

2010-01-26T00:00:00-05:00

Examining the code needed to read a file line by line is a a common way to examine the hoops a programming language makes you jump through. While Perl certainly has some one-liners for this, let's start with Ruby, which presents an elegant and clear way of doing it:

File.open("some_file.txt") do |file|
  file.readlines.each do |line|
    puts line.upcase
  end
end

It really doesn't get any clearer than that.

Here's the canonical Java way of doing it, complete with plenty of places to introduce bugs:

import java.io.*;

public class ReadFile {
  public static void main(String args[]) throws IOException {
    File file = new File("some_file.txt");
    BufferedReader reader = new BufferedReader(
      new FileReader(file));
    String line = reader.readLine();
    while (line != null) {
      System.out.println(line.toUpperCase());
      line = reader.readLine();
    }
    reader.close();
  }
}

Yech. The need to call readLine() twice kinda sucks. We could use a do-while, but that requires a second line != null check. Personally, I like to forget the second readLine() and wonder why my code runs forever :) That being said, this was extremely easy to figure out, even the very first time I did it in 1998. The class names are obvious, and the documentation is excellent.

Scala to the rescue, right?

import scala.io._

object ReadFile extends Application {
  val s = Source.fromFile("some_file.txt")
  s.getLines.foreach( (line) => {
    println(line.trim.toUpperCase)
  })
}

This was a slight pain figure out. I looked in scala.io and, of the few classes that were there (including a curiously named BytePickle), it appeared as though Source was the class to use. Of course, there's no easy way to create one from the constructor, and the scaladoc doesn't just say "Dude, look at the Source object". Once I looked through the Source object's scaladoc, the solution presented itself.

Of course, unlike every other line-traversing library in the known universe, Source leaves the line endings on. This is thankfully fixed in 2.8 (by which I mean 2.8 breaks 2.7's implementation, which is a strange thing for a point release to do). The real question is: "Is this how I'm supposed to read files in Scala?". With a class called Source?!. The scaladoc says that this class represents "an iterable representation of source files". That might explain the strange methods like reportError and reportWarning. I guess this is only for writing the Scala compiler? If so, scala.io seems an odd place to put this.

So, my answer is "No, this cannot be how to canonically read files in Scala". Since the Java way kinda, well, sucks, what alternatives are there? There's scalax.io, which seems to implement this as a class called, curiously, FileExtras. I'm not sure if this code is actively maintained, but it's documented in classic Scala style: terse and full of loaded terms like "nonstrict". Nevertheless, there seems to be some code here to easily read a file "the easy way" (despite some distracting names).

This points out a big difference between "Scala the language" and "Scala the library". Scala the language is very interesting and has a lot of potential. Scala the library is schizophrenic at best; it's not sure if it wants to be OO, functional, or what. The documentation ranges from sparse to absent, and the overall designs of the classes and package range for sublime to baffling. Years different from Java 1.1.

Unicode in Your Domain Names

2010-01-24T00:00:00-05:00

Got a few questions about how I set up ❺➠.ws (which is powered by Shorty, my Scala-based URL shortener), so I thought I'd write up how I got it working. Short answer is that it was pretty easy.

I got the idea from John Gruber, who made a similar thing for his site to post entries on @daringfireball. The trickiest part was figuring out what this was called so I could find out who could sell me a domain with unicode characters in it.

It turns out, this is called an IDN (short for Internationalized Domain Name), and not everyone will sell you one. Couple that with the need to get a non-.com domain, and I had to hunt around for a while.

I ended up going with DynaDot as they could provide the wacky hostname that I wanted as well as a .ws TLD registration. I was amazed at the number of domain regstrars whose web forms could not handle Unicode. It's been almost 7 years since Joel Spolsky wrote his screed on dealing with Unicode, so I don't know what the deal is.

At any rate, the tricky bit in actually using the domain, because a) entering ❺➠ into vim is nontrivial, and b) I doubt that Apache's config file would work with unicode characters in it. Enter Punycode, which is an asciification of any IDN. Fortunately, the domain host provides the Punycode for your IDN, so configuring apache was a matter of:

<VirtualHost XXXXX>
ServerName xn--dfi5d.ws
ServerAlias www.xn--dfi5d.ws
DocumentRoot /home/webadmin/xn--dfi5d.ws/html
<!-- whatever else goes here -->
JkMount /s* ajp13
JkMount /s ajp13
  <Directory /home/webadmin/xn--dfi5d.ws/html>
    Options Includes FollowSymLinks
    AllowOverride All
  </Directory>
</VirtualHost>

At this point, it pretty much worked, although it was sometimes difficult to get curl to work with the non-punyied name.

One thing that was weird was that I found that a lot of domains I wanted to try were taken or not available (with no explanation). Often it seemed like the punycode version was a normal looking URL that was taken; I tried several IDNs that had a unicode character with a wierd "5", and they punyied to an ascii 5. Not sure what the deal is there, but I eventually found the one I wanted.

Top 5 Shocking Omissions from Spring MVC

2010-01-09T00:00:00-05:00

Been working with Spring MVC recently. My enthusiasm has waned somewhat, as I've discovered that for all of it's tweakability and configurableosity, it omits what I believe to be incredibly obvious things:

Simple, 80/20 conventions by default - While the phrase "convention over configuration" appears numerous times in the Spring documentation, it is still unable to call the foo method of the class BarController when I request the url bar/foo without a lot of configuration, some of which subtly conflicts and causes silent failures. The default configuration is useless
Debugging the Routing - I shovel angle bracket after annotation after angle bracket into Spring and it simply can not tell me what URL patterns are mapped to where, and why a given URL isn't mapped, nor what to do to get it mapped. It's whole purpose is to map URLs to controllers. Shocking.
Simple internal routing without magic strings - If I've mapped FooController's bar method, shouldn't I be able to redirect/route to that in code via route(FooController.class,"bar"), regardless of the specific URL that FooController and bar respond to? Instead, I've got magic strings everywhere and if my urls ever change, god help me. And don't get me started about accessing this stuff via tests.
Simple web-testing - Adding a dependency on JWebUnit doesn't cut it. Jetty and Tomcat cannot hot deploy. How do people test these things??!?!
Test Fixture Support - I had to hack this up with DBUnit. With maven in the mix, it's a huge hit to productivity, but at least I have it. What project doesn't need this??!?!

Prezi on decoding the Scala Map literal

2009-11-24T00:00:00-05:00

If you haven't checked out Prezi as a means to create presentations, you really should. They have rethought the entire user experience, and it's totally awesome. To give you a flavor of it, I created a presentation of my blog on Deconstructing Scala's Map Literal. I can't create audio in the format Prezi requires, so there's no voice over, but I think it still works.

Writing a URL Shortener in Scala - The Good, The Bad, The Ugly

2009-11-23T00:00:00-05:00

I finally got around to finishing Shorty, my url-shortener for my vanity short-domain, ❺➠.ws. I did the whole thing in Scala as a way to create a fully-functining application that I would use and that I could finish in my non-work time. Scala unequivocally made this task enjoyable and quick. J2EE, on the other hand, did not help one bit.

The Good

Scala

My Scala code is so much shorter and easier to follow than the Java equivalent. Consider this code that, given the request path, finds a controller to handle it, and then calls the appropriate method based upon the HTTP method:

route(path) match {
  case Some(controller) => {
    val result = determineMethod(request) match {
      case GET => controller.get(params(request))
      case PUT => controller.put(params(request))
      case POST => controller.post(params(request))
      case DELETE => controller.delete(params(request))
    }

It's highly readable, and very concise; the Java version would've required a lot more variables, some noisier control structures, and a lot more braces and parens.

ScalaTest

ScalaTest resulted in a lot more readable code than JUnit or TestNG would've. Because of Scala's syntax, the tests are also free of weird dots and "literate" syntax that isn't quite that literate.

it ("should respond to get for a URL that is known") {
  val controller = new OneUrlController(hasher,"738ddf")
  val result = controller.get(Map())
  result.getClass should equal (classOf[URL])
  result.asInstanceOf[URL].url should equal 
    ("http://www.google.com")
}

The delineation between "expected" and "received" could not be more clear. assertEquals just isn't the same. The latest version of ScalaTest has some BDD options that look really great.

The Bad

SBT

I really wanted to like SBT, and, while it's a billion times better than maven, it's still not as easy to use as I'd like it to be.

I like:

Building code and downloading dependencies are separate
The save/run-tests loop is very handy
JavaRebel + re-deploying the webapp on file save is very handy

However:

The test output is horrid; big huge stack traces
Constant OutOfMemory errors that it traps and then doesn't exit. I had to kill -9 SBT a lot
Still more complicated than shell scripts

I believe that a build tool should be a DSL for automating software development tasks, which means it should be more concise and easier to use than UNIX shell scripts. Ant, Maven, and SBT fail miserably at this.

While SBT is light-years ahead by using an actual programming language, I found it very difficult to customize. Part of this is that the scaladoc tool gives developers no help in documenting their API, but, when it comes down to it, Scala and Java are not system automation languages.

scaladoc

Scaladoc is nowhere near as powerful as Javadoc. It makes it very hard to document how to use your code. Scala should have a more advanced documentation system than Java, but it actually has a much more primitive one; even RDoc is better. Hopefully, as Scala's popularity increases, the tools surrounding it will improve.

The Ugly

J2EE Deployment

Deployment is an underappreciated aspect of why Rails is so easy to use; copy/push your code to the production server, tell it you are running in production, and go. With J2EE, you get NONE of this.

If you want to alter configuration based upon environment, you are entirely on your own. J2EE, Ant, Maven, and SBT give you no real help or support; you have to roll it yourself. I'm just shocked at this omission; J2EE is ten years old and still has not provided a solution for something that every project needs. Amazing.

Servlet Spec

Java 5 is at end of life. The latest released Servlet Spec still doesn't support generics and is still completely schizophrenic about it's API (some methods use Enumeration, some use arrays, some use Iterable. Ugh).

The 3.0 spec looks slightly more sane, but it really doesn't do us any favors. web.xml is a trainwreck of stupidity, there's zero support for conventions, and the whole thing just feels like a solution designed for a problem that few of us ever have.

Make your own literals in Scala

2009-11-14T00:00:00-05:00

Following up from my post on Deconstructing the map literal in Scala, I thought it might be fun to go the other way around, and create a few literals of own. Since Scala provides the language features to enable literal-like syntax, and not the actual literals themselves, we can do a lot of stuff to reduce the amount of typing we have to.

In your production code, you probably wouldn't use a lot of literals, regardless of language support. However, in test cases, they are much more common. If creating objects in your tests becomes easier, you will tend to write better tests.

Suppose we are writing a Twitter app (that's all the rage these days, anyway). We might define a TwitterUser class like so:

class TwitterUser(val username:String) {
  def url = "http://www.twitter.com/" + username
  def recentTweets = // imagine some code here 
}

We might write a few test cases:

val me = new TwitterUser("davetron5000")
assertEquals(10,me.recentTweets.size)

val fake = new TwitterUser("davetron5001")
assertEquals(0,fake.recentTweets)

and so forth. It's going to get a bit tedious writing new TwitterUser over and over again. We could just make a shorter-named method in our test class, but why not use the apply() sugar along with a singleton object to make our own literal?

// Along with our TwitterUser class def
object @@ { // "@" is a reserved word :(
  def apply(username:String) = new TwitterUser(username)
}

// back to our test code

val me = @@("davetron5000")
val fake = @@("davetron5001")

It's a small bit of code, and we've saved some typing without sacrificing readability, since we take advantage of the way Twitter refers to users via the "@" symbol.

If our app is all about Twitter, we can go a step further and just rename TwitterUser to @@ like so:

// This replaces the TwitterUser class 
// and @@ singleton object
case class @@(val username:String) {
  def url = "http://www.twitter.com/" + username
}

This takes advantage of (the unfortunately named) case classes and now our Twitter user class and literal-like factory are called @@. This allows, among other things, some more readable case statements.

user match {
  case @@("davetron5000") => "it's you, dude"
  case user:@@ => "it's someone else"
}

Now, you certainly don't want to do this for every object in your domain (you'd get a horrible mess of symbols), but for key objects that you are using everywhere, an appropriately chosen literal syntax can make your code very clean and easy to read and modify.

Deconstructing the Scala Map Literal

2009-11-12T00:00:00-05:00

I find that Scala is one giant Rube Goldberg Machine that manages to do something not easily be done otherwise. By this I mean that Scala has many features that, by themselves, seem very strange, but, in combination, enable some very cool functionality. This is why I initially started my personal tour of Scala. I read stuff like explicitly typed self-references and was left scratching my head.

I thought it might be fun to deconstruct the "map literal" in Scala and observie how the features interact to create a very handy piece of code that isn't baked into the language. This assumes and understanding of some Scala basics.

Although Java 7 is getting map literals, Scala already has it (or so it appears):

val band = Map("Dave" -> "Bass",
                "Tony" -> "Guitar",
                "Greg" -> "Drums")

This is not actually a literal, but enabled by Scala features to make it look like a literal. Which means that you can use these facilities to make your own literals. So, how does this work?

Most surprising to a Java programmer is the -> operator. This makes use of two Scala features:

It turns out that the -> operator is on the class Predef.ArrowAssoc. Predef is automatically imported in every Scala program, so you don't need to prefix anything with Predef. It returns a tuple of its caller and its argument, e.g.

val dave = new ArrowAssoc("Dave")
val entry = dave -> "Bass"
// entry is now ("Dave","Bass")
// which is a Tuple2[String,String]

Of course, we aren't creating ArrowAssoc instances anywhere, so how does this get called? This is where implicits come in. Suppose we change our simple example to:

val dave = "Dave"
val entry = dave -> "Bass"
// entry is still ("Dave","Bass")
// which is a Tuple2[String,String]

Here, Scala sees that the method -> needs to be called on an ArrowAssoc, but is being called on a String. Instead of giving up, Scala notices the method:

implicit def any2ArrowAssoc[A](x: A): 
  ArrowAssoc[A] = new ArrowAssoc(x)

This means that anything at all can be converted into an ArrowAssoc if there's some reason to. And we have a reason to here.

This means our code is now effectively:

val band = Map(("Dave" , "Bass"),
                ("Tony" , "Guitar"),
                ("Greg" , "Drums"))

It's not hard to imagine a Map constructor taking Tuple2, using the first part as the key and the second part as the value, however where is the constructor? Scala creates objects via the new keyword, just as Java does. So, what's going on here?

This use two additional Scala features:

This is much simpler to decode than the -> method; there is simply an object in scope named Map, and it has an apply method that takes a variable list of Tuple2 objects. Scala interprets a method-call syntax on an object, but lacking a method name, as a call to the apply method of that object (if it exists). So, removing this, we have:

val band = Map.apply(("Dave" , "Bass"),
                      ("Tony" , "Guitar"),
                      ("Greg" , "Drums"))

That's all there is to it! A few things to note about this:

Without the application of some Scala features, it's pretty ugly
The language itself didn't need to implement a special "map literal"; it simply combines smaller features in a way to make it appear as though it does. You can even create your own "literals" rather than waiting for the language to implement them

Why Github Can Open-Source Their Libraries

2009-11-01T00:00:00-04:00

One thing I love about Github is that they open-source a lot of their internal tools that power the site. What's interesting is that, unlike SourceForge, they open source little bits and pieces; tiny libraries that do one specific thing. These things are supremely useful (I use Grit and Jekyll quite often).

This is a huge benefit to them; their products become higher-quality through contribution, and their talent-pool increases due to their contribution to the community; they are positioned as a technical leader and social force in the development community. I've often wondered why more companies don't do this and what's really involved?

There are three main hurdles to overcome in order to do this:

Usefulness - do you have code that someone will find useful?
Legalish - are you comfortable giving away some part of your company's intellectual property?
Technical - does your technical infrastructure support extraction, collaboration, and re-integration?

Usefulness

I think comparing Github to Sourceforge makes this point very clear. While SourceForge does more than, say, BERT, it's just a big huge all-or-nothing proposition; Github has extracted small parts of their infrastructure useful outside the realm of "software project hosting", resulting in many useful, fine-grained libraries.

Legalish

The issue here is essentially to determine if the gains achieved by open-sourcing some of your code are greater than the competitive advantage lost by doing so. Again, the ability to extract small, focused, and useful pieces of functionality is key. Github isn't open-sourcing their entire infrastructure; just the parts that are really useful and not incidental to their success (though one may argue that their IP has nothing to do with their success).

Technical

Here's where things get interesting. Extracting a small, useful piece of technology from your application, without revealing any trade secrets or other IP can be a challenging task. Add to that the infrastructure needed to manage and incorporate contributions, and your technical infrastructure could be the main barrier to interacting with the community (and not the lawyers!).

My company struggles with this daily, as our tools were just not designed to make extraction and management of dependencies easy. Our problem is almost entirely based on tooling and technology choice (which I'll get into in a further post).

Conclusion

The advantages to open-sourcing some of your code are obvious: you can improve the quality of your codebase while improving your standing in the community (which enahnces your ability to attract top-talent). You just need to make sure your technical infrastructure is built for it, and that the lines of communication between the development staff and the legal staff are clear and functional.

That Github routinely open-sources bits of their code speaks volumes to the choices they've made as an organization, as well as their technical prowess.

An anecdote about Joel Spolksy at DevDays DC

2009-10-27T00:00:00-04:00

I had signed up to volunteer at DC's DevDays, however My Company decided to be a local sponsor. Part of that involed us having some time for a 15-minute presentation during the lunch break. We were told that other sponsors would be doing something similar. I worked up a brief presentation on how we improve our development process with each iteration. I figured it would be of interest and not be too (obviously) self-promoting.

We get there and find out that not only are none of the other sponsors doing this, but we can't go at lunch, because Joel very-much wanted these things to be "optional" and, The State Theater being near almost nowhere to go have lunch, everyone would be stuck there and "forced" to watch our presentation. So, we'd be going after the entire conference was over.

I have no problem with this at all; it's his/their conference after all. But, we probably wouldn't have bothered if we had known this ahead of time. Nevertheless, I tweeted to the #DevDays hashtag (showing up on the big screen between speakers) and looked forward to it anyway. As a member of a crappy local band, I've played some gigs to no-one, so it's no big deal.

For most of the day the end of the schedule looked like:

4:15 - JQuery
5:30 - Goodbye!
(unspecified showcase for OPOWER (formerly Positive Energy))

Toward the end, Joel came by, introduced himself, and got my info so he could mention it in his closing remarks. We discussed some logistics and he was very polite, but obviously had a lot going on.

While Richard Worth was finishing up his JQuery talk, I head to the back. The stagehand gets me set up and Joel is about to go deliver the "goodbye and thanks" speech. Here is my mostly accurate transcription of our brief discussion:

Joel: OK, I'm gonna go out, say goodbye, and then I'll play a song that goes for about 4 minutes. After that, you can head out and start
Me: No problem, can you or someone cue me when the song is almost over? [you couldn't hear much back there and I didn't know the song]
Joel: My experience running these things is that people will start leaving pretty soon, so once it looks like everyone's left, you can head out there
Me: Um, ok [mostly amused at his comment and a bit nervious]
Joel: [slighly chuckling] We try to have a strict separation of Editorial and Advertorial content at these
Me: [not quite parsing the word "Advertorial" and also thinking about the hour-long demo on FogBugz]

As I said, it was all good, and a few people DID stick around, but I found the whole thing rather amusing in retrospect. And I think he actually paid attention to about half of my talk!