Thursday, January 28, 2010

Recovering Deleted JPEGs from a FAT File System - Part 7

Part 7 in a series of posts on recovering deleted JPEG files from a FAT file system.

After a hiatus due to the holidays, personal matters, and work-stuff I'm ready to continue the FAT Recover project. In this post I'll

  • finally demonstrate the two principal assumptions that this project is based on.
  • actually recover deleted files using a manual approach.

Follow the "read more" link for the detailed discussion.

But first a mea culpa - I've discovered that the code for long file name support in post 5 is utterly broken. The code works for files that only use a single long file name entry but does not correctly process file names spanning multiple entries. For now, I'll side-step the issue and post a fix at a later time. I'll also update post 5 to warn future readers. Apologies for the error - that's the danger of hacking in the wee hours and minimal unit testing.

Wednesday, January 20, 2010

Book Review: The Simplicity Cycle

A couple of weeks ago I came across a link to the book The Simplicity Cycle by Dan Ward. It's available as a free download so I grabbed a copy and finally got the chance to read it.

The book begins with the argument that projects follow a curve in the two-dimensional space formed by the qualities "goodness" and "complexity". Initially, all projects start off at the origin with no complexity or goodness. As they progress, complexity and goodness increase together as the initial problems are understood and suitable solutions created.

The book's primary thesis is that eventually all projects reach an inflection point where further increases in complexity yield less goodness, not more. To improve goodness beyond this point, the focus must switch from increasing complexity to reducing it. These transitions from simple, to complex, and back to simple again form the "simplicity cycle" which, when successful, results in "an elegant, graceful, and streamlined solution" capable of representing "tremendous complexity" while being "at once profound and breathtakingly simple". Inspiring stuff.

Not surprisingly, the author asserts that increasing complexity after the inflection point results in an undesirable outcome. For software, this could mean an application with too many features, options, or ways to accomplish tasks. For decision making, this could mean having too much information to draw an effective conclusion.

Unfortunately, the non-linear relationship between complexity and goodness can go unrecognized - from page 26:

One pitfall that designers, engineers, and academicians may fall prey to is the belief that continuing to increase complexity … will continue to produce increases in goodness. It just isn't the case.

In some cases, this mistaken belief can be due to a fascination with complexity itself - here goodness doesn't mean utility to the consumer but rather intellectual gratification to the producer. Also quoting from page 26:

This is genesis-gone-wild, cancerous growth. It is the product of “over-learning” and smugness, where people fall in love with complexity.

In other cases, the complexity is mastered but without achieving the deep insight needed to reduce the subject to its fundamental problems and solutions - to my mind a kind of mechanistic expertise. Quoting from page 30:

It generally means we are over-thinking a problem or over- engineering a solution. Think of it as achieving “the complexity on the other side of understanding,”

Here we find the learned academician who everyone assumes is brilliant because nobody can understand a word he says. In fact, his academics may simply be complicated and have very limited goodness. All those extra squiggles are not very useful, no matter how complex.

The book admits the difficulty of knowing when the inflection point has been reached. Guessing too late results in over complicatedness as has been discussed. However, guessing too soon results in premature optimization - an equally bad outcome as it fails to reach optimal goodness. Unfortunately, the author asserts that a method for reliably identifying the inflection point doesn't exist.

The book makes the important point that the complexity phase cannot simply be skipped. Quoting from on page 46:

However, the simplicity in this region is built upon an essential foundation of earlier complexity. We can’t just jump from simplistic to simple, skipping the complex entirely. The initial increase in complexity is as crucial to maximizing goodness as the later decrease in complexity.

The book concludes with examples of good and bath paths through the complexity vs. goodness space. I found these helpful for reflecting on how the simplicity cycle relates to past experiences.

This book was a pleasant surprise. Although short and free, I found it thought provoking and insightful. In particular, the "simplicity cycle" concept matches well with my experiences. When developing software, I often find that after writing a fair amount of code common patterns emerge that allow for a simpler design. Conversely, I've witnessed first hand the disastrous outcome of projects that fail to simplify after the inflection point. Although these experiences have given me a healthy fear of complexity, this book provided a context for that fear and a framework for managing it.

My only criticism of the simplicity cycle concept is that it may encourage the kind of over generalization that Joel Spolsky warns against in his essay "Don't Let Architecture Astronauts Scare You". Quoting from Joel's essay:

When you go too far up, abstraction-wise, you run out of oxygen. Sometimes smart thinkers just don't know when to stop, and they create these absurd, all-encompassing, high-level pictures of the universe that are all good and fine, but don't actually mean anything at all.

These are the people I call Architecture Astronauts. It's very hard to get them to write code or design programs, because they won't stop thinking about Architecture. They're astronauts because they are above the oxygen level, I don't know how they're breathing.

The point I take away from this is that simplification is not the same thing as abstraction - simplification remains focused and actionable while abstraction becomes vague and unactionable. Unfortunately, I think many mistake generalization for simplification but now that I'm consciously aware of the difference I'll be less likely to fall into the trap myself.

Saturday, January 16, 2010

New Year, New Role

With the New Year I took on a new position at work - same kind of role leading an advanced development team but with a much expanded sphere of responsibility. While exciting, this change plus some personal matters has left less time for blogging.

I expect to recover my blogging time this week and plan to finish up the FAT Recover series by the end of the month.

The bright side to this interruption is that it helped me realize how much I enjoy blogging. According to Google Analytics, this blog isn't setting the web on fire but regardless the personal benefits have more than justified the effort.

I have a lot of plans for this blog in 2010 and I intend to set aside the time required to remain active. That said, family and work take higher priority so such interruptions are probably inevitable - who knows what the year will bring.

Wednesday, January 6, 2010

Book Review: Organizing Genius

Organizing Genius: The Secrets of Creative Collaboration by Warren Bennis & Patricia Ward Biederman

I first read this book a few of years ago and recently reviewed it in preparation for a new role at work. As before, I found it to be an extremely insightful resource on the attributes of great, innovative teams.

The majority of the book provides accounts of the following "great" groups:

  • The Walt Disney Studio
  • Xerox PARC
  • The Apple Macintosh Team
  • The 1992 Clinton Campaign Team
  • Lockheed's Skunk Works Group
  • Black Mountain College
  • The Manhattan Project

As stated in the introduction, the authors attempt to systematically examine these groups in the hope of identifying their "collective magic". In the final chapter they summarize their findings into the following 15 points.

  1. Greatness starts with superb people.
  2. Great Groups and great leaders create each other.
  3. Every Great Group has a strong leader.
  4. The leaders of Great Groups love talent and know where to find it.
  5. Great Groups are full of talented people who can work together.
  6. Great Groups think they are on a mission from God.
  7. Every Great Group is an island - but an island with a bridge to the mainland.
  8. Great Groups see themselves as underdogs.
  9. Great Groups always have an enemy.
  10. People in Great Groups have blinders on.
  11. Great Groups are optimistic, not realistic.
  12. In Great Groups the right person has the right job.
  13. The Leaders of Great Groups give them what they need and free them from the rest.
  14. Great Groups ship.
  15. Great work is its own reward.

Generally speaking I agree with all of these points. The closest I've come to working in a Great Group was my first job building a big-iron ccNUMA machine. That group and experience had many of the elements listed above and a decade later I recognize how rare of an opportunity that was. I suspect others from the team feel the same way based on the reminiscing we do whenever any of us cross paths.

Clearly the leaders of Great Groups are responsible for creating an appropriate environment for the team to flourish. In my role as the leader of an advanced development engineering team I've tried my best to follow this guidance with I think moderate success. As my role and responsibility expands I need to place greater focus on creating the right environment - to that end this book will be a valuable resource.