“Don’t worry about it. Start. Begin,” he replied.

He was right. I had made a classic mistake and in doing so I was ignoring our work processes with my own pet project idea.  I had ‘gone huge’ with the idea from the very beginning. I was planning user accounts, graphs, how to handle variant covers (a recent favorite variant for Green Lantern 51), and complex relationships for keeping a single master library of existing comic issues that would be separate from each user’s collection and reading log. Don’t get me wrong, it is good to be passionate about your idea and to know what challenges may be in your future but do not let that get in the way of getting your project going. Start small. Do the simplest thing possible. Constantly adapt your plans to your application’s needs. Be Agile.

That day I stopped mulling about things like how to order numbers that sometimes had letters on the end (see Batman 700U) and took my project agile. I started by building the simplest thing possible. I got going with no users, just an entry form with blank fields for comic title, issue number and date. Since then I’ve added a pop-up date selector, notes fields, the ability to enter ranges at one time, different collection views and a few other features. It has been almost a month but I’ve only worked on my application for 15-20 hours, at least 10 of them in the last couple of days. The ComicReads application is usable. It rocks! You could clone the ComicReads github repo, rake create & migrate your database, run ‘script/server’, and be ready to log your own collection but I don’t expect you to do so.

The most exciting part is that I have stopped logging my comic reading on notecards this past weekend and am now exclusively using my new application on my virtual machine. I haven’t added users yet, switched to mysql or solved several sticky issues— but I’m on my way. By starting with the features I needed— the ability to log the simplest and most essential parts of my comic book reading— I’ve made my project more exciting. Now that I have a working version, I find myself trying to work on it more and more. One day I will add user support, grab a URL, get a better design, and throw the system up for public use. Not this week though, I’ve got too much else to do.

Go ahead and get started. Begin! Throw out the users, forget connecting your application to twitter or facebook accounts, don’t worry about which database you are going to need, just go for it. Start by solving the problems that make your application interesting.

“Tim, this is simple. Why are you writing about this?”

select count(*) from messages;

And it is simple. If this were MyISAM. See, MyISAM always stores the number of rows on the table header. So, whenever we ask “how many rows are there?”, it can just grab the count and return it. Not InnoDB.

In InnoDB (for internal reasons), the number of rows has to be counted. Every single time. One of the tables we were counting was over 1.2 million rows. On a small EC2 instance with no other queries/major processes, this takes 1 minute and 20 seconds. This is unacceptable.

We looked at another command that will estimate the number of rows, but according to MySQL, it can be off by up to 50%

show table status;

Being off by up to 50% is unacceptable, even if it is fast. So we kept looking. Most of the recommendations were “make a table that stores the count, and add to/subtract from it every time you do a write/delete from the actual table”. Also not helpful. We’ve got over a million rows, and this is a prototype that’s only been online for a few weeks. I refuse to double the number of writes to our DB.

John points out that since our table is so huge, we’ve already got indices on it. Why not make the query use one of them?

mysql> select count(*) from messages use index (index_messages_on_remote_created_at);
+----------+
| count(*) |
+----------+
|  1276831 |
+----------+
1 row in set (3.19 sec)

Win. The trick is to hint it at a specific index. So, if you’re getting poor response times from InnoDB count, it means you’ve got lots of rows that have to be counted one at a time. And since you’ve got a lot of rows, you have at least one index. Just pick an index that will never contain a NULL value, and tell MySQL to ‘use index (index_name)’. Also, thanks to Wallace, who commented on the MySQL Performance Blog

If I had to give an executive summary, I’d say Agile is a way to sustainably manage projects for the long-term, while still keeping flexibility for the short-term.

Now, that’s a great piece of marketing, suit-and-tie-speak. And it’s useless for actually figuring out how Agile works. You want to know things like: Agile makes it easy to apply today’s lessons learned, starting next week — not at the beginning of next-year’s project. This means the person with the product vision (probably you) can re-evaluate and completely change the direction of a project in under a week.

That’s not to suggest we’re wild cowboys, riding a code horse from feature to feature, shooting off wildly. We do this deliberately and methodically. This is where the “long-term” comes in. Any programmer with enough Attention Deficit Disorder can drop what they’ve been working on to start another new thing. However, this only works until around the 8-week mark. With code, once you go past about 8 weeks, you hit a wall and end up managing old code, rather than going forward. It’s tragic. We’ve all seen it before. There are three reasons:

• poor code
• poor communication
• poor planning/bad schedules

Up front, we work with smart people who produce good code. But code is tightly integrated; adding a feature to one part of the code can break another part. The only way to know something really works is to test it. Hand-testing code is monotonous, inconsistent, and unreliable; not a good use of a smart coder’s time. Instead, we build automatic tests to give us a heads up when something’s changed — and we deal with it on the spot, not in next month’s Q&A session with the review board. We don’t just know our code is good, we can prove it.

We make sure that there’s good communication among team members by having daily meetings. Every day, the coding team has a very quick meeting to discuss what they’ve been working on, what they’re doing next, and anything that’s giving them problems. We know you’re seeing value for every day we’re on the project. Also, it’s hard for problems to last long — when you’re continually looking for them. We also have regular client updates during the week. But don’t think we spend all our time in meetings; our daily is only 10 minutes long.

Finally, we plan for the project to change by making feature schedules with a client representative once every week, not once every month. Imagine driving a car with a steering wheel that only worked once every 15 seconds, and you’ll know why we plan every week.

Of course, we don’t let ourselves get tied down with planning. I’ve seen rooms of people make a hash of planning the simplest of features. Want to see 30-man-hours spent on deciding if the login button goes *right there* or *two more pixels to the left*? Not if you’re paying for those man-hours. We’re all about simple planning — but that’s my next post.

Tim has been selected to do a presentation at acts_as_conference on Saturday afternoon about OAuth. He is in good company— speakers include: David Heinemeier Hansson, Yehuda Katz and Jon Larkowski. Since OAuth is a really cool thing that not too many people know how to implement, Tim will be doing a pretty introductory talk on the subject. From Tim’s presentation description:

Tim will talk about OAuth integration into Rails projects, cover the basics of OAuth for beginners, show how to istall OAuth to protect an existing project, how to protect controller actions with OAuth, and how to accept and use OAuth tokens from other providers.

Sounds like fun. Dave, Adrian, and I will also be attending the conference. Who else is going to be there?

There is a very interesting Agile idea called an Iteration Zero. An iteration zero can be used in many different ways. You can spend the time setting up the code and project management by getting ticket and milestone management, source control, build scripts, deployment scripts, testing frameworks, and integration processes into place. Or it can be used to identify, estimate, and prioritize features.

I think its really important to remember that it is called “agile” development because it is adaptable. We didn’t exactly end up doing an Iteration Zero sprint week but did meet with Iteration Zero goals in mind. We did a few pair consulting days to flesh out our understanding of the system, create charts showing the flow of information, and talk through the technical aspects of many of the components in detail. We were able to take notes that will enable us to break the components down into small sized tickets. We also identified several small areas that we needed to do some research to fully understand their impact on the project. We even took one complete component and broke it down into tickets, all with estimated complexity ratings of two days or less.

Keeping the whole process “agile”, we’re going to do a few more pair days at the beginning of the upcoming week. We’ll be breaking up more of the components into tickets to help our new client and ourselves to get a handle on where we think complexity lies within the system and build out a road map for getting to a basic beta.

The most basic goal of agile development is to provide the most business value possible at every step. By planning down to a level of granularity that we are already familiar with (two day or less tickets) and identifying risky areas of the development process for this specific system we are able to provide a good high level view of the time line for the first stage of development. Based on this information we can then shuffle the tickets and add or prune features to make sure that release one tasks include only the essential pieces to achieve our client’s product dream. That is the most business value we could ever provide.

Besides the website and the Firefox extension, Zentact has some really cool integration features with other services such as Gmail and Yahoo Mail. Since we know you’re thinking about people when sending them email, it seemed like the perfect place to let you add and edit your contacts and their tags. Just take a look at this picture.

Thats right, for every complete email address you type into the To, Cc, or Bcc field– Zentact adds contact fields to the form so that you can add new contacts to Zentact and tag them, or like the picture above it pulls in data for your existing contacts. The right in your face integration makes it so easy to remember to do, so if you’ve got a Gmail or Yahoo Mail account, give it a try. We’re also integrated in LinkedIn.

If you aren’t using Zentact yet, sign up here, using this code will even make me your first contact. Happy networking!

There are lots of people talking about it today:

• All those crazy folks on Twitter
• ReadWriteWeb
• Mashable
• Howard Lindzon
• VentureBeat

Check it out and let us know what you think, the system is still invite only but the links in this post are set up to let you right in.

If they need even more convincing, talk to them about the speed at which the Rails group is adding amazing new extremely useful features. Default Scoping, Only & Except Routing, Object.try and even better Conditional Get Support are all post 2.0 rails features that have been released in the recent months. (Ryan Daigle’s blog “Ryan’s Scraps” is a great place to keep up with the newest Edge Rails features, even the official Ruby on Rails blog links to his new feature explanations.)

Need even more convincing? Tell them about the stunningly active Rails community of developers: the blogs, the podcasts, the screencasts, the conferences and the user groups. Even the IRC channel rocks (#rubyonrails on freenode). A community this involved is sure to thrive.