Tao Of Foo

A few weeks ago Steve Yegge posted an article about code base size and it’s negative effect on projects.  While I agree that his example of a 500k LOC project is horrid I’d have split some hairs here and say that lines of code (or bloat as he refers to it) isn’t the problem.  The problem is a horrible application architecture and lines of code is just one of the symptoms.  Other symptoms may include execution speed, memory consumption, lack of encapsulation, security vulnerabilities or a host of other issues.  I’m curious as to what political/managerial/architectural situation arose that allowed a single application code base to grow so large.

Do you have an application architect? Why didn’t the application get carved up into distinct loosely coupled systems?
Do you have sufficient business representation that prevents the “just add it to application X” problems?  How does the scope of an application grow so much with no oversight?

The application bloat I’ve seen typically takes three forms:

1. Very little code reuse is taking place.  How many separate logging infrastructures do you have?  Using multiple ORM tools in the same project?  How long does it take a developer to understand all that?
2. Reinventing the proverbial wheel.  This is loosely related to #1 but I’ll call it out here because it’s so important.  If you wrote from scratch any of the following for your project you need to seriously justify your decision IMHO: ORM, logging, web framework. If your language of choice doesn’t already include suffucuent choices for these utilities then you need to reconsider your platform choices.  Avoid the “not built here so I don’t trust it” syndrome.
3. Feature-itis.  Knowing when to say “no” is a very valuable skill.  Knowing when to say “yes” conditionally is also important.  It’s ok to add that new feature but require some refactoring time so it doesn’t just add to the collective mess.  Think of architectural trade offs as a karma based system.  In other words for every shortcut you take now you make things harder on yourself later.  For example: I can add your feature in 10 days with 40% code duplication (+ 20 days of cleanup later and every other coding task has to slow down to work around this ugliness) or I can add it in 20 days with 0% code duplication (and little to none of the other slowdowns).  It’s acceptable to tell the business that the feature, if done properly, will take 20 days to implement.  In fact it’s your duty to recommend the slower approach.  If you have a boss who doesn’t understand those trade-offs you need to start looking for another job.

I’ve seen (and worked at) both kinds of mistakes and they all contribute to code bloat but that isn’t their biggest problem.  Bringing new talented developers up to speed on the whole system is a monstrous adventure.  No single developer understands enough of the entire codebase to affect significant change.  The systems become so brittle over time that making small changes involves enormous regression testing challenges.

You are a developer.  It’s your duty to protect the integrity of the code base you are working on.  That’s part if what your company is paying you to do.  Yes there will be pressures to deliver faster.  Yes there will be pressures to do things that violate your developer principals.  That’s part of life.  Accept it.  Now that you can take a deep breath what are you doing to protect your code base?  Do you have a solid architectural plan that you can show to your PHB that lets him know how this feature fits into the greater system?  Do you have at least some high-level design documents that allow you to justify your position?  Are you prepared to defend your position to your stakeholders.  “While it’s technically possible to add feature X in 10 days that will have a negative net impact on the whole project and here’s why…”.
Also, don’t think that merely switching to/from a waterfall, agile etc shop will magically fix these issues.  It won’t I’ve seen both kinds of development shops make all of these mistakes.

The tail end of 2007 has brought with it a series of both good and bad news.

In October we learned that Amy was pregnant. Shortly thereafter her morning sickness started in earnest. At one point it was bad enough for us to spend an evening in an emergency care facility. Another time her doctor admitted her to the hospital for a few days. Thankfully is has been subsiding though lately for us to enjoy a wonderful Christmas. With some help we pulled off a spectacular feast complete with smoked turkey, cornbread and shrimp stuffed acorn squash and roasted french green beans.

In the last few months Ben has answered answered every Christmas present query with a puppy. Amy and I weren’t quite sure how to answer that request other than to say that Santa makes toys, not puppies. Then we took Daisy (our 8 year old Golden Retriever) to the vet to have a lump on her back looked at. They removed a handful of cancerous tumors from inside her back, neck and leg. X-Rays showed that the cancer has spread to her lungs and lymphatic system. Bottom line, she has roughly 6 months to live. Now we are thinking that a puppy sounds like the perfect companion to help him through losing his best friend. Santa brought (among many other cool things) a dog Encyclopedia so he and Amy have spent several days pouring through it to search for the perfect puppy. Looks like we’ll be looking for a Corgi in a few months. For now though Daisy is fine and has recovered from her surgery with flying colors.

Here’s to a strong finish to 2007 and a wonderful 2008. It may have a few rough spots but we’re looking forward to a new puppy and new baby.

I guess the title is a really geeky way of saying … we’re pregnant. Or more accurately Amy is pregnant and she’s due in mid-June.

Her morning sickness is not at all limited to the mornings but it not nearly as bad as it was with Ben. We’ve already been to an emergency clinic for an IV and she’s been on several different medications. A particular combination of them seems to be mostly doing the trick lately but she’s still having a pretty rough time of it all. It’s been bad enough that she has decided to put the business on hold until she can offer the quality photography and customer service she is committed to deliver.

Anyway, we are all happy and excited. Ben is hoping for a little brother and is already picking out names. Bob and Santa Claus are his top picks at the moment.

If you call us and we don’t answer or we don’t return your call right away, please bare with us.  It’s not that we don’t want to hear from you, it’s just not easy for Amy to talk on the phone when she feels so bad.

The Colorado Software Summit this year was a blast. Bryan, Lisa and I learned enough to make our heads spin. We drove through a bit of snow and ice from Denver to Keystone but once we arrived we were graced with very beautiful and cold weather. I took a few pictures as usual. I think that I was a little unclear if there would be an underlying theme like last year (SOA) but I would probably pin scalability and interoperability as the general topics of the event this year. Lot of people were talking about virtualization, concurrency with todays multi-core environments, general scalability, and interoperability of web services, data formats etc.
Dan Pritchett from eBay had to be one of the more impressive speakers this year in my opinion. In many ways (but certainly not all) we share a lot of scale issues with eBay so hearing Dan talk about seemingly odd architecture decisions (and the slaying of various sacred cows) really resonates with me.  Just to pick a few oddball concepts they use in various forms at eBay:

• No client-side database transactions.
• Database partitioning and sharding.
• 100% stateless application tier.
• No 2 phase commits. Don’t worry about the order or failures, clean them up later.
• Only use stored procedures for ETL-like routines.
• Use lots of small soldiers instead of large soldiers. (commodity hardware, not expensive branded stuff)
• Virtualize wherever possible so horizontal scaling is more possible.

Dan sat with us at lunch one day and he’s a simple yet captivating speaker. I talked to him for a while after the Thursday night BoF he held on the eBay architecture. We talked about monitoring and I learned a few fascinating things that have really made me think differently (or reaffirm some existing hunches) about some of the architecture decisions I’ve made on many of the applications I have managed or currently manage.

Want a few crazy facts about the eBay architecture?

• 1 Billion photos
• > 1 Billion page views / day
• 26 Billion SQL queries / day
• 100 Million items for sale
• 99.94% availability
• 1 terrabyte of log files generated per day
• around 100 code branches are active at any time
• 15,000 application servers + 300 database servers
• 300 million front-end searches per day (plus tens of millions of internal searches) == more than Google

I also learned a bit more about Continuum, ActiveMQ and Groovy.

Matt Raible (as usual) gave a couple of great talks about Java web application frameworks. During Scott Davis’s talk on ATOM and REST, Bruce Snyder made a comment that he used JMS as a SOAP transport. Fascinating. Why don’t more people do that? All the message reliability you’d expect out of a message broker (guaranteed delivery, indemnity etc) with the contract of SOAP. Great idea. AMQP also seems like a great idea for message bus architectures. If you’re at all having to deal with bus interoperability, bus vendor lock-in or anything similar, give AMQP a shot.
I went to another Scott Davis talk on Grails. Grails is a new web application framework using Groovy that has borrowed a lot of rapid development techniques from Rails. The awesome thing about Grails is that is uses all of the leading Java tools like Spring and Hibernate. I have been playing around with it on and off for the last few months and I really like it so far. I’m really interested in seeing is mature more though.

Speaking of ATOM and REST, I like the philosophy of REST but I like that WSDL has basically standardized contracts for SOAP services. I’m not saying that I think WSDL is perfect by any means, but it works. I don’t usually need to read it because it’s automatically generated and automatically consumed by most implementations I manage. REST offers no such standard right now for contracts. In order to integrate with a RESTful service I either have to read the spec documents (assuming they exist) or I have to guess as to the contract. Should I ask for “/San_Antonio” or “/San+Antonio”. Read the GData spec. It’s ATOM but it still prints out to dozens of pages. Then what data type is returned? I’d have to make a request, inspect the returned XML and build a parser. Hopefully it’s supplied a schema. What a pain. And this is supposed to be better than SOAP. If a SOAP service provides a WSDL I just point my wsdl2java client to it (or equivalent in whatever language or toolkit I’m using) and I’m literally integrating against that interface within a few seconds. The WSDL provides all the contract I need to work.

I also went to a talk by Gregor Hophe of Google (and author os the awesome Enterprise Integration Paterns book) where he demonstrated a really cool and simple integration of the Google Mashup Editor and Yahoo Pipes. I’ve been using Yahoo Pipes for several months and as a few coworkers can attest, I really love it. Think of it as all the fundamental UNIX utilities (pipes, sed/awk, grep, tee, sort, split etc) but for RSS/ATOM feeds. I have it take a handful of very chatty RSS/ATOM feeds, combine them, filter for certain topics and spit them out as a single feed. That mini-app too all of 45 seconds to create. Gragor’s example took a Google Calendar feed, fed it into Yahoo Pipes, did a little massaging of the format, added Geo Location data and spit the result back out to a feed. That feed he sucked into the Google Mashup Editor (new service, I just got my account today) and integrated it with a Google Map so he could see the location of his calendar events. It was a really slick lightweight integration technique.

Speaking of Google, the Google Gears demo was very cool. I know of a few people who are already getting interested in this project.

I’m headed to the Colorado Software Summit again this year. If I learn half as much as last year I’d still think it was worthwhile. I have been to a number of other software development and techie conferences and this one is hands down my favorite. Here’s a few quick reasons why:

1. No vendor booths. No pressure, no commercialism, no spin. Just tech.
2. Every session is worth taking notes in. They are very techie focused. I bring my laptop and between sessions find myself playing with the stuff I just learned about.
3. Several of the speakers wrote books that I own or software that I use. Perhaps that speaks to the quality of the presenters or maybe to my taste in books and software.
4. Keystone, Colorado is incredibly beautiful. We received 18 inches of snow during the conference last year.

Last year many of the session topics covered Service Oriented Architectures and various tools often used to facilitate them. Message busses, web services, mashups etc. This year there looks to be a bunch of topics on web frameworks, architecture (SOA, scaling, monitoring, grids etc) and data. Good stuff.

Here’s a few links from last year:

• My photos.
• My blog post.
• One image from the CSS site. (Bryan and me in the background)

After a decade long career as a software developer, system integrator and architect, I am moving on. I’m going to work at a mall.

Well, technically our whole company is moving into an old mall. I’m doing the same job. It is just a lot more exciting to say that I’m “working at a mall now”. Rackspace is growing so fast and we have outgrown our available space in our current San Antonio offices and after evaluating a few alternative solutions (including building a Dell-like campus from scratch, moving various offices out of state etc) we have decided to take over the old (abandoned) Windsor Park Mall as our corporate headquarters.

Here’s a few interesting links related to Windsor Park Mall:

• http://deadmalls.com/malls/windsor_park_mall.html
• http://mallsofamerica.blogspot.com/2006/03/windsor-park-mall-final-call.html

Here’s a few articles covering the recent events:

• http://www.mysanantonio.com/business/stories/MYSA080207.rackspace.EN.d699cdf1.html
• http://www.mysanantonio.com/news/neighbors/northeast/stories/MYSA081507.ne15mallimpact.135885f.html
• http://www.governor.state.tx.us/divisions/press/pressreleases/PressRelease.2007-08-02.3728
• http://digdeepertexas.com/?p=582
• http://t1ra.multiply.com/links/item/186/MySAcom_State_kicking_in_22_million_for_Rackspace_move

Many people have asked me how bad my drive from Boerne will be. After driving it during the same morning hours that I normally commute it wonlt be much worse than the drive was when we were downtown. If you are ever presented with an opportunity to tour an empty mall I highly recommend it. There’s a certain eerie quality to it that I can only liken to feeling as if zombies were right around the corner.

• My pictures from our visit
• other Rackers pictures:

• http://www.flickr.com/photos/scottsimpson/sets/72157600457390248/

I want to be a iPhone fan-boy, I seriously do. The thing is I don’t have one yet so instead … I’ll play the role of “the hater”.

Reasons not to buy an iPhone (bare with me, I’m trying to talk myself out of buying one):

1. AT&T? Yikes, Amy and I left AT&T because of how bad the coverage is out here. I’m not going back any time soon.
2. No syncing from Linux. How hard would it have been to fix this? And the OSX syncing only works with the latest and greatest version of OSX. And forget about it under 64bit Vista. Too many limitations here for me.
3. Lack of useful apps. IM? Java? Flash? SSH? Encrypted notes?  I’m not saying that these kinds of apps are used by everyone but I’m so spoiled by my Treo and the sheer number of downloadable applications.
4. No insurance? AT&T isn’t offering any for the iPhone from what I’ve heard. Is that a $500 wad of uninsured breakable cash in your pocket or are you just happy to see me?
5. Phone locking up? Reports of lockups after 24 hours of use are starting to come in. These kinds of issues are typical for first generation devices and I’m sure Apple will issue a firmware update. Rev 2 and beyone will only get better in this regard.
6. No memory card support? The only reason I can think of is that Apple REALLY wants to control what and how things get loaded to this device. iTunes is the only way to sync anything to it. (see #2)

That’s it for now. I’m sure I’ll find more as time goes on. I’d love to play with one though to see if the screen is as beautiful as I hear. If some of these issues get resolved I’d seriously consider dropping some cash but until then I’ll just be jealous the iPhone hater.

Amy is currently attending the TPPA conference in Kerrville. This is the 3rd year she has attended but the first year that she has entered their print competition. This competition is quite difficult because the competitors are all high end professional photographers from all over Texas and surrounding states.

She took 4 prints with her yesterday and all of them scored very well for a first time entrant. This morning she checked on her scores and all of her prints did very well with one earning Honor of Exhibition (a very coveted placement especially for first time entrants).

Congratulations sweetheart. I’m very proud of you.

Update: It turns out that 3 of her 4 entries won the Honor of Exhibition status!

Ben and I are headed to the Davis Mountains in a few weeks. My parents are spending about 2 weeks out there doing a bunch of work around the cabin so Ben and I are going to help out for a few days. We’ll be replacing some decking, fixing and replacing some eaves and rakes, adding flashing to a few exposed exterior walls and several other projects. I can’t believe it’s been over a year since I’ve been out there. That’s way too long for me.

We might even make it to Balmorhea State Park on our way home. Ben has never been out there and it’s going to feel so nice after a couple of days of working hard up in the mountains.

To those readers who aren’t (yet) parents, I apologize. This post is likely to disgust you. This is however precisely what life is like with a 4 year old boy.

As Ben was getting dressed for bed this evening he made me the following offer while standing in his bedroom completely naked:

Ben: “Daddy, smell my finger.”

Me: “Why? What does it smell like?”

Ben: “Like hot peppers, fresh tomatoes and yummy milk.”

Me: “Um, no thanks.”