The Colorado Software Summit this year was a blast. Bryan, Lisa and I learned enough to make our heads spin. We drove through a bit of snow and ice from Denver to Keystone but once we arrived we were graced with very beautiful and cold weather. I took a few pictures as usual. I think that I was a little unclear if there would be an underlying theme like last year (SOA) but I would probably pin scalability and interoperability as the general topics of the event this year. Lot of people were talking about virtualization, concurrency with todays multi-core environments, general scalability, and interoperability of web services, data formats etc.

Dan Pritchett from eBay had to be one of the more impressive speakers this year in my opinion. In many ways (but certainly not all) we share a lot of scale issues with eBay so hearing Dan talk about seemingly odd architecture decisions (and the slaying of various sacred cows) really resonates with me. Just to pick a few oddball concepts they use in various forms at eBay:

  • No client-side database transactions.
  • Database partitioning and sharding.
  • 100% stateless application tier.
  • No 2 phase commits. Don’t worry about the order or failures, clean them up later.
  • Only use stored procedures for ETL-like routines.
  • Use lots of small soldiers instead of large soldiers. (commodity hardware, not expensive branded stuff)
  • Virtualize wherever possible so horizontal scaling is more possible.

Dan sat with us at lunch one day and he’s a simple yet captivating speaker. I talked to him for a while after the Thursday night BoF he held on the eBay architecture. We talked about monitoring and I learned a few fascinating things that have really made me think differently (or reaffirm some existing hunches) about some of the architecture decisions I’ve made on many of the applications I have managed or currently manage.

Want a few crazy facts about the eBay architecture?

  • 1 Billion photos
  • 1 Billion page views / day

  • 26 Billion SQL queries / day
  • 100 Million items for sale
  • 99.94% availability
  • 1 terrabyte of log files generated per day
  • around 100 code branches are active at any time
  • 15,000 application servers + 300 database servers
  • 300 million front-end searches per day (plus tens of millions of internal searches) == more than Google

I also learned a bit more about Continuum, ActiveMQ and Groovy.

Matt Raible (as usual) gave a couple of great talks about Java web application frameworks. During Scott Davis’s talk on ATOM and REST, Bruce Snyder made a comment that he used JMS as a SOAP transport. Fascinating. Why don’t more people do that? All the message reliability you’d expect out of a message broker (guaranteed delivery, indemnity etc) with the contract of SOAP. Great idea. AMQP also seems like a great idea for message bus architectures. If you’re at all having to deal with bus interoperability, bus vendor lock-in or anything similar, give AMQP a shot.

I went to another Scott Davis talk on Grails. Grails is a new web application framework using Groovy that has borrowed a lot of rapid development techniques from Rails. The awesome thing about Grails is that is uses all of the leading Java tools like Spring and Hibernate. I have been playing around with it on and off for the last few months and I really like it so far. I’m really interested in seeing is mature more though.

Speaking of ATOM and REST, I like the philosophy of REST but I like that WSDL has basically standardized contracts for SOAP services. I’m not saying that I think WSDL is perfect by any means, but it works. I don’t usually need to read it because it’s automatically generated and automatically consumed by most implementations I manage. REST offers no such standard right now for contracts. In order to integrate with a RESTful service I either have to read the spec documents (assuming they exist) or I have to guess as to the contract. Should I ask for “/San_Antonio” or “/San+Antonio”. Read the GData spec. It’s ATOM but it still prints out to dozens of pages. Then what data type is returned? I’d have to make a request, inspect the returned XML and build a parser. Hopefully it’s supplied a schema. What a pain. And this is supposed to be better than SOAP. If a SOAP service provides a WSDL I just point my wsdl2java client to it (or equivalent in whatever language or toolkit I’m using) and I’m literally integrating against that interface within a few seconds. The WSDL provides all the contract I need to work.

I also went to a talk by Gregor Hophe of Google (and author os the awesome Enterprise Integration Paterns book) where he demonstrated a really cool and simple integration of the Google Mashup Editor and Yahoo Pipes. I’ve been using Yahoo Pipes for several months and as a few coworkers can attest, I really love it. Think of it as all the fundamental UNIX utilities (pipes, sed/awk, grep, tee, sort, split etc) but for RSS/ATOM feeds. I have it take a handful of very chatty RSS/ATOM feeds, combine them, filter for certain topics and spit them out as a single feed. That mini-app too all of 45 seconds to create. Gragor’s example took a Google Calendar feed, fed it into Yahoo Pipes, did a little massaging of the format, added Geo Location data and spit the result back out to a feed. That feed he sucked into the Google Mashup Editor (new service, I just got my account today) and integrated it with a Google Map so he could see the location of his calendar events. It was a really slick lightweight integration technique.

Speaking of Google, the Google Gears demo was very cool. I know of a few people who are already getting interested in this project.