I just participated in a great webminar titled “Avoiding the Fail Whale“, where Robert Scoble moderated a group of guests to talk about scalability issues and planning.
The guests, which included Paul Bucheit, Founder of Friendfeed (and who also did a lot of the development work for Gmail), Dorion Carroll, VP of engineering at Technorati and Nat Brown, CTO of iLike, touched on many aspects of how they were able to scale their applications across hundreds of servers.
Some of the things that I found really interesting:
- Application design before actual implementation is critical to scale
- They are not fans of queuing, but rather prefer to use MemcacheDB, which to be honest, I had never heard of;
- Google has always preferred doing things custom and internally;
- Code maintenance is done using Subversion and tags;
- Monitoring is done using Nagios, and –
- When asked which database engine they prefer using, they unanimously said “MySql”.
A couple of additional tips mentioned by Paul were to “always keep thing as simple as possible” and to “partition the data in the smalles chunks possible”.
Here’s the chat log: and the video is coming soon.
It’s a very interesting space that Scoble and company are exploring; I do think we need more of these, as the economy and air fare becomes more and more out of reach for entrepreneurs.
One thing I found personally encouraging is that I use all of the technologies mentioned above, but learned about MemcacheDB for the first time, and this is what makes attending these webinars so valuable. I could very well be unto the next big solution for my scalability issues.
But most important of all, I connected with a dozen or so peers with whom I lookforward to continue the conversation offline.
Update: There’s a great thread still going on about scalability in Robert’s blog
{ 3 comments… read them below or add one }
I would like to add that webinars like these help a lot not only because they work against the fact that “the economy and air fare becomes more and more out of reach for entrepreneurs” but also against the fact that, for many of us, it’s not a matter of increasing difficulty: it’s a matter of practical impossibilities.
There are lots of things to be improved (like the Skype calls) but the gist of it is simply great: the opportunity of sharing knowledge and learning directly from the experts.
More from author
Jorge, I’m glad you found the webinar useful. It was really great to be able to share some experiences with folks like you and to be able to participate on a panel with the likes of Paul and Nat.
memcached is the open source cacheing system we referred to the most: http://en.wikipedia.org/wiki/Memcached
Another persistent cache we have started using at Technorati recently is memcachedb, http://memcachedb.org/, which is a memcached protocol compliant persistent store that uses BDB as a backend. This has the benefit of being able to copy, backup, replicate, or restart your cache without having to warm it up. memcached is great for volatile data, but if you want to keep things persistent, we’ve found memcachedb provides a great alternative.
More from author
@Andres, it’s true, we can benefit developers all across the globe.
@Dorion, you were very helpful, thanks for dropping by. Hopefully the video will be up soon, a lot of developer friends are asking me about it.
More from author