In my introductory post, I mentioned that Go Daddy does things at a scale that hasn’t been achieved by many other technology companies. Here are a few examples:
- Our systems deliver about 500 million emails a day to our customers (that’s the number delivered after filtering out the spam)
- Our SSL validation systems answer the question, “Is this SSL certificate revoked?†close to 3 billion times a day
- Our DNS systems respond to about 10 billion requests per day
Those are some big numbers. There have been many scalability lessons learned along the way (and I’m sure we have more to learn). One of the most basic concepts has been applied to all three of the systems mentioned above. In Scalability Rules, by Marty Abbott and Michael Fisher, it’s Rule #9. At Go Daddy, we coined the term “podular†to describe it. Each of the systems listed is designed to be scaled horizontally in identical “cookie cutter†pods. These pods stand alone and can be geographically distributed. The result is not only a system that’s relatively easy to scale, but also one that has fault isolation (Rule #36) and can be located close to the end user to improve performance.
Simple? Yes, but it’s also an essential tool in the scalability toolbox.
One issue that comes up with a podular architecture is determining how to expose a bunch of pods to users as a single system. We haven’t found any one perfect solution and, in fact, employ at least three:
- Our DNS uses IP Anycasting, a network routing methodology that sends traffic to the nearest pod on the network. Although it works well for DNS, it has some pitfalls. Traffic is routed based on Internet topology, not on geography. So, you can end up with more network latency for some users. It’s also susceptible to Internet routing changes that can interrupt connections.
- Our email system uses redirection. This is the simplest approach. However, the client has to be a browser or something that understands HTTP redirects. POP and IMAP clients typically don’t support the concept of a redirect. Redirection can also lead to a bottleneck and single point of failure at the service to which the client initially connects in order to get redirected.
- Our SSL validation system uses DNS geolocation. In this approach, a customized DNS system is smart enough to use the client’s IP address to determine the pod to which the user should be sent. This works well until you introduce a proxy that masks the real IP address of the client. Then the DNS system can end up sending too much traffic to a single pod or it can make bad geographic decisions.
Each solution has a sweet spot. So, I suspect that we’ll continue to use all three at Go Daddy.
Good post Wayne – always interesting to see what goes on behind the scenes at a giant corp like GoDaddy.
On the topic of your DNS and SSL methodologies – I encourage you to check out what we’re doing at Cedexis; the caveats you encounter with DNS are very common, and we have developed some very cool, very fast ways to work around them.
Keep up the innovation!