Getting data to the cloud

One of the problems facing cloud computing is the difficulty in getting data from your local servers to the cloud. My home Internet connection offers me maybe 768 Kbps upstream, on a good day, if I’m standing in the right place and nobody else in my neighborhood is home. Even at the office, we have a fractional T1 connection, so we get something like 1.5 Mbps upstream. One (just one!) of the VM images I use for testing is 3.3 GB. Pushing that up to the cloud would take about five hours under ideal conditions!

I don’t know what the solution to this problem is, yet, but it’s definitely something a lot of people are working on. I thought I’d point out a couple of interesting ideas in this area. First is the Fast and Secure Protocol, a TCP replacement developed by Aspera and now integrated with Amazon Web Services. The basic idea is to improve transmission rates by eliminating some of the inefficiencies in TCP. In theory this will allow you to more reliably achieve those “ideal condition” transfer rates, and if their benchmarks are to be believed, they’ve done just that. However, all this does is help me ensure that transferring my VM image really does take “only” 5 hours — so I guess that’s good, but this doesn’t seem like a revolution.

From my perspective, a more interesting idea is LBFS, the low-bandwidth filesystem. This is a network filesystem, like NFS, but expressly designed for use over “skinny” network connections. It was developed several years ago at MIT, but I hadn’t heard of it until today, so I imagine many of you probably haven’t either. The most interesting idea in LBFS is that you can reduce the amount of data you transfer by exploiting commonalities between different files or different versions of the same file. Basically, you compute a hash for every block of every file that is transferred, and then you only send blocks that haven’t already been sent. On the client side, it takes the list of hashes and uses them to reassemble the file. This can give you a dramatic reduction in bandwidth requirements. For example, consider PDB files, the debugging information generated by the Visual C++ compiler: every time you compile another object referencing the same PDB, new symbols are added to it and some indexes are updated, but most of the data remains unchanged.

Like I said, I don’t know what the solution to this problem is, but there are already some exciting ideas out there, and I’m sure we’ll see even more as cloud computing continues to evolve.

Public Clouds

Today we announced integrations and compatibility with public cloud computing – specifically Amazon EC2. Cloud computing is a hot topic right now, and rightly so. It provides an easy to deploy, cost-effective, scalable, on-demand computing infrastructure –very timely, given shrinking or frozen IT budgets. I can’t count the number of customers who tell me that compute infrastructure is their #1 bottleneck. At Electric Cloud we have years of experience with internal or “private” clouds (after all, it’s in our name). We help customers set up private clouds, some with hundreds of machines, to accelerate and automate their software build and test tasks. It made sense for us to add public clouds to the mix. You can read the press release here.

Our customers gave us some interesting use cases for using our products in combination with the public cloud. Here are some of their ideas:
Read the rest of this entry »

Java One was Cloud Crazy

I went to Java One last week and was a bit surprised by the amount of cloud chatter. I expected to hear a lot about Larry Ellison and Scott McNealy on stage together. I expected melancholy predictions that this would be the last Java One. I did not expect to hear everyone talking about cloud computing. Don’t get me wrong, I knew that Sun was going to talk about cloud computing. The Sun booth had several stations dedicated to the subject and the Sun store was selling cloud t-shirts. The thing that got me was how many participants were talking about the cloud. All of the technical sessions that had anything to do with cloud computing were packed. Even more surprising was the fact that the sessions extolled the virtues of Amazon EC2 even though Sun is working on their own Cloud offering. I sat through three sessions where the first half hour was dedicated to “Amazon 101”. It looks like Amazon is where real solutions are being created for now but there is definitely competition on the way.