a parallel world
as you may or may not know, i have recently joined a new startup. we are in the business of storing all your data online and making sure the latest version arrives at the places you will be. needless to say, we are in the clouds. maybe because its the hype, or my software architect senses are tingling, but i’ve been looking very closely at parallelism recently. whether it is using memcached, simpledb, beanstalkd, or erlang, i find myself looking at how i can replace our current systems and build future systems with parallel or distributed solutions.
scaling out is definitely a sexy idea, especially getting nods from things like microprocessors with their multi-coredness. the idea is also natural. if you have a system that needs to perform the same function on many entities, then why shouldnt the solution mirror the problem? parallelism does have some limitations, namely latency (waiting for results to “come back”), redefinition of a problem, and new solution paradigms. we’re used to the sequential world. sequential is easy. we’re used to cause and effect. i remember having a bitch of a time doing processor design in verilog because at first i couldnt wrap my head around the fact that everything happened *at once*. the syntax was so C-like that i fell into my old imperative habits.
one logical step in the evolution of new ideas is that a group of people understand it so well that they make it accessible to the muggles. i had been thinking of how to use xgrid technology in a way that will turn my laptop into a super computer. according to the advertisement, this should be possible. unfortunately, like all things, there is no silver bullet. you get to submit jobs via a GUI or the commandline, but they warn you that you need to make your problems embarrassingly parallel. let’s say i get a mba and want that to be my only interface. things like reading email and browsing the web should be fine. what i want is to be able to borrow the processing time of my “backend machine” in an integrated way. applications that come to mind are xcode, final cut pro, and adobe photoshop. afaik, there is no way to do it magically with xgrid. there is distcc support for xcode, but i dont even know if it can utilize the xgrid technology. i would hate to have to write custom software to do anything with xgrid. maybe i just dont understand it enough.
on the other hand, i can definitely see where something like xgrid can come in handy for syncplicity. we do a lot of encryption in the backend in addition to many other things. if we can break down that encryption and parallelize it, we should be able to scale out more efficiently. someday soon i’m going to have to sit down and think about how to redesign the system for parallelism. i believe doing so will make our software “smarter”. that means money saved, which means catered lunch for everybody :).
