The big news this week is about how AMD has managed to split itself in two. On Tuesday AMD announced that is divided itself into two companies — one that that designs semiconductors and one that fabricates semiconductors. A significant part of the deal was the injection of at least $6 billion in the two companies by two Abu Dhabi investment firms. This news represents a big change in the CPU arena. Intel is now the last fully integrated (design and fabrication) CPU company. And, I don’t believe it is bad news for AMD, rather it reflects the times.
Recently someone sent me a link to a blog called The Perils Of Parallel. The subtitle has a nice message: Comments related to the book I’m writing: “The End of Computing (As We Knew It)”. It’s about the potential black hole of explicit parallelism into which the industry is heading. Yes, I shouted. An ally in my parallel programming malaise. As I looked at the blog author, I immediately recognized the name — Greg Pfister. He is the author of a great book called In Search of Clusters, 2/E. This book was first published in 1997, but is still relevant today. In particular, his discussion of software on clusters and SMP machines makes a very convincing argument that although both are “parallel” there is very little overlap with program design. Essentially, they are two very different computer architectures.
As they say, timing is everything. This past Monday (Sept 22), I attended the 5th annual High Performance on Wall Street Conference and Show. The event is a one day affair in downtown Manhattan. I usually attend this show each year and have noticed it grow in the number of visitors and vendors. This year, of course, it was a bit off. In case you have been living under a rock, the week before was one of the most traumatic in Wall Street history. And, like most I believe we are no way near the end of a long dark tunnel. From my casual conversations and estimates, there seemed to be more vendors than attendees at times. Thankfully, the show was not deserted as some may have predicted and there was still plenty of activity.
For those outside the computer business, the “power of two” thing can be quite mystifying. When we build clusters we might start at 16 nodes, but then expand to 32 nodes, or even 64. Building a cluster with 128 nodes may baffle the casual base-10 observer. Why not 100 nodes or 130 nodes? Why is everything divisible by two?
As you read this, physicists at CERN should be circulating the first batch of protons in the Large Hadron Collider (LHC). For the non-scientist, this may seem rather mundane — isn’t that what most physicists do, smash things together and see what falls out? In any case, this is no ordinary smashing. If all goes well, and if the little proton bumper cars hit each other in just the right way, there should appear something called the Higgs Boson. Now that is something you probably have not thought about for a while. There is a lot riding on these experiments. Indeed, finding the Higgs could be one of the great achievements of modern science. If the Higgs does not show up at the LHC party, there is some rewriting and head scratching that needs to be done in the world of physics. A little background may help as well.
One of my favorite computer stories is about my first web server. I had mentioned it previously, but I’ll provide a quick recap here. Back in the day, I put together a Linux web server using a 486-DX2 box I had lying around. Aside from being impressed that this “Linux thing” actually worked, I was impressed with the fact that a lowly x86 box could do a great job serving web pages. Of course, back then web pages were basic HTML 1.0, but compared to other more expensive UNIX servers this system was a real bargain.
The system was a large tower case with motherboard placed in a vertical position. The 486-DX2 ran a little warm so it had a small fan on top of it. I seem to recall that the CPU would draw about 5-6 watts of power. The system ran for at least 3 years without a hiccup. It had already seen a previous life as a workstation, so the new life as a web server was gravy. The up-times were measured in months as it was only rebooted after it was moved or upgraded. At one point, I decided to open it up to see if there was dust and dirt collecting on the inside. To my surprise, the small fan had fallen off the 486-DX2 and the system was running just fine. Based on the amount of dust on the fan, I figured this happened some time before I opened the system. I cleaned the dust out of the system, hung the fan on the 486-DX2 (some of the plastic tabs that held the fan in place were broken) and put the machine back together. It ran fine for another year or so. Now that is what I call reliability.
As clusters continue to grow in size and complexity, I often think about the programming and management issues users and administrators must face. There are currently proposals floating around that require five digit node counts. That is a lot of cores, heat, power, cables, floor space, and coffee. If I were setting up a 25,000 node cluster, I would suddenly acquire a renewed interest in statistics. Not just the nodes, but also the networking. As you may know, infrastructure parts are rated in terms of MTBF (Mean Time Between Failure). This is a statistic that is often mis-understood. If a part has a MTBF of 10,000 hours, that does not mean the part will last 10,000 hours. It is a number that is used to determine the failure rate of a collection of similar parts.
An example will help illustrate. A compute node may have a MTBF of 250,000 hours (about 28 years), which sounds impressive. If you are considering 2,500 nodes in a cluster, then you can expect one node will fail every 100 hours (Divide the MTBF for a thingby the number of things and you get the expected time between failures.) If you take it up notch — using hypothetical 25,000 nodes — then you can expect a node to fail every 10 hours. Bummer. I call it the more stuff you have, the more stuff that can break rule.1
Recently IBM announced the open availability of their xCAT software. For those that are not familiar with xCAT, I quote from the website, “xCAT is a scalable distributed computing management and provisioning tool that provides a unified interface for hardware control, discovery, and OS diskful/diskfree deployment”. In more official terms xCAT is referred to as The IBM HPC Open Software Stack®, which indicates that it is integrated and tested by IBM. In a nutshell, xCAT makes it easier to create, provision, and manage your cluster.
Many years ago, I started a “good new/bad news” joke about Linux HPC clusters. The good news is Linux HPC clusters give you the freedom to design and build your HPC machine to your needs. The bad news is Linux HPC clusters give you the freedom to design and build your HPC machine to your needs. One needs to be care with that freedom thing. In particular, there are two areas where this “freedom feature” can hurt the market place. Tools like xCAT go a long way in helping manage the “freedom” we’re all looking for.
Over the years I have made my share of cluster mistakes. Each problem presented an opportunity to learn something new, become a little smarter, get some scar tissue as it were. I had just such an opportunity this week as I as teaching Intermediate Beowulf: An Introduction to Benchmarking and Tuning as part of the ARC HPC Training at Georgetown University. I’ll get back to my teaching experience in minute, but first I wanted to talk about HPC education.
There is a big word in the title. You are either reading this because, you don’t know what it means and want to find out or you know what it means, but have no idea what it has to do with microprocessors. Either way, I’m hoping you are hooked because I think we are witnessing a second dramatic change in the computer market (the first being multi-core). As many of you know, I have had many a sleepless night when it comes to multi-core. In the fits of my obsession, however, I may have missed something.
Several years ago I was walking around the annual supercomputing exposition and noticed that many products were now Grid Enabled. This merited had a good laugh. To me, it was funny for two reasons: First, it seems there is no consensus on the definition of a grid. Of course, Ian Foster had defined a computing grid, but others took liberties with the concept of a non-local distributed computing network. Often it was associated with utility computing (i.e. you get you computing like electricity by plugging into the “grid”). Examples ranged from a cellphone network, a LAN, the Internet, even a cluster.
The lack of a concise definition did not stop the market from “enabling” everything for grid computing. All you had to do was add the phase “Grid Ready” or “Grid Enabled” to your product literature and you were “on the grid” so-to-speak. My second chuckle came from the overly obvious messaging; somewhat like labeling a car “road enabled.”
I always believed the grid concept was a good idea, but like anything else, it seemed to be oversold and misunderstood at many levels. Of late, the grid thing seems to have given way to the latest buzz word: cloud computing. Because I don’t have a good definition of grid computing, I really don’t know the difference between grid and cloud computing. Are they the same? Can you have a grid of clouds, or cloud of grids? And, let’s not forget to ask, is it green? Should I be buying products that are green cloud enabled? Somehow that sounds a little ominous.
Which brings us to today’s weather forecast. Partly cloudy with an Amazon Elastic Compute Cloud (Amazon EC2) forming quickly and a cloud from flexiscale moving in from across the ocean. There is also the Sun Grid Compute Utility at Network.com front moving in, but it has Grid in the name so maybe it’s not a cloud after all. There are others forming to be sure. Virtualization plays a big role in the cloud concept. Maybe cloud computing is grid computing with virtualization. Or, maybe it is really grid computing, but just easier to use. In any case, from what I have read, cloud computing should solve all our problems and revolutionize the computer industry. Yippee, maybe that is all that needs be said on the topic.
Ah, but not so fast. When ever anyone talks to me about “remote” anything, the first thing that pops into my mind is security. I would have no problem running recreational software in a cloud, but doing legal and financial things beyond my little router firewall requires a bit of trust. I use on-line banking with an encrypted browser and the bank does computation for me (balances, computing the enormous interest on my savings account, etc). Does that mean my bank is cloud enabled? There is always the implied promise of security but, you know, when you have root privileges in the cloud you can rain on a lot of picnics.
If I’m a big company, I probably want my private VPN-Cloud. I seriously doubt I would allow anything into the wild. I find it interesting that companies spend all kinds of money on securing their corporate IT networks only to allow laptops with anything on them to be taken out the door and lost at the airport by the truckload. Maybe that corporate cloud isn’t a bad idea from a security standpoint. Would I use cloud computing to design my next product? I would have to think about that one.
I think cloud, like grid, computing must address and ensure security or it is bound to be one of those interesting things that never quite catches on.
Then there is the performance myth. You know, the if we could use the latent PC cycles in our company at night we would have the world fastest supercomputer bit. I have been hearing about that good idea for years. One question for those HPC in the office cubicle mavens, how is it going? Of course, like any other concept it was born of good intentions. As most anyone who has run an MPI program will tell you, a bunch of PC’s and a network do not a supercomputer make. Unless you want to render a new action movie, or do many repetitive single process runs, that corporate network supercomputer is pretty much a non-starter.
As I mentioned, the cloud approach often relies on virtualization. In HPC, there has been a non-trivial amount of work to get the applications as close to the hardware as possible. Virtualization tends to move in the opposite direction. Running a cluster application in a cloud would probably not be a productive exercise. The supercomputing cloud is nice idea, but like most most good ideas, the devil lives in the details.
With all that said, there is still something I find attractive about the grid/cloud idea. The ability to tap into a huge computing resource only when you need it is attractive. Beyond running a sequential (single process) application, the challenge is software. (When is this ever not the case.) My belief is that for a cloud to be effective in HPC, software needs be dynamic adapting to resources (slow/fast and processors/interconnects) as they become available (and fail). Most HPC applications don’t have this ability. I have discussed dynamic execution before and here is yet another reason to consider this approach.
Finally, some interesting clouds are forming. I just read about a company called CherryPal that plans to sell a box and presumably a cloud. As far as I can tell, the box provides a Linux “desk top” environment (iTunes, OpenOffice, instant messenger, media player, etc.) that is maintained in the cloud. There is also 50 GB of cloud Internet storage. All this for “much less than an Asus Eee” and a claimed power draw of only 2 watts (which is why I put this announcement into the “wait and see” category). The approach does make some sense in a managed mainframe kind of way.
Finally, all the cloud goodness depends on broadband (or better) Internet access. For most companies and home owners this is not a an issue, but just to be sure, you may wan to check that your Internet connection is cloud enabled otherwise you might miss all the fun.
Over the last six days I have been listening to his holiness the Dalia Lama speak. He is teaching the The Great Treatise on the Stages of the Path to Enlightenment. First, let me say, aside from having an excellent karma rating on Slashdot, I am not a practicing Buddhist and don’t purport to be any kind of expert on the teachings (in this universe anyway). I do find Buddhist teachings (at least the ones I can understand) interesting and insightful, however. Indeed, they probably would have helped with back in grad school when I was trying to get my head around quantum mechanics.
As was noted in the program guide, the Great Treatise is part of the lamrin texts, which are considered to be free of any contradictions. This single statement held my center for quite a while. The lamrim is thousands of pages of text explaining some of the most heady concepts in the world and is free of contradiction. I read this as fully self consistent. Gödel’s first incompleteness theorem not withstanding, that was probably not an easy thing to do.
Through the miracle of broadband, and a neighbors wide-open wireless router, I’m still hanging out at the beach. By the time you read this, I’ll be heading back to the stately Eadline manor, but not to mow the stately lawn or pick up two weeks worth of newspapers in my stately driveway. I’m actually heading back to attend a series of lectures by the Dalai Lama. Really, I am. He is in town teaching this week at Lehigh University. Enlightenment is not far off my friends. Of course I would not mentioned this if it did not have something to do with parallel computing. Hopefully, I’ll have the link figured out by the end of the column.
Last week I put my Erlang flag in the ground as it were. I was expecting a raft of flaming comments. Did not happen. I doubt there was a collective head nod from my vast audience, and I wonder why so quiet? Perhaps everyone is on vacation, or getting ready for vacation and have put off reading my insightful column for when they take a vacation. Speaking of vacations and beaches, that includes me. Weekly columns are still required, however. I have set aside some beach time to continue reading Programming Erlang: Software for a Concurrent World by Joe Armstrong. I have not yet, however, brought myself to bring my Eee web-book to the beach to play with some code. Bright sun not withstanding, I fear the union of silicon, both refined and shoreline, will not be a happy affair. Although this Asus photo might seem to indicate otherwise.
This week I am going to climb back on my parallel programming soap box and make a proclamation.
If you recall, I had started a discussion on declarative programming methods. Compared to “procedural” or imperative programming (C, Fortran, Perl, etc.), the declarative approach focuses more on the “what” instead of the “how.” HTML is good example of this idea. When you create HTML, you telling the browser “what” to you want on the screen, the how is up to the browser. Because I have not told your computer how to render the information you can choose to view it using any number of browsers including a text based browser like lynx or elinks.
When one writes an procedural program (C, Fortran, Perl, etc.) you are “closer to the hardware” and farther away from your problem. When you add MPI, OpenMPI, or Pthreads you have moved closer to the hardware and further away from the problem at hand. The programmer has more to manage which also means more opportunities for bugs, but that is another issue.
These additional parallel responsibilities are why procedural parallel computing is harder than sequential computing. It is why machine code is harder than a high level language. And, finally, it is why we are facing a bit of a crises in the age of multi-core.
Back in late 90’s there was a time when clustering was the rage. A few boxes, a fast Ethernet switch, some Linux software and presto, you had a cluster. Like every new technology there are misconceptions, hype and detractors. Eventually, the price performance numbers ushered in a new HPC computing paradigm. In some cases people cobbled together clusters, just to build them, in others they built clusters and ran actual HPC programs on them. Indeed, there seemed to be story after story about a new “off the shelf system” built at low cost or from some systems that were “just lying around.” They would get an MPI or PVM program working and find that they now had their own personal supercomputer. There seemed to be several enabling factors that ran through the success stories. These included:
The performance is better than the existing system I was sharing with everyone else.
The hardware cost is really low (price-to-performance is high).
I did not have to buy any software.
I can do things I could never do before on my desktop.
This week I decided to take a break from my planned discussion of declarative languages. Mind you, this is an unplanned break as I’m at the point in my discussions where I’m about to reveal the answer to all your problems. I can’t do that this this week because I’ve been overcome with a rant. Much like that wave you see out in the ocean slowly building as it moves closer. There is simply no stopping it.
Usually my rants have a trigger. I’ll read something or hear a comment about an important issue. Sometimes I agree with the the opinion, sometimes not, often it is that bump on the waters surface. The wave has begun.
Like many people my age, my first programming experience was with the BASIC language. I recall using an ASR 33 telprinter to create/edit/run BASIC programs. I would marvel at how the cylindrical print head would dance and bang across the paper printing error messages from my ten line programs. Of course, at 300 BAUD you had plenty of time to watch things happen. Back in the day, BASIC was usually the first computer language you would learn. There were other more “advanced” languages like FORTRAN or COBOL that were deemed too difficult for the novice. By the way we capitalized Fortran back then.
In my previous column I mentioned Amdahl’s Law. Before you click away, rest assured I have no intention to talk specifically about Amdahl’s Law and I promise not to place a single equation or derivation in this column. Often times people are put off by Amdahl’s law. Such discussions usually start with an equation and talk of the limit as N goes to infinity. Not to worry. There are no formulas, no esoteric terms (sorry, no big words), just the skinny on the limits of parallel computing. I’ll even go one further, I’ll hardly mention parallel computers, multi-core, and other such over worked topics. This month’s I’ll discuss lawn care.
Sit down, take a deep breath and relax. Let your mind wander for a bit. I’m going to talk about something that runs counter to one of the basic tenants of computer performance. Something you may believe to be true for ever and always. Last time I talked about how concurrent parts of a program are those that can be computed independently. I also explained how the parallel parts of a program are those concurrent parts that should be executed at the same time in a way to increase the speed of a program. As I discussed, the differentiator is overhead. It is now time to take the next step. By the end of this column, I hope to convince you that the following is sometimes true:
In a parallel computer, faster processors are not always better than slower processors.
Before I climb back onto my multi-core soapbox, I wanted discuss some parallel computing fundamentals. Having been involved with parallel computing for almost 20 years, I often forget that those new to the cause lack a firm background in some of the basic concepts being discussed. It took hard lessons and several years for these concepts to sink into my head, hopefully your path to “fully parallel” will be far more gentle. So, let’s begin; if there is one thing to “take away” from this column, it is the following set of three rules:
Concurrency is a property of the program.
Concurrency does not necessarily imply parallel execution.
Efficient parallel execution of concurrent programs is a property of the hardware.
My recent column on the Total Cost of Parallelization (TCOP) generated some interesting feedback. I like feedback because it means people are actually reading what I write. In particular, when a response to one of my columns is from a experienced practitioner of the art, I read it carefully. Informed discussion is a great thing and the multi-core world can certainly use more ideas. With that in mind, I would like to provide equal time to Brian Dobbins of Yale University. He emailed a response directly to me with what I feel are valid points. I inserted a few comments to help clarify some of my own points referenced by Brian’s comments, but did not alter any of his views or conclusions. (Brian’s comments are in italics).
My counter point stems from a recent discussion I had with someone explaining that it’s almost always better (performance-wise), when given the choice between 150 processors at speed 1.0X vs. 100 processors at speed 1.5X, to go with the latter option. While both give the same aggregate computational “power”, the latter has fewer communication links, so more time is spent in useful computations.
The rule of thumb is a good one, however, there are some subtleties, which don’t effect my proposal, but are worth mentioning. For instance, if the interconnect and the processors are balanced, then the smaller number of processors is better, but if the processors spend time waiting on processor-to-processor then you may see better performance on more processors (provided the application scales).
I have calmed down since my previous rant. Looking at the comments, it seems I have stepped on more than a few issues. While I can’t address all the comments, I will make a general statement. There were some suggestions that solutions already exist. And, indeed, I can’t argue with that; you can write parallel programs in shell if you so desire. The big issue is the abstraction layer.
As a developer, I don’t want to think about the number of cores or nodes any more than I have to think about the addressing modes, registers, and floating point units in the processor. If parallel computing is going to truly be cost effective, then reasonable abstract layers need to exist. Many developers and most vendors will acknowledge an urgency to achieve the goal of simplified parallel programming lest we face some dire consequences going forward. The solution I have proposed is a shared GPL approach. If anyone has a different idea, I would love to hear it.
This week I’d like to take a look at the latest survey results. While the participation was not all that I wanted, it is enough to make sufficiently vague statements about the HPC cluster market.
Whenever I talk about Web surveys I always throw in a proviso that these are not well-designed, scientific surveys performed by more sophisticated marketing organizations. Web surveys are more like asking the person on the street what they thought of a movie. When I look at the survey results, think of it as a “scent” of the market and maybe suggest some area that bear watching. But, enough handwaving. Let’s look at some results.
It seems after last weeks rant, the poll participation on Today’s HPC Cluster has increased. Great job everyone! If you haven’t applied your valuable expertise to these polls, please do so as soon as possible. I plan to discuss the results next week. How as that for positive re-enforcement? Let’s move on to this weeks topic, boxes and bugs.
Let’s talk about the boxes first. There seems to be an interesting trend developing in the PC market. Wal-Mart has started selling Everex Linux computers for $200 recently. There’s also news that Shuttle will also be making a small “sub-$200″ box as well.
Of course, these systems are a little thin on resources, but there seems to be this $200 price barrier that the industry is trying to crack. My interest is not in touting the “Linux makes this possible” story, but to point out that such systems are selling quickly. If the trend continues, a large amount of these systems will be produced and anytime a large amount of “computing anything” gets produced, my “Can you build a cluster with these?” antennae goes up.
Trying to get a handle on the HPC community and market has always been difficult. The professional market forecasters seem to paint a rosy picture for HPC over the next five years. But, when I talk to people in the market, I often get a different take. For this reason, I like to ask people about how they use clusters and in particular I like to ask them about their pain points (or challenges if you prefer a more positive spin).
To help get a feel for the community’s take on the state and future of HPC the Today’s HPC Cluster site has a set of polls. I should mention that I don’t consider casual Web site polls to provide a definitive source, but rather a general feel for the issues. In addition, I like to see at least 100 responses to a poll.
As the current polls have been up for a while, I thought I would check the results (if you vote you see the results). The front page poll has two questions about physical issues and staffing. The response rate was within my “well I guess that is enough to draw a general conclusion” range.
This month, I thought I’d take a break from my usual multi-core ranting and talk about another new technology that’s starting to give me fits. No, it’s not the somewhat confusing General Purpose Graphical Processing Units (GP-GPUs) acronym, which is not to be confused with the GPU that has something to do with putting colored dots on your screen. No, the thing that’s raising my dander is the virtualization thing, or “virt” for short.
Now, virt is a good idea, even if it’s become an annoying buzzword of late. It kind of makes sense with all the extra cores and all. And while I’m not expert enough to talk about how virt will play out in the larger server market, I do have some thoughts about virtualization in high-performance computing (HPC).
It’s time for the yearly batch of retrospectives and predictions. Count me in! Let’s see, the big thing of 2007? Well, that had to be multi-core. And, the big prediction for 2008? Why that would be multi-core, once again. There, I’m done. Enjoy your year.
I’m not really done… Being a “God does play dice” (and pinball) kind of guy, I find predicting the future to be a somewhat worthless endeavor.
Allow me to elaborate. Most predictions are based on extrapolating a line of some sort. If you extrapolated the game console line, the PS3 and Xbox 360 were destined to crush the Wii. Similarly, Microsoft’s Vista was going to usher in a new era of computing and those $200 Linux desktops from Wal-Mart were never going to sell.
Because the universe is non-linear and pretty much random, one never really knows what the future holds. New developments usually come from outside the box because everyone else is sitting inside the box drinking the market driven Kool-Aid.
Therefore, instead of trying to make some lucky linear predictions to prove I’m some kind of oracle, I’ll make some wild guesses. If they turn out to be wrong, then I can say I told you so. If they are right, then I suppose I got lucky with the dice. So here goes.
As 2007 fades away, I thought I would reflect on some of the HPC events of the last twelve months. Of course there was plenty of news, new products, mergers, acquisitions, and all other kind of normal stuff that one would expect from any market. Having thought about it, though, nothing really stands out in my mind as a big breakthrough or new paradigm shifting technology.
Of course when you read the press releases they all seem to imply that the world will never be the same when you use some latest and greatest gizmo, software, or service.
Before the people who have worked so hard in various corners of the market say bad things about me, let me first say, Thank You. In my opinion, we are further along than we were last year at this time due to your efforts. I include everyone from the Linux driver writers, to the guys taping out multi-core processors. We are moving forward. This past year you could buy more FLOPS (and use less power) for your dollar than in the past.
My disappointment is based on my belief that HPC is still hard and will continue to be this way until we figure out how to create cost-effective turn-key software for this market.
With all due respect to Forest Gump, defining fast is becoming a bit harder these days. And, yes, it has to do with multicore. There certainly is no argument that a faster execution takes less time. For instance, if my program ran in 10 minutes on the old processor and now runs in five minutes on the new processor, I would be pleased. Now let’s see how this plays out in multicore land.
First, let’s assume your code(s) are “multicore” ready as they may be written using threads, OpenMP, or even MPI. Second, you’re in the market for new compute nodes and a decision must be made about the type of processor. Should you buy a larger amount of faster dual-core nodes, or a lesser amount of slightly slower quad-cores? (Note: more cores create more heat, so quad-core processors tend to come in at lower frequencies than dual-cores.) Of course, we know it all depends on the application.
No matter how you cut it, coding for multicore is really just parallel programming. Once you’ve realized that, it’s time to look at the options, whether your existing codebase will scale, or if you need to rewrite your code and how.
As stated in The Multicore Programming Challenge, parallel programming can be difficult. It moves the programmer closer to the hardware and further from their application space or problem. Fortunately, people like rocket scientists have been writing parallel software for quite some time in the HPC (High Performance Computing) sector.
As any good programmer knows, an existing code base can be valuable to current programming projects. First, the possibility of re-using existing code is a major incentive. Also, learning how someone else attacked a similar problem is very valuable.
In the HPC sector, most parallel programs are written using Message Passing Interface (MPI). While MPI is normally used on large computing systems (clusters) it can be also be used on a multicore processor. The “MPI proposition” may seem counter to conventional wisdom as MPI was designed for distributed memory (i.e. each core/processor has it own private memory), whereas OpenMP was designed for shared memory.
The lazy assumption suggests that OpenMP is a better solution because it was designed for shared memory. However, the possibility of re-using an existing MPI code base is worth considering before you spend a month(s) re-inventing the software wheel. Ultimately, the question is really about efficiency. Namely, How does the performance of MPI compare to OpenMP on a multicore system?
Earlier this year, my wife and I decided to finish our basement. Great! I thought I could finally set up a real office. I had just one thing to do before my office would become a reality — deal with the ghosts of systems past.
Over the years I have collected a substantial pile of old hardware. Vintage Pentium II servers in 2U cases, and workstations in big desk-side cases. At one point I was sure I was going to build a cluster out of these systems, but somehow I never got around to it. When you have a big empty basement, why not fill it with old hardware and build a cluster? Isn’t that what people do these days?
In any case, these systems were eBay orphans — not even worth the shipping cost, not worth testing, etc. I had checked into donating them, but they were just a little too old and slow. In addition, the systems were “white box,” unbranded systems. (Why are they still called “white box” systems when they’re in black cases?) The only solution was the recycle bin, but that meant separating the metal from the electronics. Not a problem, with a few screw drivers, some wire cutters, and the help of my teenage daughter, we quickly had a pile of metal and a pile of components.
I am standing in the middle of SC07 (Supercomputing 2007). SC07 is THE HPC event of the year. 318 exhibitors have made the trek to Reno, Nevada for SC07, and scores of attendees are here for the week-long conference. If you’re attending the show, you’re probably not reading this because you’re either completely exhausted, back-logged, depressed from losing your money at the casinos, still trying to get your shampoo back from the TSA agent at the airport, or some combination of the above.
In any case, I’m thought I might trying my turn at real-time blogging. As many of you know I write quite a bit about clusters, but not in the real-time sense. I normally to try to write clever and {insight/incite}ful articles with some take-away for the reader. Not this week. I’m blogging, baby. Plus, Linux Magazine editor-in-chief Joe ‘Zonker’ Brockmeier just came by and asked me when he can expect this week’s column…
11:00 a.m. I’m on the trade show floor standing in an vendor’s booth (Appro) as part of my duties as a “booth geek”. When not trying to invent TSA-safe shampoo, I like to test and benchmark new technology. When Appro asked me test some new Harpertown processors and write a white paper I jumped at the chance.
I find Pablo Picasso’s famous response on computers, “Computers are useless. They can only give you answers,” interesting and provocative. I also believe there is a grain of truth in that statement — namely, that it’s really important to ask the right questions of computers. I have a hard time with the useless part, however.
We ask important and world-changing questions of computers (and clusters) everyday. I call these the what questions. For instance, “What is the lowest energy state of this molecule?” or “What happens when I try an align these two genome sequences?” or “What does this seismic data tell us about possible oil in the ground?”. These questions yield important information.
In the past, I’ve put Intel quad-cores through some very simple tests. Namely, I spent quite a bit of time trying to understand the results of my “effective processor” script. Now it’s AMD’s turn.
I’ve been testing an 8-core (four-socket) Opteron machine from Appro International (http://www.appro.com). The machine can be configured as a 3U cabinet or as a deskside tower. More important, however, I now have an AMD box with 8 cores (four dual-core, 2.8 GHz Opteron processors) to compare to an Intel 8-core box (two quad-core 2.67 GHz Clovertowns) — a very apples-to-apples comparison.
However, exact parity in clock speed is not essential for my comparison. I am interested in how well the system scales when running multiple copies of the same program, so the actual clock speed is normalized out of the results. For each system, the Linux kernel finds eight processors and that is at the core of all the tests (obligatory core pun when writing about apples).
Before showing the results, I’d like to present my conclusions: In the taxonomy of apples, there are two kinds that seem to represent the results. The first is like the Jonathan — deep red, mildly tart, rich in flavor, versatile, and excellent for snacking or baking. The other is more like the Roman Beauty, slightly tart, best for baking. Of course, I will let you try and figure out which is which.
Back in the good old days of single core processors, when HPC clustering was in its infancy, getting the application as close as possible to the hardware was very important. In many cases, it still is. Communication between nodes could take place through the operating system by using TCP/IP or outside the OS using a userspace zero-copy protocol. With the exception of pinning down memory, the userspace protocol totally removes the OS from the communication. The result of userspace communication is better application performance thanks to better latency and throughput.
To see how virtualization might play in HPC, let’s consider how programs operate on a cluster. A parallel MPI program is basically a collection of individual programs (processes) running on the same or different node. Messages are essentially a way to copy memory from one program to another. Programs are “placed” on specific idle nodes/cores by a scheduling program (such as PBS, Sun Grid Engine, or Torque). The programs are managed by the all-knowing scheduler and once placed on a node, stay there until complete. Communication takes place over an interconnect, but as far as the scheduler is concerned, what you do with your nodes is your business. This model has worked well for HPC clustering although it is not the only one.
A story floating around the Internet recently claims that the Storm email worm has created the world’s largest supercomputer. I believe the genesis of this claim was this post on a security Web site. Are you amazed? Do we need James Bond to ferret out the evil villains? Hardly. Perhaps Austin Powers can handle this one.
It certainly isn’t news that Internet criminals find using other peoples computers helps with their chosen vocation. Not to mention that the most dominant OS is the world has a “Hack Me” sign hanging on its back as it walks through the Internet hallways.
The Storm worm bad guys, who presumably already had some spambots, sent spam to even more people. Once a user opened the email, the work went about installing and hiding itself on the target computer. As reported, the main use for the botnet seems to be blasting spam.
The number of machines controlled is reported to have been as many as 50 million computers, but estimates have been revised downwards to be much more conservative. Current estimates place the infected machines at about 1.5 million infected in July and only 200,000 machines available full time.
Now comes the urban legend sound bite from the post on the Full Disclosure list:
Many years ago, I set up a Web server using a spare 486DX2 tower machine and Linux. The experiment was to test Linux, and also start serving some very basic HTML from a dial-up Internet connection. The system worked beautifully — too well, in fact. It was supposed to be a demonstration, but turned into a working mail and Web server for a small company. When the machine was finally taken out of service, I removed the case and found that the tiny CPU fan had fallen off and the processor was running “naked” (my dust forensics told me it was like this for quite a while). When your processor is putting out a maximum of 7 watts, you can live with such failures. As anyone can tell you, a fan failure is not tolerable in today’s servers where the heat output is more than ten times what it was in my little Web server. Heat has become a major problem.
In the near future, you may start to hear about programming languages with some strange sounding names. Of course, it’s almost a requirement to have a unique and slightly esoteric name for a programming language. The new group of languages will be a bit different, however, as their strangeness will continue well beyond their name.
Of course, the interest in new languages, or in existing but obscure languages, is due to (ready say it with me now) multi-core. Yes, the thing that keeps me awake a night. That carnival of cores going on right now inside your computer that will make HPC mavens of us all.
In case you missed the previews, here’s the story so far. An HPC cluster is parallel computing. A multi-core processor is parallel computing. A cluster of multi-core systems is parallel computing squared. Explicit parallel programming is hard. Hard takes time. Time costs money. Now square the result. (See HPC Guru Don Becker: Why MPI Is Inadequate.)
Every time I talk about multi-core, I seem to start out with something like “back in the day” or “when things were much simpler,” or some such lament. I have decided to make it easier on me, and hopefully you too, and will now refer to two epochs in the commodity cluster computing world. The first, I will use SC for single core and the second will be MC for multi-core. That stunning insight makes me a genuine master of the obvious. I am going to add a little bit more, though.
