Fundamentals of Microsoft .NET Programming: Multiprocessing

Distributed Computing

In distributed computing, multiple computers linked by a network work together to perform a task. You can think of distributed computing as similar to multithreading except that the “threads” run on different computers.

Although coordination among threads on the same computer can be cumbersome, communication among computers in a distributed application is much slower. That means distributed computing is most useful when the problem is embarrassingly parallel. For example, several computers could be assigned the task of generating separate frames for an animated movie. Those computers could then use multithreaded ray tracing programs to calculate the pixels in each frame.

Other examples of distributed computing are “grid computing” applications, which use idle computers scattered across the Internet to perform CPU-intensive calculations while their users aren’t using them. Some of these efforts involve thousands of (or even millions of) computers. Examples include SETI@home, which tries to detect alien signals in vast amounts of radio signal data; Folding@ home, which simulates protein folding and molecular dynamics to study diseases; and the Great Internet Mersenne Prime Search (GIMPS), which tries to find Mersenne primes of the form 2^p – 1 for some number p.

Distributed computing is a specialized subtopic in a specialized field, but some of its basic ideas can be very useful when designing multithreaded (or even single-threaded) programs. One of the most important of those ideas is that each of the cooperating programs should be as independent as possible. If thousands of computers need to communicate frequently with each other or with a central computer, the network’s communications needs will quickly outweigh any potential benefits.

Similarly, keeping each thread as separate as possible (avoiding direct communication between them and avoiding locks) makes threads faster, easier to debug, and more scalable so you can easily add more if necessary.

Even if your computer has only a single core, breaking operations up into independent pieces makes writing and debugging the pieces easier. If the pieces are self-contained, then you can debug one without worrying as much about how changes to it will affect other pieces of code.

Keeping the pieces as separate as possible also can help you rewrite the program later if you decide to spread it across multiple threads.