C# skeleton application with managed number of threads that return parameter
C# template of WCF server
C# template of WFC client
Using Multiple CPUsThere are known technologies specifically dedicated to concurrent data processing: Google's MapReduce and derived from it Apache Hadoop. There is nothing wrong with them. However, the programmers frequently need immediate solution and can't afford spending time on training. Windows Communication Foundation (WCF) can provide rather elementary way of concurrent usage of hundreds of LAN CPUs. It allows perform interprocess function call and pass back and forth objects. This is more than enough to arrange coordinated processing.
Since I needed some practical application for DEMO, I chose factoring of large integers. I hope nobody will blame me for teaching how to break RSA encryption, the security of which is based on extremely long time required to factoring large integers. In RSA the public key is product of two primes that is (typically) 1024 bit long and, having few hundred computers for a brute-force attack is still not enough. The number of CPUs for successful breaking of public key in RSA should be at least 10 million, so I do not undermine the security of world financial system. This DEMO can not communicate with 10 million computers because it will create extreme network congestion, so it is only good for a few hundred computers in LAN.
The DEMO is written as two templates that can be easily converted into something else. There are two programs: client and server. Server, when started, expects the connection of client and holds the single interface function that can be called. The function name is findDivisor. The input parameter is elementary object that contains the range for possible divisors. Function returns first found divisor or NULL when divisor is not found. This server can be started on multiple computers on the network. The client interacts with multiple servers. Servers in the client code are defined by their URLs, which may have different port numbers for the same computer and different IPs or computer names for different computers in LAN.
In computational part client splits whole range of the factoring number into intervals and executes search individually in each interval in as many threads as servers are available by processing one interval after another. Only two C# library classes are used to implement this powerful and effective tool - ServiceHost and ChannelFactory. Both client and server share common code - which is interface. In some WCF solutions this interface is placed into separate DLL that is called proxy. It can be implemented this way, of course, but not necessarily.
In the DEMO each individual instance of server is called from an individual thread, one thread per each server. This is only a recommended solution. It is possible to start multiple threads per each server and it will work with rare exception throwing, when server is too busy and unable to process the request. In this case the exception should be caught and addressed, that means same data should be processed again later. Maintaining number of threads equal to number of instances of server guarantees processing free of throwing exceptions and, at the same time, keeping all server instances occupied during all time of the execution.