There are 4 tables containing about 100 M rows each, they are the tables I will look up via a column(eg. ID) in all 4 tables, we have the 5th table, which has 18M data with the(ID and SID) 1 to 1 relationship, each ID maps to about 500 records in the other 4 tables. Now I am fetching minimal date from the 4 tables with each SID and insert them in a new table.
The logic is simple but it is really time consuming so i am trying to use parallel.for() from .net. Breaking the 18 M table into 180 groups and run them concurrently, but this is limited by the hardware, I wonder if there is better way to do it.