- In the 1970's
(cold war era), there was
a lot of interest in
developing
more power nuclear weapons
There was also weather forecast applications that require speedy computations
- Common traits of
nuclear bomb (explosion)
computations and
weather forecast computations:
- The programs need to solve differential equations that use matrix and vector operations
- A vector is just an
array
of numerical values, e.g.:
+- -+ | 1 | v = | 1 | | 1 | +- -+
- A matrix is a
two-dimensional array
of numerical values, e.g.:
+- -+ | 1 0 0 | A = | 1 0 0 | | 1 0 0 | +- -+
- Programs that
operate on
matrixes and
vectors:
- Computation on matrices/vectors has a special property that we can exploit to make matrix/vector operations very fast
- Vector in
a plane (2 dimension) ---
high school
material:
Effect of adding 2 vectors:
- Normal way to
add
2 vectors:
Input: float A[N] // vector 1 in array A float B[N] // vector 2 in array B Output: float C[N] for (i = 0; i < N; i++) C[i] = A[i] + B[i]; // We must perform N addition operations
- The "vector addition" operation:
- The "vector addition" operation is a single addition operation that can perform N different addition operation simultaneously
Example:
- This vector addition (+) operation:
+- -+ +- -+ +- -+ | 1 | | 5 | | 6 | | 2 | + | 4 | = | 7 | | 3 | | 1 | | 4 | +- -+ +- -+ +- -+is a single addition operation that adds 3 pairs of numbers simultaneously !!!
- The add operation is
applied simultaneously to
3 pairs of numbers:
- Normal way to
multiply
2 vectors (= array of values):
Input: float A[N] // vector 1 in array A float B[N] // vector 2 in array B Output: float C[N] for (i = 0; i < N; i++) C[i] = A[i] * B[i]; // We must perform N multiply operations
- The "vector multiply" operation:
- The "vector multiply" operation is a single multiply operation that can perform N different multiplication operations simultaneously
Example:
- This vector multiply (*) operation:
+- -+ +- -+ +- -+ | 3 | | 2 | | 6 | | 2 | * | 4 | = | 8 | | 3 | | 3 | | 9 | +- -+ +- -+ +- -+is a single multiply operation that multiply N pairs of numbers simultaneously !!!
- The multiply operation is
applied simultaneously to
N pairs of numbers:
- "Normal" computer instruction:
- A "Normal" computer instruction has one or 2 input values and produces one output value
Example:
M68000: add R0, R1 inputs: values in R0 and R1 output: one value that is equal to the sum of the values in R0 and R1
- Vector computer instruction:
- A vector instruction has one or 2 input arrays and produces an output array of values
Example:
- Speed up achieved in
vector operations:
- The time to execute 1 vector addition/multiplication (any operation) is about same as the time to execute 1 ordinary addition/multiplication (any operation)
- Hardware needed to
execute
vector operations:
- We need multiple ALU circuits to perform multiple simultaneous operations !!!
- Fact:
- Many numerical
computer algorithms make use
of matrix multiplications
Examples:
- Weather prediction
- Modelling a nuclear explosion
- Machine learning algorithms (e.g., neural networks) make use of matrix multiplications
- Many numerical
computer algorithms make use
of matrix multiplications
- Matrix Multiplication:
+- -+ |A11 A12 A13| A = |A21 A22 A23| |A31 A32 A33| +- -++- -+ |B11 B12 B13| B = |B21 B22 B23| |B31 B32 B33| +- -+Then:
+- -+ |C11 C12 C13| C = A*B = |C21 C22 C23| |C31 C32 C33| +- -+ where: Cij = Ai1*B1j + Ai2*B2j + Ai3*B3j (for i = 1, 2, 3 and j = 1, 2, 3) - Example:
+- -+ +- -+ +- -+ | 2 3 1 | | 3 6 2 | | 2*3 + 3*2 + 1*1 2*6 + 3*2 + 1*1 ... | | 6 2 3 | * | 2 2 4 | = | 6*3 + 3*2 + 3*1 | | 4 7 1 | | 1 1 3 | | 4*3 + 7*2 + 1*1 | +- -+ +- -+ +- -+ +- -+ | 13 19 19 | = | 25 43 29 | | 27 39 39 | +- -+
- Suppose we want to multiply the following 2 matrices:
- We will go through the steps in a
parallel
matrix multiplication
using vector operations
While going through the example, make a note that the vector operations are performed on multiple values simultaneously
- (Parallel)
matrix multiplication
using vector operations:
- Initialization step (done once):
- Processing
Row 1:
- Process first column element (2)
(in row 1):
- Vector multiply:
- Vector addition:
- Vector multiply:
- Process 2nd column element (3)
(in row 1):
- Vector multiply:
- Vector addition:
- Vector multiply:
- Process 3rd column element (1)
(in row 1):
- Vector multiply:
- Vector addition:
Row 1 is done...
- Vector multiply:
- Process first column element (2)
(in row 1):
- Processing
row 2:
(similar to row 1)
- Process 1st column element (6)
in row 1:
- Vector multiply:
- Vector addition:
And so on,,,,
- Vector multiply:
- Process 1st column element (6)
in row 1:
- Initialization step (done once):
- Single (= same) Instruction on
Multiple Data items (SIMD) is
vector processing
- Facts:
- SIMD computers are
called
vector computers
- The
parallel execution
of the same operation on
multiple data items is called:
- data parallelism
- SIMD computers are
called
vector computers