- Executing instructions on a computer require 2 things:
- Instructions
- Operands
- Parallel computing
or parallel execution involves
performing
multiple computations simultaneously
that
collaboratively achieve
a common result
- This can happen in
two different ways:
- A single operation (= instruction)
is applied to
multiple (different) operands (= data)
This form of parallelism is known as:
SIMD: Single Instruction (stream) - Multiple Data (stream) - Multiple operations (= instructions)
are applied to
multiple (different) operands (= data)
This form of parallelism is known as:
MIMD: Multiple Instruction (stream) - Multiple Data (stream)
- A single operation (= instruction)
is applied to
multiple (different) operands (= data)
- From the basic building blocks of computing
(it requires instructions and operands),
the following taxonomy of computers can be formulated:
-
This is the most common computer produced - workstations, PC, etc
In fact, this is the type of computer that we have been studying all along...
- Schematically:
- One CPU present in computer
- CPU fetches 1 instruction at a time
- Instruction operates on 1 data item at a time
- The machines executes
ONE instruction
at a time
(single instruction stream) but on
MULTIPLE (different) data items
(multiple data streams)
NOTE: obviously, it's kinda stupid to execute the same instruction on the same data - you will waste CPU time getting the same result multiple times.
Properties:
- One CPU
present in computer
- The CPU has:
- one Instruction Register, but
- multiple ALUs
- uses multiple data buses to get multiple operands simultaneously
- Main memory are divided into
multiple banks
that are
accessed independently
The system also has multiple data buses connecting the CPU to the memory banks : this allow the CPU fetch multiple operands simultaneously
Operational Details:
- The CPU fetches
1 instruction at a time
- Some instruction operates on
multiple data items
at once
These instructions are known as vector instructions
- A
vector instructions
will fetch
multiple operands simultaneous
The same (vector) instruction is performed on multiple operands simultaneous
- One CPU
present in computer
- This type of computer is known as a
vector computer.
It was commonly used in speeding up numerical applications that use many vector operations (such as matrix-matrix and matrix-vector multiplications.)
- SIMD computers were the first systems to be implemented
with a massive amount of processors, and were among
the first systems to provide computational power above the
Giga FLOP range.
- MIMD parallel computers executes
multiple instruction
at once:
- multiple instruction stream.
- Each instruction
operates on
its own data
(it's not usual to apply
multiple operations
on the same data )
Executing multiple instruction at once naturally require:
- multiple data stream.
- There are
two common varieties
of
MIMD computers:
- Shared Memory MIMD
- Non-Shared (private) Memory MIMD (A.K.A.: message passing multi-processor)
- Shared Memory MIMD
- In a
Shared-Memory MIMD
computer, all memory locations are
accessible by all processors (CPUs)
- Schematically:
- Properties:
- Multiple CPUs
in computer
share the same memory
- We say that the CPU are
"tightly" coupled
- Each CPU
fetches one instruction
at a time and
different CPU can fetch a
different instructions ;
thus, there are multiple instruction streams
- The memory must be equiped with
multiple access points
(i.e., organized into multiple
independent memory banks)
to transfer
multiple instructions/operands
simultaneously
(Barring conflict of access , e.g., when multiple CPUs want to access the same memory bank ).
- An instruction operates on one data item at a time
- Multiple CPUs
in computer
share the same memory
- This type of computer is a
multi-processor computer
- They are quite common today.
Most workstations and high-end PC are multi-processor based.
- In a Message-passing MIMD,
a number of
SISD
computers are inter-connected by a
communication network :
Properties:
- Each CPU has its own
private memory
In orther words: Memory is not shared among the different CPUs.
- We say that the CPU are
"loosely" coupled
- From time to time, programs running on different CPUs
will
exchange information
each other through
messages
- Hence the name Message-passing MIMD
Message-passing MIMD computers are often called a computer cluster
They are much cheaper to manufacture than Shared-memory MIMD
- Each CPU has its own
private memory
- NOTE
- The "Network Switch" used to interconnect the different
SISD computers is usually a
dedicated high speed network switching fabric
- However, it is quite common to use
cheaper interconnection networks
- e.g., Highspeed Ethernet
A MIMD multi-processor using "cheaper interconnection networks" is typically known as networked computers
- Message-passing MIMD
computers always provide
a message-passing API (application programming interface)
to allow programmers to write statements in their programs
to exchange messages
A commonly used Message-passing API is MPI (Message Passing Interface):
- The standardization organization of MPI is here: click here
- The "Network Switch" used to interconnect the different
SISD computers is usually a
dedicated high speed network switching fabric