Slideshow:
Observation on pass 1 of the
TPMMS Algorithm
Pass 1 can partially sort an input file of any size:
...
Pass 1 can produce an unlimited number of sorted chunks - each chunk is M blocks
Capacity limitation imposed by pass 2 of
the TPMMS Algorithm
Capacity limitation imposed by pass 2 of
the TPMMS Algorithm
Inverse problem:
given file size
B(R), how many buffers
(M) do I need for
TPMMS ?
IO Cost of the TMMPS Algorithm
IO Cost of the TMMPS Algorithm
But: if we pipeline result to subsequent operation, we do not count the disk write operations
Sample file size that can be sorted by TPMMS
Multi-pass Multiway Merge Sort
- File size constraint:
- The maximum size of a
file that can be
sorted by the
TPMMS algorithm is:
B(R) ≤ M(M − 1)
Reason:
- In pass 2, we can
allocate
at most M−1 buffers to
read the
sorted chunks:
- Therefore, the number of chunks
that can be
processed by
Pass 2 is
limited to:
number of chunks in Pass 2 ≤ M − 1
- Each chunk
has the size of
M blocks:
The maximum file size is therefore:
number of chunks in Pass 2 ≤ M − 1 chunks Pass 1 produces chunk size = M blocks ====> Max file size ≤ (M − 1) × M blocks
- The maximum size of a
file that can be
sorted by the
TPMMS algorithm is:
- The
memory requirement
on
the TPMMS algorithm
if found as follows:
The maximum file size (in # blocks - B(R)) that can be sorted using TPMMS is: File size ≤ M(M − 1) // From previous analysis !! B(R) ≤ M(M − 1) ≤ M2 ==> sqrt( B(R) ) ≤ M
- The memory constraint required
to run the
TPMMS algorithm:
sqrt( B(R) ) ≤ M
------ <==> M ≥ \/ B(R) (Just reverse the relationship !!!)
- The cost of running
the TPMMS algorithm on
a relation R is
as follows:
- We need to read
B(R) blocks and
write
B(R) blocks
in Pass 1:
- We need to read
B(R) blocks and
write
B(R) blocks
in Pass 2:
But:
- We do not include
the output (write) cost
in the total
(Because we could use pipelining to pass the tuples to the next operator)
- We do not include
the output (write) cost
in the total
- We need to read
B(R) blocks and
write
B(R) blocks
in Pass 1:
- Cost of
running TPMMS of
a relation R:
cost of TPMMS on relation R = 3 B(R)Note:
- If the
result is to be
written to disk
(i.e., materialize), then:
cost of TPMMS on relation R = 4 B(R)
- If the
result is to be
written to disk
(i.e., materialize), then:
- Suppose:
- 1 block = 64 K bytes
- Memory size = 1 G bytes
- Then:
1 G 1,024,000 M = ------ buffers = ----------- = 16,000 buffers 64 K 64,000
Max file size that can be sorted: B(R) ≤ M(M − 1) blocks ≤ 16,000 (16,000 − 1) ≤ 255984000 blocks (1 block = 64 K) ≤ 255984000×64 K ≤ 16382976000 K ~= 16 Tera Bytes !!!
- Multi-Pass MMS:
- The 2-Pass
multiway merge sort
can be generalized to
multi-Pass
(3 Passes or more)
- These algorithms can sort
a file of
arbitrary size
(discussed later)
- The 2-Pass
multiway merge sort
can be generalized to
multi-Pass
(3 Passes or more)