- It turns out that:
- The unweighted sample are not appropriate for a general join operation
- But in a subsequent lecture, we will see that
the unweighted sample
is
appropriate for
a
specific join operation:
- The unweighted sample is representative for foreign key joins
- The main idea
to obtain an approximate solution
for a join operation
requires
that the
sample
represents the original
input data
Example:
- The following example shows that a
uniform sample
will in general
not produce a
representative result.
Example:
- There is one tuple (a1, ..) in R1
- And there are k tuples
(a2, ..) in R1
-
k is very large (say: 1,000,000)
- Conversely: there is one tuple (a2, ..) in R2
- And there are k tuples
(a1, ..) in R2
- The problem:
- We only get some output if
the tuple (a1, b0)
in R1 is selected
The probability that the tuple (a1, b0) in R1 is selected is extremely small !!!
- We only get some output if
the tuple
(a2, c0)
in R2 is selected
So also, the probability that the tuple (a2, c0) in R2 is selected is extremely small !!!
- We only get some output if
the tuple (a1, b0)
in R1 is selected
- The
most likely scenario
of a (uniform) sample will be:
- Buffer 1 contains only (a2, ..) tuples
- Buffer 2 contains only (a1, ..) tuples
The join operation will produce no output tuples !!!
- The result is empty and it is certainly not representative to the original output...
- Consider the distribution
of the tuples in the
input sources
and the
result:
Input 1: Input 2: Output: --------- ---------- ------------ (a1,b0) (a2,c0) (a2,b1) (a1,c1) (a1,b0,a1,c1), (a1,b0,a1,c2), ..., (a1,b0,a1,ck), (a2,b2) (a1,c2) (a2,b1,a2,c0), (a2,b2,a2,c0), ..., (a2,bk,a2,c0) ... ... (a2,bk) (a1,ck)
- The result set of the
join
R1 ▷◁ R2
contains the
source tuple
(a1,b0)
from R1
There is only a single tuple (a1,b0) in R1
- The result set of the
join
R1 ▷◁ R2
contains the
source tuple
(a2,c0)
from R2
There is only a single tuple (a1,b0) in R1
- Observation:
- Tuples in R1 and R2 are not sampled uniformly but weighted
- Weighted by what factor ???
Answer:
- A tuple x from the relation R1 is sampled with a probability that is proportional the number of tuples in R2 that join with x
- A weighted sample
is a random sample where
different items
are selected
with
different probablity
- The following figure illustrates the
difference
between:
- unweighted sampling
of the elements 1, 2, 3, and 4
- weighted sampling of 1, 2, 3 and 4 where Prob[select 1] = 1/2, Prob[select 1] = 1/6, Prob[select 1] = 1/6, and Prob[select 1] = 1/6
Example:
- Because the probablity of selecting 1 is 3 times as high as the other items, it is as if the value 1 is duplicated 3 times in the input set
- unweighted sampling
of the elements 1, 2, 3, and 4
- Definition: Weighted Sampling
- Let w(t)
be a non-negative integer
(weight) assigned to
a
tuple t
in a relation R
- A weighted WR (With Replacement) sample from relation R is the same as an unweighted WR sample from a modified relation R* where there are w(t) copies of each tuple t.
- Let w(t)
be a non-negative integer
(weight) assigned to
a
tuple t
in a relation R
- The following algorithm constructs a
weighted
sample
from an input stream:
w(i) = relative weight of item i // Must be given a priori N = 0; // N = number of tuples selected while ( not EOF ) { t = read_next_tuple(); // Get next tuple from input // Select tuple if ( random() < w(t) ) { Sample[N] = t; N++; } }
- The algorithm is adapted from
the unweighted coin toss version
(See: click here )
- Problem:
- The output size of the sample can grow arbitrarily large
- We can adapt the
concise sample method
to obtain a weight sample
- Weighted Concise Sample:
f = 1; // f*w(t) = selection probability S = empty; // S = Concise Sample while ( not EOF ) { t = next input value; if ( random() < f*w(t) ) { if ( t ∈ S ) { increase count of the t value in S } else { add (t,1) to S } } /* ------------------------------------------- Deletion step: Adjust sample when it gets too large... ------------------------------------------- */ if ( size(S) > MaxSize ) { f' = β × f; // New selection probab // β < 1 for ( each sample t ∈ S ) do { for ( i = 1; i <= t.count; i++ ) { if ( random() < 1-β ) t.count--; } if ( t.count == 0 ) delete t from S; } f = f'; } }
- Why the result will be a weighted sample:
- To have a weighted sample,
we have:
- Two different values
x1
and x2 are sampled
with their relative
"importance":
Probab[ x1 selected ] w(x1) ----------------------- = -------- Probab[ x2 selected ] w(x2) - This fact can be achieve using:
Probab[ x1 selected ] = f * w(x1) Probab[ x2 selected ] = f * w(x2)
- Two different values
x1
and x2 are sampled
with their relative
"importance":
- Denote:
- S = sample before the deletion
- S' = sample after the deletion
The sample S is clearly a weight random sample.
It is not obvious that S' will be a weighted random sample.
So, just like the uniform case, we must show that S' will also be a weighted random sample
- To show this,
let us consider how the algorithm select a value
x1.
Input stream: ... x1 ... x1 ... x1 ... x1 .... x1 ... | | | | | Selected with | | | | | probab = f*w(x1) v v v v v S: x1 x1Each value x1 is included with probability f*w(x1)
- Each value
x1 that is already in
S is
deleted with probability 1 - β
So:
- Each value x1 that is already in S is included in S' with probability β
Graphically:
Input stream: ... x1 ... x1 ... x1 ... x1 .... x1 ... | | | | | Selected with | | | | | probab = f*w(x1) v v v v v S: x1 x1 | Selected with | probab = β v S': x1
- Then the selection probability
of a value x1
(directly) into
S' is then:
Input stream: ... x1 ... x1 ... x1 ... x1 .... x1 ... | | | | | | | | | | v v v v v | Selected with | probab = β * f * w(x1) v S': x1 - So the sample preserve
the weight in the
correct proportion
The resulting sample is also weighted
- To have a weighted sample,
we have: