Processing top-k queries from samples

Edith Cohen*, Nadav Grossaug, Haim Kaplan

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


Top- k queries are desired aggregation operations on data sets. Examples of queries on network data include the top 100 source AS's, top 100 ports, or top Domain names over IP packets or over IP flow records. Since the complete dataset is often not available or not feasible to examine, we are interested in processing top-k queries from samples. If all records can be processed, the top-k items can be obtained by counting the frequency of each item. Even when the full dataset is observed, however, resources are often insufficient for such counting and techniques were developed to overcome this issue. When we can observe only a random sample of the records, an orthogonal complication arises: The top frequencies in the sample are biased estimates of the actual top- k frequencies. This bias depends on the distribution and must be accounted for when seeking the actual value. We address this by designing and evaluating several schemes that derive rigorous confidence bounds for top-k estimates. Simulations on various data sets that include IP flows data, show that schemes that exploit more of the structure of the sample distribution produce much tight confidence intervals with an order of magnitude fewer samples than simpler schemes that utilize only the sampled top-k frequencies. The simpler schemes, however, are more efficient in terms of computation. Our work is basic and is widely applicable to all applications that process top-k and heavy hitters queries over a random sample of the actual records.

Original languageEnglish
Title of host publicationProceedings of CoNEXT'06 - 2nd Conference on Future Networking Technologies
StatePublished - 2006
Event2nd Conference on Future Networking Technologies, CoNEXT'06 - Lisboa, Portugal
Duration: 4 Dec 20067 Dec 2006

Publication series

NameProceedings of CoNEXT'06 - 2nd Conference on Future Networking Technologies


Conference2nd Conference on Future Networking Technologies, CoNEXT'06


Dive into the research topics of 'Processing top-k queries from samples'. Together they form a unique fingerprint.

Cite this