Measuring the validity of peer-to-peer data for information retrieval applications

Noam Koenigstein, Yuval Shavitt, Ela Weinsberg, Udi Weinsberg*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Peer-to-peer (p2p) networks are being increasingly adopted as an invaluable resource for various information retrieval (IR) applications, including similarity estimation, content recommendation and trend prediction. However, these networks are usually extremely large and noisy, which raises doubts regarding the ability to actually extract sufficiently accurate information. This paper quantifies the measurement effort required to obtain and optimize the information obtained from p2p networks for the purpose of IR applications. We identify and measure inherent difficulties in collecting p2p data, namely, partial crawling, user-generated noise, sparseness, and popularity and localization of content and search queries. These aspects are quantified using music files shared in the Gnutella p2p network. We show that the power-law nature of the network makes it relatively easy to capture an accurate view of the popular content using relatively little effort. However, some applications, like trend prediction, mandate collection of the data from the "long tail", hence a much more exhaustive crawl is needed. Furthermore, we show that content and search queries are highly localized, indicating that location-crossing conclusions require a wide spread spatial crawl. Finally, we present techniques for overcoming noise originating from user generated content and for filtering non-informative data, while minimizing information loss.

Original languageEnglish
Pages (from-to)1092-1102
Number of pages11
JournalComputer Networks
Volume56
Issue number3
DOIs
StatePublished - 23 Feb 2012

Keywords

  • Information retrieval
  • Measurement
  • Peer-to-peer

Fingerprint

Dive into the research topics of 'Measuring the validity of peer-to-peer data for information retrieval applications'. Together they form a unique fingerprint.

Cite this