Week 11 Readings

Required Readings:

1) David Hawking , Web Search Engines: Part 1 and Part 2 IEEE Computer, June 2006.

Part 1

I found the distinction that search engines shouldnt index the entire web to be refreshing. I feel as though we focus too little on privacy and information protection.

I also found this sections discussion of crawlers to be insightful. I had no idea crawlers could be "fooled" into retrieving falsified data. What a crazy world we live in when people spend their precious time devising plans to fool robots attempting to index the web.

Part 2

This section was pretty difficult to understand. Mostly, I was able to take away the idea that ranking systems are flawed, limiting, and redundant systems. I am puzzled as to how they enhance access to the deep web.

2) Shreeves, S. L., Habing, T. O., Hagedorn, K., & Young, J. A. (2005). Current developments and future trends for the OAI protocol for metadata harvesting. Library Trends, 53(4), 576-589.

Initially developed as a means to federate access to diverse e-print archives through metadata harvesting and aggregation, the protocol has demonstrated its potential usefulness to a broad range of communities.

Current developments

Aim to facilitate dissemination of content

OAI requires that data providers expose metadata in at least unqualified Dublin Core

Promotes a structure in which data providers can focus on building collections and content.

Community and Domain Specific OAI Services

- developed additional standards, tools, and metadata schemas

Examples:

Open Language Archives Community (OLAC)- virtual library of language resources

Sheet Music Consortium- multi element media. Search feature allows for the building or virtual collections, annotation, and advanced metadata records.

National Science Digital Library (NSDL)- building an aggregate not just a series of digital collections and content but also services to use these resources and the infrastructure to support both.

Shortcomings:

Sparse records

Completeness- lack of provenance

Ongoing Challenges:

Metadata Variation- difficult to collaborate multi metadata versions

Metadata Formats- more formats, more complex

OAI Data Provider Implementation Practices- underutilization of descriptive containers.

Communication uses:

3) MICHAEL K. BERGMAN, “The Deep Web: Surfacing Hidden Value” http://www.press.umich.edu/jep/07-01/bergman.html

Most of the Web's information is buried far down on dynamically generated sites, and standard search engines never find it.

BrightPlanet's search technology automates the process of making dozens of direct queries simultaneously using multiple-thread technology and thus is the only search technology, so far, that is capable of identifying, retrieving, qualifying, classifying, and organizing both "deep" and "surface" content.

The deep web is completely HUGE!

Focused only on HTTP protocol

Most users are only aware of the web's content that is immediately searchable.

How Search Engines Work:

1. Authors may submit their own Web pages, or the search engines "crawl" or "spider" documents by following one hypertext link to another.

2. Crawlers work by recording every hypertext link in every page they index crawling.

Discrete Tasks: (taken directly from the text)

• Qualification as a deep Web site.
• Estimation of total number of deep Web sites.
• Size analysis.
• Content and coverage analysis.
• Site page views and link references.
• Growth analysis.
• Quality analysis
• The findings are almost unbelievable.
• The deep Web is about 500 times larger than the surface Web, with, on average, about three times higher quality based on our document scoring methods on a per-document basis. On an absolute basis, total deep Web quality exceeds that of the surface Web by thousands of times. Total number of deep Web sites likely exceeds 200,000 today and is growing rapidly. Content on the deep Web has meaning and importance for every information seeker and market. More than 95% of deep Web information is publicly available without restriction. The deep Web also appears to be the fastest growing information component of the Web.
• One of the more counter-intuitive results is that 97.4% of deep Web sites are publicly available without restriction; a further 1.6% are mixed (limited results publicly available with greater results requiring subscription and/or paid fees); only 1.1% of results are totally subscription or fee limited.