4 The Gatherer

• 4.1 Overview
• 4.2 Basic setup
  • Gathering News URLs with NNTP
  • Cleaning out a Gatherer
• 4.3 RootNode specifications
  • 4.3.1 RootNode filters
  • 4.3.2 Generic Enumeration filter description
  • 4.3.3 Example RootNode configuration
  • 4.3.4 Using extreme values -- ``robots''
  • 4.3.5 Gatherer enumeration vs. candidate selection
• 4.4 Generating LeafNode/RootNode URLs from a program
• 4.5 Extracting data for indexing: The Essence summarizing subsystem
  • 4.5.1 Default actions of ``stock'' summarizers
  • 4.5.2 Summarizing SGML data
    • Location of support files
    • The SGML to SOIF table
    • Errors and warnings from the SGML Parser
    • Creating a summarizer for a new SGML-tagged data type
    • The SGML-based HTML summarizer
    • Adding META data to your HTML
    • Other examples
  • 4.5.3 Summarizer components distribution
    • Using ``Rainbow'' to summarize MIF and RTF documents
    • The translation table
  • 4.5.4 Customizing the type recognition, candidate selection, presentation unnesting, and summarizing steps
    • Customizing the type recognition step
    • Customizing the candidate selection step
    • Customizing the presentation unnesting step
    • Customizing the summarizing step
• 4.6 Post-Summarizing: Rule-based tuning of object summaries
  • The Rules file
  • Rewriting URLs
• 4.7 Gatherer administration
  • 4.7.1 Setting variables in the Gatherer configuration file
  • 4.7.2 Local file system gathering for reduced CPU load
  • 4.7.3 Gathering from password-protected servers
  • 4.7.4 Controlling access to the Gatherer's database
  • 4.7.5 Periodic gathering and realtime updates
  • 4.7.6 The local disk cache
  • 4.7.7 Incorporating manually generated information into a Gatherer
• 4.8 Troubleshooting