next up previous contents index
Next: 4.5 Extracting data for Up: 4 The Gatherer Previous: 4.3.5 Gatherer enumeration vs.

4.4 Generating LeafNode/RootNode URLs from a program

   

As of version 1.4, it is now possible to generate RootNode or LeafNode URLs automatically from program output. This might be useful when gathering a large number of Usenet newsgroups, for example. The program is specified inside the RootNode or LeafNode section, preceded by a pipe symbol.

        <LeafNodes>
        |generate-news-urls.sh
        </LeafNodes>

The script must output valid URLs, such as

        news:comp.unix.voodoo
        news:rec.pets.birds
        http://www.nlanr.net/
        ...

In the case of RootNode URLs, enumeration parameters can be given after the program.

        <RootNodes>
        |my-fave-sites.pl  Depth=1  URL=5000,url-filter
        </RootNodes>



Duane Wessels
Wed Jan 31 23:46:21 PST 1996