org.apache.nutch.crawl

## Class LinkDbMerger

- java.lang.Object
- - org.apache.hadoop.conf.Configured
  - - org.apache.nutch.crawl.LinkDbMerger

- All Implemented Interfaces:
    Configurable, Tool

      public class LinkDbMerger
      extends Configured
      implements Tool

  This tool merges several LinkDb-s into one, optionally filtering URLs through the current URLFilters, to skip prohibited URLs and links.

  It's possible to use this tool just for filtering - in that case only one LinkDb should be specified in arguments.

  If more than one LinkDb contains information about the same URL, all inlinks are accumulated, but only at most `linkdb.max.inlinks` inlinks will ever be added.

  If activated, URLFilters will be applied to both the target URLs and to any incoming link URL. If a target URL is prohibited, all inlinks to that target will be removed, including the target URL. If some of incoming links are prohibited, only they will be removed, and they won't count when checking the above-mentioned maximum limit.

  Author:
    Andrzej Bialecki

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static class` | `LinkDbMerger.LinkDbMergeReducer`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `LinkDbMerger()`
    `LinkDbMerger(Configuration conf)`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `static Job` | `createMergeJob(Configuration config, Path linkDb, boolean normalize, boolean filter)`
    `static void` | `main(String[] args)`
    `void` | `merge(Path output, Path[] dbs, boolean normalize, boolean filter)`
    `int` | `run(String[] args)`

    - ### Methods inherited from class org.apache.hadoop.conf.Configured

      `getConf, setConf`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface org.apache.hadoop.conf.Configurable

      `getConf, setConf`

- - ### Constructor Detail

    - #### LinkDbMerger

          public LinkDbMerger()

    - #### LinkDbMerger

          public LinkDbMerger(Configuration conf)

  - ### Method Detail

    - #### merge

          public void merge(Path output,
                            Path[] dbs,
                            boolean normalize,
                            boolean filter)
                     throws Exception

      Throws:
        `Exception`

    - #### createMergeJob

          public static Job createMergeJob(Configuration config,
                                           Path linkDb,
                                           boolean normalize,
                                           boolean filter)
                                    throws IOException

      Throws:
        `IOException`

    - #### main

          public static void main(String[] args)
                           throws Exception

      Parameters:
        `args` -
      Throws:
        `Exception`

    - #### run

          public int run(String[] args)
                  throws Exception

      Specified by:
        `run` in interface `Tool`
      Throws:
        `Exception`

Copyright © 2020 The Apache Software Foundation