org.apache.nutch.indexer.geoip

## Class GeoIPIndexingFilter

- java.lang.Object
- - org.apache.nutch.indexer.geoip.GeoIPIndexingFilter

- All Implemented Interfaces:
    Configurable, IndexingFilter, Pluggable

      public class GeoIPIndexingFilter
      extends Object
      implements IndexingFilter

  This plugin implements an indexing filter which takes advantage of the GeoIP2-java API.

  The third party library distribution provides an API for the GeoIP2 Precision web services and databases. The API also works with the free GeoLite2 databases.

  Depending on the service level agreement, you have with the GeoIP service provider, the plugin can add a number of the following fields to the index data model:

  1. Continent
  2. Country
  3. Regional Subdivision
  4. City
  5. Postal Code
  6. Latitude/Longitude
  7. ISP/Organization
  8. AS Number
  9. Confidence Factors
  10. Radius
  11. User Type

  Some of the services are documented at the GeoIP2 Precision Services webpage where more information can be obtained.

  You should also consult the following three properties in `nutch-site.xml`

       <!-- index-geoip plugin properties -->
       <property>
         <name>index.geoip.usage</name>
         <value>insightsService</value>
         <description>
         A string representing the information source to be used for GeoIP information
         association. Either enter 'cityDatabase', 'connectionTypeDatabase',
         'domainDatabase', 'ispDatabase' or 'insightsService'. If you wish to use any one of the
         Database options, you should make one of GeoIP2-City.mmdb, GeoIP2-Connection-Type.mmdb,
         GeoIP2-Domain.mmdb or GeoIP2-ISP.mmdb files respectively available on the Hadoop classpath
         and available at runtime. This can be achieved by adding it to $NUTCH_HOME/conf
         </description>
       </property>

       <property>
         <name>index.geoip.userid</name>
         <value></value>
         <description>
         The userId associated with the GeoIP2 Precision Services account.
         </description>
       </property>

       <property>
         <name>index.geoip.licensekey</name>
         <value></value>
         <description>
         The license key associated with the GeoIP2 Precision Services account.
         </description>
       </property>

- - ### Field Summary

    - ### Fields inherited from interface org.apache.nutch.indexer.IndexingFilter

      `X_POINT_ID`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `GeoIPIndexingFilter()` Default constructor for this plugin

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `NutchDocument` | `filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)` Adds fields or otherwise modifies the document that will be indexed for a parse.
    `Configuration` | `getConf()`
    `void` | `setConf(Configuration conf)`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### GeoIPIndexingFilter

          public GeoIPIndexingFilter()

      Default constructor for this plugin

  - ### Method Detail

    - #### getConf

          public Configuration getConf()

      Specified by:
        `getConf` in interface `Configurable`
      See Also:
        `Configurable.getConf()`

    - #### setConf

          public void setConf(Configuration conf)

      Specified by:
        `setConf` in interface `Configurable`
      See Also:
        `Configurable.setConf(org.apache.hadoop.conf.Configuration)`

    - #### filter

          public NutchDocument filter(NutchDocument doc,
                                      Parse parse,
                                      Text url,
                                      CrawlDatum datum,
                                      Inlinks inlinks)
                               throws IndexingException

      Description copied from interface: `IndexingFilter`
      Adds fields or otherwise modifies the document that will be indexed for a parse. Unwanted documents can be removed from indexing by returning a null value.
      Specified by:
        `filter` in interface `IndexingFilter`
      Parameters:
        `doc` - document instance for collecting fields
        `parse` - parse data instance
        `url` - page url
        `datum` - crawl datum for the page (fetch datum from segment containing fetch status and fetch time)
        `inlinks` - page inlinks
      Returns:
        modified (or a new) document instance, or null (meaning the document should be discarded)
      Throws:
        `IndexingException`
      See Also:
        `IndexingFilter.filter(org.apache.nutch.indexer.NutchDocument, org.apache.nutch.parse.Parse, org.apache.hadoop.io.Text, org.apache.nutch.crawl.CrawlDatum, org.apache.nutch.crawl.Inlinks)`

Copyright © 2020 The Apache Software Foundation