org.apache.nutch.protocol.ftp

## Class FtpRobotRulesParser

- java.lang.Object
- - org.apache.nutch.protocol.RobotRulesParser
  - - org.apache.nutch.protocol.ftp.FtpRobotRulesParser

- All Implemented Interfaces:
    Configurable, Tool

      public class FtpRobotRulesParser
      extends RobotRulesParser

  This class is used for parsing robots for urls belonging to FTP protocol. It extends the generic `RobotRulesParser` class and contains Ftp protocol specific implementation for obtaining the robots file.

- - ### Field Summary

    - ### Fields inherited from class org.apache.nutch.protocol.RobotRulesParser

      `agentNames, CACHE, conf, EMPTY_RULES, FORBID_ALL_RULES, whiteList`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `FtpRobotRulesParser(Configuration conf)`

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `crawlercommons.robots.BaseRobotRules` | `getRobotRulesSet(Protocol ftp, URL url, List<Content> robotsTxtContent)` The hosts for which the caching of robots rules is yet to be done, it sends a Ftp request to the host corresponding to the `URL` passed, gets robots file, parses the rules and caches the rules object to avoid re-work in future.

    - ### Methods inherited from class org.apache.nutch.protocol.RobotRulesParser

      `getConf, getRobotRulesSet, isWhiteListed, main, parseRules, run, setConf`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### FtpRobotRulesParser

          public FtpRobotRulesParser(Configuration conf)

  - ### Method Detail

    - #### getRobotRulesSet

          public crawlercommons.robots.BaseRobotRules getRobotRulesSet(Protocol ftp,
                                                                       URL url,
                                                                       List<Content> robotsTxtContent)

      The hosts for which the caching of robots rules is yet to be done, it sends a Ftp request to the host corresponding to the `URL` passed, gets robots file, parses the rules and caches the rules object to avoid re-work in future.
      Specified by:
        `getRobotRulesSet` in class `RobotRulesParser`
      Parameters:
        `ftp` - The `Protocol` object
        `url` - URL
        `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
      Returns:
        robotRules A `BaseRobotRules` object for the rules

Copyright © 2020 The Apache Software Foundation