robots.txt class

Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details).

Reviews

There are no comments available yet.

Here you can write a comment


Please enter at least 10 characters.
Loading... Please wait.
* Pflichtangabe
ANG GES G P
  This debugger is build with serveral interacting technologies. The core element is a PHP based function, which can be integrated into the appl ...
ANG GES G P
php klassen zum erzeigen einer rss 1.0 datei. unterstützt dublic core und SY module
ANG GES G P
This class is a expansion to the original PHP string funcions. All functions was make to use special characters (192-223 and 224-255 in ascii ...
ANG GES G P
displays content of variables in human-readable format. just call debug_var($VAR) and an window opens with the struktur and content
ANG GES G P
A "better" CountryList class. Supports German, English and Frensh ISO 3166 Country Codes and Names.
ANG GES G P
Try to simplify and improve Sword Su's menutree class
ANG GES G P
Almost every PHP and SQL site a search. The problem is that these searches almost never support booleans. If they do it's usually pretty bad. The ...
ANG GES G P
Class to convert any document, that can be read by MS Word, to another format supported by Word. Converts extensions from/to: doc,dot,txt,rtf,ht ...
410 x viewed
0 x memorized
Details to the advertisement
ANG GES G P
Make a request

You are no longer logged in as a user. Please enter your name and your E-mail-adress!

Loading... Please wait.
Details to the seller

toon.goedhart@