iwla

iwla Commit Details

Date:2015-04-06 17:52:31 (6 years 3 months ago)
Author:Grégory Soutadé
Branch:dev, master
Commit:df78a3f4cbefd3ec26375a6bd4605a6f149fc729
Parents: 2ed7551049bc5d9de7bdffc76eed229e78b4c5c1
Message:[pre_analysis/robots] Don't checks for /robots.txt request, but endswith /robots.txt for robot detection

Changes:
Mplugins/pre_analysis/robots.py (1 diff)

File differences

plugins/pre_analysis/robots.py
100100
101101
102102
103
103
104104
105105
106106
for hit in super_hit['requests']:
# 3) /robots.txt read
if hit['extract_request']['http_uri'] == '/robots.txt':
if hit['extract_request']['http_uri'].endswith('/robots.txt'):
isRobot = True
break

Archive Download the corresponding diff file

Branches

Tags