searpl

a small php sqlite search engine. <a href="https://thunix.net/~xfnw/search">see it in action</a>
Log | Files | Refs | README

urls.sh (311B)

      1 wget -w 2 --random-wait --spider --force-html --tries 1 --timeout 1 -r -l1 -H -U 'searplbot/1.0' $@ 2>&1 | tee -a wg
      2 
      3 grep '^--' wg | awk '{ print $3 }' \
      4   | grep -v '\.\(css\|js\|png\|gif\|jpg\|txt\|ico\|ttf\|svg\)$' \
      5   | sort | uniq \
      6   | tee -a ur
      7 
      8 rm wg
      9 
     10 sleep 10
     11 
     12 php crawl.php $(cat ur | shuf)
     13 
     14 rm ur
     15 
     16