searpl

a small php sqlite search engine. <a href="https://thunix.net/~xfnw/search">see it in action</a>
Log | Files | Refs | README
commit d4d5e67da8212c7ff88b5512269247edd40bf3a2
parent 036e3addb2c21adc68e78a04e726a620c52f5a84
Author: xfnw <xfnw@thunix.net>
Date:   Thu, 31 Dec 2020 21:05:50 -0500

uniquely sort urls and put the db the gitignore

Diffstat:

A.gitignore | 2++


Murls.sh | 1+


2 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,2 @@
+db.sqlite
+
diff --git a/urls.sh b/urls.sh
@@ -2,6 +2,7 @@ wget --spider --force-html --tries 1 --timeout 1 -r -l1 -H -U 'searplbot/1.0' $@
 
 grep '^--' wg | awk '{ print $3 }' \
   | grep -v '\.\(css\|js\|png\|gif\|jpg\|txt\|ico\|ttf\|svg\)$' \
+  | sort | uniq \
   | tee -a ur
 
 rm wg