awarded
awarded
comment
Merge [web-crawler] [webcrawling] [crawler] [spider] tags
My only very slight doubt would be if there is a difference in usage - 'my site is crawled' vs 'I send out a spider' (for example).
awarded