User-agent: Googlebot User-agent: * User-agent: ArchitextSpider User-agent: FAST-WebCrawler User-agent: Lycos_Spider_(T-Rex) Allow:/ Allow: /Course_table.html Allow: /index.html Allow: /all_courses.html