ROBOTS.TXT

Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl & index pages on their website.

User-agent: *
Disallow: /ads/
Disallow: /adx/bin/
Disallow: /puzzles/leaderboards/invite/*
Disallow: /svc
Disallow: /video/embedded/*
Disallow: /search
Disallow: /multiproduct/
Disallow: /hd/
Disallow: /inyt/
Disallow: /*?*query=
Disallow: /*.pdf$
Disallow: /*?*login=
Disallow: /*?*searchResultPosition=
Disallow: /*?*campaignId=
Disallow: /*?*mcubz=
Disallow: /*?*smprod=
Disallow: /*?*ProfileID=
Disallow: /*?*ListingID=
Disallow: /phspark/wp-admin/
Disallow: /phspark/*.zip$
Disallow: /phspark/*.csv$
Disallow: /phspark/deals/beta
Disallow: /phspark/data-requests
Disallow: /phspark/search
Disallow: /phspark/*?s=
Disallow: /phspark/*&xid=
Disallow: /phspark/*?q=
Disallow: /phspark/*?l=
Disallow: /search
Disallow: /*?*smid=
Disallow: /*?*partner=
Disallow: /*?*utm_source=
Allow: /phspark/*?*utm_source=
Allow: /ads/public/
Allow: /svc/news/v3/all/pshb.rss

User-agent: Twitterbot
Allow: /*?*smid=

User-Agent: omgilibot
Disallow: /

User-Agent: omgili
Disallow: /

User-agent: ia_archiver
Disallow: /

Sitemap: https://phspark.com/sitemaps/new/news.xml.gz
Sitemap: https://phspark.com/sitemaps/new/sitemap.xml.gz
Sitemap: https://phspark.com/sitemaps/new/wire.xml.gz
Sitemap: https://phspark.com/sitemaps/new/collections.xml.gz
Sitemap: https://phspark.com/sitemaps/new/video.xml.gz
Sitemap: https://phspark.com/sitemaps/new/cooking.xml.gz
Sitemap: https://phspark.com/sitemaps/new/recipe-collects.xml.gz
Sitemap: https://phspark.com/sitemaps/new/regions.xml
Sitemap: https://phspark.com/sitemaps/phspark.com/2016_election_sitemap.xml.gz
Sitemap: https://phspark.com/phspark/sitemapindex.xml

References

• https://www.nytimes.com/robots.txt