ZeroWhen's bot

Requests from ZeroWhen's site checks say who they are:

What it does

• Only pages a person asked for. It doesn't crawl, follow links or build an index.
• Not often. A watched page is usually checked every few minutes to once a day, as the person chose.
• It reads your robots.txt (RFC 9309) before suggesting your site to anyone, and follows a group for ZeroWhen if you write one. Following it in every scheduled check is being rolled out.
• It takes no for an answer. If your site blocks it, it doesn't disguise itself or switch addresses. It tells the person to use your own alerts (price-drop or restock emails, saved searches) instead.

To ask it not to visit

User-agent: Zerowhen
Disallow: /

Or email [email protected] with your domain, and we will stop.

To learn how many sites let a polite bot in, we visited about 1,000 well-known sites once each: their robots.txt, then the home page (only if robots.txt allowed it).

On September 29 we made one second pass, to the roughly 680 of those sites that let us in: robots.txt, the home page, and one page it links to (a product, listing or article), only where robots.txt allows it. That's three requests per site at most.

On September 29–30 we make one last pass, a snapshot, to the same sites that let us in: robots.txt, the home page and one page it links to, once each, a few sites at a time. We keep a private copy of those pages and how long they took, only to test how many watchers our own servers can carry with realistic pages. The copies are never published or shared, and are deleted when the testing is done.

So that people know in advance which sites can be watched, and which can't, we are reading each site's rules: its robots.txt and its terms of use. A trial on October 1–2 covers 500 of the most visited sites in Canada: robots.txt, then the home page, one page it links to and the terms page, once each and only where robots.txt allows it. That's four requests per site at most, a few sites at a time. We keep what the terms say about automated access, with a link to them; if your terms forbid it, ZeroWhen won't watch your site.

From October 2, 2026, the same pass covers the roughly 72,000 most visited sites in the United States (the top 100,000 in Google's Chrome UX Report): robots.txt, the home page, one page it links to and the terms page, once each and only where robots.txt allows it. When the home page links no terms, we try a few usual addresses for them (such as /terms); when the terms page is only a list of legal documents, we follow its terms link once. That's eight requests per site at most, one at a time, at least a second apart (longer if your robots.txt sets a Crawl-delay). We keep what the terms say about automated access, with a link to them, the page titles and a short excerpt of the home page's text; if your terms forbid automated access, ZeroWhen won't watch your site. Adult sites are skipped without a request.

From October 2, 2026, the same pass also covers the roughly 49,000 sites in Canada's top 100,000 that the US pass did not already include, on the same terms. Shops on Shopify get only the robots.txt request for now (Shopify limits how often one address may visit its stores); they are visited later, slowly.

After that, the same pass covers the sites in the top 100,000 of the UK (about 55,000), Australia (about 47,000), New Zealand and Ireland that earlier passes did not include, one country at a time, on the same terms.

For a page of real deals from small shops, we read each shop's own sale page once a day. That covers shops in our directory whose terms say nothing against automated access: in the United States about 2,300 shops, plus about 1,200 stores on Shopify whose terms say nothing against it or that we found no terms for; from October 3, 2026 about 1,900 shops in Canada; and each other country above once its pass has finished. Shops whose terms forbid or restrict it are left out.

• Finding the sale page, once a week at most: the home page, and only if it links no sale page, the sitemap or a usual address such as /sale. At most ten requests, one at a time, at least a second apart (longer if your robots.txt sets a Crawl-delay).
• Reading it, once a day: that one page. Where its prices only appear once the page's scripts run, it is drawn once in a browser that sends the same User-Agent.
• Shopify stores, once a day: the sale collection's product list (/collections/sale/products.json), or the first page of /products.json. One store at a time, at most one request every six seconds across all stores.

Only where robots.txt allows it, never behind a login. We keep each item's name, price, the shop's own "was" price and its link, to show the day's deals and how prices change over time. Copies of the pages are kept for 14 days, then deleted. To be left out, write a ZeroWhen group in robots.txt or email [email protected]; we stop from the next day. On October 2, 2026 we tried this once on about 300 shops.