PeristyleBot

PeristyleBot reads recipes from food blogs so people can find them and turn them into a grocery cart. It crawls a site on a schedule only when the site's owner has signed up with us and turned that on. Otherwise it fetches a single page, once, because a person asked it to. It never trains a model on what it reads.

Identifying it

Requests carry this User-Agent:

PeristyleBot/1.0 (+https://peristyle.io/bot)

A User-Agent is only a claim, so PeristyleBot also signs every request using Web Bot Auth (HTTP Message Signatures, RFC 9421). Real requests carrySignature, Signature-Input and aSignature-Agent header pointing at our public key set:

https://peristyle.io/.well-known/http-message-signatures-directory

To check a request yourself, confirm its Signature-Agent names exactly that URL, fetch the keys there, and verify theSignature with the key whose thumbprint matches thekeyid in Signature-Input. The key set is itself signed with the same keys, so you can confirm it came from us too. If verification fails, the traffic is not PeristyleBot no matter what the User-Agent says. Cloudflare does this check for you if your site is behind it.

When it fetches

PeristyleBot fetches pages in three situations, and only these three:

1. Scheduled sync, for creators who turned it on. A creator who signs up with Peristyle can switch on sync. From then on, about every 15 minutes, PeristyleBot checks that site's sitemap, RSS feed or WordPress API for new and changed recipes. A site whose owner has not done this is never synced.
2. A one-time import when a creator joins. When a creator joins or upgrades, PeristyleBot reads their existing recipes once to bring them in, and checks their site to confirm they own the domain.
3. A single page, because a person asked. A signed-in shopper can paste a recipe link from any site to shop its ingredients, and a creator can run a page through our free recipe checker. PeristyleBot then fetches that one page, once — not the rest of the site. What it reads serves only the person who asked: a pasted recipe is kept only for that shopper's cart, never added to search or listings, and no one else sees it.

What it does

• Purpose: search and shopping. Recipes from creators who sync with us are indexed so a person can find them and shop the ingredients. A page a shopper pastes goes into their cart and nowhere else. Every recipe keeps its title, author and a link back to your page.
• Not model training. We do not train or fine-tune models on crawled content, and we do not sell it to anyone who does.
• Recipe pages, and the lists of them. It reads structured recipe data (schema.org Recipe JSON-LD) and the page it sits on. For a site that syncs with us, it also reads the sitemap, feed or WordPress API that lists those pages.

How often it visits

As little as it can, and it tries hard to be boring about it. On a site that syncs with us, the 15-minute check reads your sitemap or feed, not your pages. A page already indexed is re-checked only when that sitemap or feed says it changed, or when our copy is more than two weeks old. When we do re-check, we sendIf-None-Match and If-Modified-Since, so if your server answers 304 Not Modified it never sends the page at all — no bandwidth, no render.

A page a shopper pastes is fetched once, at the moment they paste it, and not again on a schedule.

Serving ETag or Last-Modified headers on your recipe pages is the single most effective way to make our crawling cheap for you. Sites Peristyle builds do it by default.

Controlling it

How to stop PeristyleBot depends on which kind of fetch you are seeing.

Scheduled sync and imports happen only because a site's owner signed up and turned them on, so the off switch is that consent. If you are a creator with a Peristyle account, turning off sync in your dashboard stops the crawl immediately. Or emailhello@peristyle.io and we stop crawling and remove anything already indexed. No form, no waiting period.

Single-page fetches happen because a person asked for that page, the way their own browser would fetch it for them. Crawl rules in robots.txt are written for bots deciding by themselves what to visit, so they do not govern these requests, and PeristyleBot does not apply them. Each one is a single page, fetched once, signed, and kept only for the person who asked. If you still want to refuse them, block PeristyleBot at your server or firewall — the User-Agent and signature make it easy to match — or email us and we will talk it through.

A note on robots.txt: PeristyleBot does not currently read it. A Disallow rule aimed at us will not stop either kind of fetch: syncing stops when you turn it off or email us, and single-page fetches are not crawling. We would rather say this plainly than let you believe a control is working when it is not.

Something looks wrong

If PeristyleBot is crawling too often, reaching pages it should not, or you are seeing traffic claiming to be us that fails signature verification, tell us at hello@peristyle.io and we will look the same day. Include a few log lines with timestamps if you can — the signature header is enough for us to confirm whether it was really us.