Caching
Fast For Crawlers
Safe For Your Site
Snapshots are cached so repeat crawls are instant, and every failure lands on your normal page.
Rendered Once
Served Fast
- The first crawl of a URL renders it live.
- A successful snapshot stays cached for the time you set on the domain: 1 hour, 6 hours, 1 day, 7 days, 14 days or 30 days.
- Crawls within that window are served straight from cache.
- Purge or recache one page or a whole domain from the app or your deploy script.
- Add your sitemap and pages are rendered before crawlers ask for them, again every 8 hours, daily or weekly.
- Mobile crawlers get the mobile layout, cached separately from desktop.

Built To
Fail Safely
Your routing rule always has a way back to your own page.
- If a render times out or fails, your rule serves your normal page instead.
- Your API keys only render domains you have verified in your workspace, so a key can never be used to crawl someone else’s site.
- If the renderer is briefly busy, your rule serves your normal page and the next crawl tries again.
- Run one routing layer. If Cloudflare forwards crawlers, your origin server does not need a rule too.
Keep Going
See the full request flow and pick a setup guide.
How it works
Your rule spots crawlers, Render Engine renders the page, people keep the live site.
Crawler coverage
Search, social, chat and AI crawlers, and the link previews they build.
Setup
A checklist, then one guide per stack: Cloudflare, nginx, Apache, Caddy, Next.js, Express, WordPress or any other server.
Fast Crawls
Without The Risk
Add Render Engine in front of your pages and keep your site as it is.