Any server
Render Engine With
ANY SERVER
The same rules every snippet follows, written out so you can add them to any server or framework.
Steps
Set It Up
Step By Step
- 01
Send Only Crawlers
Match crawler user agents on GET requests for pages, not files. Everyone else keeps getting your site.
- 02
Ask Render Engine
Send the page address, your key, the crawler's own address (from the connection or your platform, never a header a visitor can set) and its user agent, and wait at most 18 seconds.
- 03
Pass On Only Rendered Pages
Pass the answer on only when it is marked as rendered, with private cache headers. Anything else, including a timeout, gets your normal page.
- 04
Check It Works
After a real crawler visits, the page appears in the Pages tab. To check now, use Recache on a page in the Pages tab and wait until its Rendered time updates, then request that page with a crawler user agent: you get the rendered copy from the cache. A crawler user agent sent from your own machine never starts a fresh render, so a page with no cached copy still shows your normal page.
The Rules
Every Render Engine snippet follows these rules. Build them into your server or framework with your API key where it says YOUR_RENDER_KEY, and anything Render Engine does not render falls through to your normal page.
1. Send only crawlers. A GET request whose User-Agent matches (case-insensitive)
googlebot|bingbot|applebot|duckduckbot|yandexbot|baiduspider|oai-searchbot|chatgpt-user|gptbot|claude-searchbot|claude-user|claudebot|perplexity-user|perplexitybot|mistralai-user|duckassistbot|amazonbot|meta-externalagent|cohere-ai|youbot|facebookexternalhit|facebot|twitterbot|linkedinbot|pinterestbot|slackbot|discordbot|telegrambot|whatsapp
and whose path does not end in .(js|css|xml|png|jpe?g|gif|pdf|ico|svg|webp|avif|woff2?|ttf|eot|map|txt|json|mp4|webm) goes to Render Engine. Everyone else gets your site as usual.
2. Ask Render Engine for the page: GET https://api.lesuto.com/api/v2/render-engine/render with
X-Render-Key: YOUR_RENDER_KEY
X-Render-Url: the full address the crawler asked for, with its query string
X-Render-Client-IP: the address the crawler connected from (see 3)
User-Agent: the crawler's user agent
Do not follow redirects.
3. The crawler's address is your platform's verified client address where it has one (Cloudflare's
CF-Connecting-IP, Netlify's context.ip, Vercel's x-real-ip), otherwise the address of the connection.
Read X-Forwarded-For or X-Real-IP only behind a proxy you name in your server's settings, and then
the entry that proxy added. Any other header can be set by the visitor.
4. Wait at most 18 seconds. Render Engine finishes or gives up within that, and a render
you wait for is cached for the next crawler.
5. Pass the answer on only when it carries a X-Render-Cache header (every page Render Engine renders or
serves from its cache has one) and its status is 2xx or one of 301, 302, 303, 307, 308, 404, 410. Copy the status, the body,
Content-Type and any Location header, and leave Render Engine's other headers behind: its
Content-Security-Policy protects its own address, not your site.
nginx and Apache cannot read that header before answering and decide by status instead, so on those
put nothing between your server and Render Engine that answers 200 on its own (a captive proxy, a
cache serving its own pages).
6. Anything else (no X-Render-Cache header, another status such as 402 when a plan has ended, an error or
a timeout): serve your normal page as if Render Engine were not there. Your site never breaks. If that
page is fetched over HTTP, fetch it without the X-Render-Key and other X-Render headers you added.
7. Send the crawler's copy with
Cache-Control: private, max-age=0
CDN-Cache-Control: no-store
Vary: User-Agent
so no cache or CDN in front of your site gives it to people.Full Guide
Every detail, edge case and fix on the Lesuto docs.
Other Ways To Set Up
Pick the stack that matches your site, or start from the checklist.
Rendered Pages
In About 15 Minutes
Add the rules, deploy, and check one page.