Caddy
Render Engine With
CADDY
A few lines in your site block forward crawlers to Render Engine and leave everyone else on your site.
Steps
Set It Up
Step By Step
- 01
Add The Rule
Needs Caddy 2.8 or later. Replace yourbrand.com with your site, put how your site already serves its pages into the snippet's (lesuto_site) block, and use the snippet's site block in place of your current one. Keep status 200 on file_server: it makes your normal page answer 200 when it stands in for a failed render.
- 02
Add Your API Key
Replace YOUR_RENDER_KEY with your key, or copy the snippet from the Setup page in the app with your key filled in.
- 03
Check And Reload
Run caddy validate, then caddy reload.
- 04
Check It Works
After a real crawler visits, the page appears in the Pages tab. To check now, use Recache on a page in the Pages tab and wait until its Rendered time updates, then request that page with a crawler user agent: you get the rendered copy from the cache. A crawler user agent sent from your own machine never starts a fresh render, so a page with no cached copy still shows your normal page.
Copy The Rule
Replace YOUR_RENDER_KEY with your API key, or copy it from the Setup page in the app with your key filled in. Any failure falls through to your normal page.
# Caddyfile, Caddy 2.8 or later. Put how this site already serves its pages in (lesuto_site), then your site block uses it.
# {client_ip} is the address that connected unless trusted_proxies names your proxy.
(lesuto_site) {
root * /var/www/html
try_files {path} /index.html
file_server {
# 200 also when this page stands in for a failed render (handle_errors below).
status 200
}
}
yourbrand.com {
@lesuto_crawler {
method GET
header_regexp User-Agent "(?i)(googlebot|bingbot|applebot|duckduckbot|yandexbot|baiduspider|oai-searchbot|chatgpt-user|gptbot|claude-searchbot|claude-user|claudebot|perplexity-user|perplexitybot|mistralai-user|duckassistbot|amazonbot|meta-externalagent|cohere-ai|youbot|facebookexternalhit|facebot|twitterbot|linkedinbot|pinterestbot|slackbot|discordbot|telegrambot|whatsapp)"
not path_regexp "(?i)\.(js|css|xml|png|jpe?g|gif|pdf|ico|svg|webp|avif|woff2?|ttf|eot|map|txt|json|mp4|webm)$"
}
handle @lesuto_crawler {
reverse_proxy https://api.lesuto.com {
rewrite /api/v2/render-engine/render
header_up Host {upstream_hostport}
header_up X-Render-Key "YOUR_RENDER_KEY"
header_up X-Render-Url "{http.request.scheme}://{http.request.host}{http.request.orig_uri}"
header_up X-Render-Client-IP {client_ip}
transport http {
dial_timeout 5s
response_header_timeout 18s
}
# Only pages Render Engine answered carry X-Render-Cache.
@rendered {
header X-Render-Cache *
status 2xx 301 302 303 307 308 404 410
}
handle_response @rendered {
header >Cache-Control "private, max-age=0"
header >CDN-Cache-Control "no-store"
header >Vary "User-Agent"
# Render Engine's own sandbox policy is for its address, not your site's.
header -Content-Security-Policy
copy_response
}
# Anything else: your normal page.
handle_response {
import lesuto_site
}
}
}
handle {
import lesuto_site
}
# Render Engine unreachable or too slow: crawlers get your normal page.
handle_errors 502 503 504 {
@lesuto_crawler_failed {
header_regexp User-Agent "(?i)(googlebot|bingbot|applebot|duckduckbot|yandexbot|baiduspider|oai-searchbot|chatgpt-user|gptbot|claude-searchbot|claude-user|claudebot|perplexity-user|perplexitybot|mistralai-user|duckassistbot|amazonbot|meta-externalagent|cohere-ai|youbot|facebookexternalhit|facebot|twitterbot|linkedinbot|pinterestbot|slackbot|discordbot|telegrambot|whatsapp)"
}
handle @lesuto_crawler_failed {
import lesuto_site
}
}
}Full Guide
Every detail, edge case and fix on the Lesuto docs.
Other Ways To Set Up
Pick the stack that matches your site, or start from the checklist.
Rendered Pages
In About 15 Minutes
Add the rule, reload, and check one page.