Skip to content
Lesuto Render Engine

Caddy

Render Engine With
CADDY

A few lines in your site block forward crawlers to Render Engine and leave everyone else on your site.

Steps

Set It Up
Step By Step

  1. 01

    Add The Rule

    Needs Caddy 2.8 or later. Replace yourbrand.com with your site, put how your site already serves its pages into the snippet's (lesuto_site) block, and use the snippet's site block in place of your current one. Keep status 200 on file_server: it makes your normal page answer 200 when it stands in for a failed render.

  2. 02

    Add Your API Key

    Replace YOUR_RENDER_KEY with your key, or copy the snippet from the Setup page in the app with your key filled in.

  3. 03

    Check And Reload

    Run caddy validate, then caddy reload.

  4. 04

    Check It Works

    After a real crawler visits, the page appears in the Pages tab. To check now, use Recache on a page in the Pages tab and wait until its Rendered time updates, then request that page with a crawler user agent: you get the rendered copy from the cache. A crawler user agent sent from your own machine never starts a fresh render, so a page with no cached copy still shows your normal page.

Copy The Rule

Replace YOUR_RENDER_KEY with your API key, or copy it from the Setup page in the app with your key filled in. Any failure falls through to your normal page.

Caddyfilecaddyfile
# Caddyfile, Caddy 2.8 or later. Put how this site already serves its pages in (lesuto_site), then your site block uses it.
# {client_ip} is the address that connected unless trusted_proxies names your proxy.
(lesuto_site) {
	root * /var/www/html
	try_files {path} /index.html
	file_server {
		# 200 also when this page stands in for a failed render (handle_errors below).
		status 200
	}
}

yourbrand.com {
	@lesuto_crawler {
		method GET
		header_regexp User-Agent "(?i)(googlebot|bingbot|applebot|duckduckbot|yandexbot|baiduspider|oai-searchbot|chatgpt-user|gptbot|claude-searchbot|claude-user|claudebot|perplexity-user|perplexitybot|mistralai-user|duckassistbot|amazonbot|meta-externalagent|cohere-ai|youbot|facebookexternalhit|facebot|twitterbot|linkedinbot|pinterestbot|slackbot|discordbot|telegrambot|whatsapp)"
		not path_regexp "(?i)\.(js|css|xml|png|jpe?g|gif|pdf|ico|svg|webp|avif|woff2?|ttf|eot|map|txt|json|mp4|webm)$"
	}

	handle @lesuto_crawler {
		reverse_proxy https://api.lesuto.com {
			rewrite /api/v2/render-engine/render
			header_up Host {upstream_hostport}
			header_up X-Render-Key "YOUR_RENDER_KEY"
			header_up X-Render-Url "{http.request.scheme}://{http.request.host}{http.request.orig_uri}"
			header_up X-Render-Client-IP {client_ip}
			transport http {
				dial_timeout 5s
				response_header_timeout 18s
			}

			# Only pages Render Engine answered carry X-Render-Cache.
			@rendered {
				header X-Render-Cache *
				status 2xx 301 302 303 307 308 404 410
			}
			handle_response @rendered {
				header >Cache-Control "private, max-age=0"
				header >CDN-Cache-Control "no-store"
				header >Vary "User-Agent"
				# Render Engine's own sandbox policy is for its address, not your site's.
				header -Content-Security-Policy
				copy_response
			}
			# Anything else: your normal page.
			handle_response {
				import lesuto_site
			}
		}
	}

	handle {
		import lesuto_site
	}

	# Render Engine unreachable or too slow: crawlers get your normal page.
	handle_errors 502 503 504 {
		@lesuto_crawler_failed {
			header_regexp User-Agent "(?i)(googlebot|bingbot|applebot|duckduckbot|yandexbot|baiduspider|oai-searchbot|chatgpt-user|gptbot|claude-searchbot|claude-user|claudebot|perplexity-user|perplexitybot|mistralai-user|duckassistbot|amazonbot|meta-externalagent|cohere-ai|youbot|facebookexternalhit|facebot|twitterbot|linkedinbot|pinterestbot|slackbot|discordbot|telegrambot|whatsapp)"
		}
		handle @lesuto_crawler_failed {
			import lesuto_site
		}
	}
}

Full Guide

Every detail, edge case and fix on the Lesuto docs.

Rendered Pages
In About 15 Minutes

Add the rule, reload, and check one page.

Set Up Render Engine With Caddy