Setting Up ETags for Your Website

The Incremental Refresh feature in Corpus relies on correctly configured ETags.
ETags (Entity Tags) help identify changes in resources. If the ETag value of a URL remains unchanged, Corpus skips re-indexing that content.
Without proper ETag setup, Incremental Refresh will not function as intended.

This guide provides ETag setup instructions for various environments.

---

1. Apache HTTP Server

Enable ETags Based on File Properties

Avoid differences across servers by ignoring inode values.

FileETag MTime Size
Header set ETag "%{MTime}e-%{Size}e"

• Source: KeyCDN Guide on ETags

---

2. Nginx

Enable ETags Globally

etag on;

Enable ETags for Specific Files (e.g., HTML)

location ~* \.html$ {
  etag on;
  add_header Cache-Control "no-cache";
}

• Note: If gzip is enabled, ETags may be stripped.
• Source: ServerFault Thread

Advanced: Use Dynamic ETag Module

Automatically generates ETags for dynamic responses.

load_module modules/ngx_http_dynamic_etag_module.so;

• Source: GetPageSpeed Nginx Extras

---

3. Node.js / Express (REST APIs)

Enable ETags in Express

app.set('etag', 'strong'); // or 'weak'
app.use(express.static('public', { etag: true }));

• Source: Express ETag Documentation

---

4. Spring Boot (Java)

Add ETag Support with ShallowEtagHeaderFilter

Automatically adds ETags based on response content.

@Bean
public Filter shallowEtagHeaderFilter() {
    return new ShallowEtagHeaderFilter();
}

• Source: Implementing ETags in REST APIs – Medium

---

5. Cloudflare

Cloudflare Behavior

• Cloudflare preserves ETags from the origin but does not generate them.
• If content is cached at Cloudflare, ETags may not be revalidated.

Best Practices:

• Ensure your origin server sends ETag headers.
• Use:

  Cache-Control: no-cache
  to force revalidation and allow conditional requests using ETags.

---

Summary Table

Environment	Method	Notes
Apache	FileETag + Header	Use MTime and Size for consistency
Nginx	etag on or dynamic module	Gzip can interfere with ETags
Express (Node.js)	app.set('etag', 'strong')	Built-in support in Express
Spring Boot	ShallowEtagHeaderFilter	Adds ETags for REST responses
Cloudflare	Preserve origin ETags	Requires correct Cache-Control

---