Technical SEO

Technical SEO optimizes the infrastructure of your website to help search engines crawl, index, and render your pages efficiently. Learn about robots.txt, sitemaps, structured data, performance optimization, and mobile-first design.

Robots.txt File

The robots.txt file tells search engine crawlers which pages they can or cannot access on your site.

Key Concepts

  • Located at the root: https://yoursite.com/robots.txt
  • Controls crawler access before they request pages
  • Not a security mechanism—don't rely on it to hide sensitive data
  • Directives: User-agent, Allow, Disallow, Sitemap
Basic robots.txt
Advanced robots.txt

Next.js Implementation (Dynamic robots.txt)

typescript
// app/robots.ts
import { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{
userAgent: '*',
allow: '/',
disallow: ['/api/', '/admin/', '/private/'],
},
{
userAgent: 'Googlebot',
allow: '/',
disallow: ['/api/'],
},
],
sitemap: 'https://mern-docs.vercel.app/sitemap.xml',
host: 'https://mern-docs.vercel.app',
}
}
// This generates /robots.txt automatically
XML Sitemaps

XML sitemaps help search engines discover and index your pages more efficiently. They list all important URLs along with metadata like last modified date and priority.

Sitemap Best Practices

  • Include all important pages (max 50,000 URLs per sitemap)
  • Update when content changes
  • Submit to Google Search Console and Bing Webmaster Tools
  • Reference in robots.txt file
  • Use sitemap index for sites with multiple sitemaps
xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://yoursite.com/</loc>
<lastmod>2026-01-03</lastmod>
<changefreq>weekly</changefreq>
<priority>1.0</priority>
</url>
<url>
<loc>https://yoursite.com/blog/react-tutorial</loc>
<lastmod>2026-01-02</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
<url>
<loc>https://yoursite.com/about</loc>
<lastmod>2025-12-01</lastmod>
<changefreq>yearly</changefreq>
<priority>0.5</priority>
</url>
</urlset>

Next.js Sitemap Implementation

typescript
// app/sitemap.ts
import { MetadataRoute } from 'next'
export default function sitemap(): MetadataRoute.Sitemap {
const baseUrl = 'https://mern-docs.vercel.app'
const currentDate = new Date()
return [
{
url: baseUrl,
lastModified: currentDate,
changeFrequency: 'weekly',
priority: 1,
},
{
url: `${baseUrl}/tutorials/react`,
lastModified: currentDate,
changeFrequency: 'monthly',
priority: 0.8,
},
{
url: `${baseUrl}/tutorials/nodejs`,
lastModified: currentDate,
changeFrequency: 'monthly',
priority: 0.8,
},
// Add all your pages...
]
}
// Generates /sitemap.xml automatically
Structured Data (Schema Markup)

Structured data uses a standardized format (JSON-LD, Microdata) to provide explicit information about your page content. This enables rich snippets in search results like star ratings, FAQs, breadcrumbs, and more.

Benefits of Structured Data

  • Enhanced search result appearance (rich snippets)
  • Better search engine understanding of content
  • Potential for knowledge graph inclusion
  • Voice search optimization
  • Increased click-through rates

Article Schema

json
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Complete React Hooks Guide",
"description": "Learn React Hooks with practical examples",
"author": {
"@type": "Person",
"name": "John Doe"
},
"datePublished": "2026-01-03",
"dateModified": "2026-01-03",
"image": "https://yoursite.com/react-hooks.jpg",
"publisher": {
"@type": "Organization",
"name": "MERN Docs",
"logo": {
"@type": "ImageObject",
"url": "https://yoursite.com/logo.png"
}
}
}

Breadcrumb Schema

json
{
"@context": "https://schema.org",
"@type": "BreadcrumbList",
"itemListElement": [
{
"@type": "ListItem",
"position": 1,
"name": "Home",
"item": "https://yoursite.com"
},
{
"@type": "ListItem",
"position": 2,
"name": "Tutorials",
"item": "https://yoursite.com/tutorials"
},
{
"@type": "ListItem",
"position": 3,
"name": "React Hooks",
"item": "https://yoursite.com/tutorials/react-hooks"
}
]
}

FAQ Schema

json
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What are React Hooks?",
"acceptedAnswer": {
"@type": "Answer",
"text": "React Hooks are functions that let you use state and other React features in functional components."
}
},
{
"@type": "Question",
"name": "When should I use useEffect?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Use useEffect for side effects like data fetching, subscriptions, or manually changing the DOM."
}
}
]
}

Next.js Implementation (JSON-LD)

typescript
// app/blog/[slug]/page.tsx
export default function BlogPost() {
const schema = {
'@context': 'https://schema.org',
'@type': 'Article',
headline: 'React Hooks Tutorial',
description: 'Learn React Hooks...',
author: {
'@type': 'Person',
name: 'John Doe',
},
datePublished: '2026-01-03',
image: 'https://yoursite.com/image.jpg',
}
return (
<>
<script
type="application/ld+json"
dangerouslySetInnerHTML={{ __html: JSON.stringify(schema) }}
/>
<article>
{/* Your content */}
</article>
</>
)
}

Testing Tools

  • Google Rich Results Test: Test structured data
  • Schema.org Validator: Validate markup syntax
  • Google Search Console: Monitor rich result status
Canonical URLs

Canonical URLs tell search engines which version of a page is the "main" one when you have duplicate or similar content on multiple URLs. This prevents duplicate content issues.

When to Use Canonical Tags

  • Multiple URLs showing the same content (www vs non-www)
  • URL parameters (tracking codes, session IDs)
  • Printer-friendly versions
  • Paginated content
  • Syndicated content on other sites
html
<!-- In the <head> section -->
<link rel="canonical" href="https://yoursite.com/preferred-url" />
<!-- Example: Prefer HTTPS and www -->
<!-- Page: http://example.com/page -->
<link rel="canonical" href="https://www.example.com/page" />
<!-- Example: Ignore URL parameters -->
<!-- Page: /products?sort=price&color=blue -->
<link rel="canonical" href="https://yoursite.com/products" />

Next.js Implementation

typescript
// app/layout.tsx or specific page
import type { Metadata } from 'next'
export const metadata: Metadata = {
metadataBase: new URL('https://mern-docs.vercel.app'),
alternates: {
canonical: '/', // or specific path like '/blog/post-title'
},
}
// For dynamic pages
export async function generateMetadata({ params }): Promise<Metadata> {
return {
alternates: {
canonical: `/blog/${params.slug}`,
},
}
}
Page Speed Optimization

Page speed is a confirmed ranking factor. Faster sites provide better user experience and rank higher in search results.

Image Optimization

  • Use WebP or AVIF format
  • Compress images (TinyPNG, ImageOptim)
  • Implement lazy loading
  • Use responsive images (srcset)
  • Optimize with Next.js Image component

Code Optimization

  • Minify CSS, JS, HTML
  • Remove unused code (tree shaking)
  • Code splitting and lazy loading
  • Use production builds
  • Optimize third-party scripts

Server & Hosting

  • Use CDN (Cloudflare, Vercel)
  • Enable HTTP/2 or HTTP/3
  • Implement caching strategies
  • Use compression (Gzip, Brotli)
  • Choose fast hosting provider

Resource Loading

  • Defer non-critical JavaScript
  • Preload critical resources
  • Minimize render-blocking resources
  • Use resource hints (dns-prefetch)
  • Optimize font loading
typescript
// next.config.mjs - Performance optimizations
/** @type {import('next').NextConfig} */
const nextConfig = {
images: {
formats: ['image/avif', 'image/webp'],
deviceSizes: [640, 750, 828, 1080, 1200, 1920],
},
compress: true, // Enable gzip compression
swcMinify: true, // Use SWC for faster minification
// Enable HTTP/2 Server Push
async headers() {
return [
{
source: '/:all*(svg|jpg|png)',
headers: [
{
key: 'Cache-Control',
value: 'public, max-age=31536000, immutable',
},
],
},
]
},
}
export default nextConfig
Mobile-First & Responsive Design

Google uses mobile-first indexing, meaning it primarily uses the mobile version of your site for ranking. Your site must be mobile-friendly to rank well.

Mobile-First Checklist

  • Responsive design (adapts to all screen sizes)
  • Mobile viewport meta tag
  • Touch-friendly navigation (44px minimum touch targets)
  • Readable text without zooming (16px+ font size)
  • No horizontal scrolling
  • Fast mobile performance
  • Avoid mobile-unfriendly interstitials
html
<!-- Essential viewport meta tag -->
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<!-- Responsive images -->
<img
src="/image-mobile.jpg"
srcset="
/image-mobile.jpg 400w,
/image-tablet.jpg 800w,
/image-desktop.jpg 1200w
"
sizes="(max-width: 600px) 400px, (max-width: 1000px) 800px, 1200px"
alt="Responsive image"
/>
<!-- Mobile-friendly CSS -->
<style>
/* Mobile-first approach */
body {
font-size: 16px; /* Readable without zoom */
padding: 1rem;
}
button {
min-height: 44px; /* Touch-friendly */
min-width: 44px;
}
/* Tablet and up */
@media (min-width: 768px) {
body {
padding: 2rem;
}
}
/* Desktop */
@media (min-width: 1024px) {
body {
max-width: 1200px;
margin: 0 auto;
}
}
</style>
HTTPS & Security

HTTPS is a confirmed ranking signal. Google prioritizes secure sites and Chrome warns users about non-HTTPS sites.

Why HTTPS Matters

  • Data encryption between browser and server
  • SEO ranking boost
  • Trust indicators in browser (padlock icon)
  • Avoids "Not Secure" warnings
  • Required for modern web features (Service Workers, HTTP/2)

Setting Up HTTPS

  • Get an SSL/TLS certificate (Let's Encrypt offers free ones)
  • Install certificate on your server
  • Update internal links to use HTTPS
  • Set up 301 redirects from HTTP to HTTPS
  • Update canonical URLs to HTTPS
  • Enable HSTS (HTTP Strict Transport Security)

HSTS Header

Force browsers to always use HTTPS:

txt
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Core Web Vitals

Core Web Vitals are Google's metrics for measuring real-world user experience. They're part of page experience signals that affect rankings.

LCP - Largest Contentful Paint

Measures loading performance. How long until the largest content element becomes visible.

Good: <2.5 seconds
Needs Improvement: 2.5-4 seconds
Poor: >4 seconds

How to improve: Optimize images, use CDN, improve server response time, eliminate render-blocking resources

FID - First Input Delay (now INP)

Measures interactivity. Time from user interaction to browser response.

Good: <100 milliseconds
Needs Improvement: 100-300ms
Poor: >300ms

How to improve: Reduce JavaScript execution time, split long tasks, use web workers, optimize third-party scripts

CLS - Cumulative Layout Shift

Measures visual stability. How much page content shifts unexpectedly during loading.

Good: <0.1
Needs Improvement: 0.1-0.25
Poor: >0.25

How to improve: Set image/video dimensions, avoid inserting content above existing content, use transform animations

Testing Tools

  • PageSpeed Insights: Analyze performance and get recommendations
  • Chrome DevTools: Lighthouse tab for detailed audits
  • Web Vitals Extension: Real-time Core Web Vitals monitoring
  • Google Search Console: Core Web Vitals report for all pages