Technical SEO
Technical SEO optimizes the infrastructure of your website to help search engines crawl, index, and render your pages efficiently. Learn about robots.txt, sitemaps, structured data, performance optimization, and mobile-first design.
The robots.txt file tells search engine crawlers which pages they can or cannot access on your site.
Key Concepts
- Located at the root:
https://yoursite.com/robots.txt - Controls crawler access before they request pages
- Not a security mechanism—don't rely on it to hide sensitive data
- Directives:
User-agent,Allow,Disallow,Sitemap
Next.js Implementation (Dynamic robots.txt)
// app/robots.tsimport { MetadataRoute } from 'next'export default function robots(): MetadataRoute.Robots {return {rules: [{userAgent: '*',allow: '/',disallow: ['/api/', '/admin/', '/private/'],},{userAgent: 'Googlebot',allow: '/',disallow: ['/api/'],},],sitemap: 'https://mern-docs.vercel.app/sitemap.xml',host: 'https://mern-docs.vercel.app',}}// This generates /robots.txt automatically
XML sitemaps help search engines discover and index your pages more efficiently. They list all important URLs along with metadata like last modified date and priority.
Sitemap Best Practices
- Include all important pages (max 50,000 URLs per sitemap)
- Update when content changes
- Submit to Google Search Console and Bing Webmaster Tools
- Reference in robots.txt file
- Use sitemap index for sites with multiple sitemaps
<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><url><loc>https://yoursite.com/</loc><lastmod>2026-01-03</lastmod><changefreq>weekly</changefreq><priority>1.0</priority></url><url><loc>https://yoursite.com/blog/react-tutorial</loc><lastmod>2026-01-02</lastmod><changefreq>monthly</changefreq><priority>0.8</priority></url><url><loc>https://yoursite.com/about</loc><lastmod>2025-12-01</lastmod><changefreq>yearly</changefreq><priority>0.5</priority></url></urlset>
Next.js Sitemap Implementation
// app/sitemap.tsimport { MetadataRoute } from 'next'export default function sitemap(): MetadataRoute.Sitemap {const baseUrl = 'https://mern-docs.vercel.app'const currentDate = new Date()return [{url: baseUrl,lastModified: currentDate,changeFrequency: 'weekly',priority: 1,},{url: `${baseUrl}/tutorials/react`,lastModified: currentDate,changeFrequency: 'monthly',priority: 0.8,},{url: `${baseUrl}/tutorials/nodejs`,lastModified: currentDate,changeFrequency: 'monthly',priority: 0.8,},// Add all your pages...]}// Generates /sitemap.xml automatically
Structured data uses a standardized format (JSON-LD, Microdata) to provide explicit information about your page content. This enables rich snippets in search results like star ratings, FAQs, breadcrumbs, and more.
Benefits of Structured Data
- Enhanced search result appearance (rich snippets)
- Better search engine understanding of content
- Potential for knowledge graph inclusion
- Voice search optimization
- Increased click-through rates
Article Schema
{"@context": "https://schema.org","@type": "Article","headline": "Complete React Hooks Guide","description": "Learn React Hooks with practical examples","author": {"@type": "Person","name": "John Doe"},"datePublished": "2026-01-03","dateModified": "2026-01-03","image": "https://yoursite.com/react-hooks.jpg","publisher": {"@type": "Organization","name": "MERN Docs","logo": {"@type": "ImageObject","url": "https://yoursite.com/logo.png"}}}
Breadcrumb Schema
{"@context": "https://schema.org","@type": "BreadcrumbList","itemListElement": [{"@type": "ListItem","position": 1,"name": "Home","item": "https://yoursite.com"},{"@type": "ListItem","position": 2,"name": "Tutorials","item": "https://yoursite.com/tutorials"},{"@type": "ListItem","position": 3,"name": "React Hooks","item": "https://yoursite.com/tutorials/react-hooks"}]}
FAQ Schema
{"@context": "https://schema.org","@type": "FAQPage","mainEntity": [{"@type": "Question","name": "What are React Hooks?","acceptedAnswer": {"@type": "Answer","text": "React Hooks are functions that let you use state and other React features in functional components."}},{"@type": "Question","name": "When should I use useEffect?","acceptedAnswer": {"@type": "Answer","text": "Use useEffect for side effects like data fetching, subscriptions, or manually changing the DOM."}}]}
Next.js Implementation (JSON-LD)
// app/blog/[slug]/page.tsxexport default function BlogPost() {const schema = {'@context': 'https://schema.org','@type': 'Article',headline: 'React Hooks Tutorial',description: 'Learn React Hooks...',author: {'@type': 'Person',name: 'John Doe',},datePublished: '2026-01-03',image: 'https://yoursite.com/image.jpg',}return (<><scripttype="application/ld+json"dangerouslySetInnerHTML={{ __html: JSON.stringify(schema) }}/><article>{/* Your content */}</article></>)}
Testing Tools
- Google Rich Results Test: Test structured data
- Schema.org Validator: Validate markup syntax
- Google Search Console: Monitor rich result status
Canonical URLs tell search engines which version of a page is the "main" one when you have duplicate or similar content on multiple URLs. This prevents duplicate content issues.
When to Use Canonical Tags
- Multiple URLs showing the same content (www vs non-www)
- URL parameters (tracking codes, session IDs)
- Printer-friendly versions
- Paginated content
- Syndicated content on other sites
<!-- In the <head> section --><link rel="canonical" href="https://yoursite.com/preferred-url" /><!-- Example: Prefer HTTPS and www --><!-- Page: http://example.com/page --><link rel="canonical" href="https://www.example.com/page" /><!-- Example: Ignore URL parameters --><!-- Page: /products?sort=price&color=blue --><link rel="canonical" href="https://yoursite.com/products" />
Next.js Implementation
// app/layout.tsx or specific pageimport type { Metadata } from 'next'export const metadata: Metadata = {metadataBase: new URL('https://mern-docs.vercel.app'),alternates: {canonical: '/', // or specific path like '/blog/post-title'},}// For dynamic pagesexport async function generateMetadata({ params }): Promise<Metadata> {return {alternates: {canonical: `/blog/${params.slug}`,},}}
Page speed is a confirmed ranking factor. Faster sites provide better user experience and rank higher in search results.
Image Optimization
- Use WebP or AVIF format
- Compress images (TinyPNG, ImageOptim)
- Implement lazy loading
- Use responsive images (srcset)
- Optimize with Next.js Image component
Code Optimization
- Minify CSS, JS, HTML
- Remove unused code (tree shaking)
- Code splitting and lazy loading
- Use production builds
- Optimize third-party scripts
Server & Hosting
- Use CDN (Cloudflare, Vercel)
- Enable HTTP/2 or HTTP/3
- Implement caching strategies
- Use compression (Gzip, Brotli)
- Choose fast hosting provider
Resource Loading
- Defer non-critical JavaScript
- Preload critical resources
- Minimize render-blocking resources
- Use resource hints (dns-prefetch)
- Optimize font loading
// next.config.mjs - Performance optimizations/** @type {import('next').NextConfig} */const nextConfig = {images: {formats: ['image/avif', 'image/webp'],deviceSizes: [640, 750, 828, 1080, 1200, 1920],},compress: true, // Enable gzip compressionswcMinify: true, // Use SWC for faster minification// Enable HTTP/2 Server Pushasync headers() {return [{source: '/:all*(svg|jpg|png)',headers: [{key: 'Cache-Control',value: 'public, max-age=31536000, immutable',},],},]},}export default nextConfig
Google uses mobile-first indexing, meaning it primarily uses the mobile version of your site for ranking. Your site must be mobile-friendly to rank well.
Mobile-First Checklist
- Responsive design (adapts to all screen sizes)
- Mobile viewport meta tag
- Touch-friendly navigation (44px minimum touch targets)
- Readable text without zooming (16px+ font size)
- No horizontal scrolling
- Fast mobile performance
- Avoid mobile-unfriendly interstitials
<!-- Essential viewport meta tag --><meta name="viewport" content="width=device-width, initial-scale=1.0"><!-- Responsive images --><imgsrc="/image-mobile.jpg"srcset="/image-mobile.jpg 400w,/image-tablet.jpg 800w,/image-desktop.jpg 1200w"sizes="(max-width: 600px) 400px, (max-width: 1000px) 800px, 1200px"alt="Responsive image"/><!-- Mobile-friendly CSS --><style>/* Mobile-first approach */body {font-size: 16px; /* Readable without zoom */padding: 1rem;}button {min-height: 44px; /* Touch-friendly */min-width: 44px;}/* Tablet and up */@media (min-width: 768px) {body {padding: 2rem;}}/* Desktop */@media (min-width: 1024px) {body {max-width: 1200px;margin: 0 auto;}}</style>
HTTPS is a confirmed ranking signal. Google prioritizes secure sites and Chrome warns users about non-HTTPS sites.
Why HTTPS Matters
- Data encryption between browser and server
- SEO ranking boost
- Trust indicators in browser (padlock icon)
- Avoids "Not Secure" warnings
- Required for modern web features (Service Workers, HTTP/2)
Setting Up HTTPS
- Get an SSL/TLS certificate (Let's Encrypt offers free ones)
- Install certificate on your server
- Update internal links to use HTTPS
- Set up 301 redirects from HTTP to HTTPS
- Update canonical URLs to HTTPS
- Enable HSTS (HTTP Strict Transport Security)
HSTS Header
Force browsers to always use HTTPS:
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Core Web Vitals are Google's metrics for measuring real-world user experience. They're part of page experience signals that affect rankings.
LCP - Largest Contentful Paint
Measures loading performance. How long until the largest content element becomes visible.
How to improve: Optimize images, use CDN, improve server response time, eliminate render-blocking resources
FID - First Input Delay (now INP)
Measures interactivity. Time from user interaction to browser response.
How to improve: Reduce JavaScript execution time, split long tasks, use web workers, optimize third-party scripts
CLS - Cumulative Layout Shift
Measures visual stability. How much page content shifts unexpectedly during loading.
How to improve: Set image/video dimensions, avoid inserting content above existing content, use transform animations
Testing Tools
- PageSpeed Insights: Analyze performance and get recommendations
- Chrome DevTools: Lighthouse tab for detailed audits
- Web Vitals Extension: Real-time Core Web Vitals monitoring
- Google Search Console: Core Web Vitals report for all pages