Bot Traffic Tracking

TrackFox can collect AI crawler and search bot requests separately from normal visitor analytics. Bot traffic is never counted as human visitors or pageviews, and it appears in a dedicated Bot traffic card on your dashboard with provider, category, and IP-verification labels.

Why this is server-side

Most crawlers — including OpenAI's GPTBot, Anthropic's ClaudeBot, and PerplexityBot — fetch raw HTML and do not run browser JavaScript, so the regular TrackFox tracking script never sees them. Bot traffic needs to be sent from a proxy/edge function, a server handler, or request logs.

The snippet below is only a loose pre-filter: it decides which requests are worth reporting. Crawler detection and categorization happen on TrackFox's servers, so the crawler list stays current without you changing anything.

IP verification is the one exception — your server is reporting on a request it witnessed, so it never talks to TrackFox as the crawler itself. The ip field in the payload below is the only way TrackFox can see the crawler's real address; without it, that traffic can only ever show as claimed, never verified. Make sure your snippet includes it, as shown in the examples.

Next.js / Vercel

Add this as proxy.ts at the root of your project (or merge the event.waitUntil block into your existing proxy). Next.js 16 renamed the middleware.ts convention to proxy.ts — the default export is now proxy instead of middleware, but the signature and config.matcher are unchanged. If you're on an older Next.js version that still uses middleware.ts, just rename proxy back to middleware and export it the same way.

import { NextRequest, NextResponse, NextFetchEvent } from "next/server";
import { ipAddress } from "@vercel/functions";

const BOT_UA =
  /bot|crawler|spider|gpt|claude|perplexity|applebot|bytespider|amazonbot|meta-external|cohere|duckassist|grok|-user\//i;

export default function proxy(request: NextRequest, event: NextFetchEvent) {
  const userAgent = request.headers.get("user-agent") || "";

  if (BOT_UA.test(userAgent)) {
    // ipAddress() is Vercel's edge-aware IP helper. If you're not on
    // Vercel, fall back to the x-forwarded-for header your platform sets.
    const ip =
      ipAddress(request) ||
      request.headers.get("x-forwarded-for")?.split(",")[0]?.trim() ||
      null;

    event.waitUntil(
      fetch("https://trackfox.app/api/bot-traffic", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          websiteId: "YOUR_WEBSITE_ID",
          href: request.url,
          method: request.method,
          userAgent,
          referrer: request.headers.get("referer"),
          ip,
        }),
      }).catch(() => undefined),
    );
  }

  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!api|_next|.*\\..*).*)"],
};

Never await the tracking fetch — event.waitUntil runs it in the background after your response is sent, so it can't slow down your site.

@vercel/functions is a small official package (npm install @vercel/functions) and is safe to use whether or not you're deployed on Vercel — it just reads the x-real-ip header Vercel's edge sets, so it returns undefined off-platform and the x-forwarded-for fallback above still covers you.

Express

const BOT_UA =
  /bot|crawler|spider|gpt|claude|perplexity|applebot|bytespider|amazonbot|meta-external|cohere|duckassist|grok|-user\//i;

app.use((req, res, next) => {
  const userAgent = req.get("user-agent") || "";

  if (req.method === "GET" && BOT_UA.test(userAgent)) {
    res.on("finish", () => {
      fetch("https://trackfox.app/api/bot-traffic", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          websiteId: "YOUR_WEBSITE_ID",
          href: `${req.protocol}://${req.get("host")}${req.originalUrl}`,
          method: req.method,
          statusCode: res.statusCode,
          userAgent,
          referrer: req.get("referer"),
          // req.ip only reflects the real client when Express is told to
          // trust the proxy/load balancer in front of it — see below.
          ip: req.ip,
        }),
      }).catch(() => undefined);
    });
  }

  next();
});

If your Express app sits behind a reverse proxy or load balancer (nginx, an internal LB, etc.), req.ip reflects the proxy's address rather than the real client's unless you tell Express to trust it:

app.set("trust proxy", 1); // or the specific proxy IP/CIDR your infra uses

Without this, req.ip will be your proxy's own address, and that traffic can never verify — see Express's trust proxy docs for the right setting for your setup.

Sending status codes

If your server can see the final response (like the Express example above), include statusCode in the payload. TrackFox uses it to surface missing URLs crawlers requested — 404s that AI assistants and search engines keep hitting are often worth redirecting or restoring.

What gets tracked

TrackFox classifies each reported request into a category:

• AI answers: user-triggered fetches from AI assistants (ChatGPT-User, Claude-User, Perplexity-User, …)
• Indexing: search and AI-search index crawlers (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, …)
• Training: dataset and model-training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, …)
• Other bots: SEO tools, link previews, and unrecognized bots

Confidence labels

• verified: user agent matches and the IP is inside the provider's official published IP ranges.
• claimed: user agent matches but the IP could not be verified.
• suspicious: user agent claims a provider but the IP is outside the provider's official ranges — likely a spoofed crawler.
• heuristic: generic bot pattern with no known provider match.

Only verified traffic is shown on the dashboard. If your snippet doesn't send ip (or predates this doc), matching requests are still classified and stored, but they'll only ever be claimed and stay invisible — update your snippet as shown above and redeploy to start seeing them. Note this only affects traffic reported going forward; requests already stored without an ip can't be reclassified after the fact.

Requests that don't look like bot traffic at all (regular browsers, static assets, API routes) are ignored and never stored.