User Agent Database

Building the Ultimate User Agent Database: Join the Mission!

Imagine this: a comprehensive database of every browser, device, and robot (good or bad) that roams the web. Sounds ambitious? It is! And that’s exactly what we’re building with the UA DDAC Project (User Agent Dynamic Database for Analysis and Control). Right now, it's an evolving API collecting User-Agent strings, but the possibilities for the future? Endless!

What’s the UA DDAC Project About?

Currently, we’re laying the groundwork by collecting User-Agent (UA) strings to build an expansive database of browsers, devices, operating systems, and—yes—robots, crawlers, scrapers, and spiders. Each entry helps us understand the vast landscape of web traffic. Today, it’s a growing archive. Tomorrow? A tool for webmasters to:

• Monitor which browsers, devices, and operating systems are accessing their sites.
• Spot the robots, scrapers, and crawlers that visit (both the friendly and the mischievous).
• Block robots, crawlers, or specific categories from accessing their site using robots.txt or .htaccess.
• Set crawl rates for legitimate bots.
• Identify and block "spoofed" bots—those pretending to be Google or Bing but hailing from questionable IP addresses.

But for now, we need your help to collect data and grow this database. Think of it as helping the web's secret detective agency!

Why Should You Join the Mission?

Web robots and crawlers are like the paparazzi of the web—some are helpful, but others can cause havoc. By contributing to this database, you’ll be part of a larger mission to:

• Empower website owners with tools to understand and control their web traffic.
• Help identify spoofing and potential security threats.
• Create a fairer web by holding bad bots accountable while supporting the ethical ones.

And yes, this is purely about collecting data ethically. Here's exactly what gets stored:

• User-Agent string: This tells us what’s visiting—browser, robot, or device.
• IP Address: But only for robots and crawlers. For browsers, we don't store IPs—your visitors’ privacy is important to us.
• URLs (if found in the User-Agent string): These sometimes give clues about crawlers or robots.

No personal or sensitive data is collected. It's all about understanding what is browsing, not who.

Here’s How You Can Help

Adding a tiny snippet of code to your site allows us to collect valuable data. It’s lightweight, unobtrusive, and doesn’t impact your site's performance. Plus, it’s a chance to be part of something bigger!

Add This Snippet to Your Site:

Place the following PHP code in your site's header or footer so it runs on every page:

<?php
$endpoint = "https://ua.ddac.uk/public/track.php";
function getUserAgent() {
    if (!empty($_SERVER['HTTP_USER_AGENT'])) {
        return $_SERVER['HTTP_USER_AGENT'];
    }
    $headers = getallheaders();
    if (!empty($headers['User-Agent'])) {
        return $headers['User-Agent'];
    }
    return 'Unknown';
}
function getClientIp() {
    if (!empty($_SERVER['HTTP_X_FORWARDED_FOR'])) {
        $ipList = explode(',', $_SERVER['HTTP_X_FORWARDED_FOR']);
        return trim($ipList[0]);
    }
    return $_SERVER['REMOTE_ADDR'] ?? 'Unknown';
}
$userAgent = getUserAgent();
$ipAddress = getClientIp();
$ch = curl_init($endpoint);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, [
    'user_agent' => $userAgent,
    'ip_address' => $ipAddress,
]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);

curl_exec($ch);
curl_close($ch);
?>
  

This snippet sends the User-Agent string and the IP address (except for browsers) to our API. Simple as that! It’s an effortless way to help us grow the database.

A Fun Future Awaits!

Once this database reaches critical mass, its uses will be endless. Whether you’re curious about your site’s visitors, frustrated by rogue bots, or just fascinated by the secret life of web crawlers, UA DDAC will deliver.

In the meantime, let's have fun watching the web unfold—one User-Agent at a time. So, paste that snippet in your site, join the mission, and let's make the internet a better (and smarter) place. Together.

If you have any comments or questions please do not hesitate to contact me by one of my Social Media accounts or email in the footer

Happy tracking! 🐾