How should I deal with user agent parsing in logs?

Posted by Mr. Jefferson on Pro Webmasters See other posts from Pro Webmasters or by Mr. Jefferson
Published on 2012-02-17T19:41:05Z Indexed on 2012/03/23 17:41 UTC
Read the original article Hit count: 296

Filed under:

logging

|

user-agent

My web app project includes logging functionality so we can see where visitors are coming from (referrer URL), what the popular user agents are, what pages are most popular, etc. The log is stored in SQL Server, and when I query the user agents I use a large (almost 100 lines) and growing CASE statement to separate the user agents using string matching (i.e. if the user agent contains the string "Firefox/9" then it's Firefox 9). Is there a better way to do this so I don't have to continually add to that CASE statement to deal with new browser releases?

Also, how should I deal with less common, weird/unknown user agents? I've seen the following in the logs and been unable to find good information online about what they are:

WordPress/3.3.1; http://www.facecolony.org
Mozilla/4.0 ( http://www.hairirons.org redips; <a href=http://hairirons.org/>chi hair iron</a>)

I'd guess they're bots/crawlers, but the sites they point to don't appear to reference web crawlers (or even be available sometimes). I've seen other user agents aren't familiar to me, but I know they're bots because they include "bot" or "spider" or something similar in them.

© Pro Webmasters or respective owner

Related posts about logging

log file is not getting created using JDK logging with Commons-logging

as seen on Stack Overflow - Search for 'Stack Overflow'
When I run the TestJcLLoggingService class log messages are coming to Console but no log file is created, please help me if you know the answer. two source files are pasted below. TestJcLLoggingService.java package com.amadeus.psp.pasd.logging; import org.apache.commons.logging.Log; import org… >>> More
Python: combine logging and wx so that logging stream is redirectet to stdout/stderr frame

as seen on Stack Overflow - Search for 'Stack Overflow'
Here's the thing: I'm trying to combine the logging module with wx.App()'s redirect feature. My intention is to log to a file AND to stderr. But I want stderr/stdout redirected to a separate frame as is the feature of wx.App. My test code: import logging import wx class MyFrame(wx.Frame): … >>> More
Python logging before you run logging.basicConfig?

as seen on Stack Overflow - Search for 'Stack Overflow'
It appears that if you invoke logging.info() BEFORE you run logging.basicConfig, the logging.basicConfig call doesn't have any effect. In fact, no logging occurs. Where is this behavior documented? I don't really understand. >>> More
Redirect logging output using custom logging handler

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi Guys, I'm using a module in my python app that writes a lot a of messages using the logging module. Initially I was using this in a console application and it was pretty easy to get the logging output to display on the console using a console handler. Now I've developed a GUI version of my app… >>> More
Java Logger API

as seen on Oracle Blogs - Search for 'Oracle Blogs'
This is a more like a tip rather than technical write up and serves as a quick intro for newbies. The logger API helps to diagnose application level or JDK level issues at runtime. There are 7 levels which decide the detailing in logging (SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST)… >>> More

Related posts about user-agent

What does it mean when a User-Agent has another User-Agent inside it?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Basically, sometimes the user-agent will have its normal user-agent displayed, then at the end it will have teh "User-Agent: " tag displayed, and right after it another user-agent is shown. Sometimes, the second user-agent is just appended to the first one without the "User-Agent: " tag. Here are… >>> More
blocking bad bots with robots.txt in 2012 [closed]

as seen on Pro Webmasters - Search for 'Pro Webmasters'
does it still work good? I have this: # Generated using http://solidshellsecurity.com services # Begin block Bad-Robots from robots.txt User-agent: asterias Disallow:/ User-agent: BackDoorBot/1.0 Disallow:/ User-agent: Black Hole Disallow:/ User-agent: BlowFish/1.0 Disallow:/ User-agent: BotALot Disallow:/ User-agent:… >>> More
JavaScript - get detailed information about the browser

as seen on Stack Overflow - Search for 'Stack Overflow'
Basically I'm looking for something to give me easy access to information like useragentstring.com, but in JS, without me parsing the user agent and looking for each possible bit of text. The object could be something like this: browser = UserAgent.Browser; // Chrome browserVer… >>> More
User Agent in http client Android

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi everybody, I building an Https Client to send some data to a server, but I don't know what to pass in User-Agent. Should I use the webkit one's or do I have to build one explicitely for my App? I'm using this handy post Thanks for any help >>> More
Urlscan 3.1 block User Agent

as seen on Stack Overflow - Search for 'Stack Overflow'
I need to block requests from certain User Agents to our Sharepoint Environment that have been identified after going through the IIS logs. I have tried the below by amending the urlscan.ini config file and doing and iisreset, but it doesn't block anything. Am I entering the correct strings? I'm… >>> More