Inspired by Tim Bray's post on search engine blog accesses, I decided to take a look at my own logs for search engine referrals. The results aren't a terribly big surprise: here's a small image of the chart I came up with going back through April:
You should be able to click that image for a larger version of
it. The yellow line is Google - you'll note that it outpaces all
the other search engines by quite a bit - but it's been dipping of
late, with a slight rise from Yahoo. Ask Jeeves and MSN search are
nearly invisible. Getting that chart was an interesting process all
by itself - I used a simple Smalltalk script over the logs I
downloaded, building up a dictionary of weekly accesses:
logDict := Dictionary new.
refs := ApacheLogScanner scan: 'f:\logs\blog_log.0' recordSeparator: Character lf.
begin := Date readFrom: '5/8/05' readStream.
end := Date readFrom: '5/14/05' readStream.
scanned := refs entries select: [:each | each timestamp asDate >= begin and: [each timestamp asDate <= end]].
dict := Dictionary new.
dict at: 'google' put: 0.
dict at: 'yahoo' put: 0.
dict at: 'msn' put: 0.
dict at: 'askjeeves' put: 0.
dict at: 'googleImage' put: 0.
last := nil.
lastWasImage := false.
scanned do: [:each |
| referer |
referer := each referer.
('*www.google.*search*' match: referer)
ifTrue: [dict at: 'google' put: ((dict at: 'google') + 1)].
('*search.msn.com*' match: referer)
ifTrue: [dict at: 'msn' put: ((dict at: 'msn') + 1)].
('*web.ask.com*' match: referer)
ifTrue: [dict at: 'askjeeves' put: ((dict at: 'askjeeves') + 1)].
('*search.yahoo.com*' match: referer)
ifTrue: [dict at: 'yahoo' put: ((dict at: 'yahoo') + 1)].
('*http://images.google*' match: referer)
ifTrue: [(each origin ~= last and: [lastWasImage not])
ifTrue: [dict at: 'googleImage' put: ((dict at: 'googleImage') + 1).
lastWasImage := true]]
ifFalse: [lastWasImage := false].
last := each origin].
logDict at: begin put: dict.
It's pretty simple minded code, and it chews a lot of memory on large log files - but it works, and it's pretty quick. I dumped the results to a file and imported them to excel, where I made the mistake of using Pivot charts. Ack! That wasn't what I wanted. I then found the chart wizard on the toolbar, and created the simple chart above.
Then, having no idea how to export a graphic from Excel (if you can, it wasn't obvious to me), I took a screen shot, trimmed the result in Paint, and then shrunk it down with irfanview for this post. Whew. All that just to post a simple chart :)