Disk / system configuration for log collection / syslog server

Posted by Konrads on Server Fault See other posts from Server Fault or by Konrads
Published on 2012-09-09T13:24:28Z Indexed on 2012/09/09 15:39 UTC
Read the original article Hit count: 287

Filed under:

syslog

|

mongodb

|

graylog2

|

elastic-search

I am looking into building a syslog / logging infrastructure and am pondering about some architecture best practices. Essentially, I see that a syslog system needs to support two conflicting workloads:

log collection. Potentially massive streams of data need to be written quickly to disks and indexed.
log querying. logs will be queried by both fixed fields such as date and source as well as text search.

What is the best disk/system setup assuming I'd like to keep it to a single server for now? Should I use SSDs or ramdisk to off-load some processing? some disks in stripe and some in raid5?

I am particularly eyeing Graylog2 with ElasticSearch/MongoDB

© Server Fault or respective owner

Related posts about syslog

WD MBWE II (White Strip Light) 2TB - unable to access data

as seen on Super User - Search for 'Super User'
I have a WD MBWE II (White Strip Light) 2TB - (WD20000H2NC-00) Was working fine until a few days ago. I guess there was a power failure and after that I am unable to access the 'Public' or the 'Download' folder anymore. I have been searching for answers everywhere but came up empty handed. Web… >>> More
Logs are written to *.log.1 instead of *.log

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
For some reason my log files are writing to the *.log.1 files instead of the *.log files, e.g. for my Postfix log files it is writing to /var/log/mail.log.1 and not /var/log/mail.log as expected. Same goes for mail.err. It looks like it's also doing it for auth.log and syslog. Here is a ls -lt snippet… >>> More
syslog-ng and nging logs to mysql

as seen on Server Fault - Search for 'Server Fault'
So couple of days ago I asked how to log php and nginx logs to centralized MySQL database, and m0ntassar gave a perfect answer :) cheer ! The problem I am facing now is that I can not seem to get it working. syslog-ng version: # syslog-ng --version syslog-ng 3.2.5 This is my nginx log format: log_format… >>> More
Python - retrieving info from a syslog file

as seen on Stack Overflow - Search for 'Stack Overflow'
Ive been asked to do a program using python for an assignment. Ive been given a syslog file and I have to find things out about it How do I find out how many attempts were made to login to the root account? Any advice would be highly appreciated as Im very new to python an completely lost! >>> More
How can i monitor syslog messages in c# console app with TCP

as seen on Stack Overflow - Search for 'Stack Overflow'
Heya, In my application, i need to monitor all messages sent by syslog. I've tried with UDP, but after one message, i didn't respond anymore (no error, just no heads up anymore). And setting up a tcp server isn't really the solution either i think. Can anyone guide me to a solution where i can log… >>> More

Related posts about mongodb

MongoDB usage best practices

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
The project I'm working on uses MongoDB for some stuff so I'm creating some documents to help developers speedup the learning curve and also avoid mistakes and help them write clean & reliable code. This is my first version of it, so I'm pretty sure I will be adding more stuff to it, so stay tuned… >>> More
Errors trying to run MongoDB

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I'm running Ubuntu Server 12.04 (32 bit) on an old (1998) computer. Everything's working fine until I try and start MongoDB. somekittens@DLserver01:~$ mongo MongoDB shell version: 2.2.2 connecting to: test Sun Dec 16 22:47:50 Error: couldn't connect to server 127.0.0.1:27017 src/mongo/shell/mongo… >>> More
push new value to mongodb inner array - mongodb/php

as seen on Stack Overflow - Search for 'Stack Overflow'
hi i have this document in mongo: { "_id": ObjectId("4d0b9c7a8b012fe287547157"), "done_by": ["1"] } and i want to add another value to "done_by" field, so my expected document will be:: { "_id": ObjectId("4d0b9c7a8b012fe287547157"), "done_by": ["1","2","3"] } i try this: $conn… >>> More
Write-only collections in MongoDB

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm currently using MongoDB to record application logs, and while I'm quite happy with both the performance and with being able to dump arbitrary structured data into log records, I'm troubled by the mutability of log records once stored. In a traditional database, I would structure the grants for… >>> More
How to install mongoDB on windows?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi! I am trying to test out mongoDB and see if it is anything for me. I downloaded the 32bit windows version, but have no idea on how to continue from now on. I normally use the WAMP services for developing on my local computer. Can i run mongoDB on Wamp? However, what's the best (easiest!) way… >>> More