Image may be NSFW.
Clik here to view.
We have an authentication daemon on a Debian 6.0 box generating logs at a rate of about 4-6 lines\entries every second. For legal reasons this log file has to be written to disk locally first, before we can back it up or transfer it anywhere. These logs are rotated every 2 hours by the daemon itself.
I’d like to implement a system where the logs are entered into a database and looked into syslog-ng (the service logging to syslog-ng and then syslog-ng writing the entry to a db and disk) but this was rejected as if the syslog-ng process ever failed for whatever reason, we would loose the log entries.
I have been toying with using rsync and cron set to synchronise every 5 minutes or so, but the receiving end obviously requires a full copy of the log for rsync to ‘compare’ to, and really all I want is the difference between the previous check.
Is there a decent way I can pick up the changes in the log file (from a remote machine preferably) since the last check, and then feed just those lines to a script or directly to a database?
Image may be NSFW.
Clik here to view.
Any problem with using tail -F <logfile> | db-import-script? Quite honestly, the “legal reasons” you cite seem nonsensical, even for lawyers, and so I’d be inclined to seek clarification on the intent of the regulations, and then work to that rather than try to vet implementation details with people who presumably have no ability to assess the suitability of an implementation. If they do have the technical ability, then tell ’em to give you the design and you’ll implement it to spec.
Check more discussion of this question.
