System Log Analysis

SYSTEM LOG ANALYSIS PROCEDURES:

(Here are the steps to perform the log analysis for ‘slate’ and ‘quake’ and 'ics')

NOTE: the log files that need analyzing are cleared and renewed each month
NOTE2: log analysis procedures for quake and ics are the same as for slate
NOTE3: a star (*) means fill in the blank

1. Open a new terminal session. The terminal can be found in the quick link bar at the bottom of the MacOS X screen.
2. Open a connection to “fablio”, and log in using your password.
   ssh fablio
3. Change directories using the following command:
   cd /space/stadm/loganalysis
   • this directory should contain 6 log scripts (cleanall, copyall, sautolog, qautolog, iautolog, suniq) and 12 directories "conf.orig/", "iconf/", "qconf/", "sconf/", ilogs/", "qlogs/", "slogs/", "iresults/", "qresults/", "sresults/", "logclean/" and "logcopy/" (use the ‘ls’ command to list directory contents)
     
     

   Overview: slogs/ and qlogs/ and ilogs/ contain the log files to be analyzed for the slate and quake and ics machines respectively; sresults/ and qresults/ and iresults/ contain the analysis results after running the ‘sautolog’ and ‘qautolog’ and 'iautolog' scripts; sconf/ and qconf/ and iconf/ contain the .ls and .ln files used by the logsurfer script. conf.orig/ contains the default .ls and .ln files used by the logsurfer script. logclean/ contains the scripts used to remove empty log files prior to analysis. logcopy/ contains the scripts for copying all log files from their respective machines.

   • if any of the above directories do not exist, create them using “mkdir [dir_name]”
4. Verify the paths used in each *logcopy script (for copying over all logs for analysis) with Aaron.
   • DEFAULT location of .log files to be analyzed:
     see each *logcopy file within logcopy/
   • DEFAULT destination directory for .log files:
     /space/stadm/loganalysis/*logs/
5. Run copyall
   • it will take a few seconds, so be patient (you should see progress output)
   • “*logs/” should now contain the log files
6. Run cleanall
7. Verify location of logsurfer script (should be included in your PATH environment variable; if not, then ask Aaron to include it).
8. Verify that “default.ls” is in each *conf/ directory for use by logsurfer
   • if not, then use the ‘find’ command to locate it searching at the root directory or ask Aaron
9. Run the ‘*autolog’ command for the quake, ics, and slate machines.

   qautolog - log analysis for the quake machine
   

   sautolog - log analysis for the slate machine
   

   iautolog - log analysis for the ics machine
   
   

   with optional flags *autolog [-h]
   

   • for list of options and usage (AS OF 2006-02, THE LOG FILES THAT ARE COPIED DO NOT GET REPLACED EACH MONTH, SO DO NOT USE THE -NM FLAG!!!!!!!!!!!!! - you can verify this by opening the log files and checking the dates)
     
   • NOTE: use the –nm flag to reset the line counters used by logsurfer when analyzing logs of a new month for the first time. Or you can delete all files (EXCEPT default.ls) from the *conf/ directory.
   • NOTE2: useful flags are -d and -wr (see documentation or use '*autolog -h' ) (if you use these flags, you can skip step 14)
     
10. choose log # to analyze.
    • the script will prompt you
    • log analysis may take a while (depending on the size of the log file), so BE PATIENT!
    • you can ignore "ipf.log" and "http*.log"
11. when asked “Do you want to commit changes to name.log.ln? (y/n)”, select YES (only if you didn't use the -wr flag).
    • this saves and updates the last line number that was analyzed (.ln file used by logsurfer) in case log analysis was not complete, that way the next time you analyze the log file, you resume on the line you left off on rather than analyzing the entire file again.
12. repeat steps 12-14 for each log file.
    • to make this process faster, open another terminal, log into fablio, go to the loganalysis/ directory, and follow steps 12-14
13. cd into *results/ and run 'suniqall' (see documentation for more info)

    NOTE: before using 'suniqall', the suniq script must be in
    /space/stadm/loganalysis (see step #3).

    • make sure that *results/ contains another directory old_results/ and that this directory has permissions of "drwxr-xr-x" ('ls -l' to check)
14. analyze log.out.uniq files using more
    • the *.uniq files are the files that need to be analyzed and reported
15. send Aaron an email detailing any entries of interest with the appropriate subject header (like "slate log analysis results")
    
    

Do this for each machine's log entries of interest. I (RD) will write one email that includes all entries of interest.

Manual New Month Procedure

This must be done before you do the first log analysis of each month.

Some of the log files are updated monthly, while some run continuously. Because of this, the -nm tag is not enough to update the logs each month. Here is a list of the log files that update monthly (all others are continuous). The rule of thumb is that any log file that ends in .log is updated monthly, but his is not necessarily the case.

qlogs:
horde.log
rsync.log
xfer.log

slogs:
All files are continuous, so simply do not run -nm, The rest of the procedure only refers to the qlogs and ilogs.

ilogs:
maillist-autolistupdate.log
rsyncd.log
spamtraining.log
watchdog.log

1. In order to ensure that all the files work properly, go to *conf/ and use the more command to find the current line number for all of the above files. Make a note of these line numbers.
2. Run *autolog -nm.
3. Go back to *conf/ and edit the conf files for the above logs. These files will have been reset to zero, simply change this line number to the line number you wrote down.
   
   

Once you have done this, you can proceed with the normal log analysis procedure.

If you do this procedure, and one of the monthly log files has dates before the first of the month, the log file has become continuous. Add that file to the documentation.

Conversely, if you find that one of the above files is 0 bytes, this file may have become monthly. Go to *logs/ and check the file manually. If all of the entries are from the new month, this file is now monthly, and remove it from this list.

---

Mark Gorecki
modified by Joe Mount 2006-02-21
modified by Robert Dame 2006-06-02
modified by Kurt Olsson 2006-10-19