Featured Sysadmin GoAccess open source real time web log analyzer

I briefly mentioned on Google+ Community about GoAccess web log analyzer tool and it seems they had a major bump in versions since then from 0.4.2 to 1.1.1 GoAccess - Visual Web Log Analyzer.

---

Configuring GoAccess

1. You will need to configure the /etc/goaccess.conf config file as per instructions here.
2. For realtime stats you need to whitelist TCP port 7890 outlined here. Note --real-time-html flag might not work with default Centmin Mod Nginx access_log as it's configured for buffered writes. i.e.
   Code (Text):

   access_log /home/nginx/domains/newdomain.com/log/access.log main_ext buffer=256k flush=60m;

   means that writes to access.log are bufferred in memory up to 256KB before being flushed and written to access.log or every 60 minutes. This improves server performance but means writes to access.log are not in realtime.
3. Also Centmin Mod Nginx by default doesn't log static file requests via Nginx vhost site include file for /usr/local/nginx/conf/staticfiles.conf - it sets those location context matches for images, js, css and html files to access_log off by default. So won't be listed in your access.log unless you comment out access_log off line in each context location match in /usr/local/nginx/conf/staticfiles.conf and restart nginx server.

GoAccess Info

• Download and how to install GoAccess - Downloads
• Manual GoAccess - Manual Page
• Example usage listed at GoAccess - Manual Page
• Screenshots GoAccess - Screenshots
• HTML generated example Server Statistics - 2014-07-27 22:32:39
• Identifying aggressive crawlers using Go Access

GoAccess 1.1.1 updated layout

 

Configure /etc/goaccess.conf

Updated: February 23rd, 2017- Heads up seem the html generation changed as it requires specifying a date and log format first.

Full goaccess manual at GoAccess - Manual Page

So what you need to do is

If you follow official install guide, you will need to symlink the goaccess.conf installed at /usr/local/etc/goaccess/goaccess.conf to /etc/goaccess.conf first

Code (Text):

ln -s /usr/local/etc/goaccess/goaccess.conf /etc/goaccess.conf

step 1. edit the /etc/goaccess.conf file and edit and uncomment the following settings

Code (Text):

# The following time format works with any of the
# Apache/NGINX's log formats below.
#
#time-format %H:%M:%S
time-format %T

Code (Text):

# The following date format works with any of the
# Apache/NGINX's log formats below.
#
date-format %d/%b/%Y

custom format for Centmin Mod Nginx

Code (Text):

# NCSA Combined Log Format
#log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"
log-format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u"

step 2. append the path to /etc/goaccess.conf via -p option for html generation

Code:

goaccess -p /etc/goaccess.conf -f /home/nginx/domains/domainname.com/log/access.log -a > report.html

Centmin Mod specific examples

quick stats

Code:

goaccess -f /home/nginx/domains/domainname.com/log/access.log

full stats

Code:

goaccess -f /home/nginx/domains/domainname.com/log/access.log -a

html report

Code:

goaccess -f /home/nginx/domains/domainname.com/log/access.log -a > report.html

exclude certain IP addresses from reports

Code:

goaccess --exclude-ip=iptoexclude -p /etc/goaccess.conf -f /home/nginx/domains/domainname.com/log/access.log -a > report.html

i.e. the constellix.com monitoring IP list

Code:

goaccess --exclude-ip=192.73.252.209 --exclude-ip=199.38.182.28 --exclude-ip=209.177.145.67 --exclude-ip=209.177.157.208 --exclude-ip=103.25.58.42 --exclude-ip=103.6.85.32 --exclude-ip=103.6.87.219 --exclude-ip=23.239.17.158 --exclude-ip=106.186.122.163 --exclude-ip=37.235.56.13 --exclude-ip=212.71.238.144 --exclude-ip=208.111.40.251 --exclude-ip=176.58.89.83 --exclude-ip=198.58.122.143 -p /etc/goaccess.conf -f /home/nginx/domains/domainname.com/log/access.log -a > report.html

process for html generation all gzipped access-*.gz logs. For this to work with zcat you'd need to compress with gzip any uncompressed access.log files within your /home/nginx/domains/domainname.com/log/ directory in the same naming format as the rest of the gzip compressed logs.

Code:

ls -lah
total 25M
drwxr-xr-x 2 root root 4.0K Jul 26 06:16 .
drwxr-xr-x 8 root root 4.0K Jul 26 06:03 ..
-rw-r--r-- 1 root root 5.6M Jul 26 06:03 access.log-20140609.gz
-rw-r--r-- 1 root root 5.0M Jul 26 06:03 access.log-20140624.gz
-rw-r--r-- 1 root root 4.2M Jul 26 06:03 access.log-20140702.gz
-rw-r--r-- 1 root root 7.1M Jul 26 06:03 access.log-20140720.gz
-rw-r--r-- 1 root root 3.0M Jul 26 06:03 access.log-20140726.gz

Then process the logs via zcat piped into goaccess

Code:

zcat -f /home/nginx/domains/domainname.com/log/access.log-* | goaccess --exclude-ip=192.73.252.209 --exclude-ip=199.38.182.28 --exclude-ip=209.177.145.67 --exclude-ip=209.177.157.208 --exclude-ip=103.25.58.42 --exclude-ip=103.6.85.32 --exclude-ip=103.6.87.219 --exclude-ip=23.239.17.158 --exclude-ip=106.186.122.163 --exclude-ip=37.235.56.13 --exclude-ip=212.71.238.144 --exclude-ip=208.111.40.251 --exclude-ip=176.58.89.83 --exclude-ip=198.58.122.143 -p /etc/goaccess.conf -a > report.html

 

Sample HTML report http://goaccess.prosoftcorp.com/goaccess_html_report.html

 

follow instructions to install them, they'll install if not available

 

Heads up seem the html generation changed as it requires specifying a date and log format first.

Full goaccess manual at http://goaccess.io/man

So what you need to do is

step 1. create a /etc/goaccess.conf file and add the following into it

Code:

######################################
# Format Options
######################################

# The date_format variable followed by a space, specifies
# the log format date containing any combination of regular
# characters and special format specifiers. They all begin with a
# percentage (%) sign. See `man strftime`
#
#Any Apache log date format
date-format %d/%b/%Y
#
#W3C (IIS) & AWS | Amazon CloudFront (Download Distribution)
#date-format %Y-%m-%d

# The log_format variable followed by a space or \t for
# tab-delimited, specifies the log format string.
#
#Common Log Format (CLF)
log-format %h %^[%d:%^] "%r" %s %b
#
#Common Log Format (CLF) with Virtual Host
#log-format %^:%^ %h %^[%d:%^] "%r" %s %b
#
#NCSA Combined Log Format
#log-format %h %^[%d:%^] "%r" %s %b "%R" "%u"
#
#NCSA Combined Log Format with Virtual Host
#log-format %^:%^ %h %^[%d:%^] "%r" %s %b "%R" "%u"
#
#W3C
#log-format %d %^ %h %^ %^ %^ %^ %r %^ %s %b %^ %^ %u %R
#
#AWS | Amazon CloudFront (Download Distribution)
#log-format %d\t%^\t%^\t%b\t%h\t%m\t%^\t%r\t%s\t%R\t%u\t%^

######################################
# UI Options
######################################

# Prompt log/date configuration window on program start.
#
config-dialog false

# Choose among color schemes
# 1 : Default grey scheme
# 2 : Green scheme
#
color-scheme 1

# Turn off colored output. This is the  default output on
# terminals that do not support colors.
# true  : for no color output
# false : use color-scheme
#
no-color false

######################################
# File Options
######################################

# Specify the path to the input log file. If set, it will take
# priority over -f from the command line.
#
#log-file /var/log/apache2/access.log

# Send all debug messages to the specified file. Needs to configured
# with --enable-debug
#
#debug-file debug.log

# Specify a custom configuration file to use. If set, it will take
# priority over the global configuration file (if any).
#
#config-file=<filename>

# Do not load the global configuration file.
#
#no-global-config false

######################################
# Parse Options
######################################

# Consider the following extensions as static files
# The actual '.' is required and extensions are case sensitive
#
static-file .css
static-file .CSS
static-file .dae
static-file .DAE
static-file .eot
static-file .EOT
static-file .gif
static-file .GIF
static-file .ico
static-file .ICO
static-file .jpeg
static-file .JPEG
static-file .jpg
static-file .JPG
static-file .js
static-file .JS
static-file .map
static-file .MAP
static-file .mp3
static-file .MP3
static-file .pdf
static-file .PDF
static-file .png
static-file .PNG
static-file .svg
static-file .SVG
static-file .swf
static-file .SWF
static-file .ttf
static-file .TTF
static-file .txt
static-file .TXT
static-file .woff
static-file .WOFF

# Exclude an IPv4 or IPv6 from being counted.
# Ranges can be included as well using a dash in between
# the IPs (start-end).
#
#exclude-ip 127.0.0.1
#exclude-ip 192.168.0.1-192.168.0.100
#exclude-ip ::1
#exclude-ip 0:0:0:0:0:ffff:808:804-0:0:0:0:0:ffff:808:808

# Enable a list of user-agents by host. For faster parsing, do not
# enable this flag.
#
agent-list false

# Include HTTP request method if found. This will create a
# request key containing the request method + the actual request.
#
http-method true

# Include HTTP request protocol if found. This will create a
# request key containing the request protocol + the actual request.
#
http-protocol true

# Ignore request's query string.
# i.e.,  www.google.com/page.htm?query => www.google.com/page.htm
#
no-query-string false

# Disable IP resolver on terminal output.
#
no-term-resolver false

# Write output to stdout given one of the following formats:
# csv  : A comma-separated values (CSV)
# json : JSON (JavaScript Object Notation)
#
#output-format json

# Display real OS names. e.g, Windows XP, Snow Leopard.
#
real-os true

# Disable progress metrics.
#
no-progress false

# Enable mouse support on main dashboard.
#
with-mouse false

#  Enable IP resolver on HTML|JSON output.
#
with-output-resolver false

######################################
# GeoIP Options
# Only if configured with --enable-geoip
######################################

# Standard GeoIP database for less memory usage.
#
#std-geoip false

# Specify path to GeoIP City database file. i.e., GeoLiteCity.dat
# GeoLiteCity.dat needs to be downloaded from maxmind.com.
#
# wget -N http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
# gunzip GeoLiteCity.dat.gz
#
#geoip-city-data /usr/local/share/GeoIP/GeoLiteCity.dat

######################################
# Tokyo Cabinet Options
# Only if configured with --enable-tcb=btree
######################################

# On-disk B+ Tree
# Persist parsed data into disk. This should be set to
# the first dataset prior to use `load-from-disk`.
# Setting it to false will delete all database files
# when exiting the program.
#keep-db-files true

# On-disk B+ Tree
# Load previously stored data from disk.
# Database files need to exist. See `keep-db-files`.
#load-from-disk false

# On-disk B+ Tree
# Path where the on-disk database files are stored.
# The default value is the /tmp directory.
#
#db-path /tmp

# On-disk B+ Tree
# Set the size in bytes of the extra mapped memory.
# The default value is 0.
#
#xmmap 0

# On-disk B+ Tree
# Max number of leaf nodes to be cached.
# Specifies the maximum number of leaf nodes to be cached.
# If it is not more than 0, the default value is specified.
# The default value is 1024.
#
#cache-lcnum 1024

# On-disk B+ Tree
# Specifies the maximum number of non-leaf nodes to be cached.
# If it is not more than 0, the default value is specified.
# The default value is 512.
#
#cache-ncnum 512

# On-disk B+ Tree
# Specifies the number of members in each leaf page.
# If it is not more than 0, the default value is specified.
# The default value is 128.
#
#tune-lmemb 128

# On-disk B+ Tree
# Specifies the number of members in each non-leaf page.
# If it is not more than 0, the default value is specified.
# The default value is 256.
#
#tune-nmemb 256

# On-disk B+ Tree
# Specifies the number of elements of the bucket array.
# If it is not more than 0, the default value is specified.
# The default value is 32749.
# Suggested size of the bucket array is about from 1 to 4
# times of the number of all pages to be stored.
#
#tune-bnum 32749

# On-disk B+ Tree
# Specifies that each page is compressed with ZLIB|BZ2 encoding.
# Disabled by default.
#
#compression zlib

step 2. append the path to /etc/goaccess.conf via -p option for html generation

Code:

goaccess -p /etc/goaccess.conf -f /home/nginx/domains/domainname.com/log/access.log -a > report.html

 

Sample html generated from this forum

 

prefer Google Analytics for the custom dashboards you can create.. goaccess can't do this https://community.centminmod.com/threads/community-centminmod-com-journey-for-speed.3/#post-3089 but Google Analytics can

 

More interesting info to add = Identifying aggressive crawlers using Go Access

Of course instead of iptables, you can use CSF Firewall to block ips CSF Firewall - Centmin Mod - Menu based Nginx installer for CentOS servers

 

yeah either skip --enable-geoip or use addons/geoip.sh installer

 

you using GoAccess open source real time web log analyzer | Centmin Mod Community ? newer GoAccess versions require more setup as per that link

yup in manual page for zcat GoAccess - Manual Page and 2nd post GoAccess open source real time web log analyzer | Centmin Mod Community

 

yeah been using this for a while