Discover Centmin Mod today
Register Now

Featured Sysadmin GoAccess open source real time web log analyzer

Discussion in 'System Administration' started by eva2000, Jun 5, 2014.

  1. eva2000

    eva2000 Administrator Staff Member

    30,955
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,416
    Local Time:
    11:53 PM
    Nginx 1.13.x
    MariaDB 5.5
    I briefly mentioned on Google+ Community about GoAccess web log analyzer tool and it seems they had a major bump in versions since then from 0.4.2 to 1.1.1 GoAccess - Visual Web Log Analyzer.

    Configuring GoAccess


    1. You will need to configure the /etc/goaccess.conf config file as per instructions here.
    2. For realtime stats you need to whitelist TCP port 7890 outlined here. Note --real-time-html flag might not work with default Centmin Mod Nginx access_log as it's configured for buffered writes. i.e.
      Code (Text):
      access_log /home/nginx/domains/newdomain.com/log/access.log main_ext buffer=256k flush=60m;
      means that writes to access.log are bufferred in memory up to 256KB before being flushed and written to access.log or every 60 minutes. This improves server performance but means writes to access.log are not in realtime.
    3. Also Centmin Mod Nginx by default doesn't log static file requests via Nginx vhost site include file for /usr/local/nginx/conf/staticfiles.conf - it sets those location context matches for images, js, css and html files to access_log off by default. So won't be listed in your access.log unless you comment out access_log off line in each context location match in /usr/local/nginx/conf/staticfiles.conf and restart nginx server.

    GoAccess Info


    goaccess_screenshot1M-03L.png

    GoAccess 1.1.1 updated layout

    goaccess-1.1.1-screens-ssh01.png

    goaccess-1.1.1-screens-01.png

    goaccess-1.1.1-screens-02.png
     
    Last edited: Feb 23, 2017
  2. eva2000

    eva2000 Administrator Staff Member

    30,955
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,416
    Local Time:
    11:53 PM
    Nginx 1.13.x
    MariaDB 5.5

    Configure /etc/goaccess.conf



    Updated: February 23rd, 2017- Heads up seem the html generation changed as it requires specifying a date and log format first.

    Full goaccess manual at http://goaccess.io/man

    So what you need to do is

    step 1. edit the /etc/goaccess.conf file and edit and uncomment the following settings

    Code (Text):
    # The following time format works with any of the
    # Apache/NGINX's log formats below.
    #
    #time-format %H:%M:%S
    time-format %T
    

    Code (Text):
    # The following date format works with any of the
    # Apache/NGINX's log formats below.
    #
    date-format %d/%b/%Y
    

    custom format for Centmin Mod Nginx
    Code (Text):
    # NCSA Combined Log Format
    #log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"
    log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u"
    


    step 2. append the path to /etc/goaccess.conf via -p option for html generation

    Code:
    goaccess -p /etc/goaccess.conf -f /home/nginx/domains/domainname.com/log/access.log -a > report.html

    Centmin Mod specific examples



    quick stats
    Code:
    goaccess -f /home/nginx/domains/domainname.com/log/access.log
    
    full stats
    Code:
    goaccess -f /home/nginx/domains/domainname.com/log/access.log -a
    
    html report
    Code:
    goaccess -f /home/nginx/domains/domainname.com/log/access.log -a > report.html
    
    exclude certain IP addresses from reports
    Code:
    goaccess --exclude-ip=iptoexclude -p /etc/goaccess.conf -f /home/nginx/domains/domainname.com/log/access.log -a > report.html
    i.e. the constellix.com monitoring IP list
    Code:
    goaccess --exclude-ip=192.73.252.209 --exclude-ip=199.38.182.28 --exclude-ip=209.177.145.67 --exclude-ip=209.177.157.208 --exclude-ip=103.25.58.42 --exclude-ip=103.6.85.32 --exclude-ip=103.6.87.219 --exclude-ip=23.239.17.158 --exclude-ip=106.186.122.163 --exclude-ip=37.235.56.13 --exclude-ip=212.71.238.144 --exclude-ip=208.111.40.251 --exclude-ip=176.58.89.83 --exclude-ip=198.58.122.143 -p /etc/goaccess.conf -f /home/nginx/domains/domainname.com/log/access.log -a > report.html
    process for html generation all gzipped access-*.gz logs. For this to work with zcat you'd need to compress with gzip any uncompressed access.log files within your /home/nginx/domains/domainname.com/log/ directory in the same naming format as the rest of the gzip compressed logs.
    Code:
    ls -lah
    total 25M
    drwxr-xr-x 2 root root 4.0K Jul 26 06:16 .
    drwxr-xr-x 8 root root 4.0K Jul 26 06:03 ..
    -rw-r--r-- 1 root root 5.6M Jul 26 06:03 access.log-20140609.gz
    -rw-r--r-- 1 root root 5.0M Jul 26 06:03 access.log-20140624.gz
    -rw-r--r-- 1 root root 4.2M Jul 26 06:03 access.log-20140702.gz
    -rw-r--r-- 1 root root 7.1M Jul 26 06:03 access.log-20140720.gz
    -rw-r--r-- 1 root root 3.0M Jul 26 06:03 access.log-20140726.gz
    Then process the logs via zcat piped into goaccess
    Code:
    zcat -f /home/nginx/domains/domainname.com/log/access.log-* | goaccess --exclude-ip=192.73.252.209 --exclude-ip=199.38.182.28 --exclude-ip=209.177.145.67 --exclude-ip=209.177.157.208 --exclude-ip=103.25.58.42 --exclude-ip=103.6.85.32 --exclude-ip=103.6.87.219 --exclude-ip=23.239.17.158 --exclude-ip=106.186.122.163 --exclude-ip=37.235.56.13 --exclude-ip=212.71.238.144 --exclude-ip=208.111.40.251 --exclude-ip=176.58.89.83 --exclude-ip=198.58.122.143 -p /etc/goaccess.conf -a > report.html
     
    Last edited: Feb 23, 2017
  3. eva2000

    eva2000 Administrator Staff Member

    30,955
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,416
    Local Time:
    11:53 PM
    Nginx 1.13.x
    MariaDB 5.5
  4. pamamolf

    pamamolf Well-Known Member

    2,820
    253
    83
    May 31, 2014
    Ratings:
    +447
    Local Time:
    3:53 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Super thanks George !!!! :)
     
  5. pamamolf

    pamamolf Well-Known Member

    2,820
    253
    83
    May 31, 2014
    Ratings:
    +447
    Local Time:
    3:53 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    GoAccess Dependencies

    Centos:
    ncurses-devel
    glib2-devel

    Does Centminmod install them or i must install them myself?

    Thanks
     
  6. eva2000

    eva2000 Administrator Staff Member

    30,955
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,416
    Local Time:
    11:53 PM
    Nginx 1.13.x
    MariaDB 5.5
    follow instructions to install them, they'll install if not available :)
     
  7. eva2000

    eva2000 Administrator Staff Member

    30,955
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,416
    Local Time:
    11:53 PM
    Nginx 1.13.x
    MariaDB 5.5
    Heads up seem the html generation changed as it requires specifying a date and log format first.

    Full goaccess manual at http://goaccess.io/man

    So what you need to do is

    step 1. create a /etc/goaccess.conf file and add the following into it

    Code:
    ######################################
    # Format Options
    ######################################
    
    # The date_format variable followed by a space, specifies
    # the log format date containing any combination of regular
    # characters and special format specifiers. They all begin with a
    # percentage (%) sign. See `man strftime`
    #
    #Any Apache log date format
    date-format %d/%b/%Y
    #
    #W3C (IIS) & AWS | Amazon CloudFront (Download Distribution)
    #date-format %Y-%m-%d
    
    # The log_format variable followed by a space or \t for
    # tab-delimited, specifies the log format string.
    #
    #Common Log Format (CLF)
    log-format %h %^[%d:%^] "%r" %s %b
    #
    #Common Log Format (CLF) with Virtual Host
    #log-format %^:%^ %h %^[%d:%^] "%r" %s %b
    #
    #NCSA Combined Log Format
    #log-format %h %^[%d:%^] "%r" %s %b "%R" "%u"
    #
    #NCSA Combined Log Format with Virtual Host
    #log-format %^:%^ %h %^[%d:%^] "%r" %s %b "%R" "%u"
    #
    #W3C
    #log-format %d %^ %h %^ %^ %^ %^ %r %^ %s %b %^ %^ %u %R
    #
    #AWS | Amazon CloudFront (Download Distribution)
    #log-format %d\t%^\t%^\t%b\t%h\t%m\t%^\t%r\t%s\t%R\t%u\t%^
    
    ######################################
    # UI Options
    ######################################
    
    # Prompt log/date configuration window on program start.
    #
    config-dialog false
    
    # Choose among color schemes
    # 1 : Default grey scheme
    # 2 : Green scheme
    #
    color-scheme 1
    
    # Turn off colored output. This is the  default output on
    # terminals that do not support colors.
    # true  : for no color output
    # false : use color-scheme
    #
    no-color false
    
    ######################################
    # File Options
    ######################################
    
    # Specify the path to the input log file. If set, it will take
    # priority over -f from the command line.
    #
    #log-file /var/log/apache2/access.log
    
    # Send all debug messages to the specified file. Needs to configured
    # with --enable-debug
    #
    #debug-file debug.log
    
    # Specify a custom configuration file to use. If set, it will take
    # priority over the global configuration file (if any).
    #
    #config-file=<filename>
    
    # Do not load the global configuration file.
    #
    #no-global-config false
    
    ######################################
    # Parse Options
    ######################################
    
    # Consider the following extensions as static files
    # The actual '.' is required and extensions are case sensitive
    #
    static-file .css
    static-file .CSS
    static-file .dae
    static-file .DAE
    static-file .eot
    static-file .EOT
    static-file .gif
    static-file .GIF
    static-file .ico
    static-file .ICO
    static-file .jpeg
    static-file .JPEG
    static-file .jpg
    static-file .JPG
    static-file .js
    static-file .JS
    static-file .map
    static-file .MAP
    static-file .mp3
    static-file .MP3
    static-file .pdf
    static-file .PDF
    static-file .png
    static-file .PNG
    static-file .svg
    static-file .SVG
    static-file .swf
    static-file .SWF
    static-file .ttf
    static-file .TTF
    static-file .txt
    static-file .TXT
    static-file .woff
    static-file .WOFF
    
    # Exclude an IPv4 or IPv6 from being counted.
    # Ranges can be included as well using a dash in between
    # the IPs (start-end).
    #
    #exclude-ip 127.0.0.1
    #exclude-ip 192.168.0.1-192.168.0.100
    #exclude-ip ::1
    #exclude-ip 0:0:0:0:0:ffff:808:804-0:0:0:0:0:ffff:808:808
    
    # Enable a list of user-agents by host. For faster parsing, do not
    # enable this flag.
    #
    agent-list false
    
    # Include HTTP request method if found. This will create a
    # request key containing the request method + the actual request.
    #
    http-method true
    
    # Include HTTP request protocol if found. This will create a
    # request key containing the request protocol + the actual request.
    #
    http-protocol true
    
    # Ignore request's query string.
    # i.e.,  www.google.com/page.htm?query => www.google.com/page.htm
    #
    no-query-string false
    
    # Disable IP resolver on terminal output.
    #
    no-term-resolver false
    
    # Write output to stdout given one of the following formats:
    # csv  : A comma-separated values (CSV)
    # json : JSON (JavaScript Object Notation)
    #
    #output-format json
    
    # Display real OS names. e.g, Windows XP, Snow Leopard.
    #
    real-os true
    
    # Disable progress metrics.
    #
    no-progress false
    
    # Enable mouse support on main dashboard.
    #
    with-mouse false
    
    #  Enable IP resolver on HTML|JSON output.
    #
    with-output-resolver false
    
    ######################################
    # GeoIP Options
    # Only if configured with --enable-geoip
    ######################################
    
    # Standard GeoIP database for less memory usage.
    #
    #std-geoip false
    
    # Specify path to GeoIP City database file. i.e., GeoLiteCity.dat
    # GeoLiteCity.dat needs to be downloaded from maxmind.com.
    #
    # wget -N http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
    # gunzip GeoLiteCity.dat.gz
    #
    #geoip-city-data /usr/local/share/GeoIP/GeoLiteCity.dat
    
    ######################################
    # Tokyo Cabinet Options
    # Only if configured with --enable-tcb=btree
    ######################################
    
    # On-disk B+ Tree
    # Persist parsed data into disk. This should be set to
    # the first dataset prior to use `load-from-disk`.
    # Setting it to false will delete all database files
    # when exiting the program.
    #keep-db-files true
    
    # On-disk B+ Tree
    # Load previously stored data from disk.
    # Database files need to exist. See `keep-db-files`.
    #load-from-disk false
    
    # On-disk B+ Tree
    # Path where the on-disk database files are stored.
    # The default value is the /tmp directory.
    #
    #db-path /tmp
    
    # On-disk B+ Tree
    # Set the size in bytes of the extra mapped memory.
    # The default value is 0.
    #
    #xmmap 0
    
    # On-disk B+ Tree
    # Max number of leaf nodes to be cached.
    # Specifies the maximum number of leaf nodes to be cached.
    # If it is not more than 0, the default value is specified.
    # The default value is 1024.
    #
    #cache-lcnum 1024
    
    # On-disk B+ Tree
    # Specifies the maximum number of non-leaf nodes to be cached.
    # If it is not more than 0, the default value is specified.
    # The default value is 512.
    #
    #cache-ncnum 512
    
    # On-disk B+ Tree
    # Specifies the number of members in each leaf page.
    # If it is not more than 0, the default value is specified.
    # The default value is 128.
    #
    #tune-lmemb 128
    
    # On-disk B+ Tree
    # Specifies the number of members in each non-leaf page.
    # If it is not more than 0, the default value is specified.
    # The default value is 256.
    #
    #tune-nmemb 256
    
    # On-disk B+ Tree
    # Specifies the number of elements of the bucket array.
    # If it is not more than 0, the default value is specified.
    # The default value is 32749.
    # Suggested size of the bucket array is about from 1 to 4
    # times of the number of all pages to be stored.
    #
    #tune-bnum 32749
    
    # On-disk B+ Tree
    # Specifies that each page is compressed with ZLIB|BZ2 encoding.
    # Disabled by default.
    #
    #compression zlib
    step 2. append the path to /etc/goaccess.conf via -p option for html generation

    Code:
    goaccess -p /etc/goaccess.conf -f /home/nginx/domains/domainname.com/log/access.log -a > report.html
     
  8. eva2000

    eva2000 Administrator Staff Member

    30,955
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,416
    Local Time:
    11:53 PM
    Nginx 1.13.x
    MariaDB 5.5
    Sample html generated from this forum :D

    goaccess_community_260714_html.png goaccess_community_260714_html2.png goaccess_community_260714_html3.png
     
    • Like Like x 1
  9. RoldanLT

    RoldanLT Well-Known Member

    3,978
    965
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,329
    Local Time:
    9:53 PM
    1.11
    10.2
    You can get rid off Google Analytics now :D
     
  10. eva2000

    eva2000 Administrator Staff Member

    30,955
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,416
    Local Time:
    11:53 PM
    Nginx 1.13.x
    MariaDB 5.5
  11. eva2000

    eva2000 Administrator Staff Member

    30,955
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,416
    Local Time:
    11:53 PM
    Nginx 1.13.x
    MariaDB 5.5
    More interesting info to add = Identifying aggressive crawlers using Go Access

    Of course instead of iptables, you can use CSF Firewall to block ips CSF Firewall - Centmin Mod - Menu based Nginx installer for CentOS servers
     
  12. Mask

    Mask Active Member

    104
    29
    28
    Nov 10, 2014
    Ratings:
    +35
    Local Time:
    6:53 PM
    Nginx 1.9.1
    MariaDB 10.0.19
    Code:
    [root@vps goaccess-0.8.5]# ./configure --enable-geoip --enable-utf8
    checking for a BSD-compatible install... /usr/bin/install -c
    checking whether build environment is sane... yes
    checking for a thread-safe mkdir -p... /bin/mkdir -p
    checking for gawk... gawk
    checking whether make sets $(MAKE)... yes
    checking for gcc... ccache gcc
    checking whether the C compiler works... yes
    checking for C compiler default output file name... a.out
    checking for suffix of executables...
    checking whether we are cross compiling... no
    checking for suffix of object files... o
    checking whether we are using the GNU C compiler... yes
    checking whether ccache gcc accepts -g... yes
    checking for ccache gcc option to accept ISO C89... none needed
    checking for style of include used by make... GNU
    checking dependency style of ccache gcc... gcc3
    checking for pthread_create in -lpthread... yes
    checking for GeoIP_new in -lGeoIP... no
    configure: error: *** Missing development files for the GeoIP library
    
    Looks like I am missing GeoIP. Do I need to use geoip.sh from (Nginx - GeoIP Thread | Centmin Mod Community
    And will it add more load to the server?? Can I just skip --enable-geoip in the config command above??
     
  13. eva2000

    eva2000 Administrator Staff Member

    30,955
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,416
    Local Time:
    11:53 PM
    Nginx 1.13.x
    MariaDB 5.5
    yeah either skip --enable-geoip or use addons/geoip.sh installer :)
     
  14. Mask

    Mask Active Member

    104
    29
    28
    Nov 10, 2014
    Ratings:
    +35
    Local Time:
    6:53 PM
    Nginx 1.9.1
    MariaDB 10.0.19
    I used the 2nd configuration as given above.
    But my Bandwidth column says 0.0 B and my Browsers and Operating Systems says all is unknown.
    I think the log format is missing something ... Any ideas ?
     
  15. Mask

    Mask Active Member

    104
    29
    28
    Nov 10, 2014
    Ratings:
    +35
    Local Time:
    6:53 PM
    Nginx 1.9.1
    MariaDB 10.0.19
    Anyone here using GoAccess??
    Can you combine multiple log files in one report.html ?? (i.e. lets say you have 5 compressed log file and one normal file, can you combine the result in one html report file??)
    Please tell me how are u using it ?
     
  16. eva2000

    eva2000 Administrator Staff Member

    30,955
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,416
    Local Time:
    11:53 PM
    Nginx 1.13.x
    MariaDB 5.5
    you using GoAccess open source real time web log analyzer | Centmin Mod Community ? newer GoAccess versions require more setup as per that link

    yup in manual page for zcat GoAccess - Manual Page and 2nd post GoAccess open source real time web log analyzer | Centmin Mod Community
     
    • Informative Informative x 1
  17. Eduardo

    Eduardo New Member

    27
    3
    3
    Feb 7, 2015
    Ratings:
    +5
    Local Time:
    11:53 AM
    1.7.9
    I'm testing it and looks cool.

    But, I dont see any stats about 404 and "time served"
     
  18. RoldanLT

    RoldanLT Well-Known Member

    3,978
    965
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,329
    Local Time:
    9:53 PM
    1.11
    10.2
    This is WHAT I NEED! :D
     
    • Like Like x 1
  19. RoldanLT

    RoldanLT Well-Known Member

    3,978
    965
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,329
    Local Time:
    9:53 PM
    1.11
    10.2
    This is really Awesome! So easy to use :D
     
  20. eva2000

    eva2000 Administrator Staff Member

    30,955
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,416
    Local Time:
    11:53 PM
    Nginx 1.13.x
    MariaDB 5.5
    yeah been using this for a while :D