Welcome to Centmin Mod Community
Register Now

Featured Sysadmin GoAccess open source real time web log analyzer

Discussion in 'System Administration' started by eva2000, Jun 5, 2014.

  1. rdan

    rdan Well-Known Member

    5,449
    1,410
    113
    May 25, 2014
    Ratings:
    +2,204
    Local Time:
    5:04 AM
    Mainline
    10.2
    But how I can I generated a single HTML file for all the log files?

     
  2. eva2000

    eva2000 Administrator Staff Member

    55,158
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    7:04 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    manual linked in 1st post and 2nd post has examples of using zcat

    example

    Code:
    cd /home/nginx/domains/domainname.com/log
    
    zcat -f /home/nginx/domains/domainname.com/log/access.log-*.gz | cat /home/nginx/domains/domainname.com/log/access.log* | cat /home/nginx/domains/domainname.com/log/access.log-* | goaccess --exclude-ip=192.73.252.209 --exclude-ip=199.38.182.28 --exclude-ip=209.177.145.67 --exclude-ip=209.177.157.208 --exclude-ip=103.25.58.42 --exclude-ip=103.6.85.32 --exclude-ip=103.6.87.219 --exclude-ip=23.239.17.158 --exclude-ip=106.186.122.163 --exclude-ip=37.235.56.13 --exclude-ip=212.71.238.144 --exclude-ip=208.111.40.251 --exclude-ip=176.58.89.83 --exclude-ip=198.58.122.143 -p /etc/goaccess.conf -a > report.html
     
    Last edited: Sep 17, 2015
  3. Eduardo

    Eduardo Member

    38
    3
    8
    Feb 7, 2015
    Ratings:
    +5
    Local Time:
    6:04 PM
    1.7.9
    you can use

    Code:
     zcat -f /home/nginx/domains/*/log/access.log | goaccess -p /etc/goaccess.conf -a
    and goaccess.conf

    Code:
    time-format %T
    date-format %d/%b/%Y
    log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"
     
  4. eva2000

    eva2000 Administrator Staff Member

    55,158
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    7:04 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    sweet thanks for that info @Eduardo (y)
     
  5. eva2000

    eva2000 Administrator Staff Member

    55,158
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    7:04 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  6. pamamolf

    pamamolf Premium Member Premium Member

    4,100
    428
    83
    May 31, 2014
    Ratings:
    +837
    Local Time:
    12:04 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    I want to recommend to have port 7890 open by default on Centminmod installation for it as the latest version has many cool info :)
     
  7. eva2000

    eva2000 Administrator Staff Member

    55,158
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    7:04 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    just whitelist the 7890 port in TCP_IN comma separated list in CSF firewall config file /etc/csf/csf.conf and restart csf firewall CSF Firewall - CentminMod.com LEMP Nginx web stack for CentOS

    or more restricted by ip address of your ISP add to /etc/csf/csf.allow and restart csf firewall
    Code (Text):
    tcp|in|d=7890|s=11.22.33.44
    

    where 11.22.33.44 is your ISP ip address so only you can visit on port 7890
     
  8. pamamolf

    pamamolf Premium Member Premium Member

    4,100
    428
    83
    May 31, 2014
    Ratings:
    +837
    Local Time:
    12:04 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    This log format recommend it by @Eduardo doesn't work anymore with the latest version :(

    Code:
    log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"
    result:

    Code:
    Parsed 10 lines producing the following errors:
    
    Token '9.103.253.1 - - [22/Feb/2017:03:42:26 +0200]' doesn't match specifier '%x'
    Token '9.129.118.30 - - [22/Feb/2017:03:43:18 +0200]' doesn't match specifier '%x'
    Token '9.103.253.1 - - [22/Feb/2017:03:43:27 +0200]' doesn't match specifier '%x'
    Token '6.249.75.160 - - [22/Feb/2017:03:43:35 +0200]' doesn't match specifier '%x'
    Token '0.97.190.157 - - [22/Feb/2017:03:43:54 +0200]' doesn't match specifier '%x'
    Token '88.4.13.172 - - [22/Feb/2017:03:43:59 +0200]' doesn't match specifier '%x'
    Token '88.4.13.172 - - [22/Feb/2017:03:44:01 +0200]' doesn't match specifier '%x'
    Token '9.129.118.30 - - [22/Feb/2017:03:44:19 +0200]' doesn't match specifier '%x'
    Token '57.55.39.184 - - [22/Feb/2017:03:44:21 +0200]' doesn't match specifier '%x'
    Token '57.55.39.184 - - [22/Feb/2017:03:44:23 +0200]' doesn't match specifier '%x'
    
    Format Errors - Verify your log/date/time format
    :(

    From the config file i can read this:

    # NOTE: If the time/date is a timestamp in seconds or microseconds
    # %x must be used instead of %d & %t to represent the date & time.
     
  9. eva2000

    eva2000 Administrator Staff Member

    55,158
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    7:04 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Last edited: Feb 23, 2017
  10. eva2000

    eva2000 Administrator Staff Member

    55,158
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    7:04 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    ooh in goaccess FAQ
    that gives me
    Code (Text):
    ./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" '  '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for" ' 'rt=$request_time ua="$upstream_addr" ' 'us="$upstream_status" ut="$upstream_response_time" ' 'ul="$upstream_response_length" ' 'cs=$upstream_cache_status' 
    
    - Generated goaccess config:
    
    time-format %T
    date-format %d/%b/%Y
    log_format %h - %^ [%d:%t %^] "%r" 
    

    so maybe
    Code (Text):
    log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u"
    
     
  11. pamamolf

    pamamolf Premium Member Premium Member

    4,100
    428
    83
    May 31, 2014
    Ratings:
    +837
    Local Time:
    12:04 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    Working :)
     
  12. eva2000

    eva2000 Administrator Staff Member

    55,158
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    7:04 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    yay will update 1st post in this thread's instructions for goaccess.conf config
     
  13. eva2000

    eva2000 Administrator Staff Member

    55,158
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    7:04 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    updated 1st post with updated screenshots too :)

    goaccess-1.1.1-screens-01.png goaccess-1.1.1-screens-02.png goaccess-1.1.1-screens-ssh01.png
     
  14. Dnyan

    Dnyan Member

    106
    24
    18
    Sep 16, 2017
    Ratings:
    +36
    Local Time:
    2:34 AM
    1.17.8
    10.3.22
    zcat gives error

    Code:
    Parsed 1 linesproducing the following errors:
    
    Token for '%h' specifier is NULL.
     
  15. eva2000

    eva2000 Administrator Staff Member

    55,158
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    7:04 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    did you do the 1st post steps for configuraing goaccess ?
    looks like goaccess.conf symlink in that linked instruction has changed since to now
    Code (Text):
    rm -f /etc/goaccess.conf
    ln -s /usr/local/etc/goaccess/goaccess.conf /etc/goaccess.conf
    


    did you modifying Centmin Mod's default nginx log format by any chance ? the error is related to goaccess not finding a format it recognises AFAIK https://github.com/allinurl/goaccess/issues/1141 which is fixed by instructions at https://community.centminmod.com/th...rce-real-time-web-log-analyzer.286/#post-1305
     
  16. Dnyan

    Dnyan Member

    106
    24
    18
    Sep 16, 2017
    Ratings:
    +36
    Local Time:
    2:34 AM
    1.17.8
    10.3.22
    when i checked that file, it was blank one (is this OK or should i reinstall goaccess).


    i did as stated and pasted a content to etc/goaccess.conf from this post https://community.centminmod.com/th...rce-real-time-web-log-analyzer.286/#post-4100

    edited all required parameters.

    When i tried to add access.log file which is yet to be written in access.log i get error.
    For old gzip and Today's file of access.log.txt it wont throw error.

    Also the report html file wont update it after an hour or on next day too.

    What i should do to make it update (live with may be delay of 1 hour or 256k memory buffer).

    I have not changed any format of logs and using it as it is.
     
  17. eva2000

    eva2000 Administrator Staff Member

    55,158
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    7:04 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    i updated instructions at https://community.centminmod.com/th...rce-real-time-web-log-analyzer.286/#post-1305 to updated goaccess.conf symlink so need to edit /etc/goaccess.conf with the changes so goaccess recognises centmin mod nginx's log format first

    need to setup your own script to run the html log and then put that script in cronjob to schedule running
     
  18. Dnyan

    Dnyan Member

    106
    24
    18
    Sep 16, 2017
    Ratings:
    +36
    Local Time:
    2:34 AM
    1.17.8
    10.3.22
    I had setup cron in crontab with following line

    Code:
    0 * * * * /usr/bin/zcat -f /home/nginx/domains/mydomain.com/log/access.log* | goaccess -p /etc/goaccess.conf -a > /home/nginx/domains/mydomain.com/public/mydomain_report.html
    If i run this command manually, it updates the report html

    but when it run as a cron, it makes the report html to go blank without any details.

    adding >/dev/null 2>&1 to end of cron command also makes it blank only.

    can you please do help in setting up cron for this.
     
  19. eva2000

    eva2000 Administrator Staff Member

    55,158
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    7:04 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    try putting command into a script and add script to cronjob instead
     
  20. Dnyan

    Dnyan Member

    106
    24
    18
    Sep 16, 2017
    Ratings:
    +36
    Local Time:
    2:34 AM
    1.17.8
    10.3.22
    I made cron script at /root/tools/cron-goaccess.sh

    when i run this script via cli, it works and status gets updated.

    When i put that in crontab, it makes the html page go blank.

    My crontab command
    0 * * * * /root/tools/cron-goaccess.sh


    tried with and without >/dev/null 2>&1 result were always same, it make html go blank.