/ Register

Welcome to Centmin Mod Community
Register Now

Nginx Analyze which file/path is using the most Nginx traffic? [SOLVED]

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by BobbyWibowo, May 4, 2020.

Previous Thread Next Thread
Loading...
  1. May 4, 2020 #1
    BobbyWibowo

    BobbyWibowo Active Member

    197
    42
    28
    Jul 30, 2015
    Indonesia
    Ratings:
    +71
    Local Time:
    3:41 AM
    1.17.x
    10.3.x
    Hi, anybody knows what tool can be easily used to analyze Nginx logs to map which files are using the most traffic and whatnot?
    I've been using Nginx Amplify in the past month or so, but it can only be used for overall analytics as far as I'm aware.

    Not being able to figure out which files suddenly brought about this traffic is a bit headache inducing:
    [​IMG]
    [​IMG]

    For the curious, it's just one of those cheap and old server rack from Scaleway, so I think not being able to handle this kind of traffic is simply a matter of course for it.

    Oh yeah, solutions such as Google Analytics won't really mean much on my case, since I'm serving static files. I found out there was that thing where they integrate Google Analytics directly from their Nginx config, so it will indeed log every requests, but that seems too darn complicated to setup as opposed to simply analyzing existing logs, lmao.

    And ngxtop doesn't work either. Probably incompatible log format (main_ext).
     
    Last edited: May 4, 2020
  2. May 4, 2020 #2
    BobbyWibowo

    BobbyWibowo Active Member

    197
    42
    28
    Jul 30, 2015
    Indonesia
    Ratings:
    +71
    Local Time:
    3:41 AM
    1.17.x
    10.3.x
    Never mind, I found GoAccess, thanks to another wisdom eva left laying around on some other topics, lawl. It has what I needed, and it wasn't too complicated to install (their FAQ page also linked a tool with which I could easily generate GoAccess log-format from my Nginx log-format).
     
  3. May 4, 2020 #3
    eva2000

    eva2000 Administrator Staff Member

    58,905
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    6:41 AM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    Yeah main_ext log format for Nginx Amplify will break other tools log analytics as it expects it in Nginx's define format.

    But yes GoAccess Sysadmin - GoAccess open source real time web log analyzer will help for default Nginx log format or if you alter GoAccess config format for any custom Nginx formats. Or just have 2 separate Nginx log formats - 1 default and 1 for main_ext.

    Or just inspect your Nginx logs in better JSON format Nginx - How To Configure Nginx For JSON Based Access Logging

    i.e. Top 10 requested urls for just /forums/ and /threads/ excluding rss, manifest.json and sw.js files
    Code (Text):
    cat access_log.json | egrep '/forums/|/threads/' | egrep -v 'index.rss|manifest.json|sw.js' | jq -r '"\(.ssl_protocol) \(.status) \(.request_uri)"' | sort | uniq -c | sort -rn | head -n10
   2173 TLSv1.3 200 /threads/cloudflare-domain-registrar-transfers-reduced-my-costs-by-37.16038/
    945 TLSv1.3 200 /login/csrf-token-refresh
    115 TLSv1.3 200 /login/login
    113 TLSv1.3 200 /threads/php-7-3-vs-7-2-vs-7-1-vs-7-0-php-fpm-benchmarks.16090/
    106 TLSv1.3 200 /threads/php-benchmarks-7-4-vs-7-3-vs-7-2-vs-7-1-vs-7-0-php-fpm.18741/
    103 TLSv1.3 200 /threads/php-7-x-benchmarks-centmin-mod-vs-easyengine-vs-webinoly-vs-vestacp-vs-oneinstack.14988/
    103 TLSv1.3 200 /threads/13-way-vps-server-benchmark-comparison-tests-upcloud-vs-digitalocean-vs-linode-vs-vultr-vs-hetzner.17742/
    102 TLSv1.3 200 /threads/my-first-wordpress-autoptimize-gzip-companion-plugin.15314/
    100 TLSv1.3 200 /threads/python-2-7-will-reach-the-end-of-its-life-on-january-1st-2020-running-out-of-time.18806/
     98 TLSv1.3 200 /forums/centmin-mod-news.3/
     
  4. May 4, 2020 #4
    BobbyWibowo

    BobbyWibowo Active Member

    197
    42
    28
    Jul 30, 2015
    Indonesia
    Ratings:
    +71
    Local Time:
    3:41 AM
    1.17.x
    10.3.x
    lawl, after getting rid of user-files that used those unholy amount of bandwidth:
    [​IMG]
    [​IMG]

    For the curious, one of my site is a file hosting, using a Node software I co-developed.
    I've been warning my users in the past few months not to upload full-length anime or tv show episodes, and clearly listed them as forbidden files, but they never listened, lmao.
    Those huge files were, as expected, full-length anime and tv shows episodes. They were probably hosted in some kind of popular streaming websites of some sort. Geez.
     
  5. May 4, 2020 #5
    BobbyWibowo

    BobbyWibowo Active Member

    197
    42
    28
    Jul 30, 2015
    Indonesia
    Ratings:
    +71
    Local Time:
    3:41 AM
    1.17.x
    10.3.x
    @eva2000 Hi there, is there any recommendations for HTML real-time server stats dashboard?
    I mean dashboard specifically for usages of CPU, RAM, disk, network and all that jazz, but only those basically, and preferably can store historical data.
    Nginx Amplify had them, and they were pretty good, but they weren't exactly real-time, and its main feature of analyzing nginx logs couldn't seem to be toggled off (plus I'd rather have a self-hosted solution).
    GoAcess is very great in analyzing nginx logs so far, especially since it's real-time, but it doesn't have server stats.
     
  6. May 4, 2020 #6
    eva2000

    eva2000 Administrator Staff Member

    58,905
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    6:41 AM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    You could try netdata but real time and historic usage will add more server resource usage. Centmin Mod 123.09beta01 has cminfo top command and cminfo sar-cpu, sar-mem, phpstats commands Upgrade - Nginx - Insight Guide - cminfo command explained they're all moment in time (when you run them) but that would technically be considered realtime :D You can also script cminfo top etc commands to be logged at certain intervals for troubleshooting too.

    Code (Text):
    cminfo sar-cpu

------------------------------------------------------------------
 CPU Utilisation % Last 7 days (8 CPU Threads):
------------------------------------------------------------------
%CPU  min:  %user:  0.05   %nice:  0.00  %system:  0.05   %iowait:  0.00  %steal:  0.00  %idle:  8.64
%CPU  avg:  %user:  0.54   %nice:  0.00  %system:  0.15   %iowait:  0.71  %steal:  0.00  %idle:  98.60
%CPU  max:  %user:  76.08  %nice:  0.10  %system:  21.53  %iowait:  5.43  %steal:  0.00  %idle:  99.87
%CPU  50%:  %user:  0.09   %nice:  0.00  %system:  0.10   %iowait:  0.84  %steal:  0.00  %idle:  99.01
%CPU  75%:  %user:  0.09   %nice:  0.00  %system:  0.11   %iowait:  1.27  %steal:  0.00  %idle:  99.79
%CPU  90%:  %user:  0.27   %nice:  0.00  %system:  0.12   %iowait:  1.69  %steal:  0.00  %idle:  99.85
%CPU  95%:  %user:  0.61   %nice:  0.00  %system:  0.16   %iowait:  2.53  %steal:  0.00  %idle:  99.85
%CPU  99%:  %user:  13.43  %nice:  0.00  %system:  0.71   %iowait:  3.76  %steal:  0.00  %idle:  99.86

------------------------------------------------------------------
 CPU Utilisation % Daily Last 7 days (8 CPU Threads):
------------------------------------------------------------------
May 04 2020 %CPU
%CPU  min:  %user:  0.08  %nice:  0.00  %system:  0.10  %iowait:  0.01  %steal:  0.00  %idle:  96.85
%CPU  avg:  %user:  0.12  %nice:  0.00  %system:  0.11  %iowait:  0.48  %steal:  0.00  %idle:  99.28
%CPU  max:  %user:  0.63  %nice:  0.04  %system:  0.22  %iowait:  2.54  %steal:  0.00  %idle:  99.80
%CPU  95%:  %user:  0.30  %nice:  0.00  %system:  0.12  %iowait:  1.69  %steal:  0.00  %idle:  99.79
May 03 2020 %CPU
%CPU  min:  %user:  0.07   %nice:  0.00  %system:  0.08  %iowait:  0.01  %steal:  0.00  %idle:  71.95
%CPU  avg:  %user:  0.34   %nice:  0.00  %system:  0.16  %iowait:  0.62  %steal:  0.00  %idle:  98.89
%CPU  max:  %user:  24.32  %nice:  0.04  %system:  3.48  %iowait:  3.37  %steal:  0.00  %idle:  99.80
%CPU  95%:  %user:  0.49   %nice:  0.00  %system:  0.19  %iowait:  1.70  %steal:  0.00  %idle:  99.79
May 02 2020 %CPU
%CPU  min:  %user:  0.08  %nice:  0.00  %system:  0.10  %iowait:  0.01  %steal:  0.00  %idle:  92.56
%CPU  avg:  %user:  0.26  %nice:  0.00  %system:  0.12  %iowait:  0.63  %steal:  0.00  %idle:  98.99
%CPU  max:  %user:  6.97  %nice:  0.04  %system:  0.44  %iowait:  2.73  %steal:  0.00  %idle:  99.80
%CPU  95%:  %user:  0.88  %nice:  0.00  %system:  0.23  %iowait:  1.70  %steal:  0.00  %idle:  99.79
May 01 2020 %CPU
%CPU  min:  %user:  0.08   %nice:  0.00  %system:  0.09   %iowait:  0.00  %steal:  0.00  %idle:  8.64
%CPU  avg:  %user:  1.07   %nice:  0.00  %system:  0.31   %iowait:  0.61  %steal:  0.00  %idle:  98.01
%CPU  max:  %user:  76.08  %nice:  0.04  %system:  21.53  %iowait:  3.37  %steal:  0.00  %idle:  99.81
%CPU  95%:  %user:  0.44   %nice:  0.00  %system:  0.14   %iowait:  1.70  %steal:  0.00  %idle:  99.80
Apr 30 2020 %CPU
%CPU  min:  %user:  0.07  %nice:  0.00  %system:  0.07  %iowait:  0.01  %steal:  0.00  %idle:  96.44
%CPU  avg:  %user:  0.12  %nice:  0.00  %system:  0.10  %iowait:  0.66  %steal:  0.00  %idle:  99.13
%CPU  max:  %user:  1.17  %nice:  0.04  %system:  0.17  %iowait:  3.37  %steal:  0.00  %idle:  99.84
%CPU  95%:  %user:  0.27  %nice:  0.00  %system:  0.11  %iowait:  1.70  %steal:  0.00  %idle:  99.81
Apr 29 2020 %CPU
%CPU  min:  %user:  0.05   %nice:  0.00  %system:  0.05  %iowait:  0.01  %steal:  0.00  %idle:  83.67
%CPU  avg:  %user:  1.57   %nice:  0.00  %system:  0.16  %iowait:  0.61  %steal:  0.00  %idle:  97.67
%CPU  max:  %user:  13.71  %nice:  0.10  %system:  6.22  %iowait:  3.36  %steal:  0.00  %idle:  99.87
%CPU  95%:  %user:  13.42  %nice:  0.00  %system:  0.69  %iowait:  1.69  %steal:  0.00  %idle:  99.87
Apr 28 2020 %CPU
%CPU  min:  %user:  0.06   %nice:  0.00  %system:  0.06  %iowait:  0.01  %steal:  0.00  %idle:  60.00
%CPU  avg:  %user:  0.25   %nice:  0.00  %system:  0.12  %iowait:  0.91  %steal:  0.00  %idle:  98.73
%CPU  max:  %user:  31.69  %nice:  0.04  %system:  8.25  %iowait:  5.43  %steal:  0.00  %idle:  99.86
%CPU  95%:  %user:  0.59   %nice:  0.00  %system:  0.11  %iowait:  2.55  %steal:  0.00  %idle:  99.86
Apr 27 2020 %CPU
%CPU  min:  %user:  0.06   %nice:  0.00  %system:  0.06  %iowait:  0.01  %steal:  0.00  %idle:  84.36
%CPU  avg:  %user:  0.34   %nice:  0.00  %system:  0.08  %iowait:  1.02  %steal:  0.00  %idle:  98.56
%CPU  max:  %user:  13.86  %nice:  0.05  %system:  0.70  %iowait:  5.05  %steal:  0.00  %idle:  99.87
%CPU  95%:  %user:  0.60   %nice:  0.00  %system:  0.10  %iowait:  2.54  %steal:  0.00  %idle:  99.85

    Code (Text):
    cminfo sar-mem

------------------------------------------------------------------
 Memory Usage Daily Last 7 days (8 CPU Threads):
------------------------------------------------------------------

May 04 2020 Memory
Memory      min:
kbmemfree:  230600.0    kbmemused:  32346496.0  %memused:  98.8       kbbuffers:  430036.0   kbcached:  6217056.0
kbcommit:   18682480.0  %commit:    53.6        kbactive:  5142536.0  kbinact:    4527756.0  kbdirty:   24.0
Memory      avg:
kbmemfree:  275087.1    kbmemused:  32465936.9  %memused:  99.2       kbbuffers:  431126.7   kbcached:  6570137.9
kbcommit:   18715924.1  %commit:    53.7        kbactive:  5308768.1  kbinact:    4695933.2  kbdirty:   203.5
Memory      max:
kbmemfree:  394528.0    kbmemused:  32510424.0  %memused:  99.3       kbbuffers:  432088.0   kbcached:  6939108.0
kbcommit:   18739880.0  %commit:    53.8        kbactive:  5480680.0  kbinact:    4864792.0  kbdirty:   496.0
Memory      95%:
kbmemfree:  345609.0    kbmemused:  32502947.4  %memused:  99.3       kbbuffers:  431905.4   kbcached:  6904390.4
kbcommit:   18733427.4  %commit:    53.8        kbactive:  5455128.0  kbinact:    4854334.0  kbdirty:   349.4

May 03 2020 Memory
Memory      min:
kbmemfree:  228940.0    kbmemused:  32098556.0  %memused:  98.0       kbbuffers:  426992.0   kbcached:  6674212.0
kbcommit:   18597632.0  %commit:    53.4        kbactive:  5429768.0  kbinact:    4551628.0  kbdirty:   0.0
Memory      avg:
kbmemfree:  280207.0    kbmemused:  32460817.0  %memused:  99.1       kbbuffers:  439824.9   kbcached:  7614344.0
kbcommit:   18989181.8  %commit:    54.5        kbactive:  5948857.3  kbinact:    5008749.0  kbdirty:   607.2
Memory      max:
kbmemfree:  642468.0    kbmemused:  32512084.0  %memused:  99.3       kbbuffers:  448152.0   kbcached:  8314648.0
kbcommit:   19911228.0  %commit:    57.2        kbactive:  6651616.0  kbinact:    5427044.0  kbdirty:   109808.0
Memory      95%:
kbmemfree:  364846.8    kbmemused:  32504032.0  %memused:  99.3       kbbuffers:  446820.8   kbcached:  8225485.6
kbcommit:   19865805.6  %commit:    57.0        kbactive:  6580524.8  kbinact:    5302296.8  kbdirty:   471.2

May 02 2020 Memory
Memory      min:
kbmemfree:  231420.0    kbmemused:  31852684.0  %memused:  97.3       kbbuffers:  427120.0   kbcached:  6913184.0
kbcommit:   19655972.0  %commit:    56.4        kbactive:  5535808.0  kbinact:    4033628.0  kbdirty:   0.0
Memory      avg:
kbmemfree:  319047.8    kbmemused:  32421976.2  %memused:  99.0       kbbuffers:  436458.2   kbcached:  7493925.8
kbcommit:   19667884.5  %commit:    56.5        kbactive:  6083155.5  kbinact:    4732683.4  kbdirty:   517.7
Memory      max:
kbmemfree:  888340.0    kbmemused:  32509604.0  %memused:  99.3       kbbuffers:  474824.0   kbcached:  8233576.0
kbcommit:   19820512.0  %commit:    56.9        kbactive:  6539756.0  kbinact:    5521968.0  kbdirty:   87836.0
Memory      95%:
kbmemfree:  625586.8    kbmemused:  32501298.4  %memused:  99.3       kbbuffers:  474368.4   kbcached:  8132720.0
kbcommit:   19681661.6  %commit:    56.5        kbactive:  6532189.6  kbinact:    5255797.2  kbdirty:   376.0

May 01 2020 Memory
Memory      min:
kbmemfree:  240256.0    kbmemused:  26505148.0  %memused:  80.9       kbbuffers:  538824.0   kbcached:  6426496.0
kbcommit:   15191432.0  %commit:    43.6        kbactive:  5393344.0  kbinact:    1317296.0  kbdirty:   12.0
Memory      avg:
kbmemfree:  2060263.8   kbmemused:  30680760.2  %memused:  93.7       kbbuffers:  878833.6   kbcached:  6619274.6
kbcommit:   19539550.0  %commit:    56.1        kbactive:  8395524.1  kbinact:    2076909.8  kbdirty:   2462.7
Memory      max:
kbmemfree:  6235876.0   kbmemused:  32500768.0  %memused:  99.3       kbbuffers:  960880.0   kbcached:  7254512.0
kbcommit:   19984044.0  %commit:    57.4        kbactive:  9161100.0  kbinact:    4521928.0  kbdirty:   216916.0
Memory      95%:
kbmemfree:  5713924.8   kbmemused:  32481823.2  %memused:  99.2       kbbuffers:  960278.4   kbcached:  7252606.4
kbcommit:   19660165.6  %commit:    56.4        kbactive:  9158378.4  kbinact:    4042792.0  kbdirty:   412.0

Apr 30 2020 Memory
Memory      min:
kbmemfree:  6311788.0   kbmemused:  15116600.0  %memused:  46.2       kbbuffers:  919276.0   kbcached:  6248600.0
kbcommit:   19532768.0  %commit:    56.1        kbactive:  8821844.0  kbinact:    1256148.0  kbdirty:   28.0
Memory      avg:
kbmemfree:  11797979.1  kbmemused:  20943044.9  %memused:  64.0       kbbuffers:  937635.6   kbcached:  6463680.8
kbcommit:   19613818.5  %commit:    56.3        kbactive:  9046230.4  kbinact:    1307385.5  kbdirty:   210.3
Memory      max:
kbmemfree:  17624424.0  kbmemused:  26429236.0  %memused:  80.7       kbbuffers:  950928.0   kbcached:  6520340.0
kbcommit:   19663048.0  %commit:    56.4        kbactive:  9132868.0  kbinact:    1330988.0  kbdirty:   632.0
Memory      95%:
kbmemfree:  17062151.6  kbmemused:  25902642.4  %memused:  79.1       kbbuffers:  949866.8   kbcached:  6515455.2
kbcommit:   19648210.0  %commit:    56.4        kbactive:  9126419.2  kbinact:    1317015.2  kbdirty:   376.4

Apr 29 2020 Memory
Memory      min:
kbmemfree:  284532.0    kbmemused:  6784640.0   %memused:  20.7       kbbuffers:  79076.0    kbcached:  521388.0
kbcommit:   18495596.0  %commit:    53.1        kbactive:  2626400.0  kbinact:    524892.0   kbdirty:   0.0
Memory      avg:
kbmemfree:  19871830.8  kbmemused:  12869193.3  %memused:  39.3       kbbuffers:  787745.4   kbcached:  4850641.5
kbcommit:   18969667.6  %commit:    54.5        kbactive:  7071910.1  kbinact:    1346935.2  kbdirty:   275.0
Memory      max:
kbmemfree:  25956384.0  kbmemused:  32456540.0  %memused:  99.1       kbbuffers:  918748.0   kbcached:  10565440.0
kbcommit:   22857580.0  %commit:    65.6        kbactive:  9923324.0  kbinact:    9074124.0  kbdirty:   4512.0
Memory      95%:
kbmemfree:  25516101.0  kbmemused:  14760612.0  %memused:  45.1       kbbuffers:  914030.0   kbcached:  6222140.0
kbcommit:   19530650.0  %commit:    56.1        kbactive:  8729825.0  kbinact:    2338651.0  kbdirty:   440.0

Apr 28 2020 Memory
Memory      min:
kbmemfree:  245976.0    kbmemused:  32197132.0  %memused:  98.3       kbbuffers:  313472.0   kbcached:  9196244.0
kbcommit:   18593784.0  %commit:    53.4        kbactive:  7476548.0  kbinact:    4388800.0  kbdirty:   0.0
Memory      avg:
kbmemfree:  340407.1    kbmemused:  32400664.9  %memused:  99.0       kbbuffers:  314498.0   kbcached:  9327274.4
kbcommit:   18710440.1  %commit:    53.7        kbactive:  7791726.7  kbinact:    4442570.1  kbdirty:   303.5
Memory      max:
kbmemfree:  543940.0    kbmemused:  32495096.0  %memused:  99.2       kbbuffers:  386692.0   kbcached:  9393612.0
kbcommit:   18805492.0  %commit:    54.0        kbactive:  7893996.0  kbinact:    4643916.0  kbdirty:   13372.0
Memory      95%:
kbmemfree:  404569.6    kbmemused:  32453361.6  %memused:  99.1       kbbuffers:  313940.0   kbcached:  9383085.6
kbcommit:   18734136.8  %commit:    53.8        kbactive:  7871464.8  kbinact:    4488072.0  kbdirty:   398.4

Apr 27 2020 Memory
Memory      min:
kbmemfree:  237916.0    kbmemused:  31896724.0  %memused:  97.4       kbbuffers:  311232.0   kbcached:  8878192.0
kbcommit:   18547996.0  %commit:    53.2        kbactive:  6545700.0  kbinact:    4397212.0  kbdirty:   0.0
Memory      avg:
kbmemfree:  526830.5    kbmemused:  32214241.5  %memused:  98.4       kbbuffers:  312345.8   kbcached:  9241780.1
kbcommit:   18653962.1  %commit:    53.5        kbactive:  7399602.7  kbinact:    4712201.8  kbdirty:   1153.8
Memory      max:
kbmemfree:  844348.0    kbmemused:  32503156.0  %memused:  99.3       kbbuffers:  313648.0   kbcached:  9710804.0
kbcommit:   18917448.0  %commit:    54.3        kbactive:  7858672.0  kbinact:    5538224.0  kbdirty:   263536.0
Memory      95%:
kbmemfree:  815954.8    kbmemused:  32483882.8  %memused:  99.2       kbbuffers:  313500.0   kbcached:  9391798.4
kbcommit:   18831190.8  %commit:    54.1        kbactive:  7849680.8  kbinact:    5462218.4  kbdirty:   441.6
     
  7. May 4, 2020 #7
    BobbyWibowo

    BobbyWibowo Active Member

    197
    42
    28
    Jul 30, 2015
    Indonesia
    Ratings:
    +71
    Local Time:
    3:41 AM
    1.17.x
    10.3.x
    Ah yes, netdata. I'll go with that, thanks. I think I saw someone else using it a couple months ago, and even thought of using it myself, but wasn't in the mood to configure such stuff awhile back.
    I just found out they also have Nginx plugin, but fortunately it's not too extended, so I won't have to pull my hair out thinking GoAccess would become obsolete or anything, lawl.

    EDIT: Oh wait, they do have web_log plugin for more extended stats.. Basically able to replicate things that Amplify could show.. Eh, screw that jazz, lawl.
     
    Last edited: May 5, 2020
  8. May 6, 2020 #8
    BobbyWibowo

    BobbyWibowo Active Member

    197
    42
    28
    Jul 30, 2015
    Indonesia
    Ratings:
    +71
    Local Time:
    3:41 AM
    1.17.x
    10.3.x
    @eva2000 Aw shoot. So I already have Netdata installed and configured about right in the past 2 days right, but I just found this thread Beta Branch - addons/netdata.sh - new system monitor addon and pretty much just skimmed through it and its part 2 and found a few nifty parts that I hadn't configured myself. So I then immediately assumed I had simply wasted my time setting up Netdata manually, so I just purged it out of my system in preparation to instead use CM's installer,.. but I didn't notice the threads actually said the installer script was still work in progress. Lawl, this is so sad.
     
Previous Thread Next Thread
Loading...

.