Learn about Centmin Mod LEMP Stack today
Register Now

Recommend Server Specs Calculator

Discussion in 'Virtual Private Server (VPS) hosting' started by eva2000, Oct 30, 2023.

  1. eva2000

    eva2000 Administrator Staff Member

    51,957
    11,973
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,470
    Local Time:
    7:14 PM
    Nginx 1.25.x
    MariaDB 10.x
    Been playing with creating a server specs recommendation calculator that takes an existing server's resource usage stats for CPU, memory, disk and bandwidth for the past 24hrs and provides a rough ballpark recommendation of server specs needed. It probably needs more fine-tuning, but take it for a spin and see what it recommends https://centminmod.com/recommendserver/ :D


    For example, I inputted the following values:
    • Number of CPU Threads: 8
    • Average CPU Load (1 minute): 0.3
    • Average CPU Load (5 minutes): 0.6
    • Average CPU Load (15 minutes): 0.7
    • Peak CPU Load (1 minute): 0.8
    • Peak CPU Load (5 minutes): 0.9
    • Peak CPU Load (15 minutes): 1.2
    • Average CPU Usage: 22%
    • Peak CPU Usage: 66%
    • Average Memory Usage: 9.3 GB
    • Peak Memory Usage: 14.6 GB
    • Number of Local Backup Copies: 7
    • Current Disk Usage: 100 GB
    • Average Bandwidth Egress: 12.555 Mbps
    • Peak Bandwidth Egress: 99.273 Mbps
    And got recommended the following specs:
    • Recommended CPU: 10 Threads
    • Recommended Memory: 10 - 19 GB
    • Recommended Disk: 100 - 800 GB
    • Recommended Bandwidth Egress: 4 TB per month - 31 TB per month
     
  2. eva2000

    eva2000 Administrator Staff Member

    51,957
    11,973
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,470
    Local Time:
    7:14 PM
    Nginx 1.25.x
    MariaDB 10.x
  3. Colin

    Colin Premium Member Premium Member

    191
    59
    28
    Oct 7, 2015
    Sheffield UK
    Ratings:
    +153
    Local Time:
    10:14 AM
    1.19.#
    MariaDB 10.1.#
    Thanks for this.

    My semi infrequent, is it ok checks. The suggestion is roughly what I'd upgrade to, perhaps not quite so many threads. Although presently more memory bound than anything. Be a fantastic script on server that submits and gives you a 'report/uuid' link to follow :)
     
  4. eva2000

    eva2000 Administrator Staff Member

    51,957
    11,973
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,470
    Local Time:
    7:14 PM
    Nginx 1.25.x
    MariaDB 10.x
    I did say recommended :D Optimally, with Nginx + PHP-FPM + MariaDB and other services, you don't want them fighting over each CPU threads as much. So usually recommendations will be on higher end for CPU threads :)

    The ID/submission idea is interesting :)
     
  5. eva2000

    eva2000 Administrator Staff Member

    51,957
    11,973
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,470
    Local Time:
    7:14 PM
    Nginx 1.25.x
    MariaDB 10.x
    Updated formula and metrics for https://centminmod.com/recommendserver/

     
  6. MaximilianKohler

    MaximilianKohler Member

    156
    2
    18
    Jun 23, 2023
    Ratings:
    +17
    Local Time:
    2:14 AM
    My usage is very low but it recommends I upgrade to 4-7 threads?
    Code:
    Recommendations
    
    Based on your input, here are our recommendations - this is a very general recommendation based on last 24hrs or 7 days of metrics:
    
        Recommended CPU: 4 - 7 Threads
        Recommended Memory: 2 - 3 GB
        Recommended Disk: 12 - 60 GB
        Recommended Bandwidth Egress: 1 TB per month - 1 TB per month
    
    
    Your Input
    
        Number of CPU Threads: 2
        Average CPU Load (1 minute): 0.0116
        Average CPU Load (5 minutes): 0.014
        Average CPU Load (15 minutes): 0.0297
        Peak CPU Load (1 minute): 1.73
        Peak CPU Load (5 minutes): 1.4
        Peak CPU Load (15 minutes): 0.945
        Average CPU Usage: 0.62%
        Peak CPU Usage: 17.7%
        Average CPU Steal: 0.250357%
        Peak CPU Steal: 32.65%
        Peak CPU Steal 90% percentile: 0.31%
        Peak CPU Steal 95% percentile: 0.38%
        Peak CPU Steal 99% percentile: 2.7%
        Average Memory Usage: 1.32548 GB
        Peak Memory Usage: 1.60815 GB
        Number of Local Backup Copies: 4
        Current Disk Usage: 12 GB
        Average Bandwidth Egress: 0.0190751 Mbps
        Peak Bandwidth Egress: 2.33264 Mbps
    
     
  7. eva2000

    eva2000 Administrator Staff Member

    51,957
    11,973
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,470
    Local Time:
    7:14 PM
    Nginx 1.25.x
    MariaDB 10.x
    Your peak CPU load average normalised across 2 CPU cores is 1.73 - which is your system CPU load average divided by number of CPU cores you have. So peak CPU load you saw reported was 1.73x2 - which means your using 1.73 = 173% load usage of your 2 CPU cores resources and overloaded. You don't have enough CPU cores to handle your peak usage. Optimal max normalised CPU load average should be 1.0.

    So if you have 2 CPU cores 2.0 reported load average/2 cores = 1.0. So if your system reports 6.0 CPU load average /2 = normalised 6/2 cores = 3.0 CPU load average = 300% CPU load = overloaded system by 200%

    If you have 4 CPU cores 4.0 reported load average /4 cores = 1.0. So if your system reports 6.0 CPU load average /4 = normalised 6/4 cores = 1.50 CPU load average = 150% CPU load = overloaded system by 50%

    And your peak CPU steal 34% is high so your CPU resources are competing with other shared VPS neighbours which isn't good
     
    Last edited: Dec 4, 2023
  8. MaximilianKohler

    MaximilianKohler Member

    156
    2
    18
    Jun 23, 2023
    Ratings:
    +17
    Local Time:
    2:14 AM
    Why is Peak CPU Usage: 17.7% then? That seems misleading. That would be the main stat I'd be basing things on.
     
  9. eva2000

    eva2000 Administrator Staff Member

    51,957
    11,973
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,470
    Local Time:
    7:14 PM
    Nginx 1.25.x
    MariaDB 10.x
    That value is over 24hr period and measures CPU utilisation while cpu load is demand over a specific period of time and depending on if you in putted numbers for 24hrs or 7 day calculation.

    CPU Load Average (sar -q): This option reports the load average of the system. The load average represents the average system load over a period of time. It essentially shows how many processes are waiting to run (i.e., waiting for CPU time). The load average is typically presented in three numbers corresponding to the last 1, 5, and 15 minutes, reflecting short-term, medium-term, and long-term load. A high load average means a lot of processes are waiting for CPU time, which might indicate a bottleneck if the values are consistently higher than the number of CPU cores available.

    CPU Utilization (sar -u): This option reports on the utilization of CPU resources. It shows how much time the CPU spends in different states, such as user mode, system mode, idle, I/O wait, etc.The key metrics include %user (time spent running user processes), %system (time spent running kernel processes), %iowait (time waiting for I/O operations), %idle (time when the CPU was not used), and others like %nice and %steal. High CPU utilization indicates that the CPU is busy. If it's consistently near 100%, it suggests that the CPU might be a bottleneck. Conversely, a high %idle value indicates underutilisation.

    In summary, sar -q provides a measure of demand on the CPU (how many processes are waiting), while sar -u shows how the CPU time is being used. Both are essential for understanding and diagnosing system performance, but they offer different perspectives: one is about demand (load) and the other about actual usage (utilization).

    Might need to add some explanations to the recommendations output
     
  10. eva2000

    eva2000 Administrator Staff Member

    51,957
    11,973
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,470
    Local Time:
    7:14 PM
    Nginx 1.25.x
    MariaDB 10.x
    FYI, updated https://centminmod.com/recommendserver/ with explanation of the 2.

     
    Last edited: Dec 4, 2023
  11. MaximilianKohler

    MaximilianKohler Member

    156
    2
    18
    Jun 23, 2023
    Ratings:
    +17
    Local Time:
    2:14 AM
    This command for 3 values for past 24 hours only gives me 1 value output:
    Code:
    awk -v nproc=$(nproc) '/Average/ {print $4 / nproc}' <(sar -q)
    Only the
    gives me 3 values. So I wonder if the other "24 hr commands" are also giving me wrong data?

    The first time I ran it I used the "Past 7 days for EL7" as much as possible. But I was curious to see what impact the new email service (listmonk) has. It seems minimal so far for sending out a few thousand emails at 90 per second.

    This also gives me a blank output
    Code:
    sar -u | awk '/all/ && !/Average/ {if ($8 > max) max = $8} END {print max}'
     
  12. Andy

    Andy Active Member

    532
    87
    28
    Aug 6, 2014
    Ratings:
    +126
    Local Time:
    4:14 AM
    Here is my result for your record, @eva2000

    Recommendations
    Based on your input, here are our recommendations - this is a very general recommendation based on last 24hrs or 7 days of metrics:

    • Recommended CPU: 4 - 5 Threads
    • Recommended Memory: 60 - 35 GB
    • Recommended Disk: 89 - 356 GB
    • Recommended Bandwidth Egress: 1 TB per month - 2 TB per month
    Go Back To Form

    Your Input
    • Number of CPU Threads: 8
    • Normalized CPU Load Average (1 minute): 0.014 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
    • Normalized CPU Load Average (1 minute) 90% percentile: 0.0275 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
    • Normalized CPU Load Average (1 minute) 95% percentile: 0.03625 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
    • Normalized CPU Load Average (1 minute) 99% percentile: 0.095 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
    • Normalized CPU Load Average (5 minutes): 0.0142 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
    • Normalized CPU Load Average (15 minutes): 0.0147 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
    • Normalized CPU Load Average Peak (1 minute): 0.3288 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
    • Normalized CPU Load Average Peak (5 minutes): 0.1787 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
    • Normalized CPU Load Average Peak (15 minutes): 0.1663 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
    • Average CPU Utilization: 1.46788%
    • Peak CPU Utilization: 100%
    • Peak CPU Utilization 90% percentile: 2.11%
    • Peak CPU Utilization 95% percentile: 2.9%
    • Peak CPU Utilization 99% percentile: 7.17%
    • Average CPU Steal: 0.319283% (average % of CPU time stolen from a virtualized VPS system to service another virtualized VPS system on same server. Ideally <10% CPU steal overwise the VPS host node is overcrowded with busy VPS neighbours.)
    • Peak CPU Steal: 24.01% (max % of CPU time stolen from a virtualized VPS system to service another virtualized VPS system on same server. Ideally <10% CPU steal overwise the VPS host node is overcrowded with busy VPS neighbours.)
    • Peak CPU Steal 90% percentile: 0.34%
    • Peak CPU Steal 95% percentile: 0.4%
    • Peak CPU Steal 99% percentile: 1.47%
    • Average Memory Usage: 58.0979 GB
    • Peak Memory Usage: 19.8088 GB
    • Number of Local Backup Copies: 3
    • Current Disk Usage: 89 GB
    • Average Bandwidth Egress: 0.639575 Mbps
    • Peak Bandwidth Egress: 4.93592 Mbps
     
  13. eva2000

    eva2000 Administrator Staff Member

    51,957
    11,973
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,470
    Local Time:
    7:14 PM
    Nginx 1.25.x
    MariaDB 10.x
    no the data is correct I updated CPU load average at 1min intervals to ave average metric and then 2nd question input to have 90,95,99 percentile CPU load average at 1 min interval for better gauge of the spread of CPU load averages
    will double check when I am not browsing from my mobile right now
     
  14. eva2000

    eva2000 Administrator Staff Member

    51,957
    11,973
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,470
    Local Time:
    7:14 PM
    Nginx 1.25.x
    MariaDB 10.x
    does
    Code (Text):
    sar -u | awk '/all/ && !/Average/ {if ($8 > max) max = $8} END {print max}'
    

    still return empty blank results now?

    The command was correct, so the only thing I can think of is you ran it close to the server's end of the date when a new sar recorded file in /var/log/sa/sa* was rotated and there were no entries yet.
    Code (Text):
    ls -lAhrt /var/log/sa/sa*
    -rw-r--r-- 1 root root 486K Nov  6 22:53 /var/log/sa/sar06
    -rw-r--r-- 1 root root 666K Nov  6 23:55 /var/log/sa/sa06
    -rw-r--r-- 1 root root 486K Nov  7 22:53 /var/log/sa/sar07
    -rw-r--r-- 1 root root 666K Nov  7 23:55 /var/log/sa/sa07
    -rw-r--r-- 1 root root 486K Nov  8 22:53 /var/log/sa/sar08
    -rw-r--r-- 1 root root 666K Nov  8 23:55 /var/log/sa/sa08
    -rw-r--r-- 1 root root 486K Nov  9 22:53 /var/log/sa/sar09
    -rw-r--r-- 1 root root 666K Nov  9 23:55 /var/log/sa/sa09
    -rw-r--r-- 1 root root 486K Nov 10 22:53 /var/log/sa/sar10
    -rw-r--r-- 1 root root 666K Nov 10 23:55 /var/log/sa/sa10
    -rw-r--r-- 1 root root 486K Nov 11 22:53 /var/log/sa/sar11
    -rw-r--r-- 1 root root 666K Nov 11 23:55 /var/log/sa/sa11
    -rw-r--r-- 1 root root 486K Nov 12 22:53 /var/log/sa/sar12
    -rw-r--r-- 1 root root 666K Nov 12 23:55 /var/log/sa/sa12
    -rw-r--r-- 1 root root 486K Nov 13 22:53 /var/log/sa/sar13
    -rw-r--r-- 1 root root 666K Nov 13 23:55 /var/log/sa/sa13
    -rw-r--r-- 1 root root 486K Nov 14 22:53 /var/log/sa/sar14
    -rw-r--r-- 1 root root 666K Nov 14 23:55 /var/log/sa/sa14
    -rw-r--r-- 1 root root 486K Nov 15 22:53 /var/log/sa/sar15
    -rw-r--r-- 1 root root 666K Nov 15 23:55 /var/log/sa/sa15
    -rw-r--r-- 1 root root 486K Nov 16 22:53 /var/log/sa/sar16
    -rw-r--r-- 1 root root 666K Nov 16 23:55 /var/log/sa/sa16
    -rw-r--r-- 1 root root 486K Nov 17 22:53 /var/log/sa/sar17
    -rw-r--r-- 1 root root 666K Nov 17 23:55 /var/log/sa/sa17
    -rw-r--r-- 1 root root 486K Nov 18 22:53 /var/log/sa/sar18
    -rw-r--r-- 1 root root 666K Nov 18 23:55 /var/log/sa/sa18
    -rw-r--r-- 1 root root 486K Nov 19 22:53 /var/log/sa/sar19
    -rw-r--r-- 1 root root 666K Nov 19 23:55 /var/log/sa/sa19
    -rw-r--r-- 1 root root 486K Nov 20 22:53 /var/log/sa/sar20
    -rw-r--r-- 1 root root 666K Nov 20 23:55 /var/log/sa/sa20
    -rw-r--r-- 1 root root 486K Nov 21 22:53 /var/log/sa/sar21
    -rw-r--r-- 1 root root 666K Nov 21 23:55 /var/log/sa/sa21
    -rw-r--r-- 1 root root 486K Nov 22 22:53 /var/log/sa/sar22
    -rw-r--r-- 1 root root 666K Nov 22 23:55 /var/log/sa/sa22
    -rw-r--r-- 1 root root 486K Nov 23 22:53 /var/log/sa/sar23
    -rw-r--r-- 1 root root 666K Nov 23 23:55 /var/log/sa/sa23
    -rw-r--r-- 1 root root 486K Nov 24 22:53 /var/log/sa/sar24
    -rw-r--r-- 1 root root 666K Nov 24 23:55 /var/log/sa/sa24
    -rw-r--r-- 1 root root 486K Nov 25 22:53 /var/log/sa/sar25
    -rw-r--r-- 1 root root 666K Nov 25 23:55 /var/log/sa/sa25
    -rw-r--r-- 1 root root 486K Nov 26 22:53 /var/log/sa/sar26
    -rw-r--r-- 1 root root 666K Nov 26 23:55 /var/log/sa/sa26
    -rw-r--r-- 1 root root 486K Nov 27 22:53 /var/log/sa/sar27
    -rw-r--r-- 1 root root 666K Nov 27 23:55 /var/log/sa/sa27
    -rw-r--r-- 1 root root 2.5M Nov 28 23:53 /var/log/sa/sar28
    -rw-r--r-- 1 root root 3.9M Nov 28 23:59 /var/log/sa/sa28
    -rw-r--r-- 1 root root 2.5M Nov 29 23:53 /var/log/sa/sar29
    -rw-r--r-- 1 root root 3.9M Nov 29 23:59 /var/log/sa/sa29
    -rw-r--r-- 1 root root 2.5M Nov 30 23:53 /var/log/sa/sar30
    -rw-r--r-- 1 root root 3.9M Nov 30 23:59 /var/log/sa/sa30
    -rw-r--r-- 1 root root 2.6M Dec  1 23:53 /var/log/sa/sar01
    -rw-r--r-- 1 root root 4.1M Dec  1 23:59 /var/log/sa/sa01
    -rw-r--r-- 1 root root 2.5M Dec  2 23:53 /var/log/sa/sar02
    -rw-r--r-- 1 root root 3.9M Dec  2 23:59 /var/log/sa/sa02
    -rw-r--r-- 1 root root 2.5M Dec  3 23:53 /var/log/sa/sar03
    -rw-r--r-- 1 root root 3.9M Dec  3 23:59 /var/log/sa/sa03
    -rw-r--r-- 1 root root 2.5M Dec  4 23:53 /var/log/sa/sar04
    -rw-r--r-- 1 root root 3.9M Dec  4 23:59 /var/log/sa/sa04
    -rw-r--r-- 1 root root 1.5M Dec  5 09:14 /var/log/sa/sa05
    


    You can confirm if you run the full sar -u command (sed masking your server hostname for privacy)

    for past 24hrs
    Code (Text):
    sar -u | sed -e "s|$HOSTNAME|hostname|g"
    


    and check past 24 days for that sar -u command logging all entries to file excluding the Average line
    Code (Text):
    sar -u | awk '/all/ && !/Average/' > sar-u-1days.log
    
    # check how many line entries in log
    wc -l sar-u-1days.log
    

    Then can sample the log's last 10 entries and get max value of 8th column
    Code (Text):
    tail -10 sar-u-1days.log
    08:52:01 AM     all      0.10      0.00      0.08      0.00      0.02     99.80
    08:53:01 AM     all      0.55      0.00      0.47      0.12      0.00     98.86
    08:54:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.75
    08:55:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.75
    08:56:01 AM     all      0.13      0.00      0.13      0.08      0.02     99.63
    08:57:01 AM     all      0.15      0.00      0.15      0.00      0.00     99.70
    08:58:02 AM     all      0.13      0.00      0.10      0.00      0.00     99.77
    08:59:01 AM     all      0.10      0.00      0.10      0.00      0.00     99.80
    09:00:01 AM     all      0.12      0.00      0.10      0.00      0.02     99.77
    09:01:01 AM     all      0.17      0.00      0.17      0.03      0.00     99.63
    

    which shows the max value of 8th column for CPU steal % from last 10 lines is 0.02 which matches with my logged data on EL7 system
    Code (Text):
    tail -10 sar-u-1days.log | awk '/all/ && !/Average/ {if ($8 > max) max = $8} END {print max}'
    0.02
    
     
  15. eva2000

    eva2000 Administrator Staff Member

    51,957
    11,973
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,470
    Local Time:
    7:14 PM
    Nginx 1.25.x
    MariaDB 10.x
    There's a lot of sar data entry points considering sar can entries can be logged every 10 mins (usual defaults on most OSes), while Centmin Mod logs every 5 mins, so over 7 days there's at least 2016 sar entries logged at least. Probably need to wait a full 7 days to get a new average.
     
  16. MaximilianKohler

    MaximilianKohler Member

    156
    2
    18
    Jun 23, 2023
    Ratings:
    +17
    Local Time:
    2:14 AM
    So we use
    Code:
    awk -v nproc=$(nproc) '/Average/ {print $4 / nproc}' <(sar -q)
    And then paste that value into
    and leave
    and
    blank? I don't understand.

    Code:
    sar -u | awk '/all/ && !/Average/ {if ($8 > max) max = $8} END {print max}'
    Still gives me a blank output.
    Code:
    [15:45][root@domain.com ~]# sar -u | awk '/all/ && !/Average/ {if ($8 > max) max = $8} END {print max}'
    
    [15:45][root@domain.com ~]#
    
    Code:
    sar -u | sed -e "s|$HOSTNAME|hostname|g"
    Linux 3.10.0-1160.102.1.el7.x86_64 (hostname)   12/05/2023      _x86_64_        (2 CPU)
    
    12:00:02 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
    12:01:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.75
    12:02:01 AM     all      0.22      0.00      0.13      0.01      0.00     99.65
    12:03:01 AM     all      0.13      0.00      0.11      0.01      0.00     99.76
    12:04:01 AM     all      0.12      0.00      0.09      0.00      0.00     99.79
    12:05:01 AM     all      0.11      0.00      0.09      0.00      0.00     99.80
    12:06:01 AM     all      0.12      0.00      0.10      0.01      0.00     99.77
    12:07:01 AM     all      0.10      0.00      0.10      0.00      0.00     99.80
    12:08:01 AM     all      0.13      0.00      0.08      0.00      0.00     99.79
    12:09:01 AM     all      0.10      0.00      0.09      0.01      0.00     99.80
    12:10:01 AM     all      0.11      0.00      0.10      0.00      0.00     99.79
    12:11:01 AM     all      0.11      0.00      0.12      0.01      0.00     99.77
    12:12:01 AM     all      0.10      0.00      0.10      0.00      0.00     99.80
    12:13:01 AM     all      0.16      0.00      0.13      0.01      0.00     99.70
    12:14:01 AM     all      4.18      0.00      0.83      0.00      0.00     94.99
    12:15:01 AM     all      0.79      0.00      0.13      0.01      0.00     99.08
    12:16:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.75
    12:17:01 AM     all      0.12      0.00      0.10      0.01      0.00     99.77
    12:18:01 AM     all      0.14      0.00      0.15      0.01      0.00     99.70
    12:19:01 AM     all      0.12      0.00      0.08      0.00      0.00     99.80
    12:20:01 AM     all      0.11      0.00      0.12      0.01      0.00     99.77
    12:21:01 AM     all      0.44      0.00      0.18      0.00      0.00     99.38
    12:22:01 AM     all      0.10      0.00      0.10      0.00      0.00     99.80
    12:23:01 AM     all      0.10      0.00      0.10      0.01      0.00     99.79
    12:24:01 AM     all      0.13      0.00      0.11      0.00      0.00     99.77
    12:25:01 AM     all      0.10      0.00      0.12      0.01      0.00     99.77
    12:26:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.76
    12:27:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.74
    12:28:01 AM     all      0.11      0.00      0.10      0.01      0.00     99.78
    12:29:01 AM     all      0.13      0.00      0.10      0.00      0.00     99.77
    12:30:01 AM     all      0.09      0.00      0.10      0.01      0.00     99.80
    12:31:01 AM     all      0.15      0.00      0.16      0.00      0.00     99.69
    12:32:01 AM     all      0.10      0.00      0.09      0.00      0.00     99.81
    12:33:01 AM     all      0.08      0.00      0.12      0.00      0.00     99.80
    12:34:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.73
    12:35:01 AM     all      0.11      0.00      0.13      0.00      0.00     99.76
    12:36:01 AM     all      0.13      0.00      0.11      0.01      0.00     99.76
    12:37:01 AM     all      0.11      0.00      0.08      0.00      0.00     99.81
    12:38:01 AM     all      0.13      0.00      0.11      0.01      0.00     99.76
    12:39:01 AM     all      0.12      0.00      0.10      0.00      0.00     99.78
    12:40:01 AM     all      0.12      0.00      0.11      0.01      0.00     99.77
    12:41:01 AM     all      0.14      0.00      0.13      0.00      0.00     99.72
    12:42:01 AM     all      0.12      0.00      0.12      0.01      0.00     99.76
    12:43:01 AM     all      0.13      0.00      0.11      0.00      0.00     99.76
    12:44:01 AM     all      0.11      0.00      0.10      0.00      0.00     99.79
    12:45:01 AM     all      0.17      0.00      0.14      0.01      0.00     99.68
    12:46:01 AM     all      0.58      0.00      0.16      0.01      0.00     99.26
    12:47:01 AM     all      0.10      0.00      0.09      0.00      0.00     99.81
    12:48:01 AM     all      0.15      0.00      0.12      0.01      0.00     99.72
    12:49:01 AM     all      0.11      0.00      0.11      0.00      0.00     99.78
    12:50:01 AM     all      0.09      0.00      0.09      0.01      0.00     99.81
    12:51:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.74
    12:52:01 AM     all      0.10      0.00      0.11      0.00      0.00     99.79
    12:53:01 AM     all      0.12      0.00      0.12      0.01      0.00     99.76
    12:54:01 AM     all      0.11      0.00      0.11      0.00      0.00     99.78
    12:55:01 AM     all      0.11      0.00      0.12      0.01      0.00     99.77
    12:56:01 AM     all      0.12      0.00      0.10      0.01      0.00     99.77
    12:57:01 AM     all      0.10      0.00      0.10      0.00      0.00     99.80
    12:58:01 AM     all      0.11      0.00      0.08      0.00      0.00     99.81
    12:59:01 AM     all      0.12      0.00      0.11      0.01      0.00     99.77
    01:00:01 AM     all      0.10      0.00      0.11      0.00      0.00     99.79
    01:01:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.73
    01:02:01 AM     all      0.18      0.00      0.11      0.00      0.00     99.71
    01:03:01 AM     all      0.10      0.00      0.10      0.01      0.00     99.79
    01:04:01 AM     all      0.11      0.00      0.09      0.00      0.00     99.80
    01:05:01 AM     all      0.10      0.00      0.09      0.01      0.00     99.80
    01:06:01 AM     all     10.06      0.00      3.11      0.03      0.00     86.81
    01:07:01 AM     all      0.11      0.00      0.10      0.00      0.00     99.79
    01:08:01 AM     all      0.10      0.00      0.12      0.01      0.00     99.77
    01:09:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.75
    01:10:01 AM     all      0.18      0.00      0.18      0.01      0.00     99.63
    01:11:01 AM     all      0.18      0.00      0.20      0.00      0.00     99.62
    01:12:01 AM     all      0.11      0.00      0.10      0.00      0.00     99.79
    01:13:01 AM     all      0.09      0.00      0.10      0.01      0.00     99.80
    01:14:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    01:15:01 AM     all      0.10      0.00      0.11      0.01      0.00     99.78
    01:16:01 AM     all      0.11      0.00      0.11      0.00      0.00     99.78
    01:17:01 AM     all      0.12      0.00      0.09      0.00      0.00     99.79
    01:18:01 AM     all      0.10      0.00      0.11      0.01      0.00     99.78
    01:19:01 AM     all      0.08      0.00      0.08      0.00      0.00     99.83
    01:20:01 AM     all      0.11      0.00      0.13      0.00      0.00     99.77
    01:21:01 AM     all      0.15      0.00      0.13      0.01      0.00     99.71
    01:22:01 AM     all      0.10      0.00      0.09      0.00      0.00     99.81
    01:23:01 AM     all      0.18      0.00      0.17      0.00      0.00     99.66
    01:24:01 AM     all      0.11      0.00      0.10      0.01      0.00     99.78
    01:25:02 AM     all      0.10      0.00      0.12      0.01      0.00     99.77
    01:26:01 AM     all      0.11      0.00      0.13      0.00      0.00     99.76
    01:27:01 AM     all      0.13      0.00      0.09      0.01      0.00     99.77
    01:28:01 AM     all      0.11      0.00      0.10      0.00      0.00     99.79
    01:29:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.74
    01:30:01 AM     all      0.13      0.00      0.11      0.01      0.00     99.76
    01:31:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.76
    01:32:01 AM     all      1.05      0.00      0.30      0.10      0.00     98.55
    01:33:01 AM     all      0.49      0.00      0.18      0.02      0.00     99.31
    01:34:01 AM     all      1.22      0.00      0.42      0.26      0.00     98.10
    01:35:01 AM     all      0.13      0.00      0.11      0.00      0.00     99.76
    01:36:01 AM     all      0.14      0.00      0.16      0.01      0.00     99.69
    01:37:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.76
    01:38:01 AM     all      0.12      0.00      0.13      0.01      0.00     99.74
    01:39:01 AM     all      0.12      0.00      0.13      0.01      0.00     99.74
    01:40:01 AM     all      0.11      0.00      0.11      0.00      0.00     99.78
    01:41:01 AM     all      0.11      0.00      0.14      0.01      0.00     99.74
    01:42:01 AM     all      0.11      0.00      0.10      0.00      0.00     99.79
    01:43:01 AM     all      0.11      0.00      0.13      0.00      0.00     99.77
    01:44:01 AM     all      0.11      0.00      0.09      0.01      0.00     99.79
    01:45:01 AM     all      0.12      0.00      0.09      0.01      0.00     99.78
    01:46:01 AM     all      0.10      0.00      0.13      0.00      0.00     99.77
    01:47:01 AM     all      0.09      0.00      0.09      0.00      0.00     99.82
    01:48:01 AM     all      0.12      0.00      0.12      0.01      0.00     99.76
    01:49:01 AM     all      0.10      0.00      0.11      0.00      0.00     99.79
    01:50:01 AM     all      0.11      0.00      0.12      0.00      0.00     99.77
    01:51:01 AM     all      0.10      0.00      0.12      0.01      0.00     99.77
    01:52:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.76
    01:53:01 AM     all      0.13      0.00      0.10      0.01      0.00     99.77
    01:54:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    01:55:01 AM     all      0.12      0.00      0.13      0.01      0.00     99.74
    01:56:01 AM     all      0.11      0.00      0.12      0.00      0.00     99.77
    01:57:01 AM     all      0.13      0.00      0.11      0.00      0.00     99.77
    01:58:01 AM     all      0.13      0.00      0.10      0.01      0.00     99.76
    01:59:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.75
    02:00:01 AM     all      0.11      0.00      0.12      0.01      0.00     99.77
    02:01:01 AM     all      0.13      0.00      0.14      0.00      0.00     99.72
    02:02:01 AM     all      5.32      0.00      0.90      0.01      0.00     93.78
    02:03:01 AM     all      0.09      0.00      0.11      0.00      0.00     99.80
    02:04:01 AM     all      0.09      0.00      0.09      0.01      0.00     99.81
    02:05:01 AM     all      0.11      0.00      0.11      0.00      0.00     99.78
    02:06:01 AM     all      0.10      0.00      0.08      0.01      0.00     99.81
    02:07:01 AM     all      0.12      0.00      0.12      0.00      0.00     99.77
    02:08:01 AM     all      0.12      0.00      0.08      0.01      0.00     99.79
    02:09:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    02:10:02 AM     all      0.13      0.00      0.11      0.01      0.00     99.76
    02:11:01 AM     all      0.12      0.00      0.14      0.00      0.00     99.75
    02:12:01 AM     all      0.12      0.00      0.13      0.01      0.00     99.75
    02:13:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.75
    02:14:01 AM     all      0.12      0.00      0.12      0.01      0.00     99.76
    02:15:01 AM     all      0.10      0.00      0.10      0.01      0.00     99.79
    02:16:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.75
    02:17:01 AM     all      0.10      0.00      0.09      0.01      0.00     99.80
    02:18:01 AM     all      0.13      0.00      0.10      0.00      0.00     99.77
    02:19:01 AM     all      0.10      0.00      0.10      0.01      0.00     99.79
    02:20:01 AM     all      0.13      0.00      0.12      0.02      0.00     99.73
    02:21:01 AM     all      0.14      0.00      0.13      0.00      0.00     99.73
    02:22:01 AM     all      0.11      0.00      0.12      0.01      0.00     99.77
    02:23:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.75
    02:24:01 AM     all      0.10      0.00      0.11      0.00      0.00     99.79
    02:25:01 AM     all      0.49      0.00      0.15      0.02      0.00     99.34
    02:26:01 AM     all      0.12      0.00      0.13      0.01      0.00     99.75
    02:27:01 AM     all      0.13      0.00      0.09      0.00      0.00     99.78
    02:28:01 AM     all      0.12      0.00      0.11      0.01      0.00     99.77
    02:29:01 AM     all      0.13      0.00      0.09      0.00      0.00     99.78
    02:30:01 AM     all      0.11      0.00      0.11      0.01      0.00     99.77
    02:31:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.75
    02:32:01 AM     all      0.18      0.00      0.13      0.01      0.00     99.67
    02:33:01 AM     all      0.21      0.00      0.18      0.01      0.00     99.61
    02:34:01 AM     all      0.18      0.00      0.17      0.01      0.00     99.65
    02:35:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.75
    02:36:01 AM     all      0.12      0.00      0.14      0.00      0.00     99.74
    02:37:01 AM     all      0.13      0.00      0.15      0.01      0.00     99.72
    02:38:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.73
    02:39:01 AM     all      0.11      0.00      0.13      0.00      0.00     99.77
    02:40:01 AM     all      0.16      0.00      0.13      0.02      0.00     99.70
    02:41:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.73
    02:42:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    02:43:01 AM     all      0.13      0.00      0.09      0.01      0.00     99.77
    02:44:01 AM     all      0.10      0.00      0.13      0.00      0.00     99.77
    02:45:01 AM     all      0.14      0.00      0.15      0.01      0.00     99.70
    02:46:01 AM     all      0.11      0.00      0.14      0.01      0.00     99.74
    02:47:01 AM     all      0.14      0.00      0.11      0.00      0.00     99.75
    02:48:01 AM     all      0.15      0.00      0.13      0.01      0.00     99.71
    02:49:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.76
    02:50:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.74
    02:51:01 AM     all      0.13      0.00      0.16      0.00      0.00     99.72
    02:52:01 AM     all      0.16      0.00      0.13      0.00      0.00     99.71
    02:53:01 AM     all      0.12      0.00      0.11      0.01      0.00     99.77
    02:54:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.73
    02:55:01 AM     all      0.10      0.00      0.12      0.01      0.00     99.77
    02:56:01 AM     all      0.11      0.00      0.11      0.00      0.00     99.78
    02:57:01 AM     all      0.11      0.00      0.11      0.01      0.00     99.77
    02:58:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.75
    02:59:01 AM     all      0.14      0.00      0.14      0.00      0.00     99.72
    03:00:01 AM     all      0.13      0.00      0.11      0.01      0.00     99.75
    03:01:01 AM     all      0.14      0.00      0.13      0.01      0.00     99.72
    03:02:01 AM     all      0.19      0.00      0.13      0.01      0.00     99.67
    03:03:01 AM     all      0.17      0.00      0.13      0.00      0.00     99.70
    03:04:01 AM     all      0.14      0.00      0.13      0.01      0.00     99.72
    03:05:01 AM     all      0.17      0.00      0.15      0.00      0.00     99.68
    03:06:01 AM     all      0.54      0.00      0.17      0.01      0.00     99.28
    03:07:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.75
    03:08:01 AM     all      1.37      0.00      0.24      0.02      0.00     98.37
    03:09:02 AM     all      0.11      0.00      0.11      0.01      0.00     99.77
    03:10:01 AM     all      0.15      0.00      0.12      0.01      0.00     99.72
    03:11:01 AM     all      1.07      0.00      0.92      0.01      0.00     98.00
    03:12:01 AM     all      5.13      0.00      0.38      0.01      0.00     94.48
    03:13:01 AM     all      0.46      0.00      0.14      0.00      0.00     99.40
    03:14:01 AM     all      0.48      0.08      0.16      0.02      0.00     99.26
    03:15:01 AM     all      0.13      0.00      0.11      0.01      0.00     99.76
    03:16:01 AM     all      0.13      0.00      0.10      0.00      0.00     99.77
    03:17:01 AM     all      0.14      0.00      0.14      0.00      0.00     99.72
    03:18:01 AM     all      0.14      0.00      0.13      0.01      0.00     99.72
    03:19:01 AM     all      0.16      0.00      0.14      0.00      0.00     99.70
    03:20:01 AM     all      0.12      0.00      0.12      0.01      0.00     99.76
    03:21:01 AM     all      0.16      0.00      0.14      0.00      0.00     99.70
    03:22:01 AM     all      0.18      0.00      0.16      0.01      0.00     99.65
    03:23:01 AM     all      0.23      0.00      0.22      0.01      0.00     99.54
    03:24:01 AM     all      0.19      0.00      0.18      0.00      0.00     99.63
    03:25:01 AM     all      0.20      0.00      0.17      0.01      0.00     99.62
    03:26:01 AM     all      1.33      0.00      0.24      0.01      0.00     98.42
    03:27:01 AM     all      0.13      0.00      0.14      0.00      0.00     99.72
    03:28:01 AM     all      0.89      0.00      0.19      0.01      0.00     98.91
    03:29:01 AM     all      0.48      0.00      0.14      0.01      0.00     99.37
    03:30:01 AM     all      0.14      0.00      0.14      0.01      0.00     99.71
    03:31:01 AM     all      0.58      0.00      0.17      0.00      0.00     99.26
    03:32:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.76
    03:33:01 AM     all      0.12      0.00      0.09      0.01      0.00     99.78
    03:34:01 AM     all      0.49      0.00      0.16      0.00      0.00     99.35
    03:35:01 AM     all      0.35      0.00      0.12      0.01      0.00     99.52
    03:36:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.76
    03:37:02 AM     all      0.12      0.00      0.11      0.01      0.00     99.77
    03:38:01 AM     all      0.09      0.00      0.10      0.00      0.00     99.80
    03:39:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    03:40:01 AM     all      0.10      0.00      0.09      0.01      0.00     99.80
    03:41:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.73
    03:42:01 AM     all      0.11      0.00      0.10      0.00      0.00     99.79
    03:43:01 AM     all      0.13      4.99      1.05      0.30      0.00     93.53
    03:44:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.76
    03:45:01 AM     all      0.14      0.00      0.11      0.01      0.00     99.74
    03:46:01 AM     all      5.44      0.00      2.77      0.02      0.00     91.77
    03:47:01 AM     all      0.10      0.00      0.12      0.01      0.00     99.77
    03:48:01 AM     all      0.11      0.00      0.12      0.00      0.00     99.77
    03:49:01 AM     all      0.12      0.00      0.11      0.01      0.00     99.77
    03:50:01 AM     all      0.11      0.00      0.12      0.00      0.00     99.77
    03:51:01 AM     all      0.50      0.00      0.18      0.01      0.00     99.31
    03:52:01 AM     all      5.24      0.00      2.86      0.02      0.00     91.88
    03:53:01 AM     all      0.50      0.00      0.14      0.01      0.00     99.35
    03:54:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    03:55:01 AM     all      0.12      0.00      0.13      0.01      0.00     99.75
    03:56:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    03:57:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    03:58:01 AM     all      0.10      0.00      0.09      0.01      0.00     99.80
    03:59:01 AM     all      0.13      0.00      0.11      0.00      0.00     99.77
    04:00:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.75
    04:01:01 AM     all      0.18      0.00      0.15      0.00      0.00     99.67
    04:02:01 AM     all      0.17      0.00      0.13      0.00      0.00     99.71
    04:03:01 AM     all      0.13      0.00      0.11      0.01      0.00     99.76
    04:04:01 AM     all      0.11      0.00      0.10      0.00      0.00     99.79
    04:05:01 AM     all      0.14      0.00      0.14      0.01      0.00     99.71
    04:06:01 AM     all      0.11      0.00      0.12      0.00      0.00     99.77
    04:07:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    04:08:01 AM     all      0.11      0.00      0.12      0.01      0.00     99.77
    04:09:01 AM     all      0.11      0.00      0.12      0.00      0.00     99.77
    04:10:01 AM     all      0.17      0.00      0.12      0.01      0.00     99.71
    04:11:01 AM     all      0.09      0.00      0.10      0.00      0.00     99.81
    04:12:01 AM     all      0.15      0.00      0.15      0.01      0.00     99.69
    04:13:01 AM     all      0.09      0.00      0.11      0.00      0.00     99.80
    04:14:01 AM     all      0.10      0.00      0.13      0.00      0.00     99.77
    04:15:01 AM     all      5.18      0.00      0.92      0.03      0.00     93.88
    04:16:01 AM     all      0.13      0.00      0.13      0.03      0.00     99.72
    04:17:01 AM     all      0.10      0.00      0.10      0.00      0.00     99.80
    04:18:01 AM     all      0.10      0.00      0.09      0.01      0.00     99.80
    04:19:01 AM     all      0.09      0.00      0.12      0.00      0.00     99.79
    04:20:01 AM     all      0.19      0.00      0.13      0.01      0.00     99.67
    04:21:01 AM     all      0.17      0.00      0.17      0.02      0.00     99.65
    04:22:01 AM     all      0.17      0.00      0.10      0.00      0.00     99.73
    04:23:01 AM     all      0.26      0.00      0.35      0.01      0.00     99.38
    04:24:01 AM     all      0.13      0.00      0.14      0.01      0.00     99.72
    04:25:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.73
    04:26:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.75
    04:27:01 AM     all      0.11      0.00      0.08      0.00      0.00     99.81
    04:28:01 AM     all      0.12      0.00      0.13      0.01      0.00     99.75
    04:29:01 AM     all      0.11      0.00      0.08      0.00      0.00     99.81
    04:30:01 AM     all      0.14      0.00      0.12      0.01      0.00     99.74
    04:31:01 AM     all      0.14      0.00      0.13      0.00      0.00     99.73
    04:32:01 AM     all      0.12      0.00      0.10      0.00      0.00     99.78
    04:33:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.73
    04:34:01 AM     all      0.14      0.00      0.13      0.00      0.00     99.73
    04:35:01 AM     all      0.10      0.00      0.13      0.01      0.00     99.77
    04:36:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    04:37:01 AM     all      0.12      0.00      0.11      0.01      0.00     99.77
    04:38:01 AM     all      5.20      0.00      2.65      0.03      0.00     92.12
    04:39:01 AM     all      0.10      0.00      0.10      0.00      0.00     99.80
    04:40:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.72
    04:41:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.74
    04:42:01 AM     all      0.12      0.00      0.08      0.00      0.00     99.80
    04:43:01 AM     all      0.11      0.00      0.11      0.01      0.00     99.77
    04:44:01 AM     all      0.12      0.00      0.08      0.00      0.00     99.80
    04:45:01 AM     all      0.13      0.00      0.10      0.01      0.00     99.77
    04:46:01 AM     all      0.13      0.00      0.10      0.00      0.00     99.77
    04:47:01 AM     all      0.13      0.00      0.11      0.01      0.00     99.75
    04:48:01 AM     all      0.12      0.00      0.10      0.00      0.00     99.78
    04:49:01 AM     all      0.10      0.00      0.10      0.00      0.00     99.80
    04:50:01 AM     all      0.11      0.00      0.09      0.01      0.00     99.79
    04:51:01 AM     all      0.11      0.00      0.11      0.00      0.00     99.78
    04:52:01 AM     all      0.13      0.00      0.11      0.01      0.00     99.76
    04:53:01 AM     all      0.11      0.04      0.13      0.00      0.00     99.72
    04:54:01 AM     all      0.12      0.00      0.13      0.01      0.00     99.74
    04:55:01 AM     all      0.11      0.00      0.12      0.00      0.00     99.77
    04:56:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.73
    04:57:01 AM     all      0.10      0.00      0.12      0.00      0.00     99.78
    04:58:01 AM     all      0.11      0.00      0.12      0.01      0.00     99.77
    04:59:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.76
    05:00:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.74
    05:01:01 AM     all      0.13      0.00      0.16      0.00      0.00     99.72
    05:02:01 AM     all      0.18      0.00      0.11      0.02      0.00     99.69
    05:03:01 AM     all      0.95      0.00      0.18      0.01      0.00     98.86
    05:04:01 AM     all      4.86      0.00      0.85      0.00      0.00     94.29
    05:05:01 AM     all      0.14      0.00      0.13      0.02      0.00     99.72
    05:06:01 AM     all      0.12      0.00      0.12      0.00      0.00     99.77
    05:07:01 AM     all      0.11      0.00      0.11      0.01      0.00     99.77
    05:08:01 AM     all      0.10      0.00      0.12      0.01      0.00     99.77
    05:09:01 AM     all      0.16      0.00      0.13      0.00      0.00     99.72
    05:10:01 AM     all      0.12      0.00      0.14      0.01      0.00     99.73
    05:11:01 AM     all      0.12      0.00      0.11      0.01      0.00     99.77
    05:12:01 AM     all      0.10      0.00      0.10      0.00      0.00     99.80
    05:13:01 AM     all      0.10      0.00      0.11      0.01      0.00     99.78
    05:14:01 AM     all      0.12      0.00      0.10      0.00      0.00     99.78
    05:15:01 AM     all      0.11      0.00      0.12      0.01      0.00     99.77
    05:16:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.76
    05:17:01 AM     all      0.12      0.00      0.12      0.01      0.00     99.76
    05:18:01 AM     all      0.11      0.00      0.10      0.00      0.00     99.79
    05:19:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.74
    05:20:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.74
    05:21:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.76
    05:22:01 AM     all      0.13      0.00      0.11      0.01      0.00     99.75
    05:23:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.76
    05:24:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.76
    05:25:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.74
    05:26:02 AM     all      0.12      0.00      0.12      0.00      0.00     99.77
    05:27:01 AM     all      0.09      0.00      0.10      0.01      0.00     99.80
    05:28:01 AM     all      0.10      0.00      0.11      0.00      0.00     99.79
    05:29:01 AM     all      0.16      0.00      0.15      0.01      0.00     99.68
    05:30:01 AM     all      0.12      0.00      0.10      0.01      0.00     99.77
    05:31:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    05:32:01 AM     all      0.10      0.00      0.10      0.01      0.00     99.79
    05:33:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.74
    05:34:01 AM     all      0.31      0.00      0.12      0.00      0.00     99.57
    05:35:01 AM     all      0.10      0.00      0.11      0.01      0.00     99.78
    05:36:01 AM     all      0.16      0.00      0.14      0.00      0.00     99.70
    05:37:01 AM     all      0.12      0.00      0.13      0.01      0.00     99.74
    05:38:01 AM     all      0.11      0.00      0.13      0.00      0.00     99.77
    05:39:01 AM     all      0.13      0.00      0.15      0.01      0.00     99.72
    05:40:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    05:41:01 AM     all      0.13      0.00      0.14      0.01      0.00     99.72
    05:42:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.76
    05:43:01 AM     all      0.11      0.00      0.11      0.01      0.00     99.77
    05:44:01 AM     all      0.10      0.00      0.10      0.00      0.00     99.80
    05:45:01 AM     all      0.11      0.00      0.11      0.01      0.00     99.77
    05:46:01 AM     all      0.10      0.00      0.13      0.00      0.00     99.77
    05:47:01 AM     all      0.10      0.00      0.12      0.00      0.00     99.78
    05:48:01 AM     all      0.13      0.00      0.10      0.01      0.00     99.77
    05:49:01 AM     all      0.12      0.00      0.14      0.00      0.00     99.74
    05:50:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.73
    05:51:01 AM     all      0.15      0.00      0.13      0.00      0.00     99.72
    05:52:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.75
    05:53:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.75
    05:54:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.73
    05:55:01 AM     all      0.26      0.00      0.17      0.02      0.00     99.56
    05:56:01 AM     all      0.37      0.00      0.19      0.01      0.00     99.43
    05:57:01 AM     all      0.15      0.00      0.12      0.01      0.00     99.72
    05:58:01 AM     all      0.21      0.00      0.15      0.00      0.00     99.64
    05:59:01 AM     all      0.15      0.00      0.14      0.01      0.00     99.70
    06:00:01 AM     all      0.14      0.00      0.13      0.01      0.00     99.72
    06:01:01 AM     all      0.16      0.00      0.16      0.01      0.00     99.67
    06:02:01 AM     all      4.86      0.00      0.89      0.01      0.00     94.24
    06:03:01 AM     all      5.28      0.00      2.55      0.02      0.00     92.15
    06:04:01 AM     all      0.52      0.00      0.16      0.03      0.00     99.30
    06:05:01 AM     all      0.18      0.00      0.12      0.00      0.00     99.70
    06:06:01 AM     all      0.18      0.00      0.15      0.01      0.00     99.67
    06:07:01 AM     all      0.15      0.00      0.13      0.00      0.00     99.72
    06:08:01 AM     all      0.13      0.00      0.09      0.01      0.00     99.77
    06:09:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.76
    06:10:01 AM     all      0.24      0.00      0.17      0.01      0.00     99.58
    06:11:01 AM     all      0.15      0.00      0.14      0.00      0.00     99.71
    06:12:01 AM     all      0.55      0.00      0.14      0.02      0.00     99.29
    06:13:01 AM     all      0.21      0.00      0.12      0.01      0.00     99.67
    06:14:01 AM     all      0.14      0.00      0.13      0.01      0.00     99.72
    06:15:01 AM     all      0.14      0.00      0.15      0.00      0.00     99.71
    06:16:01 AM     all      0.14      0.00      0.16      0.01      0.00     99.69
    06:17:01 AM     all      0.12      0.00      0.10      0.00      0.00     99.78
    06:18:01 AM     all      0.13      0.00      0.10      0.01      0.00     99.77
    06:19:01 AM     all      0.11      0.00      0.09      0.00      0.00     99.80
    06:20:01 AM     all      0.13      0.00      0.14      0.01      0.00     99.72
    06:21:01 AM     all      0.13      0.00      0.14      0.00      0.00     99.72
    06:22:01 AM     all      0.15      0.00      0.11      0.00      0.00     99.74
    06:23:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.74
    06:24:01 AM     all      0.14      0.00      0.14      0.00      0.00     99.72
    06:25:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.74
    06:26:01 AM     all      0.15      0.00      0.16      0.00      0.00     99.69
    06:27:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.72
    06:28:01 AM     all      0.14      0.00      0.12      0.00      0.00     99.74
    06:29:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.75
    06:30:01 AM     all      0.11      0.00      0.11      0.01      0.00     99.77
    06:31:01 AM     all      0.13      0.00      0.14      0.01      0.00     99.72
    06:32:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.75
    06:33:01 AM     all      0.49      0.00      0.21      0.01      0.00     99.29
    06:34:01 AM     all      0.15      0.00      0.19      0.01      0.00     99.65
    06:35:01 AM     all      0.13      0.00      0.14      0.00      0.00     99.72
    06:36:01 AM     all      0.14      0.00      0.15      0.01      0.00     99.70
    06:37:01 AM     all      0.15      0.00      0.13      0.00      0.00     99.72
    06:38:01 AM     all      0.14      0.00      0.15      0.01      0.00     99.70
    06:39:01 AM     all      0.14      0.00      0.14      0.00      0.00     99.72
    06:40:01 AM     all      0.56      0.00      0.18      0.01      0.00     99.25
    06:41:01 AM     all      0.14      0.00      0.13      0.00      0.00     99.73
    06:42:01 AM     all      0.52      0.00      0.20      0.01      0.00     99.27
    06:43:01 AM     all      0.14      0.00      0.14      0.00      0.00     99.72
    06:44:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.73
    06:45:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.75
    06:46:01 AM     all      0.11      0.00      0.12      0.00      0.00     99.77
    06:47:01 AM     all      0.16      0.00      0.13      0.01      0.00     99.70
    06:48:01 AM     all      0.12      0.00      0.10      0.00      0.00     99.78
    06:49:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.76
    06:50:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.73
    06:51:01 AM     all      0.16      0.00      0.18      0.01      0.00     99.66
    06:52:01 AM     all      0.15      0.00      0.11      0.00      0.00     99.74
    06:53:02 AM     all      0.13      0.00      0.13      0.01      0.00     99.72
    06:54:01 AM     all      0.10      0.00      0.09      0.00      0.00     99.80
    06:55:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.73
    06:56:01 AM     all      0.12      0.00      0.14      0.00      0.00     99.74
    06:57:01 AM     all      0.18      0.00      0.18      0.01      0.00     99.63
    06:58:01 AM     all      0.12      0.00      0.13      0.01      0.00     99.74
    06:59:01 AM     all      0.18      0.00      0.17      0.01      0.00     99.65
    07:00:01 AM     all      0.15      0.00      0.19      0.00      0.00     99.66
    07:01:01 AM     all      0.18      0.00      0.19      0.01      0.00     99.62
    07:02:01 AM     all      0.28      0.00      0.21      0.01      0.00     99.51
    07:03:01 AM     all      0.23      0.00      0.18      0.01      0.00     99.59
    07:04:01 AM     all      0.16      0.00      0.18      0.01      0.00     99.66
    07:05:01 AM     all      0.21      0.00      0.17      0.01      0.00     99.62
    07:06:01 AM     all      0.20      0.00      0.18      0.00      0.00     99.62
    07:07:01 AM     all      5.67      0.00      2.64      0.05      0.00     91.64
    07:08:01 AM     all      0.15      0.00      0.15      0.01      0.00     99.69
    07:09:01 AM     all      0.18      0.00      0.18      0.01      0.00     99.62
    07:10:01 AM     all      0.26      0.00      0.18      0.01      0.00     99.56
    07:11:01 AM     all      0.17      0.00      0.19      0.00      0.00     99.64
    07:12:01 AM     all      0.16      0.00      0.18      0.01      0.00     99.65
    07:13:01 AM     all      0.27      0.00      0.20      0.01      0.00     99.52
    07:14:01 AM     all      4.92      0.00      0.93      0.01      0.00     94.14
    07:15:01 AM     all      0.16      0.00      0.19      0.01      0.00     99.64
    07:16:01 AM     all      0.16      0.00      0.18      0.01      0.00     99.65
    07:17:01 AM     all      0.16      0.00      0.17      0.00      0.00     99.67
    07:18:01 AM     all      0.18      0.00      0.19      0.01      0.00     99.62
    07:19:01 AM     all      0.14      0.00      0.13      0.00      0.00     99.73
    07:20:01 AM     all      0.17      0.00      0.20      0.01      0.00     99.62
    07:21:01 AM     all      0.18      0.00      0.20      0.01      0.00     99.61
    07:22:01 AM     all      0.53      0.00      0.22      0.00      0.00     99.25
    07:23:01 AM     all      0.66      0.00      0.25      0.01      0.00     99.08
    07:24:01 AM     all      0.16      0.00      0.17      0.00      0.00     99.67
    07:25:01 AM     all      0.18      0.00      0.18      0.01      0.00     99.62
    07:26:01 AM     all      0.15      0.00      0.18      0.00      0.00     99.67
    07:27:01 AM     all      0.17      0.00      0.17      0.00      0.00     99.67
    07:28:01 AM     all      0.15      0.00      0.17      0.01      0.00     99.67
    07:29:01 AM     all      0.13      0.00      0.15      0.01      0.00     99.72
    07:30:01 AM     all      0.13      0.00      0.17      0.01      0.00     99.69
    07:31:01 AM     all      0.17      0.00      0.20      0.00      0.00     99.63
    07:32:01 AM     all      0.15      0.00      0.13      0.01      0.00     99.71
    07:33:01 AM     all      0.16      0.00      0.15      0.00      0.00     99.69
    07:34:01 AM     all      0.16      0.00      0.17      0.01      0.00     99.67
    07:35:01 AM     all      0.13      0.00      0.18      0.00      0.00     99.69
    07:36:01 AM     all      0.16      0.00      0.20      0.01      0.00     99.63
    07:37:01 AM     all      0.18      0.00      0.16      0.01      0.00     99.66
    07:38:01 AM     all      0.17      0.00      0.18      0.00      0.00     99.65
    07:39:01 AM     all      0.15      0.00      0.16      0.00      0.00     99.69
    07:40:01 AM     all      0.43      0.00      0.29      0.09      0.00     99.18
    07:41:01 AM     all      1.23      0.00      0.82      0.07      0.00     97.88
    07:42:01 AM     all      0.99      0.00      0.62      0.08      0.00     98.32
    07:43:01 AM     all      0.24      0.00      0.20      0.02      0.00     99.54
    07:44:01 AM     all      0.21      0.00      0.18      0.02      0.00     99.59
    07:45:01 AM     all      0.19      0.00      0.20      0.02      0.00     99.59
    07:46:01 AM     all      0.49      0.00      0.22      0.03      0.00     99.26
    07:47:01 AM     all      0.53      0.00      0.24      0.02      0.00     99.22
    07:48:01 AM     all      0.23      0.00      0.20      0.02      0.00     99.56
    07:49:01 AM     all      0.22      0.00      0.22      0.01      0.00     99.56
    07:50:01 AM     all      0.23      0.00      0.20      0.02      0.00     99.55
    07:51:01 AM     all      0.23      0.00      0.24      0.02      0.00     99.51
    07:52:01 AM     all      0.23      0.00      0.19      0.01      0.00     99.57
    07:53:02 AM     all      0.18      0.00      0.19      0.01      0.00     99.62
    07:54:01 AM     all      6.06      0.00      2.79      0.02      0.00     91.14
    07:55:01 AM     all      0.15      0.00      0.22      0.02      0.00     99.62
    07:56:01 AM     all      0.18      0.00      0.16      0.00      0.00     99.67
    07:57:01 AM     all      0.21      0.00      0.18      0.01      0.00     99.60
    07:58:01 AM     all      0.14      0.00      0.15      0.01      0.00     99.70
    07:59:01 AM     all      0.18      0.00      0.18      0.01      0.00     99.63
    08:00:01 AM     all      0.18      0.00      0.18      0.01      0.00     99.63
    08:01:01 AM     all      0.21      0.00      0.22      0.02      0.00     99.56
    08:02:01 AM     all      0.48      0.00      0.23      0.03      0.00     99.27
    08:03:01 AM     all      0.78      0.00      0.27      0.02      0.00     98.94
    08:04:01 AM     all      4.63      0.00      0.88      0.01      0.00     94.48
    08:05:01 AM     all      0.16      0.00      0.18      0.01      0.00     99.65
    08:06:01 AM     all      0.20      0.00      0.22      0.03      0.00     99.56
    08:07:01 AM     all      0.18      0.00      0.18      0.01      0.00     99.62
    08:08:01 AM     all      0.25      0.00      0.21      0.01      0.00     99.53
    08:09:01 AM     all      0.20      0.00      0.18      0.01      0.00     99.61
    08:10:01 AM     all      0.18      0.00      0.20      0.01      0.00     99.62
    08:11:01 AM     all      0.15      0.00      0.13      0.01      0.00     99.72
    08:12:01 AM     all      0.17      0.00      0.18      0.01      0.00     99.64
    08:13:01 AM     all      0.19      0.00      0.18      0.01      0.00     99.62
    08:14:01 AM     all      0.20      0.00      0.18      0.00      0.00     99.62
    08:15:01 AM     all      0.14      0.00      0.18      0.01      0.00     99.67
    08:16:01 AM     all      0.20      0.00      0.23      0.01      0.00     99.57
    08:17:01 AM     all      0.15      0.00      0.16      0.00      0.00     99.69
    08:18:01 AM     all      0.17      0.00      0.18      0.01      0.00     99.64
    08:19:01 AM     all      0.15      0.00      0.16      0.01      0.00     99.68
    08:20:01 AM     all      0.17      0.00      0.18      0.01      0.00     99.65
    08:21:01 AM     all      0.19      0.00      0.20      0.01      0.00     99.60
    08:22:01 AM     all      0.18      0.00      0.20      0.01      0.00     99.61
    08:23:01 AM     all      0.18      0.00      0.23      0.00      0.00     99.60
    08:24:01 AM     all      0.16      0.00      0.18      0.01      0.00     99.65
    08:25:01 AM     all      0.18      0.00      0.16      0.01      0.00     99.66
    08:26:01 AM     all      0.17      0.00      0.19      0.01      0.00     99.63
    08:27:01 AM     all      0.15      0.00      0.14      0.01      0.00     99.70
    08:28:01 AM     all      0.16      0.00      0.18      0.00      0.00     99.67
    08:29:01 AM     all      0.22      0.00      0.18      0.01      0.00     99.60
    08:30:01 AM     all      0.16      0.00      0.17      0.01      0.00     99.67
    08:31:01 AM     all      0.21      0.00      0.23      0.01      0.00     99.56
    08:32:01 AM     all      0.18      0.00      0.21      0.00      0.00     99.62
    08:33:01 AM     all      0.16      0.00      0.15      0.01      0.00     99.68
    08:34:01 AM     all      0.19      0.00      0.21      0.01      0.00     99.59
    08:35:01 AM     all      0.19      0.00      0.18      0.01      0.00     99.62
    08:36:01 AM     all      0.16      0.00      0.15      0.01      0.00     99.68
    08:37:01 AM     all      0.17      0.00      0.18      0.00      0.00     99.65
    08:38:01 AM     all      0.19      0.00      0.18      0.03      0.00     99.60
    08:39:01 AM     all      0.17      0.00      0.18      0.01      0.00     99.64
    08:40:01 AM     all      0.29      0.00      0.22      0.02      0.00     99.47
    08:41:01 AM     all      0.19      0.00      0.22      0.01      0.00     99.58
    08:42:01 AM     all      0.18      0.00      0.16      0.01      0.00     99.65
    08:43:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.73
    08:44:01 AM     all      0.16      0.00      0.15      0.01      0.00     99.68
    08:45:01 AM     all      0.15      0.00      0.16      0.01      0.00     99.68
    08:46:01 AM     all      0.20      0.00      0.19      0.00      0.00     99.61
    08:47:01 AM     all      0.15      0.00      0.14      0.01      0.00     99.70
    08:48:01 AM     all      0.16      0.00      0.17      0.01      0.00     99.67
    08:49:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.76
    08:50:01 AM     all      0.19      0.00      0.21      0.02      0.00     99.58
    08:51:01 AM     all      0.18      0.00      0.26      0.01      0.00     99.56
    08:52:01 AM     all      0.14      0.00      0.14      0.01      0.00     99.71
    08:53:01 AM     all      0.14      0.00      0.13      0.01      0.00     99.72
    08:54:02 AM     all      0.28      0.00      0.15      0.00      0.00     99.57
    08:55:01 AM     all      0.47      0.00      0.22      0.02      0.00     99.30
    08:56:01 AM     all      0.51      0.00      0.19      0.01      0.00     99.29
    08:57:01 AM     all      0.62      0.00      0.20      0.00      0.00     99.18
    08:58:01 AM     all      5.53      0.00      2.64      0.04      0.00     91.79
    08:59:01 AM     all      0.17      0.00      0.15      0.01      0.00     99.67
    09:00:01 AM     all      0.14      0.00      0.14      0.01      0.00     99.71
    09:01:01 AM     all      0.18      0.00      0.16      0.01      0.00     99.66
    09:02:01 AM     all      0.25      0.00      0.20      0.01      0.00     99.54
    09:03:01 AM     all      0.14      0.00      0.15      0.00      0.00     99.71
    09:04:01 AM     all      0.16      0.00      0.16      0.01      0.00     99.67
    09:05:01 AM     all      0.22      0.00      0.18      0.01      0.00     99.60
    09:06:01 AM     all      0.20      0.00      0.18      0.00      0.00     99.62
    09:07:01 AM     all      0.15      0.00      0.17      0.01      0.00     99.67
    09:08:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.75
    09:09:01 AM     all      0.10      0.00      0.13      0.01      0.00     99.77
    09:10:01 AM     all      5.09      0.00      0.99      0.01      0.00     93.92
    09:11:01 AM     all      0.13      0.00      0.15      0.00      0.00     99.72
    09:12:01 AM     all      0.15      0.00      0.14      0.01      0.00     99.70
    09:13:01 AM     all      0.15      0.00      0.13      0.01      0.00     99.72
    09:14:01 AM     all      0.50      0.00      0.19      0.01      0.00     99.30
    09:15:01 AM     all      0.17      0.00      0.13      0.01      0.00     99.69
    09:16:01 AM     all      0.74      0.00      0.18      0.00      0.00     99.07
    09:17:01 AM     all      0.15      0.00      0.16      0.01      0.00     99.68
    09:18:01 AM     all      0.15      0.00      0.13      0.00      0.00     99.72
    09:19:01 AM     all      0.14      0.00      0.15      0.01      0.00     99.70
    09:20:01 AM     all      0.19      0.00      0.20      0.01      0.00     99.60
    09:21:01 AM     all      0.51      0.00      0.19      0.00      0.00     99.30
    09:22:01 AM     all      0.14      0.00      0.15      0.01      0.00     99.70
    09:23:01 AM     all      0.15      0.00      0.15      0.00      0.00     99.70
    09:24:01 AM     all      0.15      0.00      0.14      0.01      0.00     99.70
    09:25:02 AM     all      0.20      0.00      0.13      0.01      0.00     99.66
    09:26:01 AM     all      0.16      0.00      0.18      0.01      0.00     99.65
    09:27:01 AM     all      0.18      0.00      0.15      0.01      0.00     99.66
    09:28:01 AM     all      0.13      0.00      0.18      0.01      0.00     99.68
    09:29:01 AM     all      0.16      0.00      0.12      0.01      0.00     99.72
    09:30:01 AM     all      0.18      0.00      0.17      0.01      0.00     99.65
    09:31:01 AM     all      0.18      0.00      0.20      0.02      0.00     99.60
    09:32:01 AM     all      0.14      0.00      0.16      0.01      0.00     99.69
    09:33:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.74
    09:34:01 AM     all      0.14      0.00      0.14      0.00      0.00     99.72
    09:35:01 AM     all      0.20      0.00      0.18      0.01      0.00     99.61
    09:36:01 AM     all      0.14      0.00      0.17      0.01      0.00     99.68
    09:37:01 AM     all      0.14      0.00      0.15      0.01      0.00     99.70
    09:38:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.75
    09:39:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.73
    09:40:01 AM     all      0.16      0.00      0.18      0.01      0.00     99.66
    09:41:01 AM     all      0.14      0.00      0.16      0.00      0.00     99.70
    09:42:01 AM     all      0.13      0.00      0.16      0.01      0.00     99.70
    09:43:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.73
    09:44:01 AM     all      0.12      0.00      0.10      0.01      0.00     99.77
    09:45:01 AM     all      0.15      0.00      0.14      0.00      0.00     99.71
    09:46:01 AM     all      0.15      0.00      0.17      0.01      0.00     99.67
    09:47:01 AM     all      0.14      0.00      0.16      0.00      0.00     99.70
    09:48:01 AM     all      0.17      0.00      0.16      0.01      0.00     99.67
    09:49:01 AM     all      0.14      0.00      0.12      0.01      0.00     99.73
    09:50:01 AM     all      0.17      0.00      0.16      0.00      0.00     99.67
    09:51:01 AM     all      0.16      0.00      0.17      0.01      0.00     99.67
    09:52:01 AM     all      0.11      0.00      0.11      0.00      0.00     99.78
    09:53:01 AM     all      0.13      0.00      0.15      0.01      0.00     99.71
    09:54:01 AM     all      0.15      0.00      0.15      0.00      0.00     99.70
    09:55:01 AM     all      0.13      0.00      0.18      0.01      0.00     99.68
    09:56:01 AM     all      0.14      0.00      0.14      0.00      0.00     99.72
    09:57:01 AM     all      0.15      0.00      0.14      0.01      0.00     99.70
    09:58:01 AM     all      0.14      0.00      0.14      0.00      0.00     99.72
    09:59:01 AM     all      0.13      0.00      0.16      0.01      0.00     99.70
    10:00:01 AM     all      0.16      0.00      0.13      0.01      0.00     99.70
    10:01:01 AM     all      0.15      0.00      0.22      0.00      0.00     99.63
    10:02:01 AM     all      0.19      0.00      0.18      0.01      0.00     99.62
    10:03:01 AM     all      5.01      0.00      0.90      0.01      0.00     94.08
    10:04:01 AM     all      0.14      0.00      0.15      0.00      0.00     99.71
    10:05:01 AM     all      0.18      0.00      0.16      0.01      0.00     99.66
    10:06:01 AM     all      0.14      0.00      0.13      0.01      0.00     99.72
    10:07:01 AM     all      5.76      0.00      2.72      0.03      0.00     91.49
    10:08:01 AM     all      0.20      0.00      0.15      0.01      0.00     99.64
    10:09:01 AM     all      0.14      0.00      0.14      0.00      0.00     99.72
    10:10:01 AM     all      0.16      0.00      0.14      0.01      0.00     99.69
    10:11:01 AM     all      0.16      0.00      0.16      0.01      0.00     99.67
    10:12:01 AM     all      0.15      0.00      0.13      0.01      0.00     99.72
    10:13:01 AM     all      0.15      0.00      0.17      0.00      0.00     99.68
    10:14:01 AM     all      0.16      0.00      0.14      0.01      0.00     99.69
    10:15:01 AM     all      0.16      0.00      0.14      0.00      0.00     99.70
    10:16:01 AM     all      0.16      0.00      0.15      0.01      0.00     99.68
    10:17:01 AM     all      0.14      0.00      0.18      0.01      0.00     99.67
    10:18:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.74
    10:19:01 AM     all      0.80      0.00      0.16      0.02      0.00     99.02
    10:20:01 AM     all      0.13      0.00      0.10      0.01      0.00     99.77
    10:21:01 AM     all      0.13      0.00      0.09      0.01      0.00     99.77
    10:22:01 AM     all      0.11      0.00      0.12      0.00      0.00     99.77
    10:23:01 AM     all      0.11      0.00      0.11      0.00      0.00     99.78
    10:24:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.74
    10:25:01 AM     all      0.14      0.00      0.14      0.00      0.00     99.72
    10:26:01 AM     all      0.14      0.00      0.14      0.01      0.00     99.71
    10:27:01 AM     all      0.38      0.00      0.48      0.01      0.00     99.13
    10:28:01 AM     all      5.65      0.00      2.71      0.03      0.00     91.62
    10:29:01 AM     all      0.12      0.00      0.11      0.01      0.00     99.77
    10:30:01 AM     all      0.18      0.00      0.12      0.02      0.00     99.69
    10:31:01 AM     all      0.14      0.00      0.13      0.01      0.00     99.72
    10:32:01 AM     all      0.11      0.00      0.09      0.00      0.00     99.80
    10:33:01 AM     all      0.15      0.00      0.13      0.01      0.00     99.72
    10:34:02 AM     all      0.13      0.00      0.13      0.00      0.00     99.74
    10:35:01 AM     all      0.13      0.00      0.14      0.01      0.00     99.72
    10:36:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.74
    10:37:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    10:38:01 AM     all      0.10      0.00      0.09      0.00      0.00     99.81
    10:39:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.75
    10:40:01 AM     all      0.12      0.00      0.09      0.01      0.00     99.78
    10:41:01 AM     all      0.15      0.00      0.13      0.01      0.00     99.71
    10:42:01 AM     all      0.14      0.00      0.14      0.01      0.00     99.71
    10:43:01 AM     all      0.15      0.00      0.13      0.03      0.00     99.70
    10:44:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.74
    10:45:01 AM     all      0.12      0.00      0.10      0.01      0.00     99.77
    10:46:01 AM     all      0.12      0.00      0.14      0.01      0.00     99.73
    10:47:01 AM     all      0.12      0.00      0.12      0.00      0.00     99.77
    10:48:01 AM     all      0.11      0.00      0.10      0.00      0.00     99.79
    10:49:01 AM     all      0.15      0.00      0.12      0.01      0.00     99.72
    10:50:01 AM     all      0.12      0.00      0.11      0.02      0.00     99.76
    10:51:01 AM     all      0.14      0.00      0.13      0.00      0.00     99.72
    10:52:01 AM     all      0.13      0.00      0.11      0.01      0.00     99.76
    10:53:01 AM     all      0.12      0.00      0.12      0.00      0.00     99.77
    10:54:01 AM     all      0.11      0.00      0.10      0.00      0.00     99.79
    10:55:01 AM     all      1.07      0.00      0.18      0.02      0.00     98.73
    10:56:01 AM     all      0.21      0.00      0.19      0.01      0.00     99.59
    10:57:01 AM     all      0.12      0.00      0.10      0.00      0.00     99.78
    10:58:01 AM     all      0.24      0.00      0.12      0.01      0.00     99.63
    10:59:01 AM     all      0.11      0.00      0.10      0.01      0.00     99.78
    11:00:01 AM     all      0.15      0.00      0.11      0.01      0.00     99.73
    11:01:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.76
    11:02:01 AM     all      0.18      0.00      0.12      0.01      0.00     99.69
    11:03:01 AM     all      0.13      0.00      0.09      0.00      0.00     99.77
    11:04:01 AM     all      0.13      0.00      0.09      0.01      0.00     99.77
    11:05:01 AM     all      0.13      0.00      0.11      0.02      0.00     99.74
    11:06:01 AM     all      0.10      0.00      0.10      0.00      0.00     99.80
    11:07:01 AM     all      0.15      0.00      0.12      0.01      0.00     99.72
    11:08:01 AM     all      0.10      0.00      0.09      0.01      0.00     99.80
    11:09:01 AM     all      0.81      0.00      0.16      0.01      0.00     99.02
    11:10:01 AM     all      0.10      0.00      0.10      0.01      0.00     99.79
    11:11:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.74
    11:12:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.74
    11:13:01 AM     all      0.15      0.00      0.12      0.00      0.00     99.73
    11:14:01 AM     all      5.40      0.00      0.90      0.01      0.00     93.69
    11:15:01 AM     all      0.14      0.00      0.11      0.01      0.00     99.74
    11:16:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.73
    11:17:01 AM     all      0.11      0.00      0.10      0.01      0.00     99.78
    11:18:01 AM     all      0.12      0.00      0.12      0.00      0.00     99.77
    11:19:01 AM     all      0.13      0.00      0.11      0.01      0.00     99.75
    11:20:01 AM     all      0.12      0.00      0.12      0.00      0.00     99.77
    11:21:01 AM     all      0.13      0.00      0.15      0.01      0.00     99.72
    11:22:01 AM     all      0.14      0.00      0.13      0.01      0.00     99.72
    11:23:01 AM     all      0.13      0.00      0.08      0.00      0.00     99.79
    11:24:01 AM     all      0.15      0.00      0.11      0.01      0.00     99.73
    11:25:01 AM     all      0.12      0.00      0.13      0.01      0.00     99.74
    11:26:01 AM     all      0.13      0.00      0.12      0.00      0.00     99.76
    11:27:01 AM     all      0.11      0.00      0.12      0.01      0.00     99.77
    11:28:01 AM     all      0.13      0.00      0.12      0.01      0.00     99.74
    11:29:01 AM     all      0.11      0.00      0.12      0.01      0.00     99.77
    11:30:01 AM     all      0.13      0.00      0.10      0.01      0.00     99.77
    11:31:01 AM     all      0.14      0.00      0.15      0.00      0.00     99.71
    11:32:01 AM     all      0.14      0.00      0.12      0.01      0.00     99.73
    11:33:01 AM     all      0.13      0.00      0.10      0.00      0.00     99.77
    11:34:01 AM     all      0.11      0.00      0.09      0.01      0.00     99.79
    11:35:01 AM     all      0.11      0.00      0.11      0.00      0.00     99.78
    11:36:01 AM     all      0.18      0.00      0.14      0.02      0.00     99.67
    11:37:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    11:38:01 AM     all      0.11      0.00      0.09      0.00      0.00     99.80
    11:39:01 AM     all      0.11      0.00      0.11      0.01      0.00     99.77
    11:40:01 AM     all      0.13      0.00      0.13      0.01      0.00     99.73
    11:41:01 AM     all      0.12      0.00      0.11      0.00      0.00     99.77
    11:42:01 AM     all      0.11      0.00      0.12      0.01      0.00     99.77
    11:43:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.75
    11:44:01 AM     all      0.15      0.00      0.14      0.01      0.00     99.70
    11:45:01 AM     all      0.13      0.00      0.09      0.01      0.00     99.77
    11:46:01 AM     all      0.12      0.00      0.13      0.00      0.00     99.76
    11:47:01 AM     all      0.16      0.00      0.13      0.01      0.00     99.70
    11:48:01 AM     all      0.09      0.00      0.09      0.00      0.00     99.82
    11:49:01 AM     all      0.12      0.00      0.10      0.01      0.00     99.77
    11:50:01 AM     all      0.12      0.00      0.10      0.01      0.00     99.77
    11:51:01 AM     all      0.13      0.00      0.13      0.00      0.00     99.74
    11:52:01 AM     all      0.13      0.00      0.10      0.00      0.00     99.77
    11:53:01 AM     all      0.11      0.00      0.08      0.01      0.00     99.80
    11:54:01 AM     all      0.10      0.00      0.10      0.00      0.00     99.80
    11:55:01 AM     all      0.13      0.00      0.11      0.01      0.00     99.75
    11:56:01 AM     all      0.17      0.00      0.15      0.01      0.00     99.67
    11:57:01 AM     all      0.16      0.00      0.14      0.01      0.00     99.69
    11:58:01 AM     all      0.49      0.00      0.14      0.00      0.00     99.37
    11:59:01 AM     all      0.11      0.00      0.13      0.00      0.00     99.77
    12:00:01 PM     all      0.14      0.00      0.13      0.01      0.00     99.72
    12:01:01 PM     all      0.13      0.00      0.13      0.01      0.00     99.73
    12:02:01 PM     all      0.29      0.00      0.14      0.02      0.00     99.55
    12:03:01 PM     all      0.12      0.00      0.11      0.00      0.00     99.77
    12:04:01 PM     all      0.14      0.00      0.13      0.01      0.00     99.72
    12:05:01 PM     all      5.05      0.00      2.52      0.03      0.00     92.40
    12:06:01 PM     all      0.17      0.00      0.13      0.01      0.00     99.69
    12:07:01 PM     all      0.11      0.00      0.12      0.01      0.00     99.77
    12:08:01 PM     all      0.11      0.00      0.09      0.00      0.00     99.80
    12:09:01 PM     all      0.14      0.00      0.10      0.00      0.00     99.76
    12:10:01 PM     all      0.34      0.00      0.18      0.02      0.00     99.46
    12:11:01 PM     all      4.92      0.00      0.86      0.01      0.00     94.21
    12:12:01 PM     all      0.11      0.00      0.13      0.00      0.00     99.77
    12:13:01 PM     all      0.18      0.00      0.17      0.01      0.00     99.65
    12:14:01 PM     all      0.14      0.00      0.14      0.01      0.00     99.71
    12:15:01 PM     all      0.12      0.00      0.13      0.00      0.00     99.76
    12:16:01 PM     all      0.14      0.00      0.13      0.01      0.00     99.72
    12:17:01 PM     all      0.48      0.00      0.13      0.02      0.00     99.37
    12:18:01 PM     all      0.13      0.00      0.11      0.00      0.00     99.77
    12:19:01 PM     all      0.14      0.00      0.13      0.01      0.00     99.72
    12:20:01 PM     all      5.23      0.00      2.54      0.03      0.00     92.21
    12:21:01 PM     all      0.15      0.00      0.13      0.00      0.00     99.72
    12:22:01 PM     all      0.13      0.00      0.13      0.01      0.00     99.74
    12:23:01 PM     all      0.13      0.00      0.09      0.01      0.00     99.77
    12:24:01 PM     all      0.13      0.00      0.13      0.00      0.00     99.74
    12:25:01 PM     all      0.12      0.00      0.09      0.01      0.00     99.78
    12:26:01 PM     all      0.15      0.00      0.13      0.01      0.00     99.72
    12:27:01 PM     all      0.16      0.00      0.13      0.01      0.00     99.70
    12:28:01 PM     all      0.13      0.00      0.13      0.01      0.00     99.74
    12:29:01 PM     all      0.16      0.00      0.11      0.01      0.00     99.72
    12:30:01 PM     all      0.12      0.00      0.13      0.01      0.00     99.74
    12:31:01 PM     all      0.15      0.00      0.13      0.01      0.00     99.71
    12:32:01 PM     all      0.11      0.00      0.10      0.00      0.00     99.79
    12:33:01 PM     all      0.13      0.00      0.13      0.00      0.00     99.75
    12:34:01 PM     all      0.13      0.00      0.13      0.01      0.00     99.72
    12:35:01 PM     all      0.11      0.00      0.12      0.01      0.00     99.77
    12:36:01 PM     all      0.12      0.00      0.11      0.00      0.00     99.77
    12:37:01 PM     all      0.11      0.00      0.10      0.01      0.00     99.78
    12:38:01 PM     all      0.13      0.00      0.11      0.00      0.00     99.77
    12:39:01 PM     all      0.13      0.00      0.12      0.01      0.00     99.74
    12:40:01 PM     all      0.11      0.00      0.11      0.00      0.00     99.78
    12:41:01 PM     all      0.19      0.00      0.19      0.02      0.00     99.60
    12:42:01 PM     all      0.14      0.00      0.14      0.01      0.00     99.71
    12:43:01 PM     all      0.12      0.00      0.10      0.00      0.00     99.78
    12:44:01 PM     all      0.11      0.00      0.11      0.01      0.00     99.77
    12:45:01 PM     all      0.13      0.00      0.15      0.01      0.00     99.72
    12:46:01 PM     all      0.13      0.00      0.13      0.00      0.00     99.73
    12:47:01 PM     all      0.13      0.00      0.13      0.01      0.00     99.72
    12:48:01 PM     all      0.14      0.00      0.10      0.01      0.00     99.75
    12:49:01 PM     all      0.13      0.00      0.12      0.00      0.00     99.76
    12:50:01 PM     all      0.13      0.00      0.09      0.01      0.00     99.77
    12:51:01 PM     all      0.12      0.00      0.12      0.00      0.00     99.77
    12:52:01 PM     all      0.11      0.00      0.12      0.01      0.00     99.77
    12:53:01 PM     all      0.13      0.00      0.13      0.00      0.00     99.74
    12:54:01 PM     all      0.12      0.00      0.13      0.01      0.00     99.74
    12:55:01 PM     all      0.13      0.00      0.12      0.01      0.00     99.75
    12:56:01 PM     all      0.14      0.00      0.12      0.00      0.00     99.74
    12:57:01 PM     all      0.13      0.00      0.11      0.00      0.00     99.77
    12:58:01 PM     all      0.13      0.00      0.11      0.01      0.00     99.76
    12:59:01 PM     all      0.12      0.00      0.11      0.01      0.00     99.77
    01:00:02 PM     all      0.13      0.00      0.13      0.00      0.00     99.73
    01:01:01 PM     all      0.18      0.00      0.15      0.01      0.00     99.66
    01:02:01 PM     all      0.19      0.00      0.13      0.00      0.00     99.67
    01:03:01 PM     all      0.53      0.00      0.17      0.02      0.00     99.29
    01:04:01 PM     all      0.13      0.00      0.10      0.00      0.00     99.77
    01:05:01 PM     all      0.13      0.00      0.12      0.01      0.00     99.74
    01:06:01 PM     all      0.14      0.00      0.15      0.01      0.00     99.70
    01:07:01 PM     all      0.12      0.00      0.11      0.00      0.00     99.77
    01:08:01 PM     all      0.16      0.00      0.11      0.00      0.00     99.73
    01:09:01 PM     all      0.55      0.00      0.20      0.02      0.00     99.23
    01:10:01 PM     all      4.82      0.00      0.90      0.02      0.00     94.27
    01:11:01 PM     all      0.14      0.00      0.14      0.02      0.00     99.70
    01:12:01 PM     all      0.89      0.00      0.22      0.00      0.00     98.89
    01:13:01 PM     all      0.13      0.00      0.10      0.01      0.00     99.77
    01:14:01 PM     all      0.10      0.00      0.09      0.00      0.00     99.81
    01:15:01 PM     all      0.13      0.00      0.12      0.01      0.00     99.75
    01:16:02 PM     all      0.12      0.00      0.11      0.00      0.00     99.77
    01:17:01 PM     all      0.25      0.00      0.16      0.01      0.00     99.58
    01:18:01 PM     all      0.12      0.00      0.11      0.01      0.00     99.77
    01:19:01 PM     all      0.13      0.00      0.12      0.01      0.00     99.75
    01:20:01 PM     all      0.15      0.00      0.13      0.01      0.00     99.72
    01:21:01 PM     all      0.13      0.00      0.15      0.00      0.00     99.72
    01:22:01 PM     all      0.13      0.00      0.12      0.01      0.00     99.74
    01:23:01 PM     all      0.22      0.00      0.16      0.01      0.00     99.62
    01:24:01 PM     all      0.14      0.00      0.13      0.01      0.00     99.72
    01:25:01 PM     all      0.13      0.00      0.13      0.01      0.00     99.72
    01:26:01 PM     all      0.18      0.00      0.17      0.01      0.00     99.65
    01:27:01 PM     all      0.56      0.00      0.18      0.00      0.00     99.26
    01:28:01 PM     all      0.70      0.00      0.18      0.02      0.00     99.10
    01:29:01 PM     all      0.53      0.00      0.17      0.01      0.00     99.30
    01:30:01 PM     all      0.49      0.00      0.14      0.01      0.00     99.36
    01:31:01 PM     all      0.13      0.00      0.12      0.00      0.00     99.75
    01:32:01 PM     all      5.24      0.00      2.48      0.01      0.00     92.27
    01:33:01 PM     all      0.77      0.00      0.24      0.02      0.00     98.97
    01:34:01 PM     all      0.53      0.00      0.18      0.00      0.00     99.30
    01:35:01 PM     all      0.15      0.00      0.13      0.02      0.00     99.70
    01:36:01 PM     all      0.15      0.00      0.14      0.00      0.00     99.71
    01:37:01 PM     all      0.13      0.00      0.11      0.01      0.00     99.75
    01:38:01 PM     all      0.11      0.00      0.09      0.00      0.00     99.80
    01:39:01 PM     all      0.17      0.00      0.13      0.01      0.00     99.69
    01:40:01 PM     all      0.11      0.00      0.12      0.00      0.00     99.77
    01:41:01 PM     all      0.12      0.00      0.13      0.01      0.00     99.75
    01:42:01 PM     all      0.24      0.00      0.13      0.02      0.00     99.62
    01:43:01 PM     all      0.38      0.00      0.13      0.00      0.00     99.50
    01:44:01 PM     all      0.18      0.00      0.14      0.01      0.00     99.67
    01:45:01 PM     all      0.15      0.00      0.13      0.01      0.00     99.71
    01:46:01 PM     all      0.11      0.00      0.12      0.01      0.00     99.77
    01:47:01 PM     all      0.10      0.00      0.10      0.00      0.00     99.80
    01:48:01 PM     all      0.11      0.00      0.12      0.01      0.00     99.77
    01:49:01 PM     all      0.13      0.00      0.13      0.01      0.00     99.73
    01:50:01 PM     all      0.15      0.00      0.13      0.01      0.00     99.72
    01:51:01 PM     all      0.14      0.00      0.11      0.00      0.00     99.75
    01:52:01 PM     all      0.61      0.00      0.22      0.07      0.00     99.11
    01:53:01 PM     all      0.15      0.00      0.14      0.00      0.00     99.71
    01:54:01 PM     all      0.13      0.00      0.13      0.00      0.00     99.74
    01:55:01 PM     all      0.13      0.00      0.13      0.01      0.00     99.72
    01:56:01 PM     all      0.18      0.00      0.14      0.01      0.00     99.67
    01:57:01 PM     all      0.12      0.00      0.10      0.01      0.00     99.77
    01:58:01 PM     all      0.16      0.00      0.12      0.00      0.00     99.72
    01:59:01 PM     all      0.14      0.00      0.12      0.01      0.00     99.73
    02:00:01 PM     all      0.59      0.00      0.17      0.01      0.00     99.23
    02:01:01 PM     all      0.13      0.00      0.13      0.00      0.00     99.74
    02:02:01 PM     all      0.63      0.00      0.18      0.01      0.00     99.18
    02:03:01 PM     all      0.11      0.00      0.09      0.00      0.00     99.80
    02:04:01 PM     all      0.11      0.00      0.11      0.01      0.00     99.77
    02:05:01 PM     all      0.13      0.00      0.13      0.01      0.00     99.72
    02:06:01 PM     all      0.13      0.00      0.14      0.00      0.00     99.72
    02:07:01 PM     all      0.13      0.00      0.12      0.01      0.00     99.75
    02:08:01 PM     all      0.13      0.00      0.10      0.00      0.00     99.77
    02:09:01 PM     all      3.01      0.00      0.85      0.01      0.00     96.13
    02:10:01 PM     all      2.16      0.00      0.19      0.01      0.00     97.64
    02:11:01 PM     all      0.12      0.00      0.14      0.00      0.00     99.74
    02:12:01 PM     all      0.46      0.00      0.14      0.02      0.00     99.38
    02:13:01 PM     all      0.15      0.00      0.14      0.00      0.00     99.71
    02:14:01 PM     all      0.16      0.00      0.13      0.01      0.00     99.71
    02:15:01 PM     all      0.11      0.00      0.10      0.01      0.00     99.78
    02:16:01 PM     all      0.13      0.00      0.12      0.00      0.00     99.76
    02:17:01 PM     all      0.12      0.00      0.09      0.00      0.00     99.79
    02:18:01 PM     all      0.13      0.00      0.15      0.01      0.00     99.72
    02:19:01 PM     all      0.56      0.00      0.19      0.01      0.00     99.24
    02:20:01 PM     all      0.14      0.00      0.12      0.01      0.00     99.73
    02:21:01 PM     all      0.16      0.00      0.13      0.01      0.00     99.71
    02:22:01 PM     all      0.11      0.00      0.11      0.00      0.00     99.78
    02:23:01 PM     all      0.13      0.00      0.13      0.00      0.00     99.73
    02:24:01 PM     all      0.15      0.00      0.12      0.01      0.00     99.72
    02:25:01 PM     all      0.34      0.00      0.14      0.01      0.00     99.51
    02:26:01 PM     all      0.15      0.00      0.13      0.01      0.00     99.71
    02:27:01 PM     all      0.13      0.00      0.12      0.00      0.00     99.75
    02:28:01 PM     all      0.12      0.00      0.09      0.01      0.00     99.78
    02:29:01 PM     all      0.13      0.00      0.10      0.00      0.00     99.77
    02:30:01 PM     all      0.12      0.00      0.10      0.01      0.00     99.77
    02:31:01 PM     all      0.74      0.00      0.17      0.01      0.00     99.08
    02:32:01 PM     all      0.13      0.00      0.08      0.00      0.00     99.78
    02:33:01 PM     all      0.12      0.00      0.12      0.01      0.00     99.76
    02:34:01 PM     all      0.15      0.00      0.14      0.01      0.00     99.70
    02:35:01 PM     all      0.26      0.00      0.20      0.05      0.00     99.49
    02:36:01 PM     all      0.10      0.00      0.11      0.01      0.00     99.78
    02:37:01 PM     all      0.16      0.00      0.14      0.00      0.00     99.70
    02:38:01 PM     all      0.14      0.00      0.11      0.00      0.00     99.75
    02:39:01 PM     all      0.14      0.00      0.12      0.00      0.00     99.74
    02:40:01 PM     all      0.13      0.00      0.14      0.01      0.00     99.72
    02:41:01 PM     all      0.16      0.00      0.17      0.01      0.00     99.67
    02:42:02 PM     all      0.88      0.00      0.47      0.00      0.00     98.66
    02:43:01 PM     all      4.78      0.00      2.77      0.03      0.00     92.42
    02:44:01 PM     all      1.23      0.00      0.23      0.00      0.00     98.54
    02:45:01 PM     all      0.13      0.00      0.12      0.01      0.00     99.74
    02:46:01 PM     all      0.17      0.00      0.14      0.01      0.00     99.68
    02:47:01 PM     all      0.53      0.00      0.15      0.00      0.00     99.32
    02:48:01 PM     all      0.12      0.00      0.10      0.01      0.00     99.77
    02:49:01 PM     all      0.14      0.00      0.14      0.01      0.00     99.71
    02:50:01 PM     all      0.14      0.00      0.13      0.01      0.00     99.72
    02:51:01 PM     all      0.15      0.00      0.14      0.00      0.00     99.71
    02:52:01 PM     all      0.12      0.00      0.13      0.01      0.00     99.75
    02:53:01 PM     all      0.12      0.00      0.13      0.00      0.00     99.75
    02:54:01 PM     all      0.13      0.00      0.12      0.01      0.00     99.74
    02:55:01 PM     all      0.13      0.00      0.10      0.00      0.00     99.77
    02:56:01 PM     all      0.16      0.00      0.12      0.01      0.00     99.72
    02:57:01 PM     all      0.11      0.00      0.10      0.01      0.00     99.78
    02:58:01 PM     all      0.13      0.00      0.11      0.00      0.00     99.77
    02:59:01 PM     all      0.12      0.00      0.12      0.01      0.00     99.76
    03:00:01 PM     all      0.11      0.00      0.12      0.01      0.00     99.77
    03:01:01 PM     all      0.11      0.00      0.13      0.00      0.00     99.76
    03:02:01 PM     all      0.20      0.00      0.12      0.02      0.00     99.67
    03:03:01 PM     all      0.18      0.00      0.10      0.01      0.00     99.72
    03:04:01 PM     all      0.73      0.00      0.16      0.01      0.00     99.10
    03:05:01 PM     all      0.13      0.00      0.13      0.01      0.00     99.73
    03:06:01 PM     all      0.13      0.00      0.11      0.01      0.00     99.75
    03:07:01 PM     all      0.13      0.00      0.11      0.00      0.00     99.76
    03:08:01 PM     all      0.11      0.00      0.11      0.01      0.00     99.77
    03:09:01 PM     all      0.14      0.00      0.14      0.00      0.00     99.72
    03:10:01 PM     all      0.14      0.00      0.12      0.01      0.00     99.73
    03:11:01 PM     all      0.28      0.00      0.21      0.01      0.00     99.50
    03:12:01 PM     all      0.12      0.00      0.08      0.00      0.00     99.80
    03:13:01 PM     all      5.03      0.00      0.88      0.01      0.00     94.07
    03:14:01 PM     all      0.17      0.02      0.30      0.15      0.00     99.36
    03:15:01 PM     all      0.13      0.00      0.13      0.02      0.00     99.72
    03:16:01 PM     all      0.16      0.00      0.13      0.01      0.00     99.70
    03:17:01 PM     all      0.16      0.00      0.11      0.00      0.00     99.73
    03:18:01 PM     all      0.13      0.00      0.12      0.01      0.00     99.75
    03:19:02 PM     all      0.14      0.00      0.13      0.00      0.00     99.73
    03:20:01 PM     all      0.15      0.00      0.14      0.01      0.00     99.70
    03:21:01 PM     all      0.12      0.00      0.13      0.01      0.00     99.74
    03:22:01 PM     all      0.11      0.00      0.12      0.00      0.00     99.77
    03:23:01 PM     all      0.12      0.00      0.13      0.01      0.00     99.74
    03:24:01 PM     all      0.13      0.00      0.12      0.00      0.00     99.76
    03:25:01 PM     all      0.12      0.00      0.12      0.01      0.00     99.76
    03:26:01 PM     all      1.03      0.00      0.19      0.01      0.00     98.77
    03:27:01 PM     all      0.46      0.00      0.18      0.01      0.00     99.35
    03:28:01 PM     all      5.41      0.00      3.06      0.03      0.00     91.50
    03:29:01 PM     all      0.33      0.00      0.18      0.01      0.00     99.48
    03:30:01 PM     all      0.18      0.00      0.15      0.02      0.00     99.66
    03:31:01 PM     all      0.13      0.00      0.13      0.00      0.00     99.75
    03:32:01 PM     all      0.12      0.00      0.11      0.01      0.00     99.77
    03:33:01 PM     all      0.12      0.00      0.12      0.00      0.00     99.77
    03:34:01 PM     all      0.85      0.00      0.15      0.01      0.00     98.99
    03:35:01 PM     all      0.13      0.00      0.14      0.01      0.00     99.72
    03:36:01 PM     all      0.12      0.00      0.12      0.01      0.00     99.76
    03:37:01 PM     all      0.11      0.00      0.13      0.00      0.00     99.76
    03:38:01 PM     all      0.13      0.00      0.13      0.00      0.00     99.74
    03:39:01 PM     all      0.13      0.00      0.13      0.01      0.00     99.74
    03:40:01 PM     all      0.14      0.00      0.13      0.01      0.00     99.72
    03:41:01 PM     all      0.11      0.00      0.12      0.00      0.00     99.77
    03:42:01 PM     all      0.56      0.00      0.14      0.01      0.00     99.29
    03:43:01 PM     all      0.33      0.00      0.14      0.01      0.00     99.52
    03:44:01 PM     all      0.11      0.00      0.12      0.00      0.00     99.77
    03:45:01 PM     all      0.56      0.00      0.19      0.01      0.00     99.24
    03:46:01 PM     all      0.32      0.00      0.33      0.04      0.00     99.31
    03:47:01 PM     all      0.13      0.00      0.11      0.01      0.00     99.76
    03:48:01 PM     all      0.10      0.00      0.09      0.00      0.00     99.81
    Average:        all      0.34      0.01      0.19      0.01      0.00     99.45
    
    This command doesn't output anything:
    Code:
    sar -u | awk '/all/ && !/Average/' > sar-u-1days.log
    Code:
    wc -l sar-u-1days.log
    949 sar-u-1days.log
    
    Code:
    tail -10 sar-u-1days.log
    03:40:01 PM     all      0.14      0.00      0.13      0.01      0.00     99.72
    03:41:01 PM     all      0.11      0.00      0.12      0.00      0.00     99.77
    03:42:01 PM     all      0.56      0.00      0.14      0.01      0.00     99.29
    03:43:01 PM     all      0.33      0.00      0.14      0.01      0.00     99.52
    03:44:01 PM     all      0.11      0.00      0.12      0.00      0.00     99.77
    03:45:01 PM     all      0.56      0.00      0.19      0.01      0.00     99.24
    03:46:01 PM     all      0.32      0.00      0.33      0.04      0.00     99.31
    03:47:01 PM     all      0.13      0.00      0.11      0.01      0.00     99.76
    03:48:01 PM     all      0.10      0.00      0.09      0.00      0.00     99.81
    03:49:01 PM     all      0.11      0.00      0.12      0.00      0.00     99.77
    
    This gives a blank output:
    Code:
    tail -10 sar-u-1days.log | awk '/all/ && !/Average/ {if ($8 > max) max = $8} END {print max}'
    
    [15:52][root@mydomain.com ~]#
    
     
  17. eva2000

    eva2000 Administrator Staff Member

    51,957
    11,973
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,470
    Local Time:
    7:14 PM
    Nginx 1.25.x
    MariaDB 10.x
    there's 4 input boxes, 1 for average and next 3 inputs are for 90, 95, 99% percentiles

    average CPU load average 1min

    upload_2023-12-6_10-38-47.png

    percentile CPU load average 1min

    upload_2023-12-6_10-39-34.png

    Ah that might be because you CPU steal column is all 0.0 values
     
  18. MaximilianKohler

    MaximilianKohler Member

    156
    2
    18
    Jun 23, 2023
    Ratings:
    +17
    Local Time:
    2:14 AM
    Ah, I see now that the
    Code:
    awk -v nproc=$(nproc) '/Average/ {print $4 / nproc}' <(sar -q)
    command is listed twice. Maybe it shouldn't be listed in your second screenshot, and thus the only valid commands for those 3 values are over the past 7+ days?

    Perhaps a note could be added to enter "0" if the output is blank?

    I'll check again in the future but I used the "24 hr" commands (after running listmonk earlier in the day) and the result/recommendations were much lower than the first round that I did using the 7-day commands.
     
    Last edited: Dec 6, 2023
  19. eva2000

    eva2000 Administrator Staff Member

    51,957
    11,973
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,470
    Local Time:
    7:14 PM
    Nginx 1.25.x
    MariaDB 10.x
    ah my mistake!

    yes, i'm in middle of updating the commands, but this revised one should work print max+0 where max becomes empty will result in print = 0
    Code (Text):
    sar -u | awk '/all/ && !/Average/ {if ($8 > max) max = $8} END {print max+0}'
    
     
  20. MaximilianKohler

    MaximilianKohler Member

    156
    2
    18
    Jun 23, 2023
    Ratings:
    +17
    Local Time:
    2:14 AM
    My server seems to be running smoothly. My forum is fast and my listmonk has been faster with centmin + binary than it was with AWS EC2 + docker. However, I've started getting severe lag when opening up the listmonk UI to where it basically doesn't load at all and I have to try again later. I got a delay when opening the server via Putty, so it seems to be a server issue rather than a listmonk/db issue. I guess it doesn't impact the forum because it's mostly cached, whereas listmonk is directly accessing the db.

    So I ran the server spec calculator again and the recommendation it gave me was lower than the first time. With a noticeable difference being that Peak CPU Usage was 17.7% last time and this time it was 100%. And average and Peak CPU Steal are both 0% now, whereas before it was 32.65%. I guess that means the lag might be due to Hetzner not "stealing" CPU usage from other cloud servers when my CPU usage maxes out?

    Code:
    Based on your input, here are our recommendations - this is a very general recommendation based on last 24hrs or 7 days of metrics:
    
        Recommended CPU: 1 - 1 Threads
        Recommended Memory: 2 - 3 GB
        Recommended Disk: 11 - 88 GB
        Recommended Bandwidth Egress: 1 TB per month - 1 TB per month
    
    Your Input
    
        Number of CPU Threads: 2
        Normalized CPU Load Average (1 minute): 0.005 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
        Normalized CPU Load Average (1 minute) 90% percentile: 0.02 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
        Normalized CPU Load Average (1 minute) 95% percentile: 0.035 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
        Normalized CPU Load Average (1 minute) 99% percentile: 0.08 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
        Normalized CPU Load Average (5 minutes): 0.01 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
        Normalized CPU Load Average (15 minutes): 0.025 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
        Normalized CPU Load Average Peak (1 minute): 0.16 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
        Normalized CPU Load Average Peak (5 minutes): 0.06 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
        Normalized CPU Load Average Peak (15 minutes): 0.035 (reported CPU Load Average/Number Of CPU Cores should ideally be under 1.0)
        Average CPU Utilization: 0.82%
        Peak CPU Utilization: 100%
        Peak CPU Utilization 90% percentile: 1.05%
        Peak CPU Utilization 95% percentile: 1.91%
        Peak CPU Utilization 99% percentile: 9.47%
        Average CPU Steal: 0% (average % of CPU time stolen from a virtualized VPS system to service another virtualized VPS system on same server. Ideally <10% CPU steal overwise the VPS host node is overcrowded with busy VPS neighbours.)
        Peak CPU Steal: 0% (max % of CPU time stolen from a virtualized VPS system to service another virtualized VPS system on same server. Ideally <10% CPU steal overwise the VPS host node is overcrowded with busy VPS neighbours.)
        Peak CPU Steal 90% percentile: 0.3%
        Peak CPU Steal 95% percentile: 0.37%
        Peak CPU Steal 99% percentile: 3.34%
        Average Memory Usage: 1.34131 GB
        Peak Memory Usage: 1.70168 GB
        Number of Local Backup Copies: 7
        Current Disk Usage: 11 GB
        Average Bandwidth Egress: 0.0337968 Mbps
        Peak Bandwidth Egress: 2.90552 Mbps
    I checked CPU steal for the past 7 days instead of 24 hours and it was non-zero:
    Code:
    Average CPU steal (per core) over the last 7 days: 0.283359%
    Peak CPU steal (per core) over the last 7 days: 25.3600%