Welcome to Centmin Mod Community
Register Now

Beta Branch update nginxlargefile_fix function more efficient find command

Discussion in 'Centmin Mod Github Commits' started by eva2000, Oct 10, 2017.

  1. eva2000

    eva2000 Administrator Staff Member

    30,634
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    1:47 PM
    Nginx 1.13.x
    MariaDB 5.5
  2. pamamolf

    pamamolf Well-Known Member

    2,778
    245
    63
    May 31, 2014
    Ratings:
    +437
    Local Time:
    5:47 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    I just update centminmod and run it again to test it and got an error:

     
    • Informative Informative x 1
  3. eva2000

    eva2000 Administrator Staff Member

    30,634
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    1:47 PM
    Nginx 1.13.x
    MariaDB 5.5
    hmm.. what output do you get from running this command
    Code (Text):
    curl -sL https://gist.githubusercontent.com/centminmod/03433e659a81e73cfe6ca94b8dc9d047/raw/getmaxbody.sh | bash -x
    
     
  4. pamamolf

    pamamolf Well-Known Member

    2,778
    245
    63
    May 31, 2014
    Ratings:
    +437
    Local Time:
    5:47 AM
    Nginx-1.13.x
    MariaDB 10.1.x
     
  5. pamamolf

    pamamolf Well-Known Member

    2,778
    245
    63
    May 31, 2014
    Ratings:
    +437
    Local Time:
    5:47 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Also is there a way to set a maximum value as if a user has a file 20gb there to not set that value so high?
     
  6. eva2000

    eva2000 Administrator Staff Member

    30,634
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    1:47 PM
    Nginx 1.13.x
    MariaDB 5.5
    hmm does such a file even exist at /home/nginx/domains/mydomain.com/public/forum/uploads/monthly_2016_11/d4e0fd577fa66c03759cce339f09d95f1e2a3805_full.jpg.a2a5839ab6045c9c12652616a458b2fe.jpg ?

    what is output for these 2 commands
    Code (Text):
    ls -lah /home/nginx/domains/mydomain.com/public/forum/uploads/monthly_2016_11/d4e0fd577fa66c03759cce339f09d95f1e2a3805_full.jpg.a2a5839ab6045c9c12652616a458b2fe.jpg
    

    Code (Text):
    ls -lah /home/nginx/domains/mydomain.com/public/forum/uploads/monthly_2016_11/d4e0fd577fa66c03759cce339f09d95f1e2a3805*
    

    yes that's next up on the revisions allow end user to set a max size to set to
     
  7. pamamolf

    pamamolf Well-Known Member

    2,778
    245
    63
    May 31, 2014
    Ratings:
    +437
    Local Time:
    5:47 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    No such file or directory for both commands :)

    It seems that was a temporary file created from Invision forum from a user upload .... ?

    But running centminmod update and re run it i am getting the same error :(

    Don't know how is this possible as the file it seems to not be there:

    That's perfect !!!! I am waiting for it :)
     
  8. eva2000

    eva2000 Administrator Staff Member

    30,634
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    1:47 PM
    Nginx 1.13.x
    MariaDB 5.5
    added threshold variable to limit to max 5GB to start Beta Branch - update inc/cpcheck.inc nginxlargefile_fix function

    it's invalid file so problem isn't centmin mod function but that file it's invalid i think

    not sure how to deal with files that aren't actually there ? strange find command picked up on it
     
  9. eva2000

    eva2000 Administrator Staff Member

    30,634
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    1:47 PM
    Nginx 1.13.x
    MariaDB 5.5
    What about updated run of
    Code (Text):
    curl -sL https://gist.githubusercontent.com/centminmod/03433e659a81e73cfe6ca94b8dc9d047/raw/getmaxbody.sh | bash -x
    

    and
    Code (Text):
    awk '/client_max_body_size/ {print $2}' /usr/local/nginx/conf/nginx.conf
    
     
  10. pamamolf

    pamamolf Well-Known Member

    2,778
    245
    63
    May 31, 2014
    Ratings:
    +437
    Local Time:
    5:47 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Same error :(
     
  11. eva2000

    eva2000 Administrator Staff Member

    30,634
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    1:47 PM
    Nginx 1.13.x
    MariaDB 5.5
    exact outputs
     
  12. pamamolf

    pamamolf Well-Known Member

    2,778
    245
    63
    May 31, 2014
    Ratings:
    +437
    Local Time:
    5:47 AM
    Nginx-1.13.x
    MariaDB 10.1.x
     
  13. pamamolf

    pamamolf Well-Known Member

    2,778
    245
    63
    May 31, 2014
    Ratings:
    +437
    Local Time:
    5:47 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    I test it on another server with no issues :)

    So it seems that server has a ghost file there :p
     
  14. eva2000

    eva2000 Administrator Staff Member

    30,634
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    1:47 PM
    Nginx 1.13.x
    MariaDB 5.5
    what's output for
    Code (Text):
    ls -lah /home/nginx/domains/mydomain.com/public/forum/uploads/monthly_2016_11/d4e0fd*
    

    and
    Code (Text):
    ls -lah /home/nginx/domains/mydomain.com/public/forum/uploads/monthly_2016_11/ | tail -10
    

    and
    Code (Text):
    ls -lah /home/nginx/domains/mydomain.com/public/forum/uploads/monthly_2016_11/ | head -n10
    

    Input/output error might actually filesystem/disk error of some kind too
     
  15. pamamolf

    pamamolf Well-Known Member

    2,778
    245
    63
    May 31, 2014
    Ratings:
    +437
    Local Time:
    5:47 AM
    Nginx-1.13.x
    MariaDB 10.1.x
     
  16. eva2000

    eva2000 Administrator Staff Member

    30,634
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    1:47 PM
    Nginx 1.13.x
    MariaDB 5.5
    yup pretty sure your have hardware issue - you have problems with your hard disks
     
    • Informative Informative x 1
  17. pamamolf

    pamamolf Well-Known Member

    2,778
    245
    63
    May 31, 2014
    Ratings:
    +437
    Local Time:
    5:47 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    I tested also at my huge public folder with about 700GB of data and Centminmod menu open after about 25 seconds....

    I prefer to have a value for this at 1024MB as default and let users optimize that but ok i respect others that may want that automated :)
     
  18. eva2000

    eva2000 Administrator Staff Member

    30,634
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    1:47 PM
    Nginx 1.13.x
    MariaDB 5.5
    just set client_max_body_size in nginx.conf to value like 1024m. How long does it take to open centmin.sh at that value ?
     
  19. pamamolf

    pamamolf Well-Known Member

    2,778
    245
    63
    May 31, 2014
    Ratings:
    +437
    Local Time:
    5:47 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    5 seconds faster......
     
  20. eva2000

    eva2000 Administrator Staff Member

    30,634
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    1:47 PM
    Nginx 1.13.x
    MariaDB 5.5
    manual find command timed
    Code (Text):
    time find /home/nginx/domains/*/public -maxdepth 4 -size +1024M -type f -printf '%s %p\n'|sort -nr|awk 'NR == 1 {print $1}'
    

    Code (Text):
    time find /home/nginx/domains/*/public -maxdepth 4 -size +2048M -type f -printf '%s %p\n'|sort -nr|awk 'NR == 1 {print $1}'
    

    what output do you get for these 2 commands ?