Discover Centmin Mod today
Register Now

WebPerf Optimize Images

Discussion in 'Web Development & Web Performance' started by Fernando, Aug 17, 2017.

  1. Fernando

    Fernando Member

    44
    8
    8
    Jul 21, 2017
    Ratings:
    +12
    Local Time:
    3:45 PM
    1.13.3
    10.1.25
    Hi,

    I'm using optimise-images.sh for my website, thank you for this great tool! I'm planning to replace my plugins with optimise-images.sh :)

    I just have a question or maybe if you have any suggestion.
    I have set:
    Code:
    IMAGICK_WEBP='y'
    IMAGICK_WEBPLOSSLESS='n'
    And I'm planning to add:
    Code:
    UNATTENDED_OPTIMISE='y'
    So I can do this automatically.

    I can see the webp images properly in my website.
    The question that I have is my website allows customers to upload images, so at the end of the day I can have let's say 100 new images in uploads/2017/08 just as an example. I can even have more images per day in the folder.

    Is there a way that I can skip already optimized images? Or do you have any suggestion so I don't re-optimize a lot of images every time when I run the script?

    Thank you again for this great tool!
    Regards
     
  2. eva2000

    eva2000 Administrator Staff Member

    36,810
    8,060
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,414
    Local Time:
    7:45 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Is this wordpress site ? You should combine it with EWWW Image Optimizer + Imsanity i.e. Wordpress - Playing with EWWW Image Optimizer Wordpress Plugin. This allows automatic upload resize/optimisation in lossless format first in free EWWW plugin. Lossless optimisation usually only saves 2-5% overall for already ok optimized images and up to 5-20% for others.

    Then set optimise-images.sh via cronjob to work on /upload directory full path i.e. /home/nginx/domains/domain.com/public/wp-content/upload to do lossy optimisation on resulting EWWW images. The lossy optimisation boosts the image size reductions to between 20-85%.

    Currently, no way in optimise-images.sh script but if you look carefully in the optimise command verbose output you will see if you re-ran optimise mode on already optimised images, you don't do much at all

    example of re-running optimise-images.sh optimise mode on already optimised directory
    Code (Text):
    ------------------------------------------------------------------------------
    image optimisation start
    ------------------------------------------------------------------------------
    ### dslr_nikon_d7200_1.jpg (jpg) ###
    /bin/nice -n 10 /usr/bin/ionice -c2 -n7 /usr/bin/convert -define registry:temporary-path=/home/imagicktmp dslr_nikon_d7200_1.jpg -define jpeg:size=4096x4096 -filter triangle -define filter:support=2 -define jpeg:fancy-upsampling=off -unsharp 0.25x0.08+8.3+0.045 -interlace none -strip             -write mpr:dslr_nikon_d7200_1 -resize 2048x2048\> -write dslr_nikon_d7200_1.jpg +delete             mpr:dslr_nikon_d7200_1 -define webp:thread-level=1 -define webp:method=4 -define webp:alpha-quality=100 -define webp:lossless=false -quality 75 -resize 2048x2048\> dslr_nikon_d7200_1.jpg.webp
    /bin/nice -n 10 /usr/bin/ionice -c2 -n7 jpegoptim -p --max=82 dslr_nikon_d7200_1.jpg
    dslr_nikon_d7200_1.jpg 2048x1365 24bit N JFIF  [OK] 374587 --> 374561 bytes (0.01%), optimized.
    

    2nd run
    Code (Text):
    ------------------------------------------------------------------------------
    Original or Existing Images:
    ------------------------------------------------------------------------------
    | Avg width | Avg height | Avg quality | Avg size   | Total size (Bytes) | Total size (KB) |
    | --------- | ---------- | ----------- | --------   | ------------------ | --------------- |
    | 1166      | 945        | 87          | 438435     | 3507478            | 3425            |
    
    ------------------------------------------------------------------------------
    Optimised WebP Images:
    ------------------------------------------------------------------------------
    | Avg width | Avg height | Avg quality | Avg size   | Total size (Bytes) | Total size (KB) |
    | --------- | ---------- | ----------- | --------   | ------------------ | --------------- |
    | 1166      | 945        | 92          | 89689      | 717514             | 701             |
    

    3rd run
    Code (Text):
    ------------------------------------------------------------------------------
    Original or Existing Images:
    ------------------------------------------------------------------------------
    | Avg width | Avg height | Avg quality | Avg size   | Total size (Bytes) | Total size (KB) |
    | --------- | ---------- | ----------- | --------   | ------------------ | --------------- |
    | 1166      | 945        | 87          | 438415     | 3507317            | 3425            |
    
    ------------------------------------------------------------------------------
    Optimised WebP Images:
    ------------------------------------------------------------------------------
    | Avg width | Avg height | Avg quality | Avg size   | Total size (Bytes) | Total size (KB) |
    | --------- | ---------- | ----------- | --------   | ------------------ | --------------- |
    | 1166      | 945        | 92          | 89684      | 717476             | 701             |
    

    But I can see the problem you're describing. I could probably could add another mode to only look for images created/new in last XX hours i.e. 24hrs to process to limit the duplication of processing.
     
    • Like Like x 1
  3. Fernando

    Fernando Member

    44
    8
    8
    Jul 21, 2017
    Ratings:
    +12
    Local Time:
    3:45 PM
    1.13.3
    10.1.25
    Hi :)

    This would be great!: I could probably could add another mode to only look for images created/new in last XX hours i.e. 24hrs to process to limit the duplication of processing.

    If I understand this: Then set optimise-images.sh via cronjob to work on /upload directory full path i.e. /home/nginx/domains/domain.com/public/wp-content/upload to do lossy optimisation on resulting EWWW images. The lossy optimisation boosts the image size reductions to between 20-85%.

    You suggest to use:
    Code:
    IMAGICK_WEBPLOSSLESS='y'
    However, in the post it says:
    IMAGICK_WEBPLOSSLESS='n' (kept at lossless as lossy 'y' results in large WebP images for some reason)

    So I want to double check what option should I set in regarsd to IMAGICK_WEBPLOSSLESS for better optimization.

    Once again thank you!
    Regards
     
  4. eva2000

    eva2000 Administrator Staff Member

    36,810
    8,060
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,414
    Local Time:
    7:45 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    the optimise-images.sh lossy optimisation is for JPG/PNG only while keeping WebP IMAGICK_WEBPLOSSLESS='n' disabled
     
  5. Fernando

    Fernando Member

    44
    8
    8
    Jul 21, 2017
    Ratings:
    +12
    Local Time:
    3:45 PM
    1.13.3
    10.1.25
    Hi,

    Thank you for your help! Now it's clear.
    Best Regards
     
  6. eva2000

    eva2000 Administrator Staff Member

    36,810
    8,060
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,414
    Local Time:
    7:45 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Updated optimise-images.sh with new profile-age & optimise-age Mode which allow you to filter and limit profiling and optimisation of images by image timestamp age :)
     
    • Winner Winner x 1
  7. eva2000

    eva2000 Administrator Staff Member

    36,810
    8,060
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,414
    Local Time:
    7:45 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    slight correction, optimise-image.sh only operates 1 level deep so need to do some scripting as outlined at GitHub - centminmod/optimise-images: Batch image resizer, optimiser and profiler using ImageMagick convert, OptiPNG, JpegOptim and optional ZopfliPNG, Guetzli and MozJPEG. so to process directories at /home/nginx/domains/domain.com/public/wp-content/upload/2017/XX where XX is month

    i.e. new optimise-cron-age mode to filter by image timestamp age + specifically made for use as a cronjob as it saves processing time by removing the post-optimisation profile run. Set UNATTENDED_OPTIMISE='y' to disable the backup directory prompt so you can run unattended first. Edit optimise-cron-age command for command you want to use.
    Code (Text):
    find /home/nginx/domains/domain.com/public/wp-content/upload/2017/ -type d | sort | while read d; do echo "optimise directory: $d"; echo "/root/tools/optimise-images.sh optimise-cron-age $d"; /root/tools/optimise-images.sh optimise-cron-age $d; done
    chown -R nginx:nginx /home/nginx/domains/domain.com/public/wp-content/upload/2017/
    

    add a chown command to change file user/group permissions too
     
  8. Fernando

    Fernando Member

    44
    8
    8
    Jul 21, 2017
    Ratings:
    +12
    Local Time:
    3:45 PM
    1.13.3
    10.1.25
    Hi,

    This is great! Thank you for your help :)
    Best Regards
     
    • Like Like x 1
  9. Fernando

    Fernando Member

    44
    8
    8
    Jul 21, 2017
    Ratings:
    +12
    Local Time:
    3:45 PM
    1.13.3
    10.1.25
    Hi :)

    I was checking and testing. I noticed that the new filters will continue to optimize old images instead of new images. For example, if you set: FIND_IMGAGE=1440 and run the job today then everything from 8/17/2017 and older will be optimized, if you run the job again on 8/19/2017 then everything from 8/18/2017 and older will be optimized so the optimization will be performed twice, I believe the correct behaviour should be as you described:

    I could probably could add another mode to only look for images created/new in last XX hours i.e. 24hrs to process to limit the duplication of processing.

    With the current filter it just ignores X amount of time and optimize everything in the past :(

    Is this correct?
     
  10. eva2000

    eva2000 Administrator Staff Member

    36,810
    8,060
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,414
    Local Time:
    7:45 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    whoops yeah now you explain, yes that is what FIND_IMGAGE does optimises stuff older than heh :oops:

    will fix it
     
  11. eva2000

    eva2000 Administrator Staff Member

    36,810
    8,060
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,414
    Local Time:
    7:45 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
  12. eva2000

    eva2000 Administrator Staff Member

    36,810
    8,060
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,414
    Local Time:
    7:45 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    working on a new option add ADD_COMMENT='n' option · centminmod/[email protected] · GitHub
    haven't done the skipping routine yet just putting the marker comment option in place first :)
     
  13. eva2000

    eva2000 Administrator Staff Member

    36,810
    8,060
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,414
    Local Time:
    7:45 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    @Fernando @Matt will like this :)
    Code (Text):
    ./optimise-images.sh optimise /home/optimise-benchmarks
    
    ------------------------------------------------------------------------------
    image optimisation start
    ------------------------------------------------------------------------------
    ### dslr_nikon_d7200_1.jpg (jpg) skip already optimised ###
    ### webp-study-source-firebreathing.png (png) skip already optimised ###
    ### png24-interlaced-image1.png (png) skip already optimised ###
    ### dslr_nikon_d7200_2.jpg (jpg) skip already optimised ###
    ### dslr_canon_eos_m6_1.jpg (jpg) skip already optimised ###
    ### png24-image1.png (png) skip already optimised ###
    ### samsung_s7_mobile_1.jpg (jpg) skip already optimised ###
    ### bees.png (png) skip already optimised ###
    ------------------------------------------------------------------------------
    Completion Time: 2.55 seconds
    ------------------------------------------------------------------------------
     
    • Informative Informative x 1
  14. Fernando

    Fernando Member

    44
    8
    8
    Jul 21, 2017
    Ratings:
    +12
    Local Time:
    3:45 PM
    1.13.3
    10.1.25
    Amazing! :) just tested and it works perfectly
    Thank you so much for everything!
     
  15. eva2000

    eva2000 Administrator Staff Member

    36,810
    8,060
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,414
    Local Time:
    7:45 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    thanks for testing, bug reporting and feedback :D
     
  16. dcg

    dcg Member

    53
    18
    8
    Oct 17, 2015
    Florida, USA
    Ratings:
    +34
    Local Time:
    4:45 PM
    1.15.x
    10.2.x
    I noticed some extra whitespace issues. It affects the profile output tables.

    Example:
    Code:
    %-11.    0f
    %  -11.  0f
    %-   11.  0f
    Code:
    grep -n -P "(\%-\s+[0-9]|[0-9]\.\s+0f|\%\s+-?[0-9].*?0f)" optimise-images.sh
    [Bash] optimise-images whitspace - Pastebin.com

    I ran, which cleared it up.
    Code:
    sed -r 's/([0-9]\.)\s+(0f)/\1\2/g' optimise-images.sh |  sed -r 's/(\%)\s+(-[0-9][0-9]\.0f)/\1\2/g' | sed -r 's/(\%-)\s+([0-9]{2}\.0f)/\1\2/g' > optimise-images.sh
    Thanks for the script @eva2000
     
    • Like Like x 1
  17. eva2000

    eva2000 Administrator Staff Member

    36,810
    8,060
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,414
    Local Time:
    7:45 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    not sure i understand whitespace in code or output table ?

    got a diff of before and after clean up of optimise-image.sh ?

    edit: of you mean like
    Code (Text):
     grep -n -P "(\%-\s+[0-9]|[0-9]\.\s+0f|\%\s+-?[0-9].*?0f)" optimise-images.sh                    
    1024:        cat "$LOG_PROFILE" | egrep "${COMPARE_SUFFIX}.jpg :|${COMPARE_SUFFIX}.png :|${COMPARE_SUFFIX}.jpeg :" | awk -F " : " '{c3 += $3; c4 += $4; c5 += $5; c8 += $8; tb = c8; tk = c8} END {printf "| %-9.0f | %-10.0f | %-11.  0f | %-10.0f | %-18.0f | %-15.0f |\n", c3/NR, c4/NR, c5/NR, c8/NR, tb, tk/1024}'
    

    the space between
    Code (Text):
    %-11.  0f
    

    i will check thanks
     
    • Like Like x 2
  18. dcg

    dcg Member

    53
    18
    8
    Oct 17, 2015
    Florida, USA
    Ratings:
    +34
    Local Time:
    4:45 PM
    1.15.x
    10.2.x
    In the code:
    Line 1024
    Code:
            cat "$LOG_PROFILE" | egrep "${COMPARE_SUFFIX}.jpg :|${COMPARE_SUFFIX}.png :|${COMPARE_SUFFIX}.jpeg :" | awk -F " : " '{c3 += $3; c4 += $4; c5 += $5; c8 += $8; tb = c8; tk = c8} END {printf "| %-9.0f | %-10.0f | %-11.  0f | %-10.0f | %-18.0f | %-15.0f |\n", c3/NR, c4/NR, c5/NR, c8/NR, tb, tk/1024}'
    The awk command to format float number has whitespace in it: %-11. 0f

    So it doen't process float, but just prints "%-11. 0f". There are a few of them where whitespace is after % and some after - and others after the .
     
    • Like Like x 1
  19. eva2000

    eva2000 Administrator Staff Member

    36,810
    8,060
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,414
    Local Time:
    7:45 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    • Like Like x 1
  20. dcg

    dcg Member

    53
    18
    8
    Oct 17, 2015
    Florida, USA
    Ratings:
    +34
    Local Time:
    4:45 PM
    1.15.x
    10.2.x
    I figured some editor or something put in some extra whitespace ;). Thanks!
     
    • Like Like x 1
..