Join the community today
Become a Member

WebPerf PageSpeed PageSpeed Testing Via APIs: Google PageSpeed Insights, GTMetrix & WebpageTest.org

Discussion in 'Web Development & Web Performance' started by eva2000, Jul 4, 2018.

  1. eva2000

    eva2000 Administrator Staff Member

    54,548
    12,221
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,790
    Local Time:
    10:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    For fellow page speed addict you might want to check out a new script I whipped up to use APIs for Google PageSpeed Insights, GTMetrix and WebpageTest.org to run via command line pagespeed tests and optionally send the results to custom Slack channels.

    Usage help
    Code (Text):
    ./gitools.sh 
    
    Usage:
    
    Google PageSpeed Insights
    ./gitools.sh desktop https://domain.com {default|origin|site}
    ./gitools.sh mobile https://domain.com {default|origin|site}
    ./gitools.sh all https://domain.com {default|origin|site}
    
    GTMetrix
    ./gitools.sh gtmetrix https://domain.com
    
    WebpageTest
    
    supported region(s)
    dulles, california, frankfurt, singapore, sydney
    dallas, london, tokyo, hongkong, mumbia, brazil
    
    ./gitools.sh wpt https://community.centminmod.com {region} cable
    ./gitools.sh wpt https://community.centminmod.com {region} 3g
    ./gitools.sh wpt https://community.centminmod.com {region} 4g
    ./gitools.sh wpt https://community.centminmod.com {region} lte
    ./gitools.sh wpt https://community.centminmod.com {region} fios
    


    I even setup a cronjob to run the test every week and send results to my Slack Channel. So enjoy :D

    centminmod/google-insights-api-tools

    Slack channel result

    Google PageSpeed Insights

    google-pagespeed-insight-api-gitool-slack-01b.png
    WebpageTest.org

    wpt-dulles-slack-01.png


    GTMetrix

    gtmetrix-api-slack-01.png
     
  2. eva2000

    eva2000 Administrator Staff Member

    54,548
    12,221
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,790
    Local Time:
    10:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Latest version of gitools.sh can pass region on command line and supports, dulles, frankfurt, california, singapore and sydney right now. Also waterfall screenshot is included too

    Code (Text):
    ./gitools.sh wpt https://community.centminmod.com dulles
    
    --------------------------------------------------------------------------------
    Dulles:Chrome.Cable WPT Results
    --------------------------------------------------------------------------------
    Test ID: 180705_SK_ARW
    https://www.webpagetest.org/result/180705_SK_ARW/
    https://www.webpagetest.org/lighthouse.php?test=180705_SK_ARW
    /home/wptresults/wpt-dulles.chrome.cable.050718-044704.log
    Test Started 27 seconds ago
    Test Started 27 seconds ago (100)
    waiting on results...
    Test Complete
    Test Complete (200)
    waiting on results...
    Test Complete
    Test Complete (200)
    ----
    firstPaint 1235
    loadTime 1511
    domInteractive 1459
    fullyLoaded 3112
    requests 25
    TTFB 496
    domElements 1979
    visualComplete 2300
    render 1300
    SpeedIndex 1326
    visualComplete85 1300
    visualComplete90 1300
    visualComplete95 1500
    visualComplete99 1500
    chromeUserTiming.firstPaint 1235
    chromeUserTiming.firstContentfulPaint 1235
    chromeUserTiming.domInteractive 1459
    chromeUserTiming.firstMeaningfulPaintCandidate 1479
    chromeUserTiming.firstMeaningfulPaint 1479
    chromeUserTiming.domComplete 1511
    lighthouse.Performance.first-contentful-paint 1814
    lighthouse.Performance.estimated-input-latency 38
    lighthouse.Performance.speed-index 2068
    lighthouse.Performance.first-meaningful-paint 2176
    lighthouse.Performance.first-cpu-idle 4164
    https://www.webpagetest.org/results/18/07/05/SK/ARW/1_waterfall.png
    ok----
    --------------------------------------------------------------------------------
    


    wpt-dulles-slack-waterfall-01.png
     
  3. eva2000

    eva2000 Administrator Staff Member

    54,548
    12,221
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,790
    Local Time:
    10:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Example cronjob I setup to test Google PageSpeed Insights origin, GTMetrix and Webpagetest in various regions for my centmin mod main site and forums centminmod/google-insights-api-tools

    Code (Text):
    #!/bin/bash
    cd /root/tools/google-insights-api-tools
    ./gitools.sh all https://www.google.com origin
    ./gitools.sh all https://centminmod.com origin
    ./gitools.sh all https://community.centminmod.com origin
    
    ./gitools.sh gtmetrix https://centminmod.com
    ./gitools.sh gtmetrix https://community.centminmod.com
    
    ./gitools.sh wpt https://centminmod.com dulles cable
    ./gitools.sh wpt https://centminmod.com california cable
    ./gitools.sh wpt https://centminmod.com frankfurt cable
    ./gitools.sh wpt https://centminmod.com singapore cable
    ./gitools.sh wpt https://centminmod.com sydney cable
    ./gitools.sh wpt https://centminmod.com dallas cable
    ./gitools.sh wpt https://centminmod.com london cable
    ./gitools.sh wpt https://community.centminmod.com dulles cable
    ./gitools.sh wpt https://community.centminmod.com california cable
    ./gitools.sh wpt https://community.centminmod.com frankfurt cable
    ./gitools.sh wpt https://community.centminmod.com singapore cable
    ./gitools.sh wpt https://community.centminmod.com sydney cable
    ./gitools.sh wpt https://community.centminmod.com dallas cable
    ./gitools.sh wpt https://community.centminmod.com london cable
    
    ./gitools.sh wpt https://centminmod.com dulles 3g
    ./gitools.sh wpt https://centminmod.com california 3g
    ./gitools.sh wpt https://centminmod.com frankfurt 3g
    ./gitools.sh wpt https://centminmod.com singapore 3g
    ./gitools.sh wpt https://centminmod.com sydney 3g
    ./gitools.sh wpt https://centminmod.com dallas 3g
    ./gitools.sh wpt https://centminmod.com london 3g
    ./gitools.sh wpt https://community.centminmod.com dulles 3g
    ./gitools.sh wpt https://community.centminmod.com california 3g
    ./gitools.sh wpt https://community.centminmod.com frankfurt 3g
    ./gitools.sh wpt https://community.centminmod.com singapore 3g
    ./gitools.sh wpt https://community.centminmod.com sydney 3g
    ./gitools.sh wpt https://community.centminmod.com dallas 3g
    ./gitools.sh wpt https://community.centminmod.com london 3g
    
     
  4. eva2000

    eva2000 Administrator Staff Member

    54,548
    12,221
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,790
    Local Time:
    10:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Nicer formatted Slack channel messages :D

    gitools-pagespeed-insights-gtmetrix-slack-format-updated-01b.png gitools-pagespeed-insights-gtmetrix-slack-format-updated-02.png

    Slack via Android app

    gitools-pagespeed-insights-gtmetrix-slack-format-updated-mobile-01b.jpg
     
    Last edited: Jul 6, 2018
  5. eva2000

    eva2000 Administrator Staff Member

    54,548
    12,221
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,790
    Local Time:
    10:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Updated my gitools script so that Google PageSpeed Insights overall score ratings are colour coded when sent to Slack channel FAST = green, AVERAGE = yellow, SLOW = red

    gitools-pagespeed-insights-slack-message-color-coded-01.png

    edit: GTMetrix and Webpagetest slack channel messages are also colour coded too. GTMetrix is tied to Pagespeed score values while WPT is tied to TTFB recommended values for 3g, 4g, lte, fio and cable speed profiles.
     
  6. eva2000

    eva2000 Administrator Staff Member

    54,548
    12,221
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,790
    Local Time:
    10:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Working on non-origin Google PageSpeed Insight results which show specific pagespeed rule recommendations. Ran into one Slack channel message limitation as to how many lines of text are shown in one Slack message

    Code (Text):
    ./gitools.sh mobile https://xenforo.com/community/ default
    
    mobile (FAST) Score: 72
    Test url: https://xenforo.com/community/
    FCP: 1461ms (FAST) DCL: 1580ms (FAST)
    53.20% pages fast FCP (<1567ms)
    32.70% pages average FCP (<2963ms)
    14.20% pages slow FCP (>2963ms)
    65.60% pages fast DCL (<2120ms)
    28.40% pages average DCL (<4226ms)
    6.00% pages slow DCL (>4226ms)
    
    Leverage browser caching for following cacheable resources
    https://secure.gravatar.com/avatar/bd64ad41decb3e2d6d13419b904b94ed?s=48
    https://www.googletagmanager.com/gtag/js?id=UA-10855523-4
    https://www.google-analytics.com/analytics.js
    https://xenforo.com/community/data/avatars/s/0/342.jpg?1477326357
    https://xenforo.com/community/data/avatars/s/158/158213.jpg?1498339818
    https://xenforo.com/community/data/avatars/s/70/70243.jpg?1396074851
    https://xenforo.com/community/data/resource_icons/6/6023.jpg?1516685317
    https://xenforo.com/community/data/resource_icons/6/6136.jpg?1517935394
    https://xenforo.com/community/data/resource_icons/6/6281.jpg?1521698870
    https://xenforo.com/community/data/resource_icons/6/6526.jpg?1530491375
    https://xenforo.com/community/data/resource_icons/6/6537.jpg?1530920218
    https://xenforo.com/community/data/xfmg/thumbnail/0/405-885642ba1a1b9c4761eb9dc72878caf3.jpg?1504545113
    https://xenforo.com/community/data/xfmg/thumbnail/0/612-dff280b6dba486b115ddd6a5209e38a2.jpg?1504545173
    https://xenforo.com/community/data/xfmg/thumbnail/0/70-679b6cae97fe919f745f2c86dff63c7f.jpg?1504545035
    https://xenforo.com/community/data/xfmg/thumbnail/1/1114-673eabb495e17eae0988d09f0fc422ba.jpg?1504545385
    https://xenforo.com/community/data/xfmg/thumbnail/1/1168-60eb8455e79e2be6598d572a27420cfb.jpg?1504545403
    https://xenforo.com/community/data/xfmg/thumbnail/1/1382-74fcc7563f8164a6709030f9513cd4f7.jpg?1504545504
    https://xenforo.com/community/data/xfmg/thumbnail/1/1431-a5dcaa4f51e92f125024f504dfd657cc.jpg?1504545522
    https://xenforo.com/community/data/xfmg/thumbnail/1/1519-53235f9d1c949fa54e1982916a61f524.jpg?1504545574
    https://xenforo.com/community/data/xfmg/thumbnail/1/1769-bfa17a7ce54a5a15b2043af2cf8cd989.jpg?1504545657
    https://xenforo.com/community/data/xfmg/thumbnail/2/2198-506968ad6d785846a2a76b9c494e34de.jpg?1525762189
    https://xenforo.com/community/js/vendor/lightslider/lightslider.min.js?_v=b4c679f3
    https://xenforo.com/community/js/vendor/vendor-compiled.js?_v=b4c679f3
    https://xenforo.com/community/js/xf/core-compiled.js?_v=b4c679f3
    https://xenforo.com/community/js/xf/notice.min.js?_v=b4c679f3
    https://xenforo.com/community/js/xf/preamble-compiled.js?_v=b4c679f3
    https://xenforo.com/community/js/xfmg/slider.min.js?_v=b4c679f3
    https://xenforo.com/community/styles/default/xenforo/xenforo-logo2x.png
    https://xenforo.com/community/styles/fonts/fa/fontawesome-webfont.woff2?v=4.7.0
    
    ok
    


    gitools-pagespeed-insights-slack-message-leverage-browser-caching-01.png
     
  7. eva2000

    eva2000 Administrator Staff Member

    54,548
    12,221
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,790
    Local Time:
    10:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Updated gitools.sh to add webpagetest.org tokyo, hongkong, mumbia & brazil region support now :)
    Code (Text):
    ./gitools.sh 
    
    Usage:
    
    Google PageSpeed Insights
    ./gitools.sh desktop https://domain.com {default|origin|site}
    ./gitools.sh mobile https://domain.com {default|origin|site}
    ./gitools.sh all https://domain.com {default|origin|site}
    
    GTMetrix
    ./gitools.sh gtmetrix https://domain.com
    
    WebpageTest
    
    supported region(s)
    dulles, california, frankfurt, singapore, sydney
    dallas, london, tokyo, hongkong, mumbia, brazil
    
    ./gitools.sh wpt https://community.centminmod.com {region} cable
    ./gitools.sh wpt https://community.centminmod.com {region} 3g
    ./gitools.sh wpt https://community.centminmod.com {region} 4g
    ./gitools.sh wpt https://community.centminmod.com {region} lte
    ./gitools.sh wpt https://community.centminmod.com {region} fios
    
     
  8. eva2000

    eva2000 Administrator Staff Member

    54,548
    12,221
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,790
    Local Time:
    10:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Updated gitools.sh with WPT 3gfast profile test for both Webpagetest + Google Lighthouse report
    Code (Text):
    ./gitools.sh wpt https://community.centminmod.com california 3gfast
    


    Code (Text):
    ./gitools.sh wpt https://community.centminmod.com california 3gfast
    
    --------------------------------------------------------------------------------
    ec2-us-west-1:Chrome.3GFast WPT Results
    --------------------------------------------------------------------------------
    Test ID: 180709_H0_GXT
    https://www.webpagetest.org/result/180709_H0_GXT/
    https://www.webpagetest.org/lighthouse.php?test=180709_H0_GXT
    https://www.webpagetest.org/testlog.php?days=1&filter=community.centminmod.com&all=on&video=on
    /home/wptresults/wpt-california.ec2-us-west-1.chrome.3gfast.090718-070148.log
    Waiting behind 4 other tests...
    Waiting behind 4 other tests... (101)
    waiting on results...
    Waiting behind 2 other tests...
    Waiting behind 2 other tests... (101)
    waiting on results...
    Waiting at the front of the queue...
    Waiting at the front of the queue... (101)
    waiting on results...
    Test Started 23 seconds ago
    Test Started 23 seconds ago (100)
    waiting on results...
    Test Complete
    Test Complete (200)
    waiting on results...
    Test Complete
    Test Complete (200)
    ----
    bytesIn-fully-loaded 445586
    bytesInDoc-document-complete 299395
    chromeUserTiming.domComplete-time 2083
    chromeUserTiming.firstContentfulPaint 1299
    chromeUserTiming.firstMeaningfulPaint 1791
    chromeUserTiming.firstMeaningfulPaintCandidate 1791
    domComplete-time 2084
    domContentLoadedEventEnd 2077
    domContentLoadedEventStart 2077
    domElements 1981
    domInteractive 847
    firstPaint 1299.2999999988
    fullyLoaded-time 4189
    lighthouse.Accessibility 0.57
    lighthouse.BestPractices 0.71
    lighthouse.Performance 0.97
    lighthouse.Performance.estimated-input-latency 20.2873181818
    lighthouse.Performance.first-contentful-paint 1264.285
    lighthouse.Performance.first-cpu-idle 3151.611
    lighthouse.Performance.first-meaningful-paint 1780.913
    lighthouse.Performance.speed-index 1519
    lighthouse.ProgressiveWebApp 0.65
    lighthouse.SEO 1
    loadTime 2083
    start-render 1300
    requestsDoc 12
    requestsFull 25
    score_cache 75
    score_cdn 100
    score_compress 100
    score_gzip 100
    score_keep-alive 100
    SpeedIndex 1319
    TTFB 667
    visualComplete 3200
    https://www.webpagetest.org/results/18/07/09/H0/GXT/1_waterfall.png
    ok----
    --------------------------------------------------------------------------------
    


    gitools.sh webpagetest + lighthouse results sent to my custom Slack channel - updated the message formatting and added more metrics :)

    gitools-3g-fast-california-01.png
     
  9. eva2000

    eva2000 Administrator Staff Member

    54,548
    12,221
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,790
    Local Time:
    10:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Good thing about sending weekly Google PageSpeed Insight origin reports to my Slack channel is that I can now search my Slack channel as well :)

    Example Slack channel search for my mobile Google PageSpeed Insight origin page speed reports for my forums only :)

    slack-gitools-090818-01.png

    Progression of Google PageSpeed Insight origin reports and pagespeed improvements over time - especially since deploying progressive web app and service worker cache/precaching on the forums (since July 27th) and site (since July 24th) :)

    FYI, Google PageSpeed Insight origin reports are updated daily for past month's metrics. So over time if you're working on page load speed optimisations for First Contentful Paint (FCP) and Document Content Loaded (DCL) times, they will reflect in your PSI origin reported pagespeed metric times :D

    July 8th, 2018 started with mobile FCP = 1608ms and DCL = 2366ms where 48% of pages from Chrome User Experience report (CRUX) were under 1567ms for FCP and 43.9% of pages where under 2120ms

    slack-gitools-090818-02-july8-2018.png

    July 18th, 2018

    slack-gitools-090818-02-july18-2018.png

    August 1st, 2018 - first time hit FAST rating for both FCP and DCL metrics where 52+ % of pages loaded within Google PageSpeed Insights FAST threshold of <1567ms and <2120ms respectively for FCP and DCL for mobile.

    slack-gitools-090818-03-aug1-2018.png

    August 8th, 2018 with mobile FCP = 1558ms and DCL = 2140ms where 50.4% of pages from Chrome User Experience report (CRUX) were under 1567ms for FCP and 49.0% of pages where under 2120ms. Not as fast as previous week which saw me for first time hit FAST rating for both FCP and DCL metrics. Yes my DCL time is just outside the FAST rating at 2140ms versus 2120ms FAST threshold :)

    Faster averages are due to reducing the proportion of SLOW FCP/DCL pages from July 8th 14.0/15.7% FCP/DCL to 11.9/12.60% FCP/DCL page loads and increasing proportion of FAST FCP/DCL from 48/43.9% to 50.4/49.2% FCP/DCL.

    slack-gitools-090818-03-aug8-2018.png
     
    Last edited: Aug 9, 2018