Get the most out of your Centmin Mod LEMP stack
Become a Member

Wordpress Moving WordPress media to Amazon S3

Discussion in 'Blogs & CMS usage' started by Guilherme Jaccoud, Apr 28, 2015.

  1. Guilherme Jaccoud

    Guilherme Jaccoud Member

    63
    30
    18
    May 29, 2014
    Ratings:
    +30
    Local Time:
    6:26 PM
    Hey guys, it's good to be back here!

    Recently, a major dance music portal became a client of mine. They came to me with a infected/hacked WordPress site and a overloaded MediaTemple VPS. I've cleaned the site, moved it to a new CentminMod server and now I'm working on a new cloud infrastructure for them.

    I think everybody here would be pleased to know that only by moving the site from a 16GB RAM MediaTemple DV with LAMP to a 4GB DigitalOcean with CentminMod, I was able to reduce their memory and CPU usage from 96% to 16% and the site is now faster than ever according to my client.

    Long live @eva2000 !!

    The site is 3 years old and has a HUGE collection of mp3 files from it's weekly podcast. I'm planning to move all mp3 files to Amazon S3. Without it's mp3 files, the site occupies a total of 2.5GB disk space, however, including the mp3 files, the site is a 50GB monster. I believe that 47.5GB of static files is a waste of disk space on a SSD VPS, that's why I'm planning to move all mp3 files to AWS.

    What I'm looking for here, is to share experiences about this kind of migration, specially about moving static (mp3) files from WordPress to S3 without breaking the links. I'm aware of AWS WordPress plugin, but it can only point the URL of newly uploaded files and what I'm looking for is a safe (non-regex sorcery) way to move all previously uploaded mp3 files and redirect old URLs to new S3 address.

    Any ideas?

    Cheers,
    Guigo
     
    • Like Like x 1
  2. eva2000

    eva2000 Administrator Staff Member

    31,027
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    6:26 AM
    Nginx 1.13.x
    MariaDB 5.5
    @Guilherme Jaccoud glad to hear Centmin Mod LEMP stack has helped your client :)

    As to mp3 files never done it myself using S3 for wordpress. But easy way would be upload mp3 to S3 bucket then enable the bucket to use Amazon Cloudfront CDN and use your own custom cdn.yourdomain.com so you can just change the reference url from yourdomain.com/upload/yourmp3file.mp3 to cdn.yourdomain.com/upload/yourmp3file.mp3. Plenty of Wordpress plugins capable of offloading url from wordpress upload/ folder to cdn. I use Autoptimize and WordPress Super Cache for AWS Cloudfront/S3 Wordpress CDN hosting to improve Page Speed (Amazon AWS Cloudfront)

    There's also a wordpress search and replace tool at WordPress Serialized PHP Search Replace Tool | Interconnect IT - WordPress Consultants, Web Development and Web Design if need it.
     
  3. eva2000

    eva2000 Administrator Staff Member

    31,027
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    6:26 AM
    Nginx 1.13.x
    MariaDB 5.5
    googled and first result WordPress › Amazon S3 and Cloudfront « WordPress Plugins

     
  4. Guilherme Jaccoud

    Guilherme Jaccoud Member

    63
    30
    18
    May 29, 2014
    Ratings:
    +30
    Local Time:
    6:26 PM
    Nice, thanks @eva2000. I was thinking the same, but was trying to avoid search and replace the database. I've used the InterConnectIt search and replace tool in the past, but more recently, I've became a fan of WP-CLI's search-replace command which also takes care of serialized data.

    I'm aware of the above mentioned plugin, however, it says...
    I'll try the CDN approach and let you guys know about the results.

    Thanks!
     
    • Like Like x 1
  5. eva2000

    eva2000 Administrator Staff Member

    31,027
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    6:26 AM
    Nginx 1.13.x
    MariaDB 5.5
    just manually upload existing mp3 to the same s3 bucket that the plugin uses
     
  6. Guilherme Jaccoud

    Guilherme Jaccoud Member

    63
    30
    18
    May 29, 2014
    Ratings:
    +30
    Local Time:
    6:26 PM
    @eva2000 I want to try a different approach, however, I'm having trouble with the proper rewrite rule. I want to rewrite all requests to .mp3 files...

    from: http://www.example.com/wp-content/uploads/2012/08/artist-song.mp3
    to: http://mp3.example.com/artist-song.mp3

    This is what I have so far, unfortunately, it doesn't seems to work :(

    Code:
    location ~* \.mp3$ {
        rewrite ^(.*)\/(.*)\.mp3$ http://mp3.example.com/$2.mp3 last;
    }
    
    Below is the nginx vhost file (main nginx conf file is pure CentminMod).

    Code:
    server { 
    
        listen 80; listen 443;
        server_name www.example.com;
        return 301 $scheme://example.com$request_uri;
    }
    
    server {
    
        listen 80;
        server_name example.com;
       
        root         /var/www/example.com/public;
        error_log     /var/www/example.com/log/error.log;
        access_log     /var/www/example.com/log/access.log combined buffer=32k;
    
        add_header X-Frame-Options SAMEORIGIN;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
    
        rewrite ^/wp-(.*)$ /wp/wp-$1 last;
        rewrite ^/sitemap_index\.xml$ /index.php?sitemap=1 last;
        rewrite ^/([^/]+?)-sitemap([0-9]+)?\.xml$ /index.php?sitemap=$1&sitemap_n=$2 last;
    
    #    include /usr/local/nginx/conf/pagespeed.conf;
    #    include /usr/local/nginx/conf/pagespeedhandler.conf;
    #    include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
        include /usr/local/nginx/conf/wp_secure.conf;
        include /usr/local/nginx/conf/wp_supercache.conf;
    #    include /usr/local/nginx/conf/wp_fastcgicache.conf;
    
        location ~* \.mp3$ { rewrite ^(.*)\/(.*)\.mp3$ http://mp3.example.com/$2.mp3 last; }
        location ~* /wp-(admin|login|includes) { return 301 https://$server_name$request_uri; }
        location / {
    
    #        auth_basic "Private";
    #        auth_basic_user_file /var/www/example.com/conf/nginx/htpasswd;
           
            if (-f /var/www/example.com/conf/maintenance) { return 503; }
    #        try_files $uri $uri/ /index.php?$args;
            try_files /wp/wp-content/cache/supercache/$http_host/$cache_uri/index.html $uri $uri/ /index.php?$args;
            autoindex off;
           
            location ~* \.php$ {
    
                try_files $uri =404;
    
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
    #            fastcgi_pass   127.0.0.1:9000;
                fastcgi_pass   unix:/tmp/example.com.sock;
                fastcgi_index  index.php;
    #            fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
                fastcgi_param  SCRIPT_FILENAME    $request_filename;
    
                # NinjaFirewall PHPRC
                fastcgi_param PHPRC /var/www/example.com/public/.user.ini;
    
                include fastcgi_param.conf;
    #            include fastcgi_cache.conf;
            }
        }
       
        include /usr/local/nginx/conf/block.conf;
        include /usr/local/nginx/conf/drop.conf;
        include /usr/local/nginx/conf/staticfiles.conf;
        include /usr/local/nginx/conf/errorpage.conf;
    }
    
    server {
    
        listen 443 ssl spdy;
        server_name example.com;
    
        root         /var/www/example.com/public;
        error_log     /var/www/example.com/log/error.log;
        access_log     /var/www/example.com/log/access.log combined buffer=32k;
    
        ssl_certificate            /var/www/example.com/ssl/example.com.crt;
        ssl_certificate_key        /var/www/example.com/ssl/example.com.key;
       
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_session_cache shared:SSL:10m;
    
        ssl_ciphers 'AES128+EECDH:AES128+EDH:!aNULL';
        ssl_prefer_server_ciphers on;
        ssl_session_timeout 10m;
       
        add_header Alternate-Protocol 443:npn-spdy/2;
        add_header X-Frame-Options SAMEORIGIN;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
       
        rewrite ^/wp-(.*)$ /wp/wp-$1 last;
        rewrite ^/sitemap_index\.xml$ /index.php?sitemap=1 last;
        rewrite ^/([^/]+?)-sitemap([0-9]+)?\.xml$ /index.php?sitemap=$1&sitemap_n=$2 last;
        rewrite ^/wp/favicon(.*)$ $scheme://$server_name/favicon$1 last;
       
    #    include /usr/local/nginx/conf/pagespeed.conf;
    #    include /usr/local/nginx/conf/pagespeedhandler.conf;
    #    include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
        include /usr/local/nginx/conf/wp_secure.conf;
    #    include /usr/local/nginx/conf/wp_supercache.conf;
    #    include /usr/local/nginx/conf/wp_fastcgicache.conf;
    
        location / { return 301 http://$server_name$request_uri; }   
        location ~* \.mp3$ { rewrite ^(.*)\/(.*)\.mp3$ http://mp3.example.com/$2.mp3 last; }
        location ~* /wp-(admin|login|includes|content) {
    
    #        auth_basic "Private";
    #        auth_basic_user_file /var/www/example.com/conf/nginx/htpasswd;
    
            if (-f /var/www/example.com/conf/maintenance) { return 503; }
            try_files $uri $uri/ /wp/index.php?$args;
    #        try_files /wp/wp-content/cache/supercache/$http_host/$cache_uri/index.html $uri $uri/ /index.php?$args;
            autoindex off;
    
            location ~* \.php$ {
    
                try_files $uri =404;
    
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
    #            fastcgi_pass   127.0.0.1:9000;
                fastcgi_pass   unix:/tmp/$server_name.sock;
                fastcgi_index  index.php;
    #            fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
                fastcgi_param  SCRIPT_FILENAME    $request_filename;
    
                # NinjaFirewall PHPRC 
                fastcgi_param PHPRC /var/www/example.com/public/.user.ini;
    
                include fastcgi_param.conf;
    #            include fastcgi_cache.conf;
            }
        }
    
        include /usr/local/nginx/conf/block.conf;
        include /usr/local/nginx/conf/drop.conf;
        include /usr/local/nginx/conf/staticfiles.conf;
        include /usr/local/nginx/conf/errorpage.conf;
    }
    
    

    Do you have some clue of what's wrong?
    Cheers :)
     
  7. eva2000

    eva2000 Administrator Staff Member

    31,027
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    6:26 AM
    Nginx 1.13.x
    MariaDB 5.5
    well rewrites usually you're on your own as I don't provide assistance for :)

    maybe try

    Code:
    location ~ ^(/wp-content/uploads)/(.*)\.(mp3)$ {
        rewrite ^/wp-content/uploads/(.*)/(.*)\.mp3$ http://mp3.example.com/$2.mp3 break;
    }
    or some variant of it
     
    • Like Like x 1
    • Winner Winner x 1
  8. Guilherme Jaccoud

    Guilherme Jaccoud Member

    63
    30
    18
    May 29, 2014
    Ratings:
    +30
    Local Time:
    6:26 PM
    Well, it works beautifully! :D
    I had to move rewrite ^/wp-(.*)$ /wp/wp-$1 last; into the / location though.

    I've asked the same question on StackExchange two days ago and besides critics, I got no real answer. This forum, and specially @eva2000, are really awesome! Thanks one more time.

    Cheers!
     
    • Like Like x 1