/ Register

Welcome to Centmin Mod Community
Become a Member

Sysadmin Googlebot Issues

Discussion in 'System Administration' started by Jimmy, Aug 31, 2019.

Previous Thread Next Thread
Loading...
  1. Aug 31, 2019 #1
    Jimmy

    Jimmy Well-Known Member

    1,799
    391
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +999
    Local Time:
    3:52 AM
    My server keeps getting scanned by googlebot for non-existing links. I've had this VPS for awhile, so I assumed that googlebot would stop crawling my site for links which don't exist.

    I've rate limited the googlebot in the block bad bots file, but google crawls my site ever couple of seconds.

    Here is an example of what is appearing in my server log at /var/log/nginx/localhost.access.log:

    Code:
    66.249.79.2 - - [30/Aug/2019:17:49:06 +0000] "GET /tag/the-godfather-movie-subtitles-download/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.5 - - [30/Aug/2019:17:49:09 +0000] "GET /tag/in-the-heart-of-the-sea-2015-full-movie-in-hindi-download/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.2 - - [30/Aug/2019:17:49:10 +0000] "GET /tag/luke-cage-season-1-download-free/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.31 - - [30/Aug/2019:17:49:07 +0000] "GET /tag/next-720p-free-download/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.5 - - [30/Aug/2019:17:49:07 +0000] "GET /tag/resident-evil-6-full-movie-download-720p/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.31 - - [30/Aug/2019:17:49:08 +0000] "GET /tag/underworld-5-hindi-dubbed-movie-download-480p/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.2 - - [30/Aug/2019:17:49:09 +0000] "GET /tag/final-destination-1-full-movie-download-moviescounter/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.5 - - [30/Aug/2019:17:52:44 +0000] "GET /tag/the-amazing-spider-man-2-dual-audio-full-movie-download/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:53:08 +0000] "GET /tag/avengers-infinity-war-720p-altyazili-izle/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.223 - - [30/Aug/2019:17:53:26 +0000] "GET /tag/mulk-movie-free-download-in-hindi/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.31 - - [30/Aug/2019:17:52:53 +0000] "GET /tag/deadpool-2-movie-english-subtitles-free-download/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.197 - - [30/Aug/2019:17:52:58 +0000] "GET /tag/3-percent-dual-audio-amazon-prime-free-download/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.31 - - [30/Aug/2019:17:52:59 +0000] "GET /tag/free-download-skyscrapper-movie/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.7 - - [30/Aug/2019:17:53:07 +0000] "GET /tag/pirates-of-the-caribbean-2-movie-download-yify/feed/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.197 - - [30/Aug/2019:17:53:17 +0000] "GET /tag/dhadak-new-hindi-movie-mp3-songs-free-download/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.5 - - [30/Aug/2019:17:53:30 +0000] "GET /tag/fear-the-walking-dead-hindi-dubbed-300mb/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:53:36 +0000] "GET /tag/download-skyscrapper-mp4/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:53:46 +0000] "GET /tag/wrong-turn-2-dual-audio-720p-hevc/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:53:59 +0000] "GET /tag/downloads-foumovies-searching-2018-cam/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:54:02 +0000] "GET /tag/free-download-xxx-2-movie-in-hindi-hd/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:54:15 +0000] "GET /tag/kingsman-the-secret-service-all-parts-dual-audio-download/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:54:28 +0000] "GET /tag/doraemon-the-movie-doraemon-in-nobitas-great-adventure-to-the-south-seas-in-hindi-download/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.5 - - [30/Aug/2019:17:55:08 +0000] "GET /tag/x-men-4-online-subtitrat-720p/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.197 - - [30/Aug/2019:17:55:10 +0000] "GET /tag/death-note-episodes-download/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.197 - - [30/Aug/2019:17:55:18 +0000] "GET /tag/download-lost-in-space-all-episodes-in-720p/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:55:35 +0000] "GET /tag/predator-1-hindi-movie/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
168.194.109.79 - - [30/Aug/2019:17:55:44 +0000] "GET / HTTP/1.1" 200 164 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
66.249.79.5 - - [30/Aug/2019:17:56:03 +0000] "GET /tag/harry-potter-1-dual-audio-720p-watch-online/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.5 - - [30/Aug/2019:17:56:08 +0000] "GET /tag/download-movie-batman-begins-in-hindi-mp4/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:56:23 +0000] "GET /tag/download-english-subtitles-for-the-godfather-3-720p/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:56:35 +0000] "GET /tag/pirates-of-the-caribbean-4-full-movie-download-moviescounter/feed/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:56:43 +0000] "GET /category/movies/page/7/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:56:49 +0000] "GET /tag/download-altered-carbon-movie/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.5 - - [30/Aug/2019:17:57:03 +0000] "GET /tag/annabelle-1-hindi-in-hd/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:57:26 +0000] "GET /tag/harry-potter-1-movie-download-in-hindi-in-avi/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.197 - - [30/Aug/2019:17:57:38 +0000] "GET /tag/breathe-series-in-hindi-300mb/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:57:53 +0000] "GET /tag/download-scary-movie-5-full-movie-mp4/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:54:15 +0000] "GET /tag/fantastic-4-full-movie-download-1080p/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:54:24 +0000] "GET /tag/avengers-age-of-ultron-movie-english-subtitles-free-download/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:54:43 +0000] "GET /tag/mission-impossible-1-720p-yify/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:54:47 +0000] "GET /tag/death-race-3-in-hindi-720p/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:54:49 +0000] "GET /tag/spiderman-hindi-dubbed-brrip/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:55:26 +0000] "GET /tag/next-movie-free-download-in-hindi/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:55:39 +0000] "GET /tag/narcos-full-episode-youtube/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.197 - - [30/Aug/2019:17:55:46 +0000] "GET /tag/wrong-turn-hindi-ipagal/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:55:52 +0000] "GET /download-kingsman-the-secret-service-2014-hindi-480p-720p-1080p/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:56:15 +0000] "GET /tag/star-wars-1-hindi-episodes-download/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:56:29 +0000] "GET /tag/pirates-of-the-caribbean-5-hindi-hd-online/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:56:56 +0000] "GET /tag/venom-hindi-coolmoviez/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:57:13 +0000] "GET /tag/final-destination-5-in-hindi-720p-watch-online/feed/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:57:20 +0000] "GET /tag/parmanu-hd-movie-download-youtube/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.3 - - [30/Aug/2019:17:57:45 +0000] "GET /tag/the-terminator-5-720p-dual-audio/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.31 - - [30/Aug/2019:17:58:08 +0000] "GET /tag/download-naked-season-1-all-episodes-in-720p/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.197 - - [30/Aug/2019:17:58:23 +0000] "GET /tag/jai-lava-kusa-all-parts-dual-audio-300mb/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.2 - - [30/Aug/2019:17:58:36 +0000] "GET /tag/guardians-of-the-galaxy-100mb-movie-download/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.197 - - [30/Aug/2019:17:58:42 +0000] "GET /tag/next-720p-kickass/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.31 - - [30/Aug/2019:17:58:53 +0000] "GET /tag/stephen-hawkings-grand-design-720p-download/feed/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.31 - - [30/Aug/2019:17:58:59 +0000] "GET /tag/the-chronicles-of-narnia-2-full-movie-download-filmywap-com/feed/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.31 - - [30/Aug/2019:17:59:00 +0000] "GET /tag/fast-and-furious-6-full-movie-in-hindi-free-download-300mb/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.197 - - [30/Aug/2019:17:59:02 +0000] "GET /tag/sketch-movie-download-in-tamil/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.31 - - [30/Aug/2019:17:59:04 +0000] "GET /tag/scary-movie-in-hindi-300mb/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.31 - - [30/Aug/2019:17:59:05 +0000] "GET /tag/underworld-4-full-movie-download-mkv/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    Googlebot seems to be searching for a movie download type of site. I don't have movies on my site to download. The location googlebot is looking for tags is /tag/ and my site's tags are located at /tags/.

    Is there anyway to stop googlebot from searching for these non-existant links? They aren't showing in Google search console.
     
  2. Sep 1, 2019 #2
    eva2000

    eva2000 Administrator Staff Member

    58,894
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    5:52 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    /var/log/nginx/localhost.access.log is nginx access log for main hostname's /usr/local/nginx/conf/conf.d/virtual.conf which is served from /usr/local/nginx/html by default i.e. the default Centmin Mod Nginx Install page when you visit your server's IP address.

    So if the probably is there you can just block bots by user agents in your /usr/local/nginx/conf/conf.d/virtual.conf nginx vhost for your main hostname as you shouldn't be using web sites on main hostname as outlined at PHP Opcode and Memcached statistics pages

    There's also possibility someone pointed their domain DNS A record incorrectly to your server's IP address etc. But there's no real harm in search engine /crawlers looking for non existent pages. If you have Cloudflare in front it can also be configured to block fake bots or utilise cloudflare firewall rules to restrict scans at Cloudflare edge so it doesn't hit your servers etc.

    Though be careful of using cloudflare in front of main hostname as that will lower email deliverability from server's Postfix MTA as recipient mail servers won't be able to properly look up your main hostname's reverse PTR DNS record to check for spam.
     
    Last edited: Sep 1, 2019
Previous Thread Next Thread
Loading...

.