Want more timely Centmin Mod News Updates?
Become a Member

Nginx Blocking a bot

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by pamamolf, Mar 29, 2015.

  1. pamamolf

    pamamolf Premium Member Premium Member

    4,074
    427
    83
    May 31, 2014
    Ratings:
    +833
    Local Time:
    6:44 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    Hi
    I found on logs this:


    Code:
    omgilibot/0.4
    and i want to block this user agent but i can't find any info for it....

    So can i block it as user agent omgilibot/0.4 or as omgilibot ?
     
  2. rdan

    rdan Well-Known Member

    5,443
    1,402
    113
    May 25, 2014
    Ratings:
    +2,194
    Local Time:
    12:44 AM
    Mainline
    10.2
  3. pamamolf

    pamamolf Premium Member Premium Member

    4,074
    427
    83
    May 31, 2014
    Ratings:
    +833
    Local Time:
    6:44 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    I know how but is the name of the agent omgilibot/0.4 or omgilibot ?

    I have many bots that end with botname/0.1.4 or botname/4.0 and i don't know as i can't find any info on the net how to block it....with the /x.x.x or without?
     
  4. eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:44 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    just the name is enough
     
  5. pamamolf

    pamamolf Premium Member Premium Member

    4,074
    427
    83
    May 31, 2014
    Ratings:
    +833
    Local Time:
    6:44 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    One last question :)

    The user agent must match exact the name to be blocked or not?

    For example if i got this: botname-dev and i add to block it as botname without the -dev it will be blocked?
     
  6. eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:44 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    see what ~ and ~* matching does in nginx Module ngx_http_rewrite_module

     
  7. pamamolf

    pamamolf Premium Member Premium Member

    4,074
    427
    83
    May 31, 2014
    Ratings:
    +833
    Local Time:
    6:44 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    Just to be sure to archive what i ask for i should change this:

    Code:
      if ($http_user_agent ~ "GrabNet") {
      set $block_user_agents 1;
      }
    to this?

    Code:
      if ($http_user_agent ~* "GrabNet") {
      set $block_user_agents 1;
      }
     
  8. eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:44 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    yeah that would match against lower and upper case version if you want
     
  9. pamamolf

    pamamolf Premium Member Premium Member

    4,074
    427
    83
    May 31, 2014
    Ratings:
    +833
    Local Time:
    6:44 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    Only for lower and uppercase or for GrabNet-dev also?
     
  10. eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:44 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    it should but you can use curl and pass your custom user agents to test if it gets blocked i.e. for google image bot access site (curl --help for more info)
    Code:
    curl -I -A "Googlebot-Image/1.0" http://h2ohttp2.centminmod.com
     
  11. pamamolf

    pamamolf Premium Member Premium Member

    4,074
    427
    83
    May 31, 2014
    Ratings:
    +833
    Local Time:
    6:44 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    Ok great thanks :)

    After running this command i will get an output on ssh like 403 forbidden or i must check on access.log to see if it gets blocked?
     
  12. eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:44 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    curl headers -I gives the status code in output
     
  13. pamamolf

    pamamolf Premium Member Premium Member

    4,074
    427
    83
    May 31, 2014
    Ratings:
    +833
    Local Time:
    6:44 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    I got this bot: WeSEE:Ads/PageBot

    Should i block it as ~ WeSEE or ~ WeSEE:Ads ?

    Is there any general rule for this so i don't ask always about this?
    For example i use always what is before / ?

    Thanks
     
  14. eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:44 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    no general rule best to use curl to test your configured blocked user agents