Learn about Centmin Mod LEMP Stack today
Become a Member

Nginx Blocking a bot

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by pamamolf, Mar 29, 2015.

  1. pamamolf

    pamamolf Well-Known Member

    4,028
    421
    83
    May 31, 2014
    Ratings:
    +817
    Local Time:
    5:12 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Hi
    I found on logs this:

    Code:
    omgilibot/0.4
    and i want to block this user agent but i can't find any info for it....


    So can i block it as user agent omgilibot/0.4 or as omgilibot ?
     
  2. rdan

    rdan Well-Known Member

    5,426
    1,389
    113
    May 25, 2014
    Ratings:
    +2,172
    Local Time:
    11:12 PM
    Mainline
    10.2
  3. pamamolf

    pamamolf Well-Known Member

    4,028
    421
    83
    May 31, 2014
    Ratings:
    +817
    Local Time:
    5:12 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    I know how but is the name of the agent omgilibot/0.4 or omgilibot ?

    I have many bots that end with botname/0.1.4 or botname/4.0 and i don't know as i can't find any info on the net how to block it....with the /x.x.x or without?
     
  4. eva2000

    eva2000 Administrator Staff Member

    50,923
    11,806
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,264
    Local Time:
    1:12 AM
    Nginx 1.25.x
    MariaDB 10.x
    just the name is enough
     
  5. pamamolf

    pamamolf Well-Known Member

    4,028
    421
    83
    May 31, 2014
    Ratings:
    +817
    Local Time:
    5:12 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    One last question :)

    The user agent must match exact the name to be blocked or not?

    For example if i got this: botname-dev and i add to block it as botname without the -dev it will be blocked?
     
  6. eva2000

    eva2000 Administrator Staff Member

    50,923
    11,806
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,264
    Local Time:
    1:12 AM
    Nginx 1.25.x
    MariaDB 10.x
    see what ~ and ~* matching does in nginx Module ngx_http_rewrite_module

     
  7. pamamolf

    pamamolf Well-Known Member

    4,028
    421
    83
    May 31, 2014
    Ratings:
    +817
    Local Time:
    5:12 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Just to be sure to archive what i ask for i should change this:

    Code:
      if ($http_user_agent ~ "GrabNet") {
      set $block_user_agents 1;
      }
    to this?

    Code:
      if ($http_user_agent ~* "GrabNet") {
      set $block_user_agents 1;
      }
     
  8. eva2000

    eva2000 Administrator Staff Member

    50,923
    11,806
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,264
    Local Time:
    1:12 AM
    Nginx 1.25.x
    MariaDB 10.x
    yeah that would match against lower and upper case version if you want
     
  9. pamamolf

    pamamolf Well-Known Member

    4,028
    421
    83
    May 31, 2014
    Ratings:
    +817
    Local Time:
    5:12 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Only for lower and uppercase or for GrabNet-dev also?
     
  10. eva2000

    eva2000 Administrator Staff Member

    50,923
    11,806
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,264
    Local Time:
    1:12 AM
    Nginx 1.25.x
    MariaDB 10.x
    it should but you can use curl and pass your custom user agents to test if it gets blocked i.e. for google image bot access site (curl --help for more info)
    Code:
    curl -I -A "Googlebot-Image/1.0" http://h2ohttp2.centminmod.com
     
  11. pamamolf

    pamamolf Well-Known Member

    4,028
    421
    83
    May 31, 2014
    Ratings:
    +817
    Local Time:
    5:12 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Ok great thanks :)

    After running this command i will get an output on ssh like 403 forbidden or i must check on access.log to see if it gets blocked?
     
  12. eva2000

    eva2000 Administrator Staff Member

    50,923
    11,806
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,264
    Local Time:
    1:12 AM
    Nginx 1.25.x
    MariaDB 10.x
    curl headers -I gives the status code in output
     
  13. pamamolf

    pamamolf Well-Known Member

    4,028
    421
    83
    May 31, 2014
    Ratings:
    +817
    Local Time:
    5:12 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    I got this bot: WeSEE:Ads/PageBot

    Should i block it as ~ WeSEE or ~ WeSEE:Ads ?

    Is there any general rule for this so i don't ask always about this?
    For example i use always what is before / ?

    Thanks
     
  14. eva2000

    eva2000 Administrator Staff Member

    50,923
    11,806
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,264
    Local Time:
    1:12 AM
    Nginx 1.25.x
    MariaDB 10.x
    no general rule best to use curl to test your configured blocked user agents