Want to subscribe to topics you're interested in?
Become a Member

Nginx Blocking a bot

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by pamamolf, Mar 29, 2015.

  1. pamamolf

    pamamolf Well-Known Member

    3,113
    295
    83
    May 31, 2014
    Ratings:
    +530
    Local Time:
    4:34 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Hi
    I found on logs this:

    Code:
    omgilibot/0.4
    and i want to block this user agent but i can't find any info for it....

    So can i block it as user agent omgilibot/0.4 or as omgilibot ?
     
  2. rdan

    rdan Premium Member Premium Member

    4,255
    1,034
    113
    May 25, 2014
    Ratings:
    +1,486
    Local Time:
    9:34 PM
    Mainline
    10.2
    • Like Like x 1
  3. pamamolf

    pamamolf Well-Known Member

    3,113
    295
    83
    May 31, 2014
    Ratings:
    +530
    Local Time:
    4:34 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    I know how but is the name of the agent omgilibot/0.4 or omgilibot ?

    I have many bots that end with botname/0.1.4 or botname/4.0 and i don't know as i can't find any info on the net how to block it....with the /x.x.x or without?
     
  4. eva2000

    eva2000 Administrator Staff Member

    36,041
    7,907
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,189
    Local Time:
    11:34 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    just the name is enough
     
    • Like Like x 1
  5. pamamolf

    pamamolf Well-Known Member

    3,113
    295
    83
    May 31, 2014
    Ratings:
    +530
    Local Time:
    4:34 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    One last question :)

    The user agent must match exact the name to be blocked or not?

    For example if i got this: botname-dev and i add to block it as botname without the -dev it will be blocked?
     
  6. eva2000

    eva2000 Administrator Staff Member

    36,041
    7,907
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,189
    Local Time:
    11:34 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    see what ~ and ~* matching does in nginx Module ngx_http_rewrite_module

     
  7. pamamolf

    pamamolf Well-Known Member

    3,113
    295
    83
    May 31, 2014
    Ratings:
    +530
    Local Time:
    4:34 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Just to be sure to archive what i ask for i should change this:

    Code:
      if ($http_user_agent ~ "GrabNet") {
      set $block_user_agents 1;
      }
    to this?

    Code:
      if ($http_user_agent ~* "GrabNet") {
      set $block_user_agents 1;
      }
     
  8. eva2000

    eva2000 Administrator Staff Member

    36,041
    7,907
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,189
    Local Time:
    11:34 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    yeah that would match against lower and upper case version if you want
     
  9. pamamolf

    pamamolf Well-Known Member

    3,113
    295
    83
    May 31, 2014
    Ratings:
    +530
    Local Time:
    4:34 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Only for lower and uppercase or for GrabNet-dev also?
     
  10. eva2000

    eva2000 Administrator Staff Member

    36,041
    7,907
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,189
    Local Time:
    11:34 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    it should but you can use curl and pass your custom user agents to test if it gets blocked i.e. for google image bot access site (curl --help for more info)
    Code:
    curl -I -A "Googlebot-Image/1.0" http://h2ohttp2.centminmod.com
     
  11. pamamolf

    pamamolf Well-Known Member

    3,113
    295
    83
    May 31, 2014
    Ratings:
    +530
    Local Time:
    4:34 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Ok great thanks :)

    After running this command i will get an output on ssh like 403 forbidden or i must check on access.log to see if it gets blocked?
     
  12. eva2000

    eva2000 Administrator Staff Member

    36,041
    7,907
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,189
    Local Time:
    11:34 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    curl headers -I gives the status code in output
     
    • Like Like x 1
  13. pamamolf

    pamamolf Well-Known Member

    3,113
    295
    83
    May 31, 2014
    Ratings:
    +530
    Local Time:
    4:34 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    I got this bot: WeSEE:Ads/PageBot

    Should i block it as ~ WeSEE or ~ WeSEE:Ads ?

    Is there any general rule for this so i don't ask always about this?
    For example i use always what is before / ?

    Thanks
     
  14. eva2000

    eva2000 Administrator Staff Member

    36,041
    7,907
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,189
    Local Time:
    11:34 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    no general rule best to use curl to test your configured blocked user agents
     
..