Want to subscribe to topics you're interested in?
Become a Member

Nginx Blocking a bot

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by pamamolf, Mar 29, 2015.

  1. pamamolf

    pamamolf Well-Known Member

    2,724
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    2:45 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Hi
    I found on logs this:

    Code:
    omgilibot/0.4
    and i want to block this user agent but i can't find any info for it....

    So can i block it as user agent omgilibot/0.4 or as omgilibot ?
     
  2. RoldanLT

    RoldanLT Well-Known Member

    3,901
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    7:45 PM
    1.11
    10.2
    • Like Like x 1
  3. pamamolf

    pamamolf Well-Known Member

    2,724
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    2:45 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    I know how but is the name of the agent omgilibot/0.4 or omgilibot ?

    I have many bots that end with botname/0.1.4 or botname/4.0 and i don't know as i can't find any info on the net how to block it....with the /x.x.x or without?
     
  4. eva2000

    eva2000 Administrator Staff Member

    30,170
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,138
    Local Time:
    9:45 PM
    Nginx 1.13.x
    MariaDB 5.5
    just the name is enough
     
    • Like Like x 1
  5. pamamolf

    pamamolf Well-Known Member

    2,724
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    2:45 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    One last question :)

    The user agent must match exact the name to be blocked or not?

    For example if i got this: botname-dev and i add to block it as botname without the -dev it will be blocked?
     
  6. eva2000

    eva2000 Administrator Staff Member

    30,170
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,138
    Local Time:
    9:45 PM
    Nginx 1.13.x
    MariaDB 5.5
    see what ~ and ~* matching does in nginx Module ngx_http_rewrite_module

     
  7. pamamolf

    pamamolf Well-Known Member

    2,724
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    2:45 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Just to be sure to archive what i ask for i should change this:

    Code:
      if ($http_user_agent ~ "GrabNet") {
      set $block_user_agents 1;
      }
    to this?

    Code:
      if ($http_user_agent ~* "GrabNet") {
      set $block_user_agents 1;
      }
     
  8. eva2000

    eva2000 Administrator Staff Member

    30,170
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,138
    Local Time:
    9:45 PM
    Nginx 1.13.x
    MariaDB 5.5
    yeah that would match against lower and upper case version if you want
     
  9. pamamolf

    pamamolf Well-Known Member

    2,724
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    2:45 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Only for lower and uppercase or for GrabNet-dev also?
     
  10. eva2000

    eva2000 Administrator Staff Member

    30,170
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,138
    Local Time:
    9:45 PM
    Nginx 1.13.x
    MariaDB 5.5
    it should but you can use curl and pass your custom user agents to test if it gets blocked i.e. for google image bot access site (curl --help for more info)
    Code:
    curl -I -A "Googlebot-Image/1.0" http://h2ohttp2.centminmod.com
     
  11. pamamolf

    pamamolf Well-Known Member

    2,724
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    2:45 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Ok great thanks :)

    After running this command i will get an output on ssh like 403 forbidden or i must check on access.log to see if it gets blocked?
     
  12. eva2000

    eva2000 Administrator Staff Member

    30,170
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,138
    Local Time:
    9:45 PM
    Nginx 1.13.x
    MariaDB 5.5
    curl headers -I gives the status code in output
     
    • Like Like x 1
  13. pamamolf

    pamamolf Well-Known Member

    2,724
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    2:45 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    I got this bot: WeSEE:Ads/PageBot

    Should i block it as ~ WeSEE or ~ WeSEE:Ads ?

    Is there any general rule for this so i don't ask always about this?
    For example i use always what is before / ?

    Thanks
     
  14. eva2000

    eva2000 Administrator Staff Member

    30,170
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,138
    Local Time:
    9:45 PM
    Nginx 1.13.x
    MariaDB 5.5
    no general rule best to use curl to test your configured blocked user agents