Join the community today
Register Now

Quick tools to perform Server Auditing

Discussion in 'System Administration' started by latest-release, Feb 1, 2020.

  1. latest-release

    latest-release New Member

    3
    0
    1
    Feb 1, 2020
    Ratings:
    +0
    Local Time:
    12:46 PM
    1.14
    Once in a while it's important to perform Auditing on your servers to ensure that you are up to date and there is no information you are protecting lurking out on the darkweb.

    Today am going to discuss when you theHarvester one of the top OSINT tools for auditing your servers and your Organization external threats.

    TheHarvester is a tool designed to be used in the early stages of penetration testing. It is an opensource intelligence tool, that gathering information from opensource tools online, it supports of 20+ public sources some of those sources include;
    • google
    • bing
    • baidu
    • censys
    • securitytrail
    • ....and more....
    Some of the main features these OSINT tools theharvester include;

    Installing theHarvester

    Code:
    pip3 install theHarvester
    Installing theHarvester using virtualenv

    Code:
    sudo apt-get install python3-pip
    sudo pip3 install virtualenv
    
    #
    # Create virtualenv
    
    virtualenv venv
    
    # Or create a python version specific virtualenv
    
    virtualenv -p python3 myenv
    #
    # Now clone the git repo
    git clone https://github.com/laramies/theHarvester.git
    #
    pip3 install -r requirements.txt
    #
    # Wait until the installation is done.
    Usage

    Code:
    $
    $ ./theharvester.py # If you cloned from github
    $ # or
    $ theharvester # if you installed using pip3 install theHarvester
    $
    $
    *******************************************************************
    *  _   _                                            _             *
    * | |_| |__   ___    /\  /\__ _ _ ____   _____  ___| |_ ___ _ __  *
    * | __|  _ \ / _ \  / /_/ / _` | '__\ \ / / _ \/ __| __/ _ \ '__| *
    * | |_| | | |  __/ / __  / (_| | |   \ V /  __/\__ \ ||  __/ |    *
    *  \__|_| |_|\___| \/ /_/ \__,_|_|    \_/ \___||___/\__\___|_|    *
    *                                                                 *
    * theHarvester 3.1.1dev3                                          *
    * Coded by Christian Martorella                                   *
    * Edge-Security Research                                          *
    * [email protected]                                   *
    *                                                                 *
    *******************************************************************
    
    
    usage: theHarvester.py [-h] -d DOMAIN [-l LIMIT] [-S START] [-g] [-p] [-s]
                           [-v] [-e DNS_SERVER] [-t DNS_TLD] [-n] [-c]
                           [-f FILENAME] [-b SOURCE]
    theHarvester.py: error: the following arguments are required: -d/--domain

    Code:
    $
    $ ./theHarvester.py -d google.com -b google
    $
    *******************************************************************
    *  _   _                                            _             *
    * | |_| |__   ___    /\  /\__ _ _ ____   _____  ___| |_ ___ _ __  *
    * | __|  _ \ / _ \  / /_/ / _` | '__\ \ / / _ \/ __| __/ _ \ '__| *
    * | |_| | | |  __/ / __  / (_| | |   \ V /  __/\__ \ ||  __/ |    *
    *  \__|_| |_|\___| \/ /_/ \__,_|_|    \_/ \___||___/\__\___|_|    *
    *                                                                 *
    * theHarvester 3.1.1dev3                                          *
    * Coded by Christian Martorella                                   *
    * Edge-Security Research                                          *
    * [email protected]                                   *
    *                                                                 *
    *******************************************************************
    
    
    [*] Target: google.com
     
    [*] Searching Google.
        Searching 0 results.
        Searching 100 results.
        Searching 200 results.
        Searching 300 results.
        Searching 400 results.
        Searching 500 results.
    
    [*] No IPs found.
    
    [*] No emails found.
    
    [*] Hosts found: 23
    ---------------------
    aboutme.google.com:216.58.223.110
    accounts.google.com:216.58.223.109
    adservice.google.com:216.58.223.98
    .......
    There are more to this tool you can explore by yourself
     
  2. eva2000

    eva2000 Administrator Staff Member

    43,132
    9,792
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,122
    Local Time:
    7:46 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    wow thanks for heads up on this tool. Looks very interesting. Seems you're instructions are for Ubunut/Debian ? Problem for Centmin Mod users is they'd use CentOS and CentOS 7 only has native Python 2.7 and addons/python36_install.sh is highest version installed via IUS Community Yum repo for Python 3.6.

    From their github repo's wiki install guide you need Python 3.7+ for it to work laramies/theHarvester. So Centmin Mod and CentOS 7 won't have the required Python 3.7+ versions for such an install. However, you can install Python 3.7 manually too and seems to work in running theHarvester tool.

    If folks want to try this on CentOS 7 with Centmin Mod, I'd test on a test VPS these steps rather than on live site VPS so as to not screw anything up. Easy enough just spin up a test hourly billed VPS host like vultr, digitalocean, linode or like highly recommended Upcloud - signees also get US$25 credits to use as well :)

    Install Python 3.7 first.

    The compilation can take a long time to install especially on slow servers or single cpu threaded servers - could take hours on very slow servers!
    Code (Text):
    # update Centmin Mod code
    cmupdate
    
    # pip doesn't like ccache so disable it
    export CC='gcc'
    export CXX="g++"
    
    # pip needs a tmp directory that doesn't have noexec restrictions
    mkdir -p /home/piptmp
    chmod 1777 /home/piptmp
    export TMPDIR=/home/piptmp
    
    # install python 3.7.6 or 3.8.1
    yum -y install libffi-devel
    python_ver=3.7.6
    python_prefixver=$(echo $python_ver | cut -d . -f1,2)
    cd /svr-setup
    wget https://www.python.org/ftp/python/${python_ver}/Python-${python_ver}.tgz
    tar xzf Python-${python_ver}.tgz
    cd Python-${python_ver}
    make clean
    if [[ "$(nproc)" -le '2' ]]; then time ./configure --prefix=/opt/python${python_prefixver} --with-openssl=/usr; else time ./configure --enable-optimizations --prefix=/opt/python${python_prefixver} --with-openssl=/usr; fi
    time make -j$(nproc)
    time make altinstall
    ls -lah /opt/python${python_prefixver}/bin
    ln -s /opt/python${python_prefixver}/bin/python${python_prefixver} /opt/python${python_prefixver}/bin/python3
    ln -s /opt/python${python_prefixver}/bin/python${python_prefixver} /opt/python${python_prefixver}/bin/python
    ln -s /opt/python${python_prefixver}/bin/python${python_prefixver} /usr/bin/python${python_prefixver}
    ln -s /opt/python${python_prefixver}/bin/pip${python_prefixver} /opt/python${python_prefixver}/bin/pip3
    ln -s /opt/python${python_prefixver}/bin/pip${python_prefixver} /opt/python${python_prefixver}/bin/pip
    ln -s /opt/python${python_prefixver}/bin/easy_install-${python_prefixver} /opt/python${python_prefixver}/bin/easy_install
    ln -s /opt/python${python_prefixver}/bin/idle${python_prefixver} /opt/python${python_prefixver}/bin/idle
    ln -s /opt/python${python_prefixver}/bin/2to3-${python_prefixver} /opt/python${python_prefixver}/bin/2to3
    ln -s /opt/python${python_prefixver}/bin/pydoc${python_prefixver} /opt/python${python_prefixver}/bin/pydoc
    ln -s /opt/python${python_prefixver}/bin/pyvenv-${python_prefixver} /opt/python${python_prefixver}/bin/pyvenv
    rm -f /svr-setup/Python-${python_ver}.tgz
    /opt/python${python_prefixver}/bin/python${python_prefixver} --version
    /opt/python${python_prefixver}/bin/pip${python_prefixver} --version

    Python 3.7 will be installed at
    Code (Text):
    /opt/python3.7/bin/python --version
    Python 3.7.6
    
    /opt/python3.7/bin/pip --version  
    pip 19.2.3 from /opt/python3.7/lib/python3.7/site-packages/pip (python 3.7)
    

    Code (Text):
    ls -lah /opt/python3.7/bin
    total 27M
    drwxr-xr-x 2 root root 4.0K Jan 31 20:15 .
    drwxr-xr-x 6 root root   56 Jan 31 18:58 ..
    lrwxrwxrwx 1 root root   27 Jan 31 19:02 2to3 -> /opt/python3.7/bin/2to3-3.7
    -rwxr-xr-x 1 root root  105 Jan 31 20:15 2to3-3.7
    lrwxrwxrwx 1 root root   35 Jan 31 19:02 easy_install -> /opt/python3.7/bin/easy_install-3.7
    -rwxr-xr-x 1 root root  245 Jan 31 18:58 easy_install-3.7
    lrwxrwxrwx 1 root root   26 Jan 31 19:04 idle -> /opt/python3.7/bin/idle3.7
    -rwxr-xr-x 1 root root  103 Jan 31 20:15 idle3.7
    lrwxrwxrwx 1 root root   25 Jan 31 19:02 pip -> /opt/python3.7/bin/pip3.7
    lrwxrwxrwx 1 root root   25 Jan 31 19:02 pip3 -> /opt/python3.7/bin/pip3.7
    -rwxr-xr-x 1 root root  227 Jan 31 18:58 pip3.7
    -rwxr-xr-x 1 root root  218 Jan 31 19:24 pipenv
    -rwxr-xr-x 1 root root  229 Jan 31 19:24 pipenv-resolver
    lrwxrwxrwx 1 root root   27 Jan 31 19:04 pydoc -> /opt/python3.7/bin/pydoc3.7
    -rwxr-xr-x 1 root root   88 Jan 31 20:15 pydoc3.7
    lrwxrwxrwx 1 root root   28 Jan 31 19:02 python -> /opt/python3.7/bin/python3.7
    lrwxrwxrwx 1 root root   28 Jan 31 19:02 python3 -> /opt/python3.7/bin/python3.7
    -rwxr-xr-x 2 root root  14M Jan 31 20:12 python3.7
    -rwxr-xr-x 2 root root  14M Jan 31 20:12 python3.7m
    -rwxr-xr-x 1 root root 2.9K Jan 31 20:15 python3.7m-config
    lrwxrwxrwx 1 root root   29 Jan 31 19:02 pyvenv -> /opt/python3.7/bin/pyvenv-3.7
    -rwxr-xr-x 1 root root  445 Jan 31 20:15 pyvenv-3.7
    -rwxr-xr-x 1 root root  224 Jan 31 19:24 virtualenv
    -rwxr-xr-x 1 root root  229 Jan 31 19:24 virtualenv-clone
    

    Install theHarvester
    Code (Text):
    python_ver=3.7.6
    python_prefixver=$(echo $python_ver | cut -d . -f1,2)
    mkdir -p /home/python_projects
    cd /home/python_projects
    git clone https://github.com/laramies/theHarvester theharvester
    cd theharvester
    /opt/python${python_prefixver}/bin/python${python_prefixver} -m venv /home/python_projects/theharvester/pipenv
    source /home/python_projects/theharvester/pipenv/bin/activate
    pip install -U pip
    pip install -U setuptools
    pip install -r requirements.txt
    pip list
    python theHarvester.py -h

    Code (Text):
    pip list
    Package            Version
    ------------------ ----------
    aiodns             2.0.0
    atomicwrites       1.3.0
    attrs              19.3.0
    beautifulsoup4     4.8.0
    certifi            2019.11.28
    cffi               1.13.2
    chardet            3.0.4
    Click              7.0
    click-plugins      1.1.1
    colorama           0.4.3
    dnspython          1.16.0
    entrypoints        0.3
    flake8             3.7.8
    gevent             1.4.0
    greenlet           0.4.15
    grequests          0.4.0
    idna               2.8
    importlib-metadata 1.5.0
    mccabe             0.6.1
    more-itertools     8.2.0
    mypy               0.740
    mypy-extensions    0.4.3
    netaddr            0.7.19
    packaging          20.1
    pip                20.0.2
    plotly             4.2.1
    pluggy             0.13.1
    py                 1.8.1
    pycares            3.1.1
    pycodestyle        2.5.0
    pycparser          2.19
    pyflakes           2.1.1
    pyparsing          2.4.6
    pytest             5.2.0
    PyYAML             5.1.2
    requests           2.22.0
    retrying           1.3.3
    setuptools         45.1.0
    shodan             1.19.0
    six                1.14.0
    soupsieve          1.9.5
    texttable          1.6.2
    typed-ast          1.4.1
    typing-extensions  3.7.4.1
    urllib3            1.25.8
    wcwidth            0.1.8
    XlsxWriter         1.2.7
    zipp               2.1.0
    

    run from /home/python_projects/theharvester directory
    Code (Text):
    cd /home/python_projects/theharvester/
    source /home/python_projects/theharvester/pipenv/bin/activate
    python theHarvester.py -h
    

    Code (Text):
    python theHarvester.py -h
    *******************************************************************
    *  _   _                                            _             *
    * | |_| |__   ___    /\  /\__ _ _ ____   _____  ___| |_ ___ _ __  *
    * | __|  _ \ / _ \  / /_/ / _` | '__\ \ / / _ \/ __| __/ _ \ '__| *
    * | |_| | | |  __/ / __  / (_| | |   \ V /  __/\__ \ ||  __/ |    *
    *  \__|_| |_|\___| \/ /_/ \__,_|_|    \_/ \___||___/\__\___|_|    *
    *                                                                 *
    * theHarvester 3.1.1dev3                                          *
    * Coded by Christian Martorella                                   *
    * Edge-Security Research                                          *
    * [email protected]                                   *
    *                                                                 *
    *******************************************************************
    usage: theHarvester.py [-h] -d DOMAIN [-l LIMIT] [-S START] [-g] [-p] [-s]
                           [-v] [-e DNS_SERVER] [-t DNS_TLD] [-n] [-c]
                           [-f FILENAME] [-b SOURCE]
    theHarvester is used to gather open source intelligence (OSINT) on a company
    or domain.
    optional arguments:
      -h, --help            show this help message and exit
      -d DOMAIN, --domain DOMAIN
                            company name or domain to search
      -l LIMIT, --limit LIMIT
                            limit the number of search results, default=500
      -S START, --start START
                            start with result number X, default=0
      -g, --google-dork     use Google Dorks for Google search
      -p, --port-scan       scan the detected hosts and check for Takeovers
                            (21,22,80,443,8080)
      -s, --shodan          use Shodan to query discovered hosts
      -v, --virtual-host    verify host name via DNS resolution and search for
                            virtual hosts
      -e DNS_SERVER, --dns-server DNS_SERVER
                            DNS server to use for lookup
      -t DNS_TLD, --dns-tld DNS_TLD
                            perform a DNS TLD expansion discovery, default False
      -n, --dns-lookup      enable DNS server lookup, default False
      -c, --dns-brute       perform a DNS brute force on the domain
      -f FILENAME, --filename FILENAME
                            save the results to an HTML and/or XML file
      -b SOURCE, --source SOURCE
                            baidu, bing, bingapi, certspotter, crtsh, dnsdumpster,
                            dogpile, duckduckgo, github-code, google, hunter,
                            intelx, linkedin, linkedin_links, netcraft, otx,
                            securityTrails, spyse(disabled for now), threatcrowd,
                            trello, twitter, vhost, virustotal, yahoo, all
    

    Installation at your own risk, so try only on test VPS and not live servers.
     
  3. latest-release

    latest-release New Member

    3
    0
    1
    Feb 1, 2020
    Ratings:
    +0
    Local Time:
    12:46 PM
    1.14
    Great compilation instructions. Though I thought python2.7 has already been dropped by all
     
  4. eva2000

    eva2000 Administrator Staff Member

    43,132
    9,792
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,122
    Local Time:
    7:46 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x