Quick tools to perform Server Auditing

latest-release · Feb 1, 2020

Once in a while it's important to perform Auditing on your servers to ensure that you are up to date and there is no information you are protecting lurking out on the darkweb.

Today am going to discuss when you theHarvester one of the top OSINT tools for auditing your servers and your Organization external threats.

TheHarvester is a tool designed to be used in the early stages of penetration testing. It is an opensource intelligence tool, that gathering information from opensource tools online, it supports of 20+ public sources some of those sources include;

google
bing
baidu
censys
securitytrail
....and more....

Some of the main features these OSINT tools theharvester include;

Names ( for people from social networks)
Virtualhost
Subdomain finder
IPs
Port scanning
Emails

Installing theHarvester

Code:

pip3 install theHarvester

Installing theHarvester using virtualenv

Code:

sudo apt-get install python3-pip
sudo pip3 install virtualenv

#
# Create virtualenv

virtualenv venv

# Or create a python version specific virtualenv

virtualenv -p python3 myenv
#
# Now clone the git repo
git clone https://github.com/laramies/theHarvester.git
#
pip3 install -r requirements.txt
#
# Wait until the installation is done.

Usage

Code:

$
$ ./theharvester.py # If you cloned from github
$ # or
$ theharvester # if you installed using pip3 install theHarvester
$
$
*******************************************************************
*  _   _                                            _             *
* | |_| |__   ___    /\  /\__ _ _ ____   _____  ___| |_ ___ _ __  *
* | __|  _ \ / _ \  / /_/ / _` | '__\ \ / / _ \/ __| __/ _ \ '__| *
* | |_| | | |  __/ / __  / (_| | |   \ V /  __/\__ \ ||  __/ |    *
*  \__|_| |_|\___| \/ /_/ \__,_|_|    \_/ \___||___/\__\___|_|    *
*                                                                 *
* theHarvester 3.1.1dev3                                          *
* Coded by Christian Martorella                                   *
* Edge-Security Research                                          *
* cmartorella@edge-security.com                                   *
*                                                                 *
*******************************************************************


usage: theHarvester.py [-h] -d DOMAIN [-l LIMIT] [-S START] [-g] [-p] [-s]
                       [-v] [-e DNS_SERVER] [-t DNS_TLD] [-n] [-c]
                       [-f FILENAME] [-b SOURCE]
theHarvester.py: error: the following arguments are required: -d/--domain

Code:

$
$ ./theHarvester.py -d google.com -b google
$
*******************************************************************
*  _   _                                            _             *
* | |_| |__   ___    /\  /\__ _ _ ____   _____  ___| |_ ___ _ __  *
* | __|  _ \ / _ \  / /_/ / _` | '__\ \ / / _ \/ __| __/ _ \ '__| *
* | |_| | | |  __/ / __  / (_| | |   \ V /  __/\__ \ ||  __/ |    *
*  \__|_| |_|\___| \/ /_/ \__,_|_|    \_/ \___||___/\__\___|_|    *
*                                                                 *
* theHarvester 3.1.1dev3                                          *
* Coded by Christian Martorella                                   *
* Edge-Security Research                                          *
* cmartorella@edge-security.com                                   *
*                                                                 *
*******************************************************************


[*] Target: google.com
 
[*] Searching Google.
    Searching 0 results.
    Searching 100 results.
    Searching 200 results.
    Searching 300 results.
    Searching 400 results.
    Searching 500 results.

[*] No IPs found.

[*] No emails found.

[*] Hosts found: 23
---------------------
aboutme.google.com:216.58.223.110
accounts.google.com:216.58.223.109
adservice.google.com:216.58.223.98
.......

There are more to this tool you can explore by yourself

eva2000 · Feb 1, 2020

wow thanks for heads up on this tool. Looks very interesting. Seems you're instructions are for Ubunut/Debian ? Problem for Centmin Mod users is they'd use CentOS and CentOS 7 only has native Python 2.7 and addons/python36_install.sh is highest version installed via IUS Community Yum repo for Python 3.6.

From their github repo's wiki install guide you need Python 3.7+ for it to work laramies/theHarvester. So Centmin Mod and CentOS 7 won't have the required Python 3.7+ versions for such an install. However, you can install Python 3.7 manually too and seems to work in running theHarvester tool.

If folks want to try this on CentOS 7 with Centmin Mod, I'd test on a test VPS these steps rather than on live site VPS so as to not screw anything up. Easy enough just spin up a test hourly billed VPS host like vultr, digitalocean, linode or like highly recommended Upcloud - signees also get US$25 credits to use as well

Install Python 3.7 first.

The compilation can take a long time to install especially on slow servers or single cpu threaded servers - could take hours on very slow servers!

Code (Text):

# update Centmin Mod code
cmupdate

# pip doesn't like ccache so disable it
export CC='gcc'
export CXX="g++"

# pip needs a tmp directory that doesn't have noexec restrictions
mkdir -p /home/piptmp
chmod 1777 /home/piptmp
export TMPDIR=/home/piptmp

# install python 3.7.6 or 3.8.1
yum -y install libffi-devel
python_ver=3.7.6
python_prefixver=$(echo $python_ver | cut -d . -f1,2)
cd /svr-setup
wget https://www.python.org/ftp/python/${python_ver}/Python-${python_ver}.tgz
tar xzf Python-${python_ver}.tgz
cd Python-${python_ver}
make clean
if [[ "$(nproc)" -le '2' ]]; then time ./configure --prefix=/opt/python${python_prefixver} --with-openssl=/usr; else time ./configure --enable-optimizations --prefix=/opt/python${python_prefixver} --with-openssl=/usr; fi
time make -j$(nproc)
time make altinstall
ls -lah /opt/python${python_prefixver}/bin
ln -s /opt/python${python_prefixver}/bin/python${python_prefixver} /opt/python${python_prefixver}/bin/python3
ln -s /opt/python${python_prefixver}/bin/python${python_prefixver} /opt/python${python_prefixver}/bin/python
ln -s /opt/python${python_prefixver}/bin/python${python_prefixver} /usr/bin/python${python_prefixver}
ln -s /opt/python${python_prefixver}/bin/pip${python_prefixver} /opt/python${python_prefixver}/bin/pip3
ln -s /opt/python${python_prefixver}/bin/pip${python_prefixver} /opt/python${python_prefixver}/bin/pip
ln -s /opt/python${python_prefixver}/bin/easy_install-${python_prefixver} /opt/python${python_prefixver}/bin/easy_install
ln -s /opt/python${python_prefixver}/bin/idle${python_prefixver} /opt/python${python_prefixver}/bin/idle
ln -s /opt/python${python_prefixver}/bin/2to3-${python_prefixver} /opt/python${python_prefixver}/bin/2to3
ln -s /opt/python${python_prefixver}/bin/pydoc${python_prefixver} /opt/python${python_prefixver}/bin/pydoc
ln -s /opt/python${python_prefixver}/bin/pyvenv-${python_prefixver} /opt/python${python_prefixver}/bin/pyvenv
rm -f /svr-setup/Python-${python_ver}.tgz
/opt/python${python_prefixver}/bin/python${python_prefixver} --version
/opt/python${python_prefixver}/bin/pip${python_prefixver} --version

Python 3.7 will be installed at

Code (Text):

/opt/python3.7/bin/python --version
Python 3.7.6

/opt/python3.7/bin/pip --version  
pip 19.2.3 from /opt/python3.7/lib/python3.7/site-packages/pip (python 3.7)

Code (Text):

ls -lah /opt/python3.7/bin
total 27M
drwxr-xr-x 2 root root 4.0K Jan 31 20:15 .
drwxr-xr-x 6 root root   56 Jan 31 18:58 ..
lrwxrwxrwx 1 root root   27 Jan 31 19:02 2to3 -> /opt/python3.7/bin/2to3-3.7
-rwxr-xr-x 1 root root  105 Jan 31 20:15 2to3-3.7
lrwxrwxrwx 1 root root   35 Jan 31 19:02 easy_install -> /opt/python3.7/bin/easy_install-3.7
-rwxr-xr-x 1 root root  245 Jan 31 18:58 easy_install-3.7
lrwxrwxrwx 1 root root   26 Jan 31 19:04 idle -> /opt/python3.7/bin/idle3.7
-rwxr-xr-x 1 root root  103 Jan 31 20:15 idle3.7
lrwxrwxrwx 1 root root   25 Jan 31 19:02 pip -> /opt/python3.7/bin/pip3.7
lrwxrwxrwx 1 root root   25 Jan 31 19:02 pip3 -> /opt/python3.7/bin/pip3.7
-rwxr-xr-x 1 root root  227 Jan 31 18:58 pip3.7
-rwxr-xr-x 1 root root  218 Jan 31 19:24 pipenv
-rwxr-xr-x 1 root root  229 Jan 31 19:24 pipenv-resolver
lrwxrwxrwx 1 root root   27 Jan 31 19:04 pydoc -> /opt/python3.7/bin/pydoc3.7
-rwxr-xr-x 1 root root   88 Jan 31 20:15 pydoc3.7
lrwxrwxrwx 1 root root   28 Jan 31 19:02 python -> /opt/python3.7/bin/python3.7
lrwxrwxrwx 1 root root   28 Jan 31 19:02 python3 -> /opt/python3.7/bin/python3.7
-rwxr-xr-x 2 root root  14M Jan 31 20:12 python3.7
-rwxr-xr-x 2 root root  14M Jan 31 20:12 python3.7m
-rwxr-xr-x 1 root root 2.9K Jan 31 20:15 python3.7m-config
lrwxrwxrwx 1 root root   29 Jan 31 19:02 pyvenv -> /opt/python3.7/bin/pyvenv-3.7
-rwxr-xr-x 1 root root  445 Jan 31 20:15 pyvenv-3.7
-rwxr-xr-x 1 root root  224 Jan 31 19:24 virtualenv
-rwxr-xr-x 1 root root  229 Jan 31 19:24 virtualenv-clone

Install theHarvester

Code (Text):

python_ver=3.7.6
python_prefixver=$(echo $python_ver | cut -d . -f1,2)
mkdir -p /home/python_projects
cd /home/python_projects
git clone https://github.com/laramies/theHarvester theharvester
cd theharvester
/opt/python${python_prefixver}/bin/python${python_prefixver} -m venv /home/python_projects/theharvester/pipenv
source /home/python_projects/theharvester/pipenv/bin/activate
pip install -U pip
pip install -U setuptools
pip install -r requirements.txt
pip list
python theHarvester.py -h

Code (Text):

pip list
Package            Version
------------------ ----------
aiodns             2.0.0
atomicwrites       1.3.0
attrs              19.3.0
beautifulsoup4     4.8.0
certifi            2019.11.28
cffi               1.13.2
chardet            3.0.4
Click              7.0
click-plugins      1.1.1
colorama           0.4.3
dnspython          1.16.0
entrypoints        0.3
flake8             3.7.8
gevent             1.4.0
greenlet           0.4.15
grequests          0.4.0
idna               2.8
importlib-metadata 1.5.0
mccabe             0.6.1
more-itertools     8.2.0
mypy               0.740
mypy-extensions    0.4.3
netaddr            0.7.19
packaging          20.1
pip                20.0.2
plotly             4.2.1
pluggy             0.13.1
py                 1.8.1
pycares            3.1.1
pycodestyle        2.5.0
pycparser          2.19
pyflakes           2.1.1
pyparsing          2.4.6
pytest             5.2.0
PyYAML             5.1.2
requests           2.22.0
retrying           1.3.3
setuptools         45.1.0
shodan             1.19.0
six                1.14.0
soupsieve          1.9.5
texttable          1.6.2
typed-ast          1.4.1
typing-extensions  3.7.4.1
urllib3            1.25.8
wcwidth            0.1.8
XlsxWriter         1.2.7
zipp               2.1.0

run from /home/python_projects/theharvester directory

Code (Text):

cd /home/python_projects/theharvester/
source /home/python_projects/theharvester/pipenv/bin/activate
python theHarvester.py -h

Code (Text):

python theHarvester.py -h
*******************************************************************
*  _   _                                            _             *
* | |_| |__   ___    /\  /\__ _ _ ____   _____  ___| |_ ___ _ __  *
* | __|  _ \ / _ \  / /_/ / _` | '__\ \ / / _ \/ __| __/ _ \ '__| *
* | |_| | | |  __/ / __  / (_| | |   \ V /  __/\__ \ ||  __/ |    *
*  \__|_| |_|\___| \/ /_/ \__,_|_|    \_/ \___||___/\__\___|_|    *
*                                                                 *
* theHarvester 3.1.1dev3                                          *
* Coded by Christian Martorella                                   *
* Edge-Security Research                                          *
* cmartorella@edge-security.com                                   *
*                                                                 *
*******************************************************************
usage: theHarvester.py [-h] -d DOMAIN [-l LIMIT] [-S START] [-g] [-p] [-s]
                       [-v] [-e DNS_SERVER] [-t DNS_TLD] [-n] [-c]
                       [-f FILENAME] [-b SOURCE]
theHarvester is used to gather open source intelligence (OSINT) on a company
or domain.
optional arguments:
  -h, --help            show this help message and exit
  -d DOMAIN, --domain DOMAIN
                        company name or domain to search
  -l LIMIT, --limit LIMIT
                        limit the number of search results, default=500
  -S START, --start START
                        start with result number X, default=0
  -g, --google-dork     use Google Dorks for Google search
  -p, --port-scan       scan the detected hosts and check for Takeovers
                        (21,22,80,443,8080)
  -s, --shodan          use Shodan to query discovered hosts
  -v, --virtual-host    verify host name via DNS resolution and search for
                        virtual hosts
  -e DNS_SERVER, --dns-server DNS_SERVER
                        DNS server to use for lookup
  -t DNS_TLD, --dns-tld DNS_TLD
                        perform a DNS TLD expansion discovery, default False
  -n, --dns-lookup      enable DNS server lookup, default False
  -c, --dns-brute       perform a DNS brute force on the domain
  -f FILENAME, --filename FILENAME
                        save the results to an HTML and/or XML file
  -b SOURCE, --source SOURCE
                        baidu, bing, bingapi, certspotter, crtsh, dnsdumpster,
                        dogpile, duckduckgo, github-code, google, hunter,
                        intelx, linkedin, linkedin_links, netcraft, otx,
                        securityTrails, spyse(disabled for now), threatcrowd,
                        trello, twitter, vhost, virustotal, yahoo, all

Installation at your own risk, so try only on test VPS and not live servers.

latest-release · Feb 4, 2020

Great compilation instructions. Though I thought python2.7 has already been dropped by all

eva2000 · Feb 4, 2020

Nope IIRC Redhat/CentOS 7.x has python 2.7 until 2024 and Redhat/CentOS 8.x is python 3.6 Python in RHEL 8 - Red Hat Developer

Log in / Register

Forums

Learn about Centmin Mod LEMP Stack today

Quick tools to perform Server Auditing

latest-release New Member

eva2000 Administrator Staff Member

latest-release New Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Learn about Centmin Mod LEMP Stack today

Quick tools to perform Server Auditing

latest-release New Member

eva2000 Administrator Staff Member

latest-release New Member

eva2000 Administrator Staff Member

.