EXIST is a web application for aggregating and analyzing CTI (cyber threat intelligence). It is written by the following software.
It automatically fetches data from several CTI services and Twitter via their APIs and feeds. You can cross-search indicators via the web interface and the API.
If you have servers logging network behaviors of clients (e.g., logs of DNS and HTTP proxy servers, etc.), you will be able to analyze the logs by correlating with data on EXIST. If you implement some programs by using the API, you will realize automated CTI-driven security operation center.
Case1: Investigate domain detected by IDS
Just type domain in the search form.
Case2: Access the malicious URL on behalf of the user and acquire the display image of the browser and the contents to be downloaded
Just type url in the search form.
Case3: Monitor cyber threats
Just add keywords in the Threat Hunter or Twitter Hunter.
Lookup retrieves information related to specific information (e.g. IP
address, domain) from several internet services (e.g. whois).
Web API
Provide data stored in the EXIST database by Web API.
Getting Started
After that I assume the environment of CentOS 7 or Ubuntu 18.04 LTS. Please at your own when deploying to other environment.
Install python modules
$ sudo pip install -r requirements.txt
Install MariaDB
$ curl -sS https://downloads.mariadb.com/MariaDB/mariadb_repo_setup | sudo bash
$ sudo yum install MariaDB-server MariaDB-client
$ sudo apt install mariadb-server mariadb-client
$ sudo systemctl start mariadb
$ sudo systemctl enable mariadb
Database Setting
Migrate Database
$ python manage.py makemigrations exploit reputation threat threat_hunter twitter twitter_hunter news news_hunter vuln
$ python manage.py migrate
Install Redis server
Reputation tracker uses redis as the Celery cache server backend.
$ sudo yum install redis
$ sudo systemctl start redis
$ sudo systemctl enable redis
$ sudo apt install redis-server
$ sudo systemctl start redis-server
$ sudo systemctl enable redis-server
Setup Celery
Reputation tracker uses Celery as an asynchronous task job queue.
$ sudo mkdir /var/log/celery; sudo chown YOUR_USER:YOUR_GROUP /var/log/celery
$ sudo mkdir /var/run/celery; sudo chown YOUR_USER:YOUR_GROUP /var/run/celery
#Type Path Mode UID GID Age Argument
d /var/run/celery 0755 YOUR_USER YOUR_GROUP –
$ sudo systemctl start celery.service
$ sudo systemctl enable celery.service
Run web server
$ python manage.py runserver 0.0.0.0:8000
Note: I recommend to use Nginx and uWSGI when running in production environment.
Collect feed
Scripts for inserting feed into database are scripts/insert2db/*/insert2db.py.
Configure insert2db
$ python scripts/insert2db/reputation/insert2db.py
$ python scripts/insert2db/twitter/insert2db.py
$ python scripts/insert2db/exploit/insert2db.py
$ python scripts/insert2db/threat/insert2db.py
$ python scripts/insert2db/news/insert2db.py
$ python scripts/insert2db/vuln/insert2db.py
Note: To automate information collection, write them to your cron.
Twitter Hunter can detect tweets containing specific keywords and user ID. And you can notify slack if necessary.
Threat Hunter
Threat Hunter can detect threat events containing specific keywords. And you can notify slack if necessary.
Other Requirement Tools & Settings
EXIST uses VirusTotal API.
Note: You get more information if you have private API key.
wkhtmltopdf and Xvfb
Lookup URL uses wkhtmltopdf and Xvfb.
$ sudo yum install xorg-x11-server-Xvfb
If you deploy EXIST on Ubuntu 18.04 LTS, you can install these packages by using apt.
$ sudo apt install wkhtmltopdf xvfb
Flush Old Data
This content was originally published here.
