Elastic Stack
Install Java
- install java runtime too >= version 8
- set JAVA_HOME environment variable to 'c:\ProgramFiles\Java\jre8'
Elastic search (Database)
steps
- Download and unzip Elasticsearch
- Run bin/elasticsearch (or bin\elasticsearch.bat on Windows)
- Open URL in browser or via curl or via powershell
curl http://localhost:9200/
//PowerShell
Invoke-RestMethod http://localhost:9200
- config/elasticsearch.yml contains configuration
- modify IP address, port & data directory location
- also add 'node.name' for machine identification, 'cluster.name' for group identification
Sample config
take from live server
Usage
(Normally auto creation)
- Better do querying from kibana
- In IT, using for centralized logging. Actually built for full text search with different analyzers
Kibana (Viewer)
steps
- Download and unzip Kibana
- Open config/kibana.yml in an editor
- modify IP address & server name
- Set elasticsearch.hosts to point at your Elasticsearch instance
- Run bin/kibana (or bin\kibana.bat on Windows)
- Point your browser at http://localhost:5601
Sample config
take from live server
ELBK Stack (Centralized logging)
Elastic search : Database
Logstash : collects, alters (for consistency) & forward logs to elastic search
Beats : Log collectors from various sources
- File beat : Text log files
- Metric beat : OS & applications
- Packet beat : Network monitoring
- Winlog beat : Windows event log
Kibana : UI for visualizing data
Logstash
Steps
Download and unzip Logstash
Prepare a logstash.conf file link
Run "bin/logstash -f logstash.conf"
Note : Enable logstash as a service in linux server
Sample config
take from live server
Winlog beat
Steps
- Download and unzip Winlogbeat
- Edit the winlogbeat.yml configuration file
- Run in PowerShell: winlogbeat.exe -c winlogbeat.yml (better install as a service using bat file)
Note : winlogbeat.yml for important settings, winlogbeat.full.yml for larger list of features & options
Sample config
take from live server
Say to create template schema for all DBs staring with winlog name
Invoke-WebRequest -Method PUT -InFile .\winlogbeat.template.json -Uri http://localhost:9200/_template/winlogbeat
Metric beat
Steps
- Download and unzip Metricbeat
- Metricbeat can also be installed from our package repositories using apt or yum. See Repositories in the Guide.
- Edit the metricbeat.yml configuration file
- Start the daemon by running sudo ./metricbeat -e -c metricbeat.yml (Better install as a service)
Sample config
take from live server
Say to create template schema for all DBs staring with metric name
Invoke-WebRequest -uri http://localhost:9200/_template/metricbeat -Method PUT -infile .\metricbeat.template.json
File beat
Steps
- Download and unzip Filebeat
- Edit the filebeat.yml configuration file
- Start the daemon by running sudo ./filebeat -e -c filebeat.yml
Sample config
take from live server
Say to create template schema for all DBs staring with file name
Invoke-WebRequest -uri http://localhost:9200/_template/filebeat -Method PUT -infile .\filebeat.template.json
Packet beat
Steps
- Download and unzip Packetbeat
- Edit the packetbeat.yml configuration file
- Start the daemon by running sudo ./packetbeat -e -c packetbeat.yml (better install as a service)
Sample config
take from live server
Say to create template schema for all DBs staring with file name
Invoke-WebRequest -uri http://localhost:9200/_template/packetbeat -Method PUT -infile .\packetbeat.template.json
Other (IT)
http://10.10.10.xxx:920X
To check all indices (databases)
http://10.10.10.xxx:920x/_cat/indices
http://10.10.10.xxx:920x/_cat/indices?s=status //sort by status
To see mapping/ schema of particular index
http://10.10.10.xxx:920x/filebeat-2019.09.07/_mapping
To get all rows of table log(takes 100)
http://10.10.10.xxx:920x/filebeat-2019.09.07/log/_search
To search
http://10.10.10.xxx:920x/filebeat-2019.09.07/log/_search?q=host:ASL3
https://www.elastic.co/guide/en/elasticsearch/guide/current/retiring-data.html
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/installation.html
Closing Old Indices
POST http://10.10.10.xxx:920x/filebeat-2018*/_flush
POST http://10.10.10.xxx:920x/filebeat-2018*/_close
POST http://10.10.10.xxx:920x/filebeat-2018*/_open
Deleting old Indices
DELETE http://10.10.10.xxx:920x/filebeat-2018*