ELK Stack - Kurindeta Limited

ELK Stack

What is ELK Stack?

“ELK” is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch.

Kibana

Set Up Kibana
- Installing Kibana
  - Install Kibana with .tar.gz
  - Install Kibana with Debian Package
  - Install Kibana with RPM
  - Install Kibana on Windows
- Starting and stopping Kibana
- Configuring Kibana
- Running Kibana on Docker
- Accessing Kibana
- Connect Kibana with Elasticsearch
- Using Kibana in a production environment
- Upgrading Kibana
- Configuring monitoring
- Configuring security
Getting Started
- Get up and running with sample data
- Explore Kibana using the Flight dashboard
- Building your own dashboard
Discover
- Setting the time filter
- Searching Your Data
- Filtering by Field
- Viewing Document Data
- Viewing Document Context
- Viewing Field Data Statistics
Visualize
- Creating a Visualization
- Line, Area, and Bar charts
- Controls Visualization
- Data Table
- Markdown Widget
- Metric
- Goal and Gauge
- Pie Charts
- Coordinate Maps
- Region Maps
- Time Series Visual Builder
- Tag Clouds
- Heatmap Chart
- Vega Graphs
- Inspecting Visualizations
Dashboard
- Building a Dashboard
- Arranging Dashboard Elements
- Inspecting a Visualization from the Dashboard
- Sharing a Dashboard
Timelion
- Getting Started
- Inline Help and Documentation
Canvas
- Getting started with Canvas
- Canvas function reference
Machine Learning
- Creating machine learning jobs
Maps
- Getting started with Maps
- Heat map layer
- Tile layer
- Vector layer
- Searching your data
- Connecting to Elastic Maps Service
Infrastructure
- Monitor your infrastructure
- Using the Infrastructure UI
Logs
- Using the Logs UI
APM
- Getting Started
- Visualizing Application Bottlenecks
- Using APM
Uptime
- Overview
- Monitor
- Use with Elasticsearch Security
Graphing Connections in Your Data
- Getting Started
- Configuring Graph
- Troubleshooting
- Limitations
Dev Tools
- Console
  - Auto formatting
  - Keyboard shortcuts
  - Configuring Console
- Profiling your Queries and Aggregations
  - Getting Started
  - Profiling a more complicated query
  - Rendering pre-captured profiler JSON
  - Index and Type filtering
- Debugging Grok Expressions
  - Getting Started
Monitoring
- Beats Metrics
- Cluster Alerts
- Elasticsearch Metrics
- Kibana Metrics
- Logstash Metrics
- Troubleshooting
Management
- License Management
- Index Patterns
  - Cross-cluster search
- Working with rollup indices
  - Create and manage rollup jobs
  - Create a visualization using rolled up data
- Index lifecycle policies
  - Creating an index lifecycle policy
  - Managing index lifecycle policies
  - Adding a policy to an index
  - Example of using an index lifecycle policy
- Managing Fields
  - String Field Formatters
  - Date Field Formatters
  - Geographic Point Field Formatters
  - Numeric Field Formatters
  - Scripted Fields
- Managing Indices
- Setting advanced options
- Managing Saved Objects
- Managing Beats
- Working with remote clusters
- Spaces
  - Getting Started
  - Managing spaces
  - Securing spaces
  - Moving saved objects between spaces
- Security
  - Authorization
  - Kibana privileges
- Watcher UI
  - Getting Started
  - Create Threshold Alert
  - Create Advanced Watch
  - Watcher UI Security
- Upgrade Assistant
- Kibana Dashboard Only Mode
  - Advanced Configuration for Dashboard Only Mode
Reporting from Kibana
- Getting Started
- Automating Report Generation
- Deprecated Report URLs
- PDF Layout Modes
- Configuring Reporting
  - Encryption Keys for Multiple Kibana Instances
  - Reporting Indices for Multiple Kibana Workspaces
  - Using Reverse Proxies
  - Reporting and Security
  - Securing the Reporting Endpoints
  - Chromium Sandbox
- Troubleshooting
- Reporting Integration
REST API
- Kibana Spaces API
  - Create Space
  - Update Space
  - Get Space
  - Delete space
- Kibana Role Management API
  - Create or Update Role
  - Get Role
  - Delete role
- Saved Objects API
  - Get Object
  - Bulk Get Objects
  - Find Objects
  - Create Object
  - Bulk Create Objects
  - Update Object
  - Delete Object
- Dashboard Import API
  - Import Dashboard
  - Export Dashboard
- Logstash Configuration Management API
  - Create Pipeline
  - Retrieve Pipeline
  - Delete Pipeline
  - List Pipelines
- URL Shortening API
  - Shorten URL
- Upgrade Assistant API
  - Upgrade Readiness Status
  - Reindex API
  - Add Default Field API
Kibana Plugins
- Installing Plugins
- Updating & Removing Plugins
- Disabling Plugins
- Configuring the Plugin Manager
- Known Plugins
Contributing to Kibana
- Core Development
  - Considerations for basePath
  - Managing Dependencies
  - Modules and Autoloading
  - Communicating with Elasticsearch
  - Functional Testing
- Plugin Development
  - Plugin Resources
  - UI Exports
  - Functional Tests for Plugins
  - Localization for plugins
- Developing Visualizations
  - Embedding Visualizations
  - Developing Visualizations
  - Visualization Factory
  - Visualization Editors
  - Visualization Request Handlers
Visualization Response Handlers
- - Vis object
  - AggConfig object
- Add Data Guide
- Security
  - Role-based access control
- Pull request review guidelines

Logstash

Logstash Introduction
Getting Started with Logstash
- Installing Logstash
- Stashing Your First Event
- Parsing Logs with Logstash
- Stitching Together Multiple Input and Output Plugins
How Logstash Works
- Execution Model
Setting Up and Running Logstash
- Logstash Directory Layout
- Logstash Configuration Files
- logstash.yml
- Secrets keystore for secure settings
- Running Logstash from the Command Line
- Running Logstash as a Service on Debian or RPM
- Running Logstash on Docker
- Configuring Logstash for Docker
- Logging
- Shutting Down Logstash
- Setting Up X-Pack
Upgrading Logstash
- Upgrading Using Package Managers
- Upgrading Using a Direct Download
- Upgrading between minor versions
- Upgrading Logstash to 7.0
- Upgrading with the Persistent Queue Enabled
Configuring Logstash
- Structure of a Config File
- Accessing Event Data and Fields in the Configuration
- Using Environment Variables in the Configuration
- Logstash Configuration Examples
- Multiple Pipelines
- Pipeline-to-Pipeline Communication (Beta)
- Reloading the Config File
- Managing Multiline Events
- Glob Pattern Support
- Converting Ingest Node Pipelines
- Logstash-to-Logstash Communication
- Centralized Pipeline Management
- X-Pack monitoring
- X-Pack security
- X-Pack Settings
Managing Logstash
- Centralized Pipeline Management
Working with Logstash Modules
- Using Elastic Cloud
- ArcSight Module
- Netflow Module
- Azure Module
Working with Filebeat Modules
- Use ingest pipelines for parsing
- Use Logstash pipelines for parsing
- Example: Set up Filebeat modules to work with Kafka and Logstash
Data Resiliency
- Persistent Queues
- Dead Letter Queues
Transforming Data
- Performing Core Operations
- Deserializing Data
- Extracting Fields and Wrangling Data
- Enriching Data with Lookups
Deploying and Scaling Logstash
Performance Tuning
- Performance Troubleshooting Guide
- Tuning and Profiling Logstash Performance
Monitoring Logstash
- Overview
- Monitoring UI
- Pipeline Viewer UI
- Troubleshooting
Monitoring APIs
- Node Info API
- Plugins Info API
- Node Stats API
- Hot Threads API
Working with plugins
- Generating Plugins
- Offline Plugin Management
- Private Gem Repositories
- Event API
Input plugins
Output plugins
Filter plugins
Codec plugins

Elastic Search

Getting started
- Basic Concepts
- Installation
- Exploring Your Cluster
- Modifying Your Data
- Exploring Your Data
- Conclusion
Set up Elasticsearch
- Installing Elasticsearch
  - Install Elasticsearch from archive on Linux or MacOS
  - Install Elasticsearch with .zip on Windows
  - Install Elasticsearch with Debian Package
  - Install Elasticsearch with RPM
  - Install Elasticsearch with Windows MSI Installer
  - Install Elasticsearch with Docker
- Configuring Elasticsearch
  - Setting JVM options
  - Secure settings
  - Logging configuration
  - Auditing settings
  - Cross-cluster replication settings
  - Index lifecycle management settings
  - License settings
  - Machine learning settings
  - Monitoring settings
  - Security settings
  - SQL access settings
  - Watcher settings
- Important Elasticsearch configuration
  - Discovery and cluster formation settings
  - Setting the heap size
  - JVM heap dump path
  - GC logging
  - Temp directory
  - JVM fatal error logs
- Important System Configuration
  - Configuring system settings
  - Disable swapping
  - File Descriptors
  - Virtual memory
  - Number of threads
  - DNS cache settings
  - JNA temporary directory not mounted with noexec
- Bootstrap Checks
- Starting Elasticsearch
- Stopping Elasticsearch
- Adding nodes to your cluster
- Set up X-Pack
- Configuring monitoring
  - Collecting monitoring data
  - Collecting monitoring data with Metricbeat
  - Configuring indices for monitoring
- Configuring security
- Configuring X-Pack Java Clients
- Bootstrap Checks for X-Pack
Upgrade Elasticsearch
- Rolling upgrades
- Full cluster restart upgrade
- Reindex before upgrading
  - Reindex in place
  - Reindex from a remote cluster
API conventions
- Multiple Indices
- Date math support in index names
- Common options
- URL-based access control
Document APIs
Search APIs
- Search
- URI Search
- Request Body Search
- Search Template
- Multi Search Template
- Search Shards API
- Suggesters
  - Term suggester
  - Phrase Suggester
  - Completion Suggester
  - Context Suggester
  - Returning the type of the suggester
- Multi Search API
- Count API
- Validate API
- Explain API
- Profile API
- Field Capabilities API
- Ranking Evaluation API
Aggregations
- Metrics Aggregations
- Bucket Aggregations
- Pipeline Aggregations
- Matrix Aggregations
- Caching heavy aggregations
- Returning only aggregation results
- Aggregation Metadata
- Returning the type of the aggregation
Indices APIs
- Create Index
- Delete Index
- Get Index
- Indices Exists
- Open / Close Index API
- Shrink Index
- Split Index
- Rollover Index
- Put Mapping
- Get Mapping
- Get Field Mapping
- Types Exists
- Index Aliases
- Update Indices Settings
- Get Settings
- Analyze
  - Explain Analyze
- Index Templates
- Indices Stats
- Indices Segments
- Indices Recovery
- Indices Shard Stores
- Clear Cache
- Flush
  - Synced Flush
- Refresh
- Force Merge
cat APIs
Cluster APIs
Query DSL
- Query and filter context
- Match All Query
- Full text queries
- Term level queries
- Compound queries
- Joining queries
- Geo queries
- Specialized queries
- Span queries
- Minimum Should Match
- Multi Term Query Rewrite
Mapping
- Removal of mapping types
- Field datatypes
- Meta-Fields
- Mapping parameters
  - Dynamic Mapping
Analysis
- Anatomy of an analyzer
- Testing analyzers
- Analyzers
- Normalizers
- Tokenizers
- Token Filters
- Character Filters
Modules
- Discovery and cluster formation
- Shard allocation and cluster-level routing
- Local Gateway
  - Dangling indices
- HTTP
- Indices
- Network Settings
- Node
- Plugins
- Scripting
- Snapshot and Restore
- Thread Pool
- Transport
- Remote clusters
- Cross-cluster search
Index modules
- Analysis
- Index Shard Allocation
  - Total shards per node
- Mapper
- Merge
- Similarity module
- Slow Log
- Store
  - Pre-loading data into the file system cache
- Translog
- Index Sorting
  - Use index sorting to speed up conjunctions
Ingest node
- Pipeline Definition
- Ingest APIs
- Accessing Data in Pipelines
- Conditional Execution in Pipelines
- Handling Failures in Pipelines
- Processors
Managing the index lifecycle
- Getting started with index lifecycle management
- Policy phases and actions
  - Timing
  - Phase Execution
  - Actions
  - Full Policy
- Set up index lifecycle management policy
  - Applying a policy to an index template
  - Apply a policy to a create index request
- Using policies to manage index rollover
  - Skipping Rollover
- Update policy
  - Updates to policies not managing indices
  - Updates to executing policies
  - Switching policies for an index
- Index lifecycle error handling
- Restoring snapshots of managed indices
- Start and stop index lifecycle management
SQL access
- Overview
- Getting Started with SQL
- Conventions and Terminology
  - Mapping concepts across SQL and Elasticsearch
- Security
- SQL REST API
- SQL Translate API
- SQL CLI
- SQL JDBC
  - API usage
- SQL ODBC
- SQL Client Applications
  - DBeaver
  - DbVisualizer
  - Microsoft Excel
  - Microsoft Power BI Desktop
  - Microsoft PowerShell
  - MicroStrategy Desktop
  - Qlik Sense Desktop
  - SQuirreL SQL
  - SQL Workbench/J
  - Tableau Desktop
- SQL Language
- Lexical Structure
- SQL Commands
- Data Types
- Index patterns
- Functions and Operators
- Reserved keywords
- SQL Limitations
Monitoring Elasticsearch
- Collectors
- Exporters
  - Local Exporters
  - HTTP Exporters
- Pausing Data Collection
Rolling up historical data
- Overview
- API Quick Reference
- Getting Started
- Understanding Groups
- Rollup Aggregation Limitations
- Rollup Search Limitations
Frozen indices
- Best practices
- Searching a frozen index
- Monitoring frozen indices
X-Pack APIs
- Info API
- Cross-cluster replication APIs
- Explore API
- Freeze index
- Index lifecycle management API
- Licensing APIs
- Migration APIs
- Machine learning APIs
- Rollup APIs
- Security APIs
- Unfreeze index
- Watcher APIs
- Definitions
Command line tools