Learning Session Summary

Elasticsearch & Kibana

Everything you learned — from zero to searching, visualising, and investigating

①

What Are They?

Elasticsearch
A search and analytics engine. Stores JSON documents and makes them searchable in milliseconds. Not a traditional database — it's optimised for finding and analysing data fast.
Kibana
The visual interface for Elasticsearch. Build dashboards, explore data, run queries — all without writing code. Like Prisma Studio is to PostgreSQL.
Key analogy

Elasticsearch is a phone book — it holds data, but its real job is helping you find things fast. The actual records (source of truth) live in your primary database like PostgreSQL.

PostgreSQL
Source of truth
→
Elasticsearch
Search copy
→
Kibana
Visual layer
②

Core Concepts

Cluster
The entire Elasticsearch system. One per project/domain. Like the whole library building.
Node
A server within the cluster. More nodes = faster and more redundant. Like floors in the building.
Index
A collection of documents — equivalent to a database table. Like a bookshelf.
Document
A single JSON record within an index. Flexible — each document can have different fields.
③

Docker Setup

What you did

Created a docker-compose.yml that runs Elasticsearch (port 9200) and Kibana (port 5601) in containers. Docker keeps your system clean — nothing is "installed," just run and stop.

# Start everything
docker compose up

# Check if ES is running
curl http://localhost:9200

# Stop everything
docker compose down

# See running containers
docker ps
④

Talking to Elasticsearch (curl)

curl command anatomy
curl -X POST "http://localhost:9200/my-index/_doc" \
  -H "Content-Type: application/json" \
  -d '{"name": "John", "city": "Lisbon"}'
  • curl — sends HTTP requests from terminal
  • -X POST/PUT/DELETE — HTTP method (GET is default)
  • -H — headers (multiple allowed)
  • -d — request body (your data)
  • ?pretty — formats JSON response nicely
  • \ — continue command on next line
  • Always quote URLs with ? or & characters
Essential operations
# Create an index
curl -X PUT "http://localhost:9200/my-index"

# Add a document
curl -X POST "http://localhost:9200/my-index/_doc" ...

# Search everything
curl "http://localhost:9200/my-index/_search?pretty"

# Quick URL search
curl "http://localhost:9200/my-index/_search?pretty&q=city:Lisbon"

# List all indices
curl "http://localhost:9200/_cat/indices?v"
⑤

Query DSL vs KQL

Query DSL
JSON-based, used by developers in code and API calls. Powerful, verbose, handles complex queries.
KQL
Simple text syntax for Kibana's search bar. Gets translated to Query DSL behind the scenes.
Query DSL building blocks you learned
  • match — find documents matching a value
  • bool + must — combine conditions with AND
  • bool + should — combine conditions with OR
  • bool + must_not — exclude results (NOT)
  • bool + filter — like must but skips scoring (faster)
  • range + gt/gte/lt/lte — numeric and date ranges
Relevance scoring

_score ranks how well each document matches your query. Use must when relevance matters (text search). Use filter for exact yes/no conditions (city, status, age) — it's faster because it skips scoring.

⑥

Kibana Navigation

📊 Dashboard

Collection of visualisations showing different views of the same data. Supports cross-filtering — click on one chart and all others update. Start here when investigating problems (helicopter view).

🔍 Discover

Raw data explorer — see individual documents, expand entries, run KQL queries. Go here to dig into details after spotting a problem on the dashboard (ground investigation).

📈 Visualise Library

Build individual charts (bar, line, pie, etc.) — the building blocks that go onto dashboards.

🎨 Canvas

Presentation-style reports with custom layouts and branding. Like a mix of dashboard and PowerPoint.

🗺️ Maps

Dedicated section for complex geographic visualisations with multiple layers.

⚡ APM (Traces)

Distributed tracing — shows waterfall diagrams of requests flowing across microservices. Requires APM agents installed in your application code. Read the waterfall bottom-up to find the real bottleneck.

Important

Filters do NOT carry across sections. Each tab has its own independent filters. But within a Dashboard, all charts share the same filters (cross-filtering).

⑦

Annotations (What You Built)

You created a separate deployments index with timestamp, service, and version fields. Then added an annotation layer to a chart that pulls from this index, showing vertical lines on the timeline wherever a deployment happened.

1. Create deployments index 2. Create data view in Kibana 3. Add annotation layer to chart 4. Set to Custom Query
⑧

Investigating a Problem

1. Dashboard
Spot the anomaly — when did it start? Spike in errors? Drop in traffic?
→
2. Dashboard + Cross-filters
Narrow down — which service? Which endpoint? Which server?
→
3. Discover
Dig into individual documents — error messages, exact timestamps, raw details
→
4. Cross-index correlation
Compare with infrastructure metrics, deployment logs, database performance
→
5. Root cause identified
Annotations on charts show deployments that correlate with the problem
⑨

Distributed Tracing (APM)

How trace data flows
Your App
+ APM Agent
→
APM Server
→
Elasticsearch
→
Kibana APM
Reading a waterfall — go bottom-up!
Browser
598ms
  API Gateway
400ms
    Payment Svc
300ms
      DB: SELECT
150ms ← bottleneck

The outer bars are waiting for inner bars. The deepest bar doing actual work is your real bottleneck.

⑩

The Golden Rule

Kibana can only show what exists in your documents

Every chart, metric, map, and trace in Kibana comes from fields inside your Elasticsearch documents. No field = no visualisation. The quality of your monitoring depends entirely on what data you collect. Discover is the best place to check what fields are available — it is your source of truth.

🔄 Parallels you discovered