If you are familiar with Kibana Alerts, using ElasticSearch Watcher is a natural progression. Watchers give us complete flexibility when it comes to constructing an alert query. We can customize everything, from the search query, the aggregations on the search results, the filters we want to alert, the template of the action email and much more. One point to note, watchers run in ElasticSearch rather than being invoke from Kibana. So if you want to send emails in your watcher action, you need to update your elastic.yml. This is fairly easy, just a few lines of xpack settings and you should be on our way. This is well described in the ElasticSearch documentation and can be found here.
Watcher Trigger
A Watcher is made up of 4 main parts, the trigger, the input search queries, the condition to check, the action. When a watcher runs is controller by the trigger json. For example to make the watcher run every 15minutes the trigger json will look like
"trigger": {
"schedule": {
"interval": "15m"
}
}And if you need a daily report to be sent out at 1am every morning, we can use the below trigger schedule
"trigger" : {
"schedule" : {
"daily" : { "at" : "01:00" }
}
}Can read more about trigger scheduler here.
Watcher Query
Next is the input section which is the elastic query. The easiest way I’ve found to do this is to get the desired results in Kibana Discover and then Inspect > Request. Or use the Dev Tools console to create the query. Here is a query to find all the elastic documents that have a temperature greater than 30 in the last 24hrs.
"query": {
"bool": {
"must": [],
"filter": [
{
"match_all": {}
},
{
"range": {
"Temperature": {
"gte": 30,
"lt": null
}
}
},
{
"range": {
"@timestamp": {
"gte": "now-24h",
"lte": "now"
}
}
}
],
"should": [],
"must_not": []
}
}It is even possible to run two or more separate queries in a single watcher using the chain syntax. Here we have a temperature search and a humidity search and both are done separate but the results are available to check in the condition and in the action steps
"input" : {
"chain" : {
"inputs" : [
{
"temperature_search": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"device-live-*"
],
"rest_total_hits_as_int": true,
"body": {
"size": 0,
"query": {
"bool": {
"must": [],
"filter": [
{
"match_all": {}
},
{
"range": {
"Temperature": {
"gte": 30,
"lt": null
}
}
},
{
"range": {
"@timestamp": {
"gte": "now-24h",
"lte": "now"
}
}
}
],
"should": [],
"must_not": []
},
"humidity_search": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"device-live-*"
],
"rest_total_hits_as_int": true,
"body": {
"size": 0,
"query": {
"bool": {
"must": [],
"filter": [
{
"match_all": {}
},
{
"range": {
"Humidity": {
"gte": 70,
"lt": null
}
}
},
{
"range": {
"@timestamp": {
"gte": "now-24h",
"lte": "now"
}
}
}
],
"should": [],
"must_not": []
},
}
]
}
}Can read more about it here.
Watcher Condition
After this comes the condition check. This is fairly simple, the results are available in the payload section. If you had one query the payload will look like this. I’ve used the painless language cause it give us the programming syntax and this is quiet powerfull
"condition": {
"script": {
"source": "ctx.payload.live_search.hits.total > 0",
"lang": "painless"
}
},If your search had multiple queries then the results can be accessed on the payload too. Using our temperature and humidity queries from above the query will become
"condition": {
"script": {
"source": "ctx.payload.temperature_query.hits.total > 0 || ctx.payload.humidity_query.hits.total > 0",
"lang": "painless"
}
},The condition is used to decide whether the action needs to be carried out. The above check means if there are any elastic documents with temperatures > 30 or humidity > 70 in the last 24hrs then carry out the action part of the watcher.
Watcher Action
The action can be anything from logging back to elastic, to posting to Slack, to emailing the maintenance team and even creating at ticket in Jira. I’m going to send an email to myself and thus had add the email setup in the elastic yml. Can see more about it here
"actions": {
"send_email": {
"email": {
"profile": "standard",
"to": [
"schubert...@gmail.com"
],
"subject": "URGENT: Found a high temperature and humidity reading in the last 15mins",
"body": {
"html": """
<h5>Summary</h5>
Temperature had {{ctx.payload.temperature_query.hits.total}} in the last 15mins<br/>
Humidity had {{ctx.payload.humidity_query.hits.total}} in the last 15mins<br/>
<br/>
Dashboard <a href="https://...:5601/app/dashboards#/view/...">click here</a>
"""
}
}
}Final thoughts
Putting all the pieces together you should get an elasticsearch watcher that can perform some intricate queries and send out emails that keep the team updated of any issues
Happy Coding!