1. Visualizing streaming data
Welcome to the last lesson of this course!
2. Our pipeline so far
So far in this course, we have built an AWS pipeline that collects Twitter data, sends it to Firehose, enriches the tweets with Sentiment and then sends it to Elasticsearch.
We are now meeting all of our requirements except for the last two - we need to be able to visualize our data in real time and trigger an alert!
3. Kibana Discover view
While we are sending our data to Elasticsearch - a storage engine - we are going to visualize it using Kibana a UI for Elasticsearch.
The strength of Elasticsearch is that it lets us define the schema when we read the data. Kibana also has several key views.
The Kibana discover view will let us see the raw data coming in, and let us slice and dice it as well as create a schema.
4. Kibana visualizations
The Kibana Visualizations view lets us create charts and visualize the data.
5. Dashboards and alerts
Lastly, Kibana Dashboards and Alerts allow us to combine the visualizations and set alerts when they pass thresholds.
6. Elasticsearch vs CloudWatch
Let's get this out of the way. Elasticsearch and CloudWatch are very similar, providing monitoring, alerts, visualizations and dashboards.
Yet, they're different. CloudWatch is AWS centric. It can accept custom data, but it's not the best tool for general visualization. CloudWatch is also great for working with logs explicitly.
Elasticsearch on the other hand is open source, and can accept a variety of datasets. The visualizations are more robust than CloudWatch, and there is a wide plugin ecosystem.
7. Let's practice!
Now, let's put Elasticsearch to use monitoring our tweets!