1. Real-world use case: sensor data
Great work with the last scenario. Let's consider another common use case for streaming data processes: sensor data handling.
2. What is sensor data?
What is meant by sensor data? Sensors are automated devices that monitor some aspect of their environment.
This can include temperature sensors,
electricity usage monitors,
vehicle presence detection (like a highway monitoring congestion)
and many others.
These devices tend to communicate with centralized services for management and data reporting. This allows the devices to be fairly simple, needing only to monitor the attribute in question and being able to communicate with the service. They can avoid requiring large local data storage and management mechanisms.
Sensor data networks can range wildly in scale, from a few systems communicating in a home automation system, to the hundreds of sensors streaming data in a vehicle, to the millions or even billions of devices for a wide-spread service such as in weather monitoring.
3. Connected doorbell
For our example, we're going to look at a connected doorbell service.
The device is primarily designed to replace a doorbell, and is monitoring for button presses.
The doorbell also contains audio and camera hardware, which can record or transmit data for storage of audio / video data.
This does allow for live remote communication for when owners can't come to the door or are away from the home (meaning the owner can see who's at the door and can speak with the visitor.)
Finally, the doorbell has additional sensors (temperature, humidity, light) that can be used with the audio / video capabilities for further functionality.
4. What are we monitoring?
As with our previous use case, we need to consider what we're most interested in monitoring.
The company in question has said they want streaming style transmission anytime someone presses the button on the doorbell. This is primarily to make sure a notification is sent to the owner with minimal delay (consider if this was batched vs streamed).
They would also like to have movement detection done within the device, but further analyzed by their services, with as little delay as possible.
Finally, they want updates of sound detection, perhaps of a car or fire alarm, or of a child crying nearby.
These requirements will need more intensive processing than just logging the events. Let's consider some aspects to this now.
5. Data handling
While there are many aspects to a product like this, let's focus on how we're going to store the data and what issues may appear.
For general event data, such as the button press, we'd like to store that data as quickly as we can, and likely have high priority update systems in place. This is to keep the time from button press to notification as short as possible.
We may also have sensor based events, such as a changing light source or microphone detection. These are defined to simply alert the user of changes if desired. This information may be of different priority, but is still important to process fairly quickly.
Finally, we do need to store the raw data (audio / video) for analysis as well. This should be done quickly, but not necessarily immediately.
The important thing to note is that the same product may have different SLAs for different aspects of its design.
6. Let's practice!
Now that we've looked at the basics of a sensor data network, let's consider some of the implications in the exercises ahead.