Imagine you’ve invited an important person to your home for dinner – say for a first date. Naturally, you’re excited. You want everything to be perfect. The food has to be ready. The table has to be set. Now there are two ways you can handle the situation: Either you run to the door every 10 minutes in your excitement and check if someone is already there. Or you get everything ready and open the door only when the doorbell actually rings. Pixels in a camera don’t usually have dates, but they have a similar choice of how to behave when triggered in a recording situation.
What Are Event Cameras?
Event cameras are relatively new sub-category within Computer Vision, a field of study that focuses on enabling machines to interpret visual information, like images or videos. Computer Vision involves techniques such as image processing, pattern recognition, and machine learning. Event cameras intend to mimick the human brain technologically. As the interest in thie human brain and how it works is growing, neuromorphic engineering is gaining more and more traction. Event cameras are only the first step in the direction of imitating the human retina on a chip.These cameras are also known as neuromorphic- or DVS (Dynamic Vision System) cameras. They capture visual information based on the dynamics of the recorded environment.
This is how it works: Each pixel in those event cameras works as an independent processing unitthat allows them to asynchronously output intensity changes. A change in a one-pixel intensity is referred to as an event. An event represents a movement within the recorded scene, a change of brightness, and a timestamp. Events are timestamped with microsecond resolution and are transmitted with sub-millisecond latency, which makes these sensors react quickly to visual stimuli.
The image to the right visualizes the output from an event camera in comparison to a conventional camera. You can see a disc containing a rotating black dot. The conventional camera records the complete frame in fixed intervals. In each frame, the black dot moves quite a distance. The information in between those recorded frames gets lost. Whereas pixels in the event camera are only triggered by the movement of the black dot. This means they provide information in a continuous stream.
So, Why Event Cameras?
For decades, conventional cameras have represented the default – or sometimes even the only –way of obtaining visual information, which has forced the user to adapt to certain limitations. These limitations include low frame rates, high latencies, poor adjustment to extreme light conditions, or high power consumption. Some manufacturers managed to create hardware that compensates for some of these limitations, while not considering changes to the core technology. An event camera has a fundamentally different hardware setup than a conventional camera and allows extremely fast frame rates of ~ 10.000 fps, very low power consumption of ~1mW, and a dynamic range that resembles that of the human eye. The dynamic range makes the event camera resistant to changing light circumstances, as seen in the image below, where a car is leaving a tunnel and the image obtained by a conventional camera is overexposed by the change in brightness while in the reconstructed image from the event camera one can clearly see the details outside of the tunnel.
An event camera only gives an output where there is a change in intensity, and the images as we know them from conventional cameras can then be reconstructed using different deep learning algorithms. For many applications, the information on static objects in the scene is irrelevant, adding noise to the image, and are essentially just producing overhead in the processing pipeline.
Checklist: Do I Have a Use Case for Event Cameras?
There is a huge amount of reasons for choosing event cameras over a conventional camera, but in favor of not writing an endless list, ask yourself the following questions:
- Do I have uncontrolled light conditions in my scene?
- Do I have poor light conditions – very dark or very bright?
- Do I record something that is moving?
- Do I need to record something with a very high framerate?
- Do I use moving equipment for recording (e.g. a car)?
- Do I have power limitations – e.g. because I’m using drones?
- Do I like new technology and think mimicking the human brain is pretty awesome?
If you can answer yes to any or at least some of the questions above, then you should definitely look into event cameras for your purposes.
What Are Possible Applications For Event Cameras?
The possible use cases of Event Cameras are actually infinite. The question is really why they are not yet being used across the board. From our point of view, an application would be particularly promising in:
Robotics and Manufacturing:
- Simultaneous Localization and Mapping (SLAM)
- High-speed obstacle avoidance (also in drone applications)
- Drone applications
- High-speed interaction between machine and environment
- Monitoring of production lines
- Visual detections with uncontrollable light sources
Automotive and General Applications:
- Fast detection of pedestrians and bicyclists
- Detection of environments with changing light
- Gesture recognition
- Night vision
- Depth estimation
- High-speed detection with no latency
How Have We Used Them?
At Motius, we have gained a lot of knowledge in state-of-the-art event cameras and the processing of their data. This allows us to realize their potential and the value they can add to projects. During our quarterly Discovery Conference, we devoted time to a project all about event cameras, where we used a simulation environment, called ESIM, that was provided by the ETH Zurich. ESIM gives you the simulated output from event cameras as well as that from the conventional cameras in different renderings. This quickly shows the benefits of an event camera.
Where Will We See Event Cameras Next?
In several of our Motius projects, an event camera would provide a great addition and provide a true benefit. Naturally, the next step for Motius is to explore incoming and previous projects to assess the value of their addition. Through event cameras, we will be able to solve complex tasks that were previously not feasible and extend our knowledge to new domains of Computer Vision.