Summary
In the mission to protect the nation from ever-evolving threats, DHS requires an automated way to detect anomalous activity in large amounts of video data, and share this information securely across organizational boundaries without compromising privacy. This project’s proposed approach will generate a cumulative, agent-based machine learning model to detect suspicious activity and improve detection accuracy across video sources, without the need for sensitive video data to be shared between sites. The system and user-friendly interface developed in this project can be integrated into existing systems of stakeholders such as USSS, USCIS, USCG, CBP, TSA and ICE, allowing local video to be processed onsite and prioritized for review by a human analyst more efficiently and effectively.
Problem addressed
The Department of Homeland Security (DHS) obtains and retrieves a high amount of video data from various sources (e.g., CCTV cameras and vehicle-mounted cameras). These video data are difficult to process manually by human agents for anomaly detection. Moreover, the manual detection process is prone to human error. The main purpose of this project is to develop an agent-based learning method that can leverage local computational resources to identify anomalous events or unusual behaviors using various local camera data at a certain location, (e.g., sporting event venues). We also propose to develop a user-friendly dashboard for appropriate visualization and analytics of the data.
Approach
To detect anomalous events, we propose a framework that utilizes a pretrained model based on the ImageNet large-scale dataset. Initially, we extract the features from the publicly available video data using the pretrained model. We then process this data for instance segmentation (30 frames per instance). We also summarize the instance segmentation to calculate instance difference. The instance difference is further processed for normalization. Based on the instance difference, we can detect anomalies where there is a significant difference in between the instances. The next stage of the project is dedicated to visualization of the results produced by the developed tools.
Results
Anomalous events are hard to detect, even by human agents. During the first 7 months of the project, we were able to analyze the results of our anomaly detection framework. According to the initial results, our proposed framework can localize the anomalous events accurately in most scenarios with sufficient data availability and acceptable video quality. However, due to noise in the dataset, we experienced some false alarms. During the rest of the project, we plan to improve the accuracy and to implement the visualization tool.
Anticipated Impact for DHS
Our proposed framework can detect anomalous events and enhance the security of our nation. Moreover, it increases the accuracy as compared with the manual detection process. In order to monitor for threats, there are surveillance cameras in various locations, such as airports, checkpoints, and rail stations. Human agents check video and when they see something unusual, they report the incident to the law enforcement agencies to take necessary actions. Our framework automates this process, optimizes human involvement, and increases the efficiency of anomaly detection. This is aligned with the DHS mission to secure the nation’s borders to prevent illegal activity while facilitating lawful travel, and directly increases border and public security through more efficient anomalous event detection.
Research Products:
Publications:
A Federated Learning Framework for Automated Decision Making with Microscopic Traffic Simulation
Presentations:
CINA Research Results Briefing: Agent-based Learning to Utilize Local Data for Activity Recognition