ViWi Dataset
Overview
ViWi Scenarios
Applications
ML Competition
ViWi-BT Challenge

Applications on Vision-Aided Wireless Communications


1) Vision-Aided 6G Wireless Communications: Blockage Prediction and Proactive Handoff


Snow
Forest
The illustrative figure on the left shows a mmWave basestation equipped with a mmWave array and a camera serving multiple mobile users. One user is LOS while the other is blocked by a bus. The figure on the right presents an overall block diagram illustrating the future blockage prediction problem and its solution. The proposed solution takes in a sequence of bimodal image-beam tuples and produces a prediction of the future link status.


  • Paper: Gouranga Charan, Muhammad Alrabeiah, and Ahmed Alkhateeb, "Vision-Aided 6G Wireless Communications: Blockage Prediction and Proactive Handoff," submitted to IEEE Transactions on Vehicular Technology, Feb. 2021, arXiv e-prints, p. arXiv:2102.,

  • Key Idea: The sensitivity to blockages is a key challenge for the high-frequency (5G millimeter wave and 6G sub-terahertz) wireless networks. Since these networks mainly rely on line-of-sight (LOS) links, sudden link blockages highly threaten the reliability of the networks. Further, when the LOS link is blocked, the network typically needs to hand off the user to another LOS basestation, which may incur critical time latency, especially if a search over a large codebook of narrow beams is needed. A promising way to tackle the reliability and latency challenges lies in enabling proaction in wireless networks. Proaction basically allows the network to anticipate blockages, especially dynamic blockages, and initiate user hand-off beforehand. This paper presents a complete machine learning framework for enabling proaction in wireless networks relying on visual data captured, for example, by RGB cameras deployed at the base stations. In particular, the paper proposes a vision-aided wireless communication solution that utilizes bimodal machine learning to perform proactive blockage prediction and user hand-off. The bedrock of this solution is a deep learning algorithm that learns from visual and wireless data how to predict incoming blockages. The predictions of this algorithm are used by the wireless network to proactively initiate hand-off decisions and avoid any unnecessary latency. The algorithm is developed on a vision-wireless dataset generated using the ViWi data-generation framework. Experimental results on two basestations with different cameras indicate that the algorithm is capable of accurately detecting incoming blockages more than ~ 90% of the time. Such blockage prediction ability is directly reflected in the accuracy of proactive hand-off, which also approaches 87%. This highlights a promising direction for enabling high reliability and low latency in future wireless networks.

  • To reproduce the blockage prediction results, please follow the steps in the following Github link

  • If you have any questions regarding the code and used dataset, please contact Gouranga Charan.





2) Vision-Aided Millimeter Wave Beam and Blockage Prediction


cam_beam
A block diagram of a vision-aided dual-band BS. Two ResNet18 models are deployed to learn beam prediction and user detection, respectively. Each network has a customized fully-connected layer that suits the task it handles. A network is trained to directly predict the beam index while the other predicts the user existence (detection) which is, then, converted to blockage prediction using the sub-6 GHz channels.


  • Paper: Muhammad Alrabeiah, Andrew Hredzak, and Ahmed Alkhateeb, "Millimeter Wave Base Stations with Cameras: Vision Aided Beam and Blockage Prediction," submitted to IEEE Vehicular Technology Conference, Oct. 2019, arXiv e-prints, p. arXiv:1911.,

  • Key Idea: This paper investigates a novel research direction that leverages vision to help overcome the critical wireless communication challenges. In particular, this paper considers millimeter wave (mmWave) communication systems, which are principal components of 5G and beyond. These systems face two important challenges: (i) the large training overhead associated with selecting the optimal beam and (ii) the reliability challenge due to the high sensitivity to link blockages. Interestingly, most of the devices that employ mmWave arrays will likely also use cameras, such as 5G phones, self-driving vehicles, and virtual/augmented reality headsets. Therefore, we investigate the potential gains of employing cameras at the mmWave base stations and leveraging their visual data to help overcome the beam selection and blockage prediction challenges. To do that, this paper exploits computer vision and deep learning tools to predict mmWave beams and blockages directly from the camera RGB images and the sub-6GHz channels. The experimental results reveal interesting insights into the effectiveness of such solutions. For example, the deep learning model is capable of achieving over 90% beam prediction accuracy, which only requires snapping a shot of the scene and zero overhead.

  • To reproduce the beam prediction results, please follow the steps in the following Github link

  • If you have any questions regarding the code and used dataset, please contact Muhammad Alrabeiah.





Join the ViWi mailing list!

Please provide your contact information to get the latest updates about the ViWi dataset