Immersive Space to Think

Two colocated AR users and one VR user collaborating with annotations made both in the real and virtual world (left). Part of the final layout of the collaborative literature review (top-right). LaTeX file output from the layout (bottom-right).

Collaboration plays a vital role in both academia and industry whenever we need to browse through a big amount of data to extract meaningful insights. These collaborations often involve people living far from each other, with different levels of access to technology. Effective cross-border collaborations require reliable telepresence systems that provide support for communication, cooperation, and understanding of contextual cues. In the context of collaborative academic writing, while immersive technologies offer novel ways to enhance collaboration and enable efficient information exchange in a shared workspace, traditional devices such as laptops still offer better readability for longer articles. We propose the design of a hybrid cross-reality cross-device networked system that allows the users to harness the advantages of both worlds. Our system allows users to import documents from their personal computers (PC) to an immersive headset, facilitating document sharing and simultaneous collaboration with both colocated colleagues and remote colleagues. Our system also enables a user to seamlessly transition between Virtual Reality, Augmented Reality, and the traditional PC environment, all within a shared workspace. We present the real-world scenario of a global academic team conducting a comprehensive literature review, demonstrating its potential for enhancing cross-reality hybrid collaboration and productivity.

Here's a sneak peek into our system (to be released on October 20, 2023).

Overview of the EyeST prototype. a) An analyst views and interacts with text documents in Augmented Reality (AR) while eye-tracking data is collected. b) A hand-registered menu allows the analyst to annotate and search the dataset. c) The system retrieves a word that the analyst paid attention to, and the analyst rates it in terms of relevance, complexity, and familiarity.

Eye gaze patterns vary based on reading purpose and complexity, and can provide insights into a reader's perception of the content. We hypothesize that during a complex sensemaking task with many text-based documents, we will be able to use eye-tracking data to predict the importance of documents and words, which could be the basis for intelligent suggestions made by the system to an analyst. We introduce a novel eye-gaze metric called `GazeScore' that predicts an analyst's perception of the relevance of each document and word when they perform a sensemaking task. We conducted a user study to assess the effectiveness of this metric and found strong evidence that documents and words with high GazeScores are perceived as more relevant, while those with low GazeScores were considered less relevant.  We explore potential real-time applications of this metric to facilitate immersive sensemaking tasks by offering relevant suggestions.

Here's a sneak peek into our system.