StreamInput - Cross-platform advanced sensor processing and user interaction

The Khronos StreamInput working group is driving industry consensus to create a cross-platform API to enable applications to discover and use new generation sensors to create sophisticated user interactions. The new API will support a general-purpose framework for consistently handling advanced sensors such as depth cameras, touch screens and motion and orientation sensors as well as traditional input devices. StreamInput will provide flexible device discovery to enable an application to select and process high-level semantic input from low-level device capabilities, enabling significant innovations by sensor and device manufacturers while simplifying portable application development. The API will also provide system-wide sensor synchronization for advanced multi-sensor applications such as augmented reality, and will use Khronos’ proven extension mechanisms to enable new types of input devices to be easily added and supported.

Compute Power Driving Sensor Innovation

StreamInput Sensor Processing

• Application discovers what sensors can provide required semantic input
  • “Am I in an elevator, in a moving car, or being carried in a pocket”
  • “Need full body tracking and gestures”
• Enables application portability – don‟t have to know sensor details
  • Different devices can have different sensors that generate required input
  • Developers don‟t have to be sensor experts
  • Enables innovative holistic multi-sensor processing under the API
• Handles almost any sensor
  • Multi-axis motion/position sensors, capacitive multi-touch surfaces
  • RGB and Depth sensing cameras – including mono and stereo
  • Microphone arrays for speech recognition etc.
  • Haptic devices, Biometric sensors etc. etc.
• OS Agnostic
  • Can be built over system libraries and resources

Core StreamInput Concepts

• Graph-based API
  • Application sets up a graph of input device nodes
  • Filter nodes modify and process input data
  • Graph generates high or low level data for app
• Filter Nodes can include image processing
  • Can use APIs such as OpenMAX or OpenCL
  • Can receive data from external media frameworks Filter
• Multi-sensor synchronization Node
  • Required for applications such as augmented reality
  • UST Timestamp EVERYTHING in a system
  • Camera frames, Audio samples, Sensor samples, Display buffer switch times
  • App can then detect and compensate for sensor/pipeline/rendering delays

Aiming for Spec in under 12 Months

• Khronos welcomes new members to define and drive StreamInput
  • Initiated and chaired by TransGaming
  • .(JavaScript must be enabled to view this email address)