VirtualKeyMouse turns an Android phone into a wireless keyboard and mouse. The phone connects to an ESP32-S3 over Bluetooth, and the ESP32-S3 presents itself to the target device as a standard USB HID keyboard and mouse.
The target device does not need custom drivers or a proprietary receiver. It only sees normal USB input, which makes the approach useful for PCs, smart TVs, kiosks, embedded systems, and other devices that accept USB keyboard or mouse input.
Architecture
- 1Android phone over Bluetooth to ESP32-S3
- 2ESP32-S3 over USB HID to target device
- 3Target device receives standard keyboard and mouse reports
Current app features
- Trackpad and keyboard interface
- QR scanner for quick pairing or configuration
- TV-style controls such as arrows, home, and volume keys
- Macro system for triggering multiple actions with one button
- Custom buttons for user-defined key mappings and behaviors
Runtime Behavior
The protocol uses Nordic UART Service as a soft real-time stream. The Android app continuously pushes small text commands into the BLE RX characteristic. There is no strict request-response model, no guaranteed one-to-one mapping between app sends and BLE packets, and no application-level delivery acknowledgment.
- Small delays are acceptable.
- Occasional packet drops are tolerable, especially for mouse movement.
- Predictability comes from frequent lightweight updates, not heavyweight reliability.
Command Processing
On the ESP32-S3, the parsing pipeline is intentionally small: receive bytes, append them to a buffer, detect a command boundary, parse the command, and dispatch to a handler. This avoids dynamic allocation, complex parsing trees, and heavy string processing in the hot path.
- Mouse movement commands can be sent many times per second.
- Short commands such as MOUSE:5,3 fit efficiently within BLE payload constraints.
- Handlers should avoid blocking work so HID reports stay responsive.
HID Mapping
Commands are converted into USB HID reports rather than direct operating system events. Keyboard commands become keyboard usage reports, mouse movement becomes X/Y deltas, and click commands set and clear mouse button bitmasks.
- Key commands are usually press and release sequences.
- Mouse button press and release commands can maintain hold state.
- The device avoids cursor tracking and keeps most state on the app side.
Protocol Tradeoffs
The protocol is text-based even though binary would be more compact. Text keeps iteration fast, serial debugging easy, logs readable, and backward compatibility simpler while the command set is still evolving.
- Unknown commands can be ignored and logged.
- Invalid parameters can be skipped without breaking the session.
- Small atomic commands are easier to compose and debug than large complex gestures.
Future Direction
The project can evolve without losing its simple base protocol. Higher-frequency input can get a binary fast path, macros can gain optional acknowledgments, and devices can report supported capabilities for safer versioning.
- Hybrid text and binary command modes.
- Command batching to reduce BLE overhead.
- Sequence IDs, acknowledgments, and capability negotiation for critical actions.