To achieve computer control of Android devices, I developed a cross-platform desktop client using Tarui + Vue 3 + Rust. This client provides visual mouse and keyboard mapping configuration, enabling multi-touch operations similar to Android emulators through key mapping, with millisecond-level response time. This tool can be widely used for controlling Android devices from computers to play mobile games, providing a smooth touch experience.
I express my deep respect and gratitude to the developers of the Scrcpy project. Scrcpy is a powerful and efficient open-source tool that greatly facilitates control over Android devices. This project is built upon the excellent architecture of Scrcpy, with optimizations and adjustments for mouse and keyboard control.
Because screen mirroring may involve latency and blurriness issues, this project takes a different approach by directly abandoning screen mirroring and instead using a transparent mask to display the content behind the window (which can be AVD, low-latency screen mirroring provided by your phone manufacturers, etc.), Completely eliminates the problem of poor screen casting experience inherent in Scrcpy.
**If you really need screen mirroring and don't mind the latency and performance issues,** you can use the Android app [ScreenStream](https://github.com/dkrivoruchko/ScreenStream) for LAN screen mirroring. Scrcpy MAsk has been adapted to work with ScreenStream since version `v0.5.0`. While its performance may leave something to be desired, it is ready to use out of the box.
Furthermore, to better support interaction between Scrcpy Mask and Android devices, I have made some modifications to the scrcpy-server, leading to the creation of a separate branch project called [scrcpy-mask-server](https://github.com/AkiChase/scrcpy-mask-server).
1. You need to solve the problem of screen casting on your own. Recommend using the official screen mirror method of your device brand to achieve the minimum delay. Since `v0.5.0` version, it can be used with [ScreenStream](https://github.com/dkrivoruchko/ScreenStream) to cast screen under the same LAN.
2. Enable ADB debugging on your device via USB or wirelessly, then connect it to your computer.
2. For emulator, you don't need screen mirror, and emulator generally default to enabling ADB wired debugging. So this is the best way for game, I think.
4. Navigate to the Settings page -> Mask Settings, set the width and height of the mask to the same ratio of the device screen size and ensure that the mask size is appropriate.
5. Navigate to the Mask page where you can see a transparent mask. Next, adjust and move your emulator window or screen mirroring window to align the displayed content area with the transparent mask area.
6. Navigate to the Key mapping page and switch or edit the key mapping configs.
Currently, the structure of macros is simply a JSON object with limited functionality, serving as a transitional solution. **Please refrain from investing too much time in writing macros, as the specifications for macro creation may change with version updates.**
An example of macros can be found in the `async function execMacro` function in [hotkey.ts](https://github.com/AkiChase/scrcpy-mask/blob/master/src/hotkey.ts) file.
For instance, the `key-input-mode` macro can switch from key mapping mode to key input mode, commonly used for text input. An example is as follows:
When asking a question, please provide as much detailed information as possible regarding the issue, including the operating system and software version. Specifically, if there is an error output, please be sure to include the relevant logs.
There are two sources of logs that might help in identifying and solving the error. Generally, the error output can be found in the Web logs.
1. Web Logs: Open Developer Tools by pressing `Ctrl+Shift+I` or `Cmd+Opt+I`, click on the console tab, and check the information output in the console.
2. Rust Logs:
1. On macOS or Linux, navigate to the installation directory, use the **terminal** to run `scrcpy-mask`, and you can see the program's output in real-time in the terminal.
2. On Windows, you need to clone the project and run it yourself to view the Rust output.
If you are interested in this project, you are welcome to submit pull request or issue.But my time and energy is limited, so I may not be able to deal with it all.
