I began my contributions for the Google Summer of Code with Waycrate at the beginning of June, with a project called “Improve RDP support for xdg-desktop-portal-luminous”.

The goal of the project was to fix several issues with the implementation of xdg-desktop-portal-luminous, an xdg-desktop-portal backend written in Rust for wlroots compositors, to ultimately enhance remote control. The issues, related to methods in the RemoteDesktop and ScreenCast portals, are the following:

The first week was spent understanding the codebase, especially the areas that were relevant for the issues I was about to work on, as well as tools useful for debugging, such as d-feet/d-spy, which are utilities for simulating individual D-Bus calls. They are very useful for testing changes on single methods, although they do not fit every use case, since some of the issues I worked on appeared on sequences of function calls.

The first issue I addressed was the simulation of key combinations. It turns out that when RemoteDeskop’s NotifyKeyboardKeycode method was called to notify key presses, the internal implementation simply forwarded these calls to a ZwpVirtualKeyboardV1 object. However, key combinations are not obtained by setting the state of the keys involved to “pressed” one after the other, but rather by simulating a key press with a modifier: for example, the A character is produced by simulating the press of a with the Shift modifier, not by simulating Shift followed by a. I added a Modifiers enum to map each modifier key to its bitflag, and I obtained the keycodes representing the modifier keys from the header file /usr/include/linux/input-event-codes.h. Pressing a modifier key now sets its bit in the modifiers bitflag to 1, while releasing it sets the bit back to 0. The CapsLock modifier is handled separately since it is the only modifier key that remains active after being released: each time it is pressed, its bit in the modifiers bitflag toggles.

To understand these concepts, I found a project called wvkbd, a virtual keyboard for wlroots compositors implemented in C, to be very useful. I built it and looked into it to understand how the problem was tackled.

I then fixed simulated cursor movements: the error was related to the implementation of RemoteDesktop’s NotifyPointerMotionAbsolute method, which is employed to update the cursor position during remote connections. To simulate the cursor movements, the motion_absolute method of ZwlrVirtualPointerV1 is used, which accepts two unsigned integers as parameters, x_extent and y_extent, that represent the dimension of the WlOutput. These parameters were hardcoded, and this produced an error because they are internally used for scaling purposes. I refactored the code in order to pass correct values to the function, obtained from libwaysip.

I also fixed RemoteDesktop’s NotifyKeyboardKeysym method, which is used to simulate keyboard presses using keysyms. Keysyms, which are related to X11 key management, have a different semantic meaning compared to keycodes, since a keysym can represent a keycode with a modifier (i.e. there is a keysym that represents a and another that represents a+AltGr or a+Shift). This means a keycode is mapped to different keysysms depending on the modifiers active. /usr/share/X11/xkb/symbols/ contains the layout mappings from keycodes to keysyms depending on the level, and through libxkbcommon I was able to first convert the keysyms to keycodes and then use the already implemented NotifyKeyboardKeycode method.

I have also found and fixed an error about the keyboard layout of the virtual keyboard, that was statically created with the US layout. I managed to listen to wl_keyboard::keymap events from the compositor to obtain the current layout dynamically at the beginning of each remote session.

The last issue was related to how PipeWire streams are handled to enable screensharing. When a PipeWire stream is started, a list of available color formats is advertised. The PipeWire server then sends a param-changed event, which includes the chosen format among the available ones. Buffers then need to be filled using the negotiated format. The previous implementation employed a hardcoded list of available formats, ignored the param-changed event and performed the screen capture with a random format. To fix this issue I also had to make changes in wayshot, Waycrate’s screenshot tool, which includes libwayshot, a library published as a standalone crate that is employed by xdg-desktop-portal-luminous to perform screen capture.

Future work#

During the course of the project I have also looked into RemoteDesktop’s methods that simulate touch events(NotifyTouchDown, NotifyTouchMotion and NotifyTouchUp). wlroots compositors do not currently implement any Wayland protocol similar to wlr virtual pointer or virtual keyboard to simulate touch events, therefore these methods were only added as empty functions. I have also looked into RemoteDesktop’s ConnectToEIS method, to send input events through an EIS connection. As I found that wlroots does not currently provide an EIS implementation, I opened an issue to gather information about the EI protocol. I also documented the attempts by other xdg-desktop-portal backends to support input sharing through EIS connections and looked into reis, a library that provides a Rust version of EI and EIS, which could be employed in the future to implement the ConnectToEIS method. Future advances could enable seamless and correct implementation of the aforementioned methods. An additional improvement would be to listen to wl_keyboard::keymap events directly in the event loop to react to keyboard layout changes while a connection is active. This approach was attempted but resulted in the event loop freezing, requiring further debugging as documented in xdg-desktop-portal-luminous/pull/89.