Skip to main content

Device Communication

  • Each device needs to know if other devices are online before doing any operation

Namespace: /device​

Payload Type Definition​

enum Feature {
  ClipboardSync = "clipboard_sync",
  MouseShare = "mouse_share",
  FileTransfer = "file_transfer",
}

type WebRTCInfo {
    // TODO
}

type DeviceConnectPayload = {
  features: Array<Feature>;
  user_id: number;
  device_id: number;
  device_name: string;
  utc_timestamp: Date;
  socket_id?: string;
  sync_socket_id?: string;
  file_transfer_socket_id?: string;
//   webrtc_info?: WebRTCInfo
};

type DeviceDisconnectPayload = {
  socket_id: string; // this is enough for other devices to find the rest of the info
};

type DeviceUpdatePayload = {
    user_id: number,
    device_id: number,
    device_name: number,
    features: Array<Feature>
    sync_socket_id?: string;
    file_transfer_socket_id?: string;
    // webrtc_info: WebRTCInfo // for example, make a new connection
}

Events​

  • connect
    • connect is a built-in event of socket-io, everything happens when a device first connect to server
  • Client-side connect
    • Client listens on connect. When successfully connected with server on namespace /device, client will also need to connect to /clipboard_sync and /mouse_share namespaces if the features are enabled
    • We will talk about the connect event later in the corresponding namespaces
    • Note: TODO
  • device_connect
    • Emitted after connect
    • When a device gets online, client emit device_connect to server with DeviceConnectPayload
    • Server join socket to room user-room/<user_id> of namespace /device
    • Server broadcast event device_connect to room user-room/<user_id> (namespace /device) with received DeviceConnectPayload
      • Note: the broadcast event should include socket_id (add it if not added initially), so other devices can associate socket id with device (sometimes only socket id is available)
    • Friend Status
      • When a user device gets online, user's friends should all be notified so that they know the user is ready for communication
      • When a device gets online, not only its fellow devices (same owner) are notified, all devices that belong to the friends of the owner should also be notified. This is why we have the user_id field in DeviceConnectPayload
      • This is done by querying all friends of the current user and broadcast the device_connect event to all of their user rooms in the /device namespace.
  • device_disconnect
    • Server listens on disconnect event, when a device disconnects, broadcast to all 3 rooms (in different namespaces) mentioned above with event device_disconnect, and payload DeviceDisconnectPayload
    • When other devices receives the event, they find the device list stored locally, find the device with the socket_id, and set the device status to offline (should reflect in UI).
  • device_update
    • When a device enables or disables a feature, DeviceUpdatePayload is send to server, and broadcast to all devices. WebRTC connects can be established or dropped depending on this update.

Note: From now on, we will call room id user-room/<user_id> "user room" for convenience.

Workflow​

In the sequence diagram below, I illustrate the workflow using 2 devices, when I say broadcast I only draw one arrow to one device, but it's actually broadcasting to all devices in room

Now, every device knows the status of every other device.

Friends Status​

We want to have a QQ-like friend status feature, so that we know whether our friends are ready to communicate.

  1. When a device connects to SocketIO Server, with device_connect event and DeviceConnectPayload
  2. Server should find all of the user's friends, find their user rooms (user-room/<user_id>)
    1. If a user room exists, broadcast the

Namespace: /file_transfer​

Given that all devices know the status (online or not) of all other devices, choose a file and the target device.

Payload Type Definition​

type FileTransferRequestPayload = {
    webrtc: {
        info: string,
    },
    file: {
        filename: string.
        size: number
    },
    device: {
        socket_id: string,
        id: number,
        platform: PlatformEnum // this DeviceEnum should be defined as a common varaible with all platforms
    },
    target: {
        socket_id: string,
        device_id: number
    }
}

type FileTransferResponsePayload = {
    accept: boolean,
    message: string,
    webrtc: {
        info: string
    },
    device: {
        socket_id: string,
        id: number,
        platform: PlatformEnum // this DeviceEnum should be defined as a common varaible with all platforms
    },
    source: {
        socket_id: string,
        device_id: number
    }
}

Event​

  • file_transfer_request
    • Create WebRTC connection and emit file_transfer_request to server with FileTransferRequestPayload
    • Server on file_transfer_request event: emit file_transfer_request event to target based on socket id specified in payload
  • file_transfer_response
    • When target device get file_transfer_request, respond with file_transfer_response event
    • Respond with FileTransferResponsePayload
    • If file too large for target device platform (e.g. web and file size is 2G), or for some other reason target device canont do file transfer, set accept to false
      • Sample reasons
        1. Storage not enough
        2. Device is on cellular data and file size exceed limit set for cellular data
    • When server gets file_transfer_response event, server will emit file_transfer_response to source device by the socket id included in FileTransferResponsePayload

Problem: Different Socket ID from Different Namespaces​

In /device namespace, devices already exchanged socket id with each other, but notice that socket id for different namespaces are different. The /device socket id cannot be used in /file-transfer namespace. In the WebRTC version, each 2 devices have to be able to reach each other directly using socket id in the /file-transfer namespace.

What are the solutions?

One solution is simply repeat the "greeting" (share socket id, features, ...) in namespace /device in /file-transfernamespace, but doesn't this sound stupid?

The purpose of /device namespace is to get devices to know each other and keep all devices updated with other devices' pubic information. So why not use it as a central namespace to also share /file-transfer namespace socket io?

Solution: Add an optional file-transfer_socket_id field to DeviceConnectPayload and DeviceUpdatePayload.

When a device connects to namespace /file-transfer, it can emit event device_update in /device namespace to update its /file-transfer socket id. This also helps other devices to know wheter one device is ready to transfer.

Once everyone knows everyone else's /file-transfer socket id, the WebRTC connection can proceed.

Workflow​

Namespace: /sync​

Sync namespace will be responsible for syncing both clipboard and mouse sharing. The original plan is to use 2 namespaces and 2 separate groups of WebRTC connections. If not combined, the number of WebRTC connections will double, which is stupid design.

Since devices can enable one or both features, we need 1 user rooms for each feature.

  • Room clipboard_sync/user_room/<user_id>
  • Room mouse_share/user_room/<user_id>

Payload Type Definition​

Current payload design is just a draft, we have to think more carefully what to include in payload so every device knows whether it has the "focus"

type ConnectionPayload = {
  features: Array<Feature>; // Feature is a enum defined earlier
};

enum UpdateType {
  Clipboard = "clipboard",
  MouseMove = "mouse_move",
  KeyboardMove = "keyboard_move",
}

type MouseMovePayload = {
  x: number;
  y: number;
};

type KeyboardMovePayload = {
  key: string;
};

enum ClipboardDataType {
  Text = "text",
  Image = "image",
}

type NewClipboardDataPayload = {
  type: ClipboardDataType;
  value: string;
  uuid: string; // used to avoid duplicate
  utc_time: Date;
};

type UpdatePayload = {
  type: UpdateType;
  device_id: number;
  value: MouseMovePayload | KeyboardMovePayload | NewClipboardDataPayload;
};

SocketIO Version​

Event​

  • connect
    • When client connects to /sync namespace with ConnectionPayload, server will join socket to the rooms clipboard_sync/user_room/<user_id> or mouse_share/user_room/<user_id> depending on features enabled. How to join room is a problem we will discuss.
  • update
    • When there is an update/move in key, mouse or clipboard, emit update to server with UpdatePayload
    • Server will broadcast the update payload to all devices in the corresponding room

How to decide room during broadcasting?​

This is a problem because some devices may enable only one feature and some enable both. It's possible that one want mouse and keyboard share but not clipboard share. To avoid broadcasting the same payload to the same device twice, we need a solution.

The root problem is, clipboard sync is part of mouse share, otherwise we can simply use 2 separate rooms.

Solution: We can remove the clipboard sync function from mouse room. If we remove the overlap, the problem is solved.

Room Joining​
if clipboard and mouse both enabled {
    join socket to both clipboard and mouse room
} else {
    if only clipboard enabled {
        join clipboard room
    } else if only mouse enabled {
        join mouse room
    } else {
        // none enabled
    }
}
Broadcasting​

Pseudocode

// when a payload is received
switch payload.type {
    case mouse:
        broadcast to mouse room
        break;
    case clipboard:
        broadcast to clipboard
        break;
    default:
        Unexpected
}

Workflow​

WebRTC Version​

Since WebRTC uses P2P connection, we don't have the broadcasting problem above. Every connection is one-to-one, and clients can decide whether or not to send a payload. Remember that in /device namespace connection, devices share their enabled features with all connected devices.

Assumption: Mouse share is a superset of clipboard sync in terms of functionality

Analysis​

  • If device A enables only clipboard sync and device B enables only mouse share, they don't need a connection

  • If a device enables both features, it will need to establish a connection with every other device in the /sync user room.

  • This is like a XOR operation, features are represented in one-hot vector.

    • One-Hot Vector definition
      • 11 means both enabled
      • 01 means only mouse share is enabled
      • 10 means only clipboard sync is enabled
      • 00 means none enabled (not possible in /sync namespace as devices who connect to the namespace enables either feature)
    • The function to decide if 2 devices need a connection is expressed as
      • XOR(onehot(A), onehot(B)) > 0
    • XOR is the idea, we don't really have to implement the bitwise XOR operation, it's too hard to understand for other readers. The following logic is the equivalent.
      •   def need_connection(A: Device, B: Device):
              return (A.mouse and B.mouse) or (A.clipboard and B.clipboard)

Now, how to establish so many connections efficiently using SocketIO?

Since in /device namespace connection stage, we know the features enabled (as well as device id and socket id) for all devices, each device can compute whether it needs a connection with another device. They will use SocketIO to share WebRTC connection info.

Problem 1: How to solve conflict?​

2 devices which need a connection don't know who to start the request, they need to reach a concensus.

This is similar to Block Chain. The problem can be easily solved with public information (both device have). e.g. larger device id starts the connection (this is not a good solution, the device with largest id will start all connections, not sure this is slow, but this sounds unfair). We can use math to introduce some randomness.

  • For example, both devices know the device id of each other, then we can implement the following logic

    • def i_should_start_connection(self_device: Device, other_device: Device):
          device_id_product = self_device.device_id * other_device.device_id
          if self_device.device_id > other_device.device_id:
              return is_odd(device_id_product)
          else:
              return is_even(device_id_product)
    • Two devices starts to gamble blindly.
      • Then device with larger device id bet the product is even, the device with smaller device id bet the product is odd. Only one of them is true.
    • Product introduces a little bit of randomness. Makes the "game" fair.

Another solution is simply let server decide who to start, but this may be more complicated as server needs to store the feature info for each connected device.

Problem 2: Different Socket ID from Different Namespaces​

I've discussed this in /file-transfer namespace.

Same solution for /sync namespace, simply add optional sync_socket_id field to DeviceConnectPayload and DeviceUpdatePayload. Devices can update their sync socket id after connecting to /sync namespace.

This also helps devices know when a device is ready to sync.

Algorithm​

  1. Every time a new device joins the /device namespace user room (means "online"), all devices in the user room are notified with its features enabled.
    1. Every device will start to compute whether it needs a connection with the new device using the previously described XOR logic.
    2. If a connection is needed, a device will compute who should start the connection, itself or the new device. The chosen requester will start the request.
  2. Once connection is established, they will start to broadcast data to connected devices.
    1. The type of data to send can also be computed easily. If the payload type is enabled on the device, then send it.
      1. If A has a clipboard payload to send, send it if B has enabled clipboard syncing
      2. If A has a mouse/keyboard payload to send, send it if B has enabled mouse sharing

Payload Type Definition​

type NewWebRTCConnectionRequestPayload = {
  webrtc_info: WebRTCInfo;
  device: {
    socket_id: string;
    id: number;
    platorm: PlatformEnum;
  };
  target: {
    socket_id: string;
    device_id: number;
  };
};

type NewWebRTCConnectionResponsePayload = {
  accept: boolean;
  message: string;
  webrtc_info: WebRTCInfo;
  device: {
    socket_id: string;
    id: number;
    platorm: PlatformEnum;
  };
  target: {
    socket_id: string;
    device_id: number;
  };
};

Event​

All there is left for SocketIO to do is helping to establish WebRTC connection. The process and payload will be pretty much the same as WebRTC-based File Transfer

Let's say A starts a connection to B.

  • new_webrtc_connection_request
    • A emits new_webrtc_connection_request to server with NewWebRTCConnectionRequestPayload, server will emit to target device (socket_id included in payload)
  • new_webrtc_connection_response
    • When the target device get new_webrtc_connection_request, respond with new_webrtc_connection_response event and NewWebRTCConnectionResponsePayload
  • When server gets new_webrtc_connection_response event, server will emit new_webrtc_connection_response to source device by the socket id included in NewWebRTCConnectionResponsePayload

Stranger​

How to communicate with a stranger (user that is not logged in)? How should a user room be created?

Magic Wormhole Style​

  1. Device A starts a file transfer request to server with some file metadata. Without knowing the destination.
  2. Server will create a 3-word passcode such as 1-apple-banana
    1. The first number 1 is a channel id to uniquely identify the current transfer.
    2. apple and banana should be 2 easy-to-recognize words as the actual passcode.
  3. Server creates a SocketIO room with anounymous_transfer/1-apple-banana
  4. When another user enters the passcode 1-apple-banana on another device (B), server will first look for anounymous_transfer/1-* room to check if this file transfer event is going on.
    1. If the room exists, then check the passcode.
      1. If the passcode is correct, Device A and B can now exchange their WebRTC information and start transferring.
      2. If the passcode is incorrect, notify both devices. If the passcode is wrong for too many times, the transfer request should be cancelled.
    2. If the room doesn't exist, notify the user.

One thing to worry about​

  • What if "hackers" keep sending request to our service with all ranges of channel id causing all transfers to fail on purpose?