Wormhole Style Anonymouse File Transfer Design

I want to design a magic-wormhole-style file transfer system that is anonymous and secure. This is a design document for the system.

However instead of using magic-wormhole's rendezvous server and protocol, we will use WebRTC.

• Requirements
• Usage
• Algorithm
  • Events
• Further Considerations
  • Multiple File Transfer Design Using the Same Connection
    • Same RTCPeerConnection
    • Same Socket.io Connection

Requirements

• Use WebRTC
• Design should be as simple as possible, no auth required
• The connection should be atomic, don't prepare anything when the other side isn't ready
• Supports both text and file

Usage

xc send <file>
# gives you a code: e.g. 5-great-fox
xc receive <code>
# downloads the file

Algorithm

1. Clint 1 runs xc send <file>

   1. CrossCopy reads file info (size, name, shasum) and send a file transfer request (HTTP) to the server
      1. shasum will be used later to verify the file integrity in the destination
      2. shasum can be computed asynchronously in the background and upload to the server later. In case of a large file, the client can start uploading the file while the server is computing the shasum.
      3. Client info should also be included (e.g. platform). This is because web platform has size limitation (max 200MB)

2. Server generates an unique passphrase (e.g. 5-great-fox), and creates a new file transfer session in database (e.g. redis)

   1. store the file info in the session (e.g. redis hashes or json)

   hset anonymous:file-transfer:5-great-fox size 1234
   hset anonymous:file-transfer:5-great-fox name app.zip
   hset anonymous:file-transfer:5-great-fox shasum 2b40c047855d459576fd685d058fe21c1278808b
   EXPIRE anonymous:file-transfer:5-great-fox 600       # 10 minutes expiry

3. Server sends the passphrase back to client 1

4. Client 1 displays passphrase, url to copy and QR Code

   1. User 1 should somehow tell User 2 the passphrase

5. Client 1 connects to server through Socket.io and wait for Client 2

   1. Use a custom socketio namespace (e.g. anonymous-file-transfer)
   2. Connection handshake query should include the passphrase
   3. Listen to receiver-online event, once receiver is online, start connection and transfer. let's call it start() function

6. Server get's connection from client 1

   1. Store socketid in database

      hset anonymous:file-transfer:5-great-fox sender-socketid <socket-id>

   2. Wait for Client 2's connection

7. Client 2 runs xc receive 5-great-fox

   1. CrossCopy sends a post request with the passprase in request body to get file transfer information
      1. If Client 2 is web platform and the file size exceeds the limit, the server should return an error to client 2
   2. CrossCopy connects to server with socket.io and the passphrase in handshake query

8. Servers receives connection

   1. Look for the passphrase in database

      1. If not found, return an error to client 2 and close connection
      2. If found, store the socketid in database

        hset anonymous:file-transfer:5-great-fox receiver-socketid <socket-id>
      3. Send receiver-online event to client 1

9. Client 1 receives receiver-online event and start()

   1. Create RTCPeerConnection, RTCDataChannel and offer
   2. Set up ice candidate listener
   3. Set offer to local description
   4. Emit offer event to client 2 (server simply forwards the event to client 2)
   5. Wait for answer event from client 2

10. Client 2 receives offer event

11. Create RTCPeerConnection, RTCDataChannel and answer

    1. Also set up ice candidate listener

12. Set offer to remote description

13. Set answer to local description

14. Emit answer event to client 1 (server simply forwards the event to client 1)

15. Client 1 receives answer event

    1. Set answer to remote description
    2. When data channel's on open is triggered
       1. Use something like BufReader to read file chunk by chunk (huge file may overflow memory)
       2. For every chunk read, send the data through RTCDataChannel
       3. After all chunks are sent, close the data channel.
          1. Consider whether data loss is possible if the data channel is closed before all data is received.
          2. If possible, wait for a socketio event from client 2 to close the data channel. Client 2 knows the file size and can verify if all data is received. But it's possible the transfer hangs as client 2 keeps waiting for new data if unexpected error happens. Think carefully.

16. Client 2 receives file data

    1. When on message triggered
       1. Record the timestamp the first chunk arrives
       2. Append received data to the target file
       3. Accumulate the received data size
       4. If the accumulated data size is equal to the file size, the file transfer is complete, but let's wait for data channel close event
    2. After data channel closed
    3. Compare the received file size with the expected file size.
       1. If they match, the file transfer is successful. Emit a success event to Client 1, and display a message.
       2. If not, there was an error in the file transfer. Emit a failure event to Client 1, and display a message.
    4. Verify file integrity with shasum
       1. Display a message saying the file is being verified.
       2. Emit success or failure event to Client 1 and display a message.

17. Once the file transfer is complete

    1. Close RTCPeerConnection.
    2. Close the socket.io connection.

Events

• receiver-online
• offer
• answer
• success
• failure
• verify-success
• verify-failure
• error

Further Considerations

Multiple File Transfer Design Using the Same Connection

Same RTCPeerConnection

For transferring multiple files using the same RTCPeerConnection, we can consider the following design:

1. Client 1 sends a list of files (with their metadata) to Client 2 before starting the transfer. This can be done using the RTCDataChannel.

2. Client 2 acknowledges the receipt of the file list and prepares for receiving multiple files.

3. Client 1 starts sending files one by one. For each file, it sends a 'start-file' message with the file's metadata (name, size, etc.) before sending the file data.

4. Client 2, upon receiving a 'start-file' message, prepares to receive a new file and starts writing the incoming data to a new file.

5. Once a file is completely transferred, Client 1 sends an 'end-file' message. Client 2, upon receiving this message, finalizes the current file and prepares for the next one.

6. This process continues until all files are transferred. After the last file, Client 1 sends an 'end-transfer' message to signal the end of the transfer.

7. Client 2, upon receiving the 'end-transfer' message, finalizes the last file and closes the RTCDataChannel.

This design allows for the transfer of multiple files over a single RTCPeerConnection, but it also introduces additional complexity. Error handling becomes more important and complex, as errors can now occur at the file level as well as at the connection level. For example, if a file fails to transfer correctly, the system needs to decide whether to abort the entire transfer or to skip the problematic file and continue with the next one.

Same Socket.io Connection