W2

Week 2. Cross-Device Interaction (WebSocket)

1. Mobile Modality & Parametric Interfaces For Web

Mobile Modality: The Phone as an Instrument

• Think of the classic "2001: A Space Odyssey" scene: a bone becomes a satellite. This represents the evolution of a tool.
  • 2001: A Space Odyssey (1968) - From Bone to Satellite Scene (1/6) | Movieclips
• Steve Jobs introducing the iPhone in 2007 was a similar moment. It put a powerful new tool in our hands.
  • Steve Jobs introduces iPhone in 2007
• The best tool we have is our fingers. The iPhone/smartphone is an instrument played by our fingers.
• Our Goal: To build interfaces that respond to this "performance."
  • Instrumental Interface

Two Types of Interaction

• 1. Semantic Interaction (Meaning-based)
  • What it is: Discontinuous, categorical. You make a clear choice.
  • Examples: Clicking a button, selecting an option from a menu.
  • Used in: Most practical interfaces (apps, websites).
• 2. Phenomenological Interaction (Experience-based)
  • What it is: Continuous, flowing.
  • Examples: Moving your body, dragging your finger, scrolling.
  • Used in: Interactive art, games.
• Combining Them
  • It's not "either/or." We can—and should—use both.
  • How? By using a sum of discontinuous UIs (like buttons, toggles) to create a feeling of continuous control.
    • Example: http://internetinental.herokuapp.com/term6/delta/mobile/1 (Access from mobile)

Parametric Web Interface

• What is it? An interface where your mobile interactions (inputs) are used to change parameters (variables).
• Think of it like playing an instrument: your actions change the sound. Here, your actions change the context, the time, or the space.
• Example: Changing Time
  • Semantic way: Typing "8:00 AM" in a box.
  • Phenomenological way: Dragging a dial, like the iOS Alarm clock. (iOS Wheel Picker - inspired from Las Vegas)
  • Question: Which is more intuitive? More fun?
• Example: Changing Season
  • Semantic way: 4 separate buttons ("Spring," "Summer," "Fall," "Winter").
  • Phenomenological way: A continuous X-axis or Y-axis slider that blends between seasons.
• Example: Changing Job
  • Semantic way: A dropdown list of pre-set jobs.
  • Phenomenological way: A "blurry" selection space where you can slide between different job categories.

Input Materials (The "Keys" of Your Instrument)

• I. Selection / Navigation (Broad Control)
  • Scrolling
  • Swiping (e.g., Tinder-style card swipe)
  • Zoom-in / Zoom-out
• II. Fine-Tuning (Precise Control)
  • Finger-Skating / Sketching (drawing on the screen)
  • Set of Togglers (on/off switches)
  • Set of Sliders
  • Set of Buttons
  • Accelerometer (tilting the phone)
  • Voice/Speech
    • Semantic: Speech-to-Text (what you say)
    • Non-Semantic: Volume size (how loud you are)
  • Camera Tracking (Eye tracking, Object detection)
  • Rotating & Zooming an object

Single Device vs. Multi-Device

• Single Device (e.g., Laptop only)
  • Structure: User Input (Mouse/Keyboard) → Parameter Conversion → Visual Output
  • Inputs: Mouse click duration, mouse move velocity/path, keyboard typing speed.
  • Artistic Strategy: "Verfremdungseffekt" (Defamiliarization effect) – making a normal interaction (like typing) create an unexpected result.
• Multi-Device Web Artwork (Our Focus)
  • Input Device: Mobile (iOS / Android)
  • Output Device: Laptop (1920 x 1080 screen)
  • Why? Phone interactions (touch, tilt) have a different phenomenological feel than laptop interactions (mouse, keyboard).

Creating Your 'Visual Instrument'

• This is the core design structure for your project.
• The Flow: Mobile Input → Parameter Conversion + System Rules → Laptop Output
• Which Inputs to Use? (Mobile)
  • Button 'Tab'
  • Finger 'Swipe'
  • Zoom, Pinch, Rotate
  • Voice Input, Camera Input
  • TTS/STT (Text-to-Speech, Speech-to-Text)
  • LLM API (sending prompts to an AI)
  • Decibel (sound level)
  • External Data (weather, traffic, stock prices)
• Which Outputs to Use? (Laptop)
  • Creative Coding (Canvas, GLSL)
  • Text / Typography (changing font, size, layout)
  • Collage of Images / Videos
  • 3D Graphics (e.g., using Three.js)
  • Audio Elements (e.g., using Tone.js)

2. Multi-Device Interaction & Parametric Interface

• Multi-Device Interaction: Using two or more devices
  • Ex. Mobile: Input, Screen/Laptop: Output
• Parametric Interface: Designing your output visuals/interfaces in terms of parameters
• Understanding Web Modalities & Designing Multi-Device Interaction is Crucial! (See section 1)

3. Socket.io

• WebSocket: Bi-directional real-time communication btw client/server, enabling real-time inter-device interaction in web applications
• Socket.io: Javascript library, high-level abstraction of WebSocket, beginner-friendly (Automatic reconnection, Default fallback option)
• Strategy
  • Let's say we have Client A (Interaction Input) and Client B (Interaction Output)
  • Client A → Send Message to Relay Server → Relay Server Propagates Message to all other clients in the same room → Client B Receives a message
• Already installed in the template: socket.io (server), socket.io-client (browser)

How it is structured in this repo

• socket-server.mjs → The relay server. One file, port 4000
  • join(room): A device joins a room
  • message({ type, data }): Forwarded as-is to every other device in the same room
  • It is not part of Next.js. It is a separate Node process (that's why Vercel can't host it, see below)
• hooks/useSocket.ts → The client-side hook
  • const { send, connected } = useSocket(room, onMessage)
  • send(type, data) → emits to the room
  • onMessage(type, data) → called for every message from the other devices
  • Address: NEXT_PUBLIC_SOCKET_URL if set, otherwise the current host at port 4000
• Route convention for one experiment
  • Client A Mobile: app/experiments/w2/<name>/mobile/page.tsx → localhost:3000/experiments/w2/name/mobile
  • Client B Screen: app/experiments/w2/<name>/screen/page.tsx → localhost:3000/experiments/w2/name/screen
  • Both call useSocket("<name>", ...) with the same room name

Overall workflow

1. Screen page mounts → useSocket connects to the relay → join("name")
2. Mobile page mounts → same
3. Mobile: send("action", { x, y })
4. Relay: forwards { type: "action", data: { x, y } } to the rest of the room
5. Screen: onMessage("action", { x, y }) → update your state → render

• This is the base code. You can make it more complicated/add modularity on top of it
• What will be your speculative inter-device interaction? How will you use this technology to design your interactive/immersive audio-visual design practice?
• You can also connect Socket.io/WebSocket with TouchDesigner or Unreal (W3)

Socket.io examples

1. Simple: Tap on mobile, respond on screen
2. Continuous: Touch move / tilt on mobile, parameter on screen
3. Multiple mobiles, one screen

Deployment

• pnpm dev (http://localhost:3000) is enough when the phone is NOT involved
• Vercel hosts the Next.js app only. The relay is a long-running process, so a Vercel deployment has no socket
• For WA3 and the final, the laptop runs the relay locally. Same in the Black Room: the installation laptop is the server
• If you ever need the relay online, socket-server.mjs runs as-is on any Node host and the client follows NEXT_PUBLIC_SOCKET_URL

4. Debugging & QA

A. Mobile Debugging

• Mobile Debugging: Hard
• Cross-Device (Mobile + Desktop) Debugging: VERY HARD
  • Glad that this job won't be taken by AI very soon!

A1. Local Network Connection (Using ifconfig)

Your mobile device cannot access localhost:3000 because "localhost" refers to the device itself. You must connect your mobile to your computer's local network IP address.

• Find Your Computer's IP: On your computer, open the Terminal.
  • On macOS or Linux, type ifconfig | grep "inet ".
  • On Windows, type ipconfig.
  • pnpm dev:mobile also prints it (see A3).
• Identify the IP: Look for an IP address in the list that looks like 192.168.1.XX or 10.0.0.XX. This is your computer's local IP address.
• Connect Devices: Ensure your mobile phone and your computer are connected to the same Wi-Fi network. If the school Wi-Fi blocks device-to-device traffic, use your phone's hotspot.
• Access on Mobile: On your phone's browser (e.g., Chrome or Safari), type https://<YOUR_IP_ADDRESS>:3000. For example: https://192.168.1.10:3000/experiments/w2/simple/mobile

A2. Remote Debugging (iOS vs. Android)

This is how you "Google it" to see console.log() messages, inspect the DOM, and check network requests from your phone on your computer's screen.

Android (Chrome Debugging)

1. On Your Android Device: Enable "Developer Options" (this is usually hidden; search how to enable it for your specific phone). Inside Developer Options, toggle on "USB Debugging".
2. On Your Computer: Connect your phone to your computer with a USB cable.
3. Open desktop Chrome and type chrome://inspect#devices into the address bar.
4. You should see your device and any open tabs. Click "Inspect" on the tab you want to debug.
5. A full DevTools window will open, allowing you to inspect your mobile browser session.

iOS (Safari Web Inspector)

Note: This process requires a Mac.

1. On Your iPhone: Go to Settings > Safari > Advanced and toggle on "Web Inspector".
2. On Your Mac: Open Safari. Go to Safari > Preferences > Advanced and tick the box at the bottom: "Show Develop menu in menu bar".
3. Connect: Connect your iPhone to your Mac with a USB cable.
4. On your Mac's Safari, click the new "Develop" menu. You should see your iPhone's name.
5. Hover over your iPhone's name to see a list of open tabs. Click the tab you wish to debug to open a Web Inspector window.

A3. HTTPS for Local Dev: pnpm dev:mobile

Modern browsers block sensitive APIs (like the camera, microphone, or accelerometer) on insecure http:// connections for security reasons. localhost is an exception (it counts as a secure context), which is why pnpm dev was enough in W1. The moment your phone connects via 192.168.x.x, it is not localhost anymore, so you must serve your local site over https://.

What pnpm dev:mobile does

• Runs next dev --experimental-https -H 0.0.0.0 → Next.js generates a self-signed certificate into certificates/ (first time only, it may ask for your password) and listens on https://localhost:3000, reachable from your phone via your IP
• Runs node socket-server.mjs → the relay reuses the same certificate and listens on https://localhost:4000. It prints the phone URL

CRITICAL: Bypassing the Security Warning (twice)

This uses a self-signed certificate, which your phone does not trust by default. You will see a large security warning page (e.g., "Your connection is not private" or NET::ERR_CERT_AUTHORITY_INVALID).

• This is normal and expected.
• On the warning screen, tap "Advanced" → "Proceed (unsafe)".
• You have to do this for BOTH ports:
  1. https://<YOUR_IP>:3000 → the page
  2. https://<YOUR_IP>:4000 → the relay. Open it once in the phone browser, pass the warning, you should see "socket relay ok"
• If you skip step 2 the symptom is: "the page loads but nothing arrives on the screen". The socket is silently blocked.

You only need to do this once per device (until the certificate changes). Your phone will then load the https:// version of your site, and your sensitive APIs (camera, accelerometer, etc.) will now work for testing.

B. Other Debugging Strategies