AR-Menu/.agents/skills/needle-engine/references/networking.md

Needle Engine — Networking Reference

Table of Contents

• Core: context.connection
• Persistent vs ephemeral messages
• SyncedRoom
• @syncField
• SyncedTransform
• PlayerSync + PlayerState
• Voip and ScreenCapture

---

Some APIs documented here (e.g. Voip volume/speaking detection, PlayerSync.setupFrom with Object3D) may require the latest pre-release version of @needle-tools/engine.

Needle Engine networking is layered. The lowest level is this.context.connection (WebSocket rooms + messages). Higher-level components build on it.

Core: this.context.connection

// Connect and join a room — no SyncedRoom needed
this.context.connection.connect();
this.context.connection.joinRoom("my-room");

// Room state
this.context.connection.isConnected      // boolean
this.context.connection.isInRoom         // boolean
this.context.connection.connectionId     // this client's ID
this.context.connection.usersInRoom()    // all user IDs in current room

// Send and receive custom messages
this.context.connection.send("my-event", { score: 10, name: "Alice" });
this.context.connection.beginListen("my-event", (msg: { score: number; name: string }) => {
  console.log(msg.score, msg.name);
});
this.context.connection.stopListen("my-event", handler);  // always clean up in onDisable/onDestroy

// Room lifecycle events (import { RoomEvents } from "@needle-tools/engine")
this.context.connection.beginListen(RoomEvents.JoinedRoom, () => { ... });
this.context.connection.beginListen(RoomEvents.LeftRoom, () => { ... });
this.context.connection.beginListen(RoomEvents.UserJoinedRoom, (evt) => {
  console.log("User joined:", evt.userId);   // evt: { userId: string }
});
this.context.connection.beginListen(RoomEvents.UserLeftRoom, (evt) => {
  console.log("User left:", evt.userId);     // evt: { userId: string }
});
this.context.connection.beginListen(RoomEvents.RoomStateSent, () => { ... }); // all persisted state received

// Always check connection state before sending:
if (this.context.connection.isInRoom) {
  this.context.connection.send("my-event", { data: 42 });
}

Important: send() broadcasts to all users in the room except yourself — you won't receive your own messages. Custom messages do NOT automatically include a sender ID. If you need to identify who sent a message (e.g. to map data to a specific player), include connectionId yourself:

this.context.connection.send("player-score", {
  senderId: this.context.connection.connectionId,
  score: 42,
});

this.context.connection.beginListen("player-score", (msg) => {
  // msg.senderId tells you which player sent this
  playerScores.set(msg.senderId, msg.score);
});

userId is only available in room lifecycle events (UserJoinedRoom, UserLeftRoom), not in custom messages.

---

Networked Instantiation and Destruction

For spawning objects that should appear on all clients, use syncInstantiate / instantiateSynced instead of manually sending custom events.

import { instantiate, syncInstantiate, syncDestroy, registerPrefabProvider } from "@needle-tools/engine";

// Local clone (only on this client)
const clone = instantiate(prefabObject, { parent: this.gameObject });

// Networked spawn (appears on all connected clients)
const networked = syncInstantiate(prefabObject, {
  parent: this.gameObject,
  position: [x, y, z],
  deleteOnDisconnect: true,    // removed when the spawning user disconnects
});

// Persistent networked spawn (survives disconnects, replayed to late joiners)
const persistent = syncInstantiate(prefabObject, {
  parent: this.gameObject,
  deleteOnDisconnect: false,
});

// Via AssetReference
const synced = await myAssetRef.instantiateSynced({
  parent: this.gameObject,
  deleteOnDisconnect: false,
});

// Networked destroy (removed on all clients)
syncDestroy(obj, this.context.connection, true);

// Listen for remote syncInstantiate events (get references to objects spawned by other users)
import { onSyncInstantiate } from "@needle-tools/engine";
const unsub = onSyncInstantiate((instance, model) => {
  console.log("Remote object created:", instance.name, model.originalGuid);
});
// later: unsub();

Runtime prefabs with registerPrefabProvider

For runtime-created prefabs (not loaded from GLB), every client must register the prefab in their setup code — not just the client that calls syncInstantiate. This is because late joiners receive state replay and need to resolve the prefab by guid locally.

import { registerPrefabProvider, ObjectUtils, syncInstantiate } from "@needle-tools/engine";

// ALL clients run this setup code:
const cookiePrefab = ObjectUtils.createPrimitive("Cube", { color: 0xff8c00 });
cookiePrefab.guid = "cookie-prefab";
registerPrefabProvider("cookie-prefab", async () => cookiePrefab);

// Only the first player calls syncInstantiate — late joiners get state replay
syncInstantiate(cookiePrefab, { parent: ctx.scene, deleteOnDisconnect: false });

Note: syncInstantiate auto-registers the prefab on the calling client, but remote clients and late joiners still need the explicit registerPrefabProvider call in their setup code.

PlayerSync with runtime-created avatars (no GLB)

Since Needle Engine 5.0.1, PlayerSync.setupFrom accepts an Object3D directly — no GLB URL needed:

import { PlayerSync, PlayerState, ObjectUtils } from "@needle-tools/engine";

// Create your avatar template
const avatarPrefab = ObjectUtils.createPrimitive("Sphere", { color: 0x4488ff });

// Pass the Object3D directly — PlayerState is added automatically
const ps = await PlayerSync.setupFrom(avatarPrefab, { guid: "player-avatar" });
ctx.scene.add(ps.gameObject);

// onPlayerSpawned fires for both local and remote players
ps.onPlayerSpawned?.addEventListener((playerObj) => {
  // playerObj is the spawned avatar Object3D
  // Use PlayerState.isLocalPlayer(playerObj) to check if it's yours
});

For older versions, use registerPrefabProvider with a URL key:

import { PlayerSync, PlayerState, registerPrefabProvider, ObjectUtils, GameObject } from "@needle-tools/engine";

const avatarPrefab = ObjectUtils.createPrimitive("Sphere", { color: 0x4488ff });
GameObject.addComponent(avatarPrefab, PlayerState);

const avatarKey = "runtime://player-avatar";
registerPrefabProvider(avatarKey, async () => avatarPrefab);
const ps = await PlayerSync.setupFrom(avatarKey);
ctx.scene.add(ps.gameObject);

World-building pattern (first player seeds, late joiners receive)

let shouldBuildWorld = false;

connection.beginListen(RoomEvents.JoinedRoom, () => {
  const inRoom = connection.usersInRoom();
  shouldBuildWorld = inRoom.length === 1; // I'm the only one here
});

connection.beginListen(RoomEvents.RoomStateSent, () => {
  // State replay complete — only build if we're first AND no objects exist yet
  if (!shouldBuildWorld) return;

  for (let i = 0; i < 100; i++) {
    syncInstantiate(cookiePrefab, {
      parent: ctx.scene,
      position: [x, 0, z],
      deleteOnDisconnect: false, // persists for late joiners
    });
  }
});

---

Persistent vs ephemeral messages (guid)

When a message's data contains a guid field, the server stores it as room state. New users joining later receive all stored state via RoomStateSent. Messages without a guid are fire-and-forget — only currently connected users see them.

// Ephemeral — only users currently in the room receive this
this.context.connection.send("chat", { text: "hello", sender: "Alice" });

// Persistent — server stores this by guid; late joiners get it automatically
this.context.connection.send("object-color", { guid: this.guid, color: "#ff0000" });

// Read cached state for a guid (received from server or local sends)
const state = this.context.connection.tryGetState(this.guid);

// Delete persisted state (removes from server so new joiners won't get it)
this.context.connection.sendDeleteRemoteState(this.guid);

// Delete ALL room state (use with caution)
this.context.connection.sendDeleteRemoteStateAll();

Any JSON message with a guid can also include these optional fields:

this.context.connection.send("my-state", {
  guid: this.guid,                // persists on server
  dontSave: false,                // set true to prevent server storage (ephemeral but with guid for identity)
  deleteOnDisconnect: true,       // auto-delete when sender disconnects
  // ...your data
});

This is how @syncField() and SyncedTransform work under the hood — they send messages with the component's guid, so state persists for late joiners. Understanding this lets you build custom networking that also persists correctly.

---

SyncedRoom (convenience component)

Wraps context.connection with auto-join, URL params, random rooms, auto-reconnect, and a join/leave menu button. Add to any object — no code needed for basic room management.

import { SyncedRoom } from "@needle-tools/engine";

// Add at runtime:
myObject.addComponent(SyncedRoom, { roomName: "my-room" });
// or join a random room:
myObject.addComponent(SyncedRoom, { joinRandomRoom: true });
// or with a prefix (useful for multiple apps on same server):
myObject.addComponent(SyncedRoom, { joinRandomRoom: true, roomPrefix: "myApp_" });

Key properties:

Property	Default	Description
roomName	""	Room to join
urlParameterName	"room"	URL param for room name (?room=xyz)
joinRandomRoom	undefined	Join random room if no name set
autoRejoin	true	Auto-reconnect on disconnect
requireRoomParameter	false	Only join if URL has room param
createJoinButton	true	Show join/leave button in menu

---

@syncField (auto-sync fields)

@syncField() score: number = 0;                  // auto-syncs on reassignment

// With change callback:
@syncField(MyClass.prototype.onHealthChange)
health: number = 100;

private onHealthChange(newVal: number, oldVal: number) {
  console.log(`Health changed: ${oldVal} → ${newVal}`);
}

// Complex types — must reassign to trigger sync:
@syncField() items: string[] = [];
this.items.push("sword");
this.items = this.items;   // ← triggers sync

---

SyncedTransform (sync position/rotation)

Syncs an object's position, rotation, and scale across clients. Ownership is automatic — when a user interacts (e.g. via DragControls), they take ownership. Always prefer SyncedTransform over manually sending position via custom events — it handles interpolation, ownership, and late-joiner state automatically.

import { SyncedTransform } from "@needle-tools/engine";

// Add to ANY Object3D — cameras, player objects, scene objects, anything
myObject.addComponent(SyncedTransform);

// IMPORTANT: SyncedTransform only sends updates if you have ownership.
// Request ownership before modifying the transform:
const sync = myObject.getComponent(SyncedTransform);
sync?.requestOwnership();           // fire-and-forget (ownership arrives async ~100ms)
myObject.worldPosition = newPos;    // may not broadcast immediately — ownership is async

// For interactive objects (e.g. DragControls), ownership is taken automatically on interaction.

Critical: add SyncedTransform to the prefab BEFORE networking, not after spawning. SyncedTransform uses its component guid to match state across clients. When added via addComponent independently on each client, each gets a random guid — they'll never match. Add it to the prefab before syncInstantiate or PlayerSync.setupFrom so the seeded InstantiateIdProvider generates matching deterministic guids on all clients.

// CORRECT — add SyncedTransform to the prefab before networking
const avatarPrefab = ObjectUtils.createPrimitive("Sphere");
avatarPrefab.guid = "player-avatar";
avatarPrefab.addComponent(SyncedTransform);  // part of the prefab — guid will be deterministic

const ps = await PlayerSync.setupFrom(avatarPrefab);
ctx.scene.add(ps.gameObject);

// onPlayerSpawned only fires for the LOCAL player's avatar.
// To detect ALL players (local + remote), use PlayerState.OwnerChanged:
PlayerState.addEventListener(PlayerStateEvent.OwnerChanged, (evt) => {
  const { playerState } = evt.detail;
  const avatar = playerState.gameObject;
  if (playerState.isLocalPlayer) {
    avatar.visible = false; // hide own avatar (we see through the camera)
    avatar.getComponent(SyncedTransform)?.requestOwnership();
  } else {
    // Remote player — color/customize their avatar
  }
});

// WRONG — adding SyncedTransform after spawn gives each client a random component guid
// avatar.addComponent(SyncedTransform);  // DON'T DO THIS — guids won't match

Timing: Set up PlayerSync (add to scene) before SyncedRoom connects. If SyncedRoom joins a room before PlayerSync is enabled, the join events fire before PlayerSync is listening — onPlayerSpawned will never be called. Either add PlayerSync to the scene first, or set up SyncedRoom after PlayerSync is ready.

Do NOT manually replicate position with connection.send("player-position", { x, y, z }) — use SyncedTransform instead. It uses efficient binary messages (flatbuffers) rather than JSON, making it much faster for high-frequency transform updates. Custom events are for gameplay data (scores, actions, chat), not for transform replication.

---

PlayerSync + PlayerState (player avatar management)

PlayerSync instantiates a prefab for each player joining a room and destroys it on leave. The prefab must have a PlayerState component. This is the recommended approach for multiplayer player objects and WebXR avatars.

import { PlayerSync, PlayerState } from "@needle-tools/engine";

// Runtime setup — load a GLB as the player prefab:
const ps = await PlayerSync.setupFrom("assets/avatar.glb");
scene.add(ps.gameObject);
// The GLB should have a PlayerState component. setupFrom() adds one if missing.

// Events:
ps.onPlayerSpawned  // EventList<Object3D> — fires when any player instance spawns

PlayerState — attached to each spawned player instance:

// Static helpers:
PlayerState.isLocalPlayer(obj)   // true if obj belongs to this client
PlayerState.all                  // all PlayerState instances in the scene
PlayerState.local                // only local player's PlayerState instances
PlayerState.getFor(obj)          // find PlayerState for an Object3D or Component

// Instance:
state.isLocalPlayer              // boolean
state.owner                      // connection ID of the owning player

// Events:
PlayerState.addEventListener(PlayerStateEvent.OwnerChanged, (evt) => {
  // evt.detail: { playerState, oldValue, newValue }
});

---

Voice & Video: Voip and ScreenCapture

Both require an active networked room and HTTPS.

import { Voip, ScreenCapture } from "@needle-tools/engine";

// Voice chat — auto-connects when joining a room
const voip = myObject.addComponent(Voip, { autoConnect: true, createMenuButton: true });
voip.connect();       // manual start
voip.disconnect();    // manual stop
voip.setMuted(true);  // mute mic
voip.volume = 0.5;    // set incoming audio volume (0–1)

// Access raw audio elements for spatial audio / Web Audio API routing
const audioEl = voip.getAudioElement(userId);
if (audioEl) {
  const audioCtx = new AudioContext();
  const source = audioCtx.createMediaElementSource(audioEl);
  const panner = audioCtx.createPanner();
  source.connect(panner).connect(audioCtx.destination);
}

// Speaking detection — fires when a user starts/stops speaking
voip.onSpeakingChanged.addEventListener((evt) => {
  console.log(evt.userId, evt.isSpeaking, evt.volume); // volume: 0–1
});
voip.speakingThreshold = 30;  // amplitude threshold (0–255, default 30)

// Iterate all incoming streams
for (const [userId, audioEl] of voip.incomingStreams) { /* ... */ }

// Screen/camera/microphone sharing
const sc = myObject.addComponent(ScreenCapture);
sc.share({ device: "Screen" });    // "Screen", "Camera", "Microphone", "Canvas"
sc.close();                         // stop sharing
// Receiving clients see the video on a VideoPlayer component on the same object

Voip property	Default	Description
autoConnect	true	Start when joining a room
runInBackground	true	Stay connected when tab loses focus
createMenuButton	true	Show mute/unmute button in menu
volume	1	Incoming audio volume (0–1, applies to all streams)
speakingThreshold	30	Amplitude threshold for speaking detection (0–255)

---

Syncing Animations

For syncing Animator state across clients, see the SyncedAnimator sample — it listens for Animator parameter changes and broadcasts them via context.connection.

---

Typical multiplayer setup

1. Add SyncedRoom to an object (or call context.connection.joinRoom() manually)
2. For player avatars: add PlayerSync with a prefab that has PlayerState
3. For synced objects: add SyncedTransform to movable objects
4. For custom state: use @syncField() on component properties
5. For custom events: use context.connection.send() / beginListen()
6. For voice chat: add Voip — for screen sharing: add ScreenCapture