agentsclimarketplace

Describe surroundings

Skill sumitaich1998/jarvisvr/skills/perception/describe-surroundings

An AI agentic operating system for mixed reality on the Meta Quest 3 — your own J.A.R.V.I.S. Multi-agent orchestration, multimodal perception (sight/hearing/gaze), 42 holographic widgets, 20 LLM providers.

Install
npx -y skills add sumitaich1998/jarvisvr --skill describe-surroundings

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Describe the user's surroundings from the passthrough camera — the main objects and layout of the space — and label a few of them in place. Use for "what's around me?", "describe this room", "what's on my desk?", scene overviews, or accessibility narration. Triggers: describe, what's around me, what's here, look around, what's on the desk, scan the room.

The file declares its own license as MIT. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

2.8 KB, as published. Nobody here has run it

Describe Surroundings

Give a concise, useful overview of what Jarvis currently sees and pin labels on the most relevant few objects (the backend caps annotations at ~3 so the view stays readable).

Steps

  1. Ensure a fresh frame. If vision is active the buffer already has one; for a cold start request perception.request{stream:"vision", action:"once", reason:"user asked to describe the room"}.
  2. Call describe_view. Returns data.speech, data.objects (top labels), and an observation{text, annotations} plus up to three vision_annotation directives. It also records what it saw to episodic memory (so locate-remembered-object can recall it later).
  3. Narrate the scene with agent.observation; render the labels.
  4. Optionally open show_vision_feed ("show me what you see") so the user can confirm the camera view with detection overlays.

Output

agent.observation with spatial annotations:

{ "text": "I can see a coffee mug, a laptop, and a notebook on your desk.",
  "final": true,
  "annotations": [
    { "label": "coffee mug", "position": [0.3,0.8,0.7], "anchor": "world" },
    { "label": "laptop", "position": [0.0,0.78,0.65], "anchor": "world" },
    { "label": "notebook", "position": [-0.25,0.79,0.7], "anchor": "world" }
  ] }

Each annotation is realized as a vision_annotation holo.spawn (annotate_object). Optional feed (show_vision_feed → vision_feed):

{ "widget_type": "vision_feed",
  "props": { "title": "What Jarvis sees", "source": "rgb_center", "show_detections": true, "fps": 2.0 } }

Edge cases

  • Empty / featureless view → "It's pretty bare from here — I mostly see a wall and the floor."
  • Too cluttered → describe categories ("a desk covered in cables and a few mugs") rather than enumerating everything; cap labels at ~3.
  • Specific subset asked ("just the desk") → describe only objects near the named surface from client.scene.surfaces.
  • Single object focus → use identify-object instead.
  • Privacy/battery → stop a once stream when finished; honor perception.state thermal/battery hints by lowering fps.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.