Smart Factory Lab · PhysicalAI
Edgar Patzel from Smart Factory Lab asked: “What if we connect robots to agents?”
That question sparked an ongoing exploration: giving a robot an agent it can understand, while keeping people in control of every movement.
Challenge: Connect an agent to a robot and get it to write text from a prompt.
How we approached it: We connected the chat agent to the robot through a controlled interface. The agent turns the requested text into pen movements, which are previewed for a person to approve before the robot moves.
What we learned: Even a simple request like writing depends on translating language into safe, accurate robot movements.
Challenge: Turn an image into a drawing the robot can trace with a pen.
How we approached it: We converted the image into drawing paths and checked that the resulting movements fit the robot’s workspace.
What we learned: An image needs to become a set of clear, bounded strokes before the robot can draw it.
Challenge: Let the robot use a camera to see a tic-tac-toe board and play through the agent.
How we approached it: We used an overhead camera and a calibrated view of the workspace. Tic-tac-toe gave us a clear task: find the board, recognize a new move, and plan the robot’s reply. The person approves each robot move.
What we learned: Camera calibration and interpreting what changed on the board are key challenges when connecting what a robot sees to what it does.
Each task builds on the last: first writing from a prompt, then drawing from an image, then using camera input to guide an action. We’re continuing to explore how agents can make robots more useful while keeping their actions visible and under human control.