Lomo Agent: Architecture and Engineering of a Creative, Continuously Collaborating Agent

Making a short product film usually means going through requirement discussions, gathering reference material, storyboarding, generating stills, generating video, and revising over and over. Some of that work takes seconds; some of it means waiting for minutes. Some decisions can be left to the model; others must be confirmed by the user. And once the first cut is done, the work is far from over.

Since the AI era arrived, I have rarely written technical articles. Most of my time has gone into exploring how AI can be put to work in different real-world scenarios.

Many people believe product managers (PMs) are the biggest winners of the AI era, because the distance between an idea and a shipped product has shrunk dramatically. In my view, though, AI may empower engineers even more profoundly. It is gradually removing the constraints that technical implementation used to place on creativity, giving engineers the chance to step out of code and engineering details and rethink the essence of products, architecture, and even human–computer interaction.

In the past, we spent enormous amounts of time figuring out how to make machines execute instructions correctly. Now we can finally devote more energy to a far more interesting question: how do we make machines truly understand human intent and collaborate with people in a more natural way?

We recently launched Lomo Agent on Lomote AI. I'd like to take this opportunity to share some of the thinking, architecture, and engineering practice that went into taking it from concept to production.

This article covers task organization, message routing, durable execution, asynchronous jobs, context engineering, long-term memory, autonomous progress, and the shared creative workspace. Some of these designs are already in production; others are still being explored and validated. Along the way, I'll draw on excellent products and engineering work such as DeepSeek, Manus, and Claude Code, and share how I weighed the trade-offs between different approaches.

Lomote AI and Lomo Agent

Lomote AI is the all-in-one AI creation platform we've been building. Interestingly, on this project I've mostly played the role of product manager (PM), while also helping design and build the underlying infrastructure. That has given me the chance to rethink, from both a product and an engineering perspective, what shape and experience an AI creation tool should have.

Today, Lomote AI integrates a range of image, video, and 3D generation and enhancement models, and offers an asset library, Moodboards, and Flows visual workflows. Our goal is to let creators manage assets, combine models, and orchestrate creative processes in a single workspace, so that AI creation evolves from scattered model calls into a complete, coherent creative experience.

These capabilities go a long way toward solving the fragmentation of AI creation tools. But as usage deepened, we gradually realized that integrating tools is only the first step. The more interesting question is how to make them genuinely work together to free up users' productivity.

In a traditional AI creation workflow, users still have to plan the steps themselves, pick models, organize assets, and repeatedly pass context and intermediate outputs between different tasks. As the process gets more complex, users end up spending a great deal of effort just coordinating the pieces.

Lomo Agent was born out of this context. I want AI to evolve from a mere creation tool into an agent that understands intent, plans tasks, and collaborates continuously. The user only needs to describe the goal; the Agent can call the models and tools already on the platform, organize the creative process, and keep adjusting based on user feedback as it goes.

Delivering that experience clearly takes more than single-turn tool calls. How tasks are organized, how context is managed, how asynchronous execution is coordinated, and how the Agent keeps moving work forward after the user walks away all became architectural problems we had to solve.

Click to enlarge

Click to enlarge

Starting with a creative task that keeps getting revised

Suppose that on Monday a user says, "Make me a set of retro-style brand posters." The Agent sorts out the requirements and starts generating reference images. Meanwhile, the user slips in another request: "Also, turn yesterday's cat picture into a five-second video." Then on Wednesday, the user brings it up again: "Make that set of posters a bit warmer, and move the text around too."

This is perfectly ordinary in everyday collaboration between people. For an Agent, though, it involves managing multiple tasks in parallel, asynchronous generation, linking context, and retrieving past assets. As the work progresses, cancellation, retries, budget control, shifting user preferences, and multi-device collaboration all come into play.

About a month ago, I started asking myself a question: why does an Agent have to be centered on sessions?

Most Agent products today still follow the chatbot interaction paradigm: users have to actively create, find, and maintain sessions. For heavy users, sessions pile up, and the context management that AI should be handling becomes a burden on the user instead.

In real life, people don't deliberately maintain "sessions." We switch topics whenever we like, and we can naturally pick up where we left off days later. An ideal agent should probably be able to do the same.

So in Lomo Agent we tried to de-emphasize the user-facing concept of a session, handing task recognition, context isolation, and historical linking over to the system. Of course, this is just one piece of the overall architecture. Truly supporting ongoing creative work also requires solving a whole series of engineering problems: execution recovery, asynchronous scheduling, long-term memory, budget authorization, and shared output management.

While building the demo, I happened to read the DeepSeek team's architectural vision for a Super-agent. Interestingly, they had run into problems similar to ours in task organization and context management. The Router / Worker division of responsibilities in their article Exploring Super-agent Design: Toward More Autonomous Agent Systems, along with their subsequent message routing design, inspired me a great deal and helped me refine the overall architecture of Lomo Agent.

Below, I'll walk through the actual design and implementation and share my thinking and trade-offs on several of these problems.

1. Task organization: from user requests to tasks that keep moving forward

Back to the earlier example: a user might switch between several topics within a few minutes, or come back to a task after several days. For the Agent, receiving a message is only the beginning. What matters more is understanding which goal the message belongs to, and how to find the previous progress and related assets.

This raises an architectural trade-off. If all tasks share a single context, information from different goals easily interferes with each other, and the model's context window balloons quickly. If everything is isolated strictly by session, the historical connections between tasks get severed.

So in Lomo Agent, we split the interaction entry point, the execution context, and long-term memory into three relatively independent layers:

The user simply expresses intent naturally through a single entry point. The system identifies which task a message belongs to and creates a separate task for any work that needs to keep moving forward. Each task has its own context, execution state, and references to its outputs, and can, when needed, obtain information from other tasks through controlled retrieval and sharing mechanisms.

In this design, we also looked at how existing products approach the problem.

For example, ChatGPT uses cross-session memory to improve continuity across conversations, while Claude uses mechanisms like Projects to let users organize knowledge and context around a specific project. Each has its strengths: explicit session and project boundaries are easy to understand, manage, and share, and they give users more direct control over the scope of context.

But in creative scenarios that involve frequent task-switching over long periods, users still have to shoulder part of the context organization work. Cross-session memory helps the Agent understand the user's preferences and history, but it can't fully represent a task that is in progress, along with where it stands.

Lomo Agent therefore hands more of the task organization work to the system, freeing users from day-to-day session management as much as possible. Naturally, automatic recognition also means tasks can occasionally be misattributed. To address this, we kept explicit task entry points and correction mechanisms, and when the target is clear, we prefer deterministic routing to cut down on unnecessary model judgment.

flowchart TD
    U["User"] --> E["Unified interaction entry point"]
    E --> R["Task recognition and routing"]
    R --> A["Task A: brand posters"]
    R --> B["Task B: video production"]
    R --> C["Simple request: handled directly by the main thread"]
    A <--> M["Long-term memory"]
    B <--> M
    C <--> M
    A --> S["Shared assets and outputs"]
    B --> S

Figure 1: Lomo Agent's task organization model

2. System overview: coordination layer, execution layer, and shared workspace

Once we'd settled on how to organize tasks, the next step was figuring out how to actually make those tasks run.

An Agent capable of ongoing collaboration has to handle new user requests, push existing tasks forward, wait for external generation results, and respond to users' edits to their work, all at the same time. If all of this is handed to a single Agent, the context balloons quickly and tasks interfere with one another. If every request spins up its own independent Agent, you introduce extra coordination costs.

With that in mind, I designed a Main / Task two-tier structure for Lomo Agent and introduced a durable runtime in the execution layer, so different tasks can progress independently while accessing and modifying creative outputs through a shared workspace.

2.1 Overall architecture

The system breaks down into four main parts:

• Main (main thread): the entry point the user continuously interacts with. It understands new requests, links them to existing tasks, handles simple questions, and coordinates execution and feedback across tasks.

• Task: an independent execution space that carries a specific creative goal. Each task maintains its own brief, context, execution plan, message inbox, and run state.

• Agent Runtime: manages model inference, tool calls, asynchronous waits, state persistence, and execution recovery, and checks budget, permissions, and stop conditions at every step.

• Shared Workspace: stores creative outputs such as images, videos, 3D models, documents, asset collections, and Flows workflows, and lets users and different tasks access and modify them within their permissions.

In terms of roles, Main primarily acts as the Router, while inside each Task a Worker drives the actual work forward. Router and Worker are logical runtime roles; they don't need to run on different base models.

flowchart TB
    U["User: natural language / card actions / workspace edits"] --> UI["Unified collaboration interface"]
    UI --> GW["Access layer: in-app API / open API / real-time events"]
    GW --> CMD["Command and message intake"]
    subgraph CORE["Agent Core"]
      ROUTER["Main / Router: understanding and coordination"]
      ROUTING["Goal attribution and deterministic routing"]
      W1["Task A / Worker: poster series"]
      W2["Task B / Worker: short video"]
      CTX["Context orchestration / compaction / on-demand tools"]
      RT["Durable execution engine"]
      GUARD["Authorization / budget / autonomy level"]
    end
    CMD --> ROUTING
    ROUTING --> ROUTER
    ROUTING --> W1
    ROUTING --> W2
    ROUTER --> RT
    W1 --> RT
    W2 --> RT
    RT --> CTX
    RT --> GUARD
    RT --> TOOLS["Creative tools: image / video / audio / 3D / canvas / assets"]
    RT <--> DB[("Persistent database: execution facts and state")]
    RT <--> M["Long-term memory: profile / facts / episodic summaries"]
    TOOLS --> ASSET["Shared outputs and workspace"]
    ASSET --> UI
    DB --> EVENTS["Queues / event publishing / real-time push"]
    EVENTS --> UI

Figure 2: Lomo Agent's overall layered architecture

A key property of this architecture is that task execution and creative outputs each have their own independent lifecycle.

For example, when a user kicks off poster creation in the main thread, the system creates a dedicated task for it. Subsequent reference images, generation records, and revision history are maintained by that task, while the final outputs are stored in the shared workspace. Even if the user switches to another task, the poster task can keep waiting for generation results and resume execution when needed.

The main thread only needs to keep a summary and the current status of each task; it doesn't have to retain the full trace of every model call and tool execution. This prevents the main thread's context from growing without bound as usage accumulates.

What we took from DeepSeek's Super-agent

During the design process, the DeepSeek team's Super-agent architectural vision gave us plenty of inspiration.

It likewise separates coordination from execution, organizing different goals through a Router / Worker split and using mechanisms like Ticket / Executor to maintain task ownership. This design is especially well suited to Agents that run for long periods and repeatedly receive new instructions.

However, the research prototype DeepSeek published is aimed mainly at code repositories. Repository tasks usually have a fairly clear scope, file changes, and deliverables, whereas the boundaries of AI creative tasks are much fuzzier.

For example, a simple image generation might take only a few dozen seconds, but a few follow-up revisions from the user can gradually turn it into a long-running creative project. Meanwhile, the same asset may be referenced and modified by multiple tasks.

So we kept the basic Router / Worker division of labor, and added designs tailored to creative work: lightweight tasks, shared output identity, and asynchronous waiting.

This approach also introduces new complexity, such as deciding task ownership, permissions on shared objects, and cross-task state synchronization. We'll cover each of these in later sections.

2.2 Not every request needs its own Worker

Multi-agent architectures easily fall into a trap: since tasks can be split up, why not hand every single thing to its own Agent?

But an independent execution space has a cost. Every new Worker means maintaining extra context, run state, and message routing relationships, and model-call and coordination overhead may grow along with it.

Take Lomo Agent's creative scenarios as an example:

If a user asks "What resolutions does this video model support?", Main can answer directly;

If a user asks to "generate an image of an orange cat", that may only require a single generation job;

But "create a set of brand posters with a consistent style" involves planning, asset management, multiple rounds of generation, and follow-up revisions, so it's better suited to its own task.

For this reason, we further split execution into Lightweight Tasks and Persistent Tasks.

Lightweight tasks mainly carry a single generation, its execution state, and the result card, without taking up a permanent spot in the user's task list. When the user keeps iterating on the result, or the work gradually grows into a complex creative goal, the system can upgrade it to a persistent task, retaining the context and execution records needed for further collaboration.

Persistent tasks have a relatively complete lifecycle: they can keep receiving messages, execute plans, wait on external jobs, and resume work across days or even longer.

This split also complements the de-sessionized design. We want to reduce the burden of managing sessions, and we certainly don't want to simply replace it with another task list that needs constant tidying.

2.3 How a collaboration flows

Let's go back to the brand posters and cat video example.

The user first kicks off poster production in Main. The Router determines that this is a goal that needs ongoing work, so it creates a corresponding task, and a Worker takes over the subsequent planning and generation.

While the posters are in progress, the user asks to turn the earlier cat picture into a video. The system recognizes this as separate work and creates a new execution space for it. The two tasks each maintain their own context and can progress independently according to their own state.

When a poster proposal needs confirmation, the Agent presents the candidate options to the user via a task card. The user picks one directly on the card, and the message is routed back to the corresponding Worker based on the explicit task identity, with no need to redo any semantic judgment.

A few days later, the user says in the main thread, "Make that set of posters a bit warmer." The Router uses the task index and history to link this new instruction to the original poster task.

sequenceDiagram
    participant U as User
    participant R as Main / Router
    participant A as Task A / posters
    participant B as Task B / video
    U->>R: Make a series of brand posters
    R->>A: Create task and brief
    R-->>U: Show task card in the main thread
    U->>R: Also turn yesterday's cat into a video
    R->>B: Create another task
    Note over A,B: Independent contexts, progressing in parallel
    A-->>U: Poster proposals awaiting confirmation
    U->>A: Pick option B on the card
    B-->>U: Video generation complete
    U->>R: Make last time's posters a bit warmer
    R->>A: Route the follow-up back to the existing task

Figure 3: Sequence diagram of Main / Task multi-task collaboration

There's a design detail worth pointing out here: attributing a message and executing a message are two different stages.

The Router can assign a message to the corresponding task, but the Worker won't necessarily process it immediately. It might be waiting on a video generation, or paused at a user confirmation point. New messages first go into a durable inbox, and the runtime then schedules their consumption based on the current state.

This also means task organization is only the first step. As the number of tasks grows, accurately recognizing the user's follow-up intent, avoiding misrouting, and handling new messages that arrive mid-execution become central problems for the whole architecture.

3. Message routing: getting every message to the right execution unit

A unified entry point lets users switch tasks naturally, but it also hands the system a job users used to do themselves: deciding who should handle each new message.

For example, the user has just generated an image and then suddenly says, "Make that one blue."

"That one" might refer to the most recently generated image, or to an asset from some earlier poster task.

If attribution goes wrong, at best a duplicate task gets created; at worst, the Agent edits the wrong piece of work.

So the accuracy of message routing directly affects the reliability of the whole system. We didn't want every message to depend on the model guessing its owner from scratch. Instead, we adopted a fairly simple principle: whatever can be determined through explicit relationships is handled directly by code; only when there's no explicit owner do we hand it to the model for semantic judgment.

3.1 Deterministic routing and semantic routing

Message intake in Lomo Agent roughly goes through three stages.

Layer 1: direct routing for explicit targets.

When a user replies in a task panel, clicks an action button on a task card, or requests changes to a piece of work that is explicitly bound to a task, the system already knows which goal the message belongs to and can deliver it directly by Task ID or object association.

In these cases, asking the model to judge ownership again would only add latency and the risk of misjudgment.

Layer 2: semantic judgment by the Router.

For messages sent from the main thread without an explicit target, the Router has to decide based on the current request, recent interactions, and the index of existing tasks.

Possible outcomes include:

• A simple request handled directly by Main.

• A lightweight task that needs only a single generation.

• A follow-up instruction that can be linked to an existing task.

• A new goal that warrants creating a persistent task.

• Clearly ambiguous ownership that requires further confirmation from the user.

There's another easily overlooked point here: the Router doesn't need to read the full history of every task. Usually it only needs each task's goal, summary, latest status, and related object references; once it has identified candidate targets, it can read specific details on demand.

This keeps routing itself from turning into yet another ever-growing context as the number of tasks increases.

For now, we prioritize providing index information for recently active tasks, such as their goals, summaries, and status. If the user brings up earlier work, we retrieve it through keyword matching and summary vector similarity, then hand a small number of relevant candidates to the Router for judgment. This way, a growing number of tasks mainly affects the scope of retrieval, not the length of context every inference has to carry.

Another common mistake is treating work the user actually wants to continue as a new goal. To guard against this, before creating a persistent task, the system also runs a similarity check against the 50 most recent in-progress tasks. If it finds a possibly duplicate goal, it hands the few most similar candidates to the model and lets it decide whether to reuse an existing task or confirm creating a new one. This check catches some duplicate creations, but it can't replace clarification for every semantic ambiguity. When the evidence is insufficient, the user still needs to make an explicit choice.

Layer 3: ownership and state checks before execution.

Once a message's target is determined, the system still has to check whether the task is valid, whether it can accept the current operation, and whether the related objects are still accessible.

For example, the user might reference a task that has been archived, or try to modify an asset that has been deleted. The Router getting the semantic ownership right doesn't guarantee that the subsequent operation can actually be carried out.

flowchart TD
    M["User message or interaction event received"] --> A{"Does it carry an explicit task identity?"}
    A -->|Yes| V["Check existing target and access permissions"]
    A -->|No| B{"Is there an explicit object association?"}
    B -->|Yes| V
    B -->|No| R["Router semantic judgment"]
    R --> Q["Simple request: handled directly by Main"]
    R --> L["Single-step generation: create lightweight task"]
    R --> E["Existing goal: link to existing task"]
    R --> N["Independent goal: prepare to create persistent task"]
    R --> C["Ambiguous: ask the user to confirm"]
    N --> S{"Does it match an in-progress task in the duplicate check?"}
    S -->|Yes| H["Provide similar candidates for the model to judge"]
    H -->|Link to existing task| E
    H -->|Confirm new task| CREATE["Create persistent task"]
    S -->|No| CREATE
    CREATE --> NV["Validate the new task and operation permissions"]
    L --> NV
    NV --> I["Set up execution state and deliver input"]
    E --> V
    V --> D{"Is the existing target state valid?"}
    D -->|Yes| T["Write to the task's Inbox"]
    D -->|No| F["Notify the user or reselect target"]
    C --> U["Re-evaluate after the user replies"]
    U --> R

Figure 4: Lomo Agent's message routing flow.

3.2 Delivered doesn't mean executed

There's another easily overlooked detail in message routing: a message being assigned to a Worker and the Worker actually starting to process it can happen at different times.

For example, while the Agent is generating a video, the user suddenly adds: "Make the background indoors."

At this point, the video generation request may already have been submitted to the upstream service and can't be changed right away. The system needs to save this new instruction first, then, once the Worker reaches a suitable execution boundary, decide what to do next in light of current progress.

That's why we maintain a durable message inbox (Inbox) for each task. The Router is responsible for delivering messages to the right inbox; the runtime decides when to read them, how to merge them into context, and whether the existing plan needs to change.

A few cases need to be distinguished here:

• Messages that the Worker hasn't consumed yet can be withdrawn or replaced when conditions allow;

• Messages that have already entered model inference must wait for the current execution boundary;

• External generation requests that have already been submitted can't currently be withdrawn. The new instruction has to wait until the Worker reaches a suitable execution boundary, and then, based on task progress, the Worker decides whether to regenerate or apply the new constraints to subsequent steps.

This separation also lays the groundwork for recovery after interruption, asynchronous jobs, and letting users step in at any time.

3.3 Borrowing message ownership design from collaboration products

On the interaction side, we also looked at how collaboration products such as Slack and Feishu (Lark) organize messages.

Take threads, for example: when a user replies within a particular thread, the system inherently knows which discussion the message belongs to, without having to re-analyze the whole chat to work out ownership. The advantage of this kind of explicit association is that it's stable, predictable, and costs almost nothing in semantic recognition.

Lomo Agent's interactions, however, are more flexible. A user might reply directly on a task card, or go back to the main thread and casually mention some work from a few days ago.

So we kept both approaches: explicit task actions are routed directly by structured identity, while natural language in the main thread is linked to past tasks by the Router.

This hybrid strategy reduces how often the model gets involved in routing, but it can't eliminate ambiguity entirely. For example, if the user is working on two sets of posters at once and just says "Change last time's set," the system may not have enough evidence to determine the target. In that case, asking proactively is usually more reliable than picking whichever task looks most relevant.

As the number of tasks grows, balancing recall, routing accuracy, and interaction cost will still require continuous tuning based on real-world usage.

That said, ownership judgment isn't the last check in the pipeline. An extra task can usually be merged or closed later; but if a message is wrongly linked to editing a piece of work, deleting assets, or a paid generation, the cost can be much higher. So we separate message routing from actual execution authorization: the former decides which task the work should go to, while the latter still independently checks permissions and cost boundaries.

For example, irreversible operations such as deleting assets or forgetting memories require user confirmation every time. Paid generations first show a quote, and ask the user to confirm when the cost exceeds the confirmation-free allowance, the daily confirmation-free cap, or an approved plan budget. Paid operations initiated through the open API are also confirmed one by one. Even when the Router picks the right task, that doesn't grant execution rights beyond what the user has authorized.

Even after a message reaches the right task, execution may still span minutes or even days and run into process restarts, network timeouts, or user cancellations. Reliable recovery is the job of the durable execution mechanism.

4. Durable execution: recovery after interruption, retries, and concurrency control

Once a message has been routed to the right task, the next problem is making the execution itself reliable.

With an ordinary chat completion, a request usually finishes within a few seconds to a few dozen seconds. A creative Agent's workflow is far more complex: producing a single video can involve multiple rounds of model inference, dozens of tool calls, and several external generation jobs that each take minutes. During execution, the user may also change requirements or stop the task at any time.

Consider a very common failure scenario: the Agent submits a request to a video generation service, the upstream actually creates the job successfully, but the service process happens to restart just before receiving the response.

What should happen after recovery? Retrying blindly might generate the video twice and charge twice; assuming the request succeeded might mean waiting forever for a job that was never actually created.

Problems like this made us realize that an Agent's execution needs persistent state that is independent of both the model context and the service process.

So I introduced a Durable Execution mechanism in Lomo Agent. The database records key execution state, model decisions, and tool call results; the task queue handles scheduling and recovery; and real-time events report progress to the user. Even if the executing process is interrupted, the system can still determine what to do next from the facts it has saved.

In the design, we drew on the ideas behind Temporal's Durable Execution. Through workflow history, task scheduling, and retry mechanisms, Temporal lets business processes outlive any single process, and its advantages are especially clear in complex, long-running workflows.

That said, general-purpose workflow engines come with their own modeling and operational costs. Lomo Agent's execution also has to manage model inference, dynamic tool calls, creative state, and user interaction, so we built an execution ledger and event scheduling mechanism around the Agent's own data structures. What we borrowed is the engineering idea of durable execution; we don't use Temporal itself.

4.1 Persist the decision first, then execute the tool calls

A traditional agent loop usually follows a simple flow:

Read context → model inference → call tools → get results → continue inference.

For short tasks, this implementation is fairly straightforward. But when execution spans multiple processes, external services, and a billing system, an interruption at any step can leave state inconsistent.

We divide execution into a series of Steps. Each Step can contain one model inference plus the set of tool calls the model proposes.

Before executing any tools, the runtime first persists the model's decision and the pending calls. Every tool call has a stable business identifier, and its status, inputs, and results are recorded in the execution ledger.

That way, when a Worker recovers, it can use the ledger to determine which operations have completed, which can be retried, and which require checking external state first, without calling the model again to regenerate the same round of tool decisions.

flowchart TD
    S["Read Inbox and current execution state"] --> L["Model inference: generate reply and tool call intents"]
    L --> P["Atomically commit model result and pending calls"]
    P --> X{"Does the call already have a persisted result?"}
    X -->|Yes| REUSE["Reuse existing result"]
    X -->|No| K{"Might it have been executed before?"}
    K -->|No| EXEC["Submit tool call"]
    K -->|Yes| I{"Does it support idempotent retry?"}
    I -->|Yes| EXEC
    I -->|No| UNKNOWN["Mark result unknown and hand back to the model"]
    EXEC --> SAVE["Persist tool result and business state"]
    REUSE --> NEXT["Continue to next call or Step"]
    SAVE --> NEXT
    UNKNOWN --> CHECK["Query current state first, then decide whether to retry"]
    CHECK --> NEXT

Figure 5: The Agent's durable execution flow.

A key concept here is idempotency.

Suppose a user approves generating a video, and a network timeout causes the system to submit the same request twice. Ideally, both submissions should map to the same generation order: there must not be two duplicate videos, and certainly not a double charge.

For downstream services that support idempotency keys, a stable call identifier can be used for deduplication. For the platform's internal orders and billing, call IDs, order IDs, and unique constraints can likewise ensure that duplicate requests produce no new business effects.

But if an upstream generation API supports neither idempotency keys nor querying request results, then after a network timeout the system genuinely cannot confirm whether the job was created.

In that situation, durable execution cannot conjure up an exactly-once guarantee out of thin air.

We need to further distinguish between two cases. For the few tools that aren't idempotent, if the execution ledger has recorded started but no result ever arrives, the runtime doesn't simply re-execute the call. Instead, it hands the model a result marked outcome_unknown, so the model first checks the current state before deciding what to do next. Calls that support idempotency can be replayed with the original call identifier.

Another layer of uncertainty sits between the generation service and the upstream API. Here we take a more conservative approach: if the outcome of a submission is unclear, the submission is treated as failed, and no new request is automatically sent upstream. Both may look like "unknown outcome," but one is about recovering an Agent tool call, while the other is about avoiding duplicate generation jobs that incur real charges.

Temporal faces the same problem with Activity retries. A workflow engine can reliably schedule and retry execution, but side effects on external APIs still need business-level idempotency to back them up.

For paid generation services, this boundary is especially important. A failed execution can be recovered; a double charge can't necessarily be undone automatically.

4.2 Worker takeover and Fencing Tokens

Durable execution solves recovery after a process is interrupted, but there's a subtler case: the original Worker may not have actually exited.

For example, Worker A temporarily loses connectivity mid-execution due to network jitter. The system detects a timeout, and Worker B takes over the task and continues. A few seconds later, Worker A reconnects and tries to commit the result it just finished.

At this point there may be two Workers in the system that each believe they own execution. Without safeguards, the old Worker could overwrite the new state, or even push a task that has already been stopped back into execution.

To handle this, we use an execution-generation mechanism similar to a Fencing Token.

Whenever a task is taken over by a new executor, the runtime bumps that task's execution generation. When a Worker commits a state change, it must carry and validate the generation it holds.

If the generation in the database has already moved on, the old Worker's commit is rejected.

You can think of this mechanism as: the system allows a new executor to take over the work while revoking the old executor's right to keep modifying internal state.

When a user deliberately stops a task, the same run-state and execution-generation checks prevent an old Worker from pushing forward after it recovers.

However, Fencing Tokens protect the consistency of the system's own state. If the old Worker has already sent a video generation request to an external platform, invalidating its internal execution rights won't automatically undo the remote operation.

So even after the user has stopped a task, external generations submitted earlier may keep running. The generation service continues tracking their results and settles the corresponding costs, but a late completion event won't wake the stopped task back up, nor will it change an execution state that has already been canceled.

4.3 Handling new user messages during execution

Another class of problems comes from users stepping in in real time.

Suppose the Agent is calling the model, about to generate a set of posters. The user then sends: "Not a white background, make it dark blue."

This message may reach the server before the model finishes inference, but the current model request never actually read it.

If the system simply stitches history together by message creation time, a kind of state confusion can arise: it looks as if the model made its decision after the user changed the requirement, when in fact it knew nothing about the new instruction.

To solve this, we handle message receipt and message consumption separately.

New messages first go into the task's durable Inbox. The Worker reads these inputs at the next safe execution boundary and organizes them into the model context together with the current task state.

The system records which Step actually consumed a message, rather than relying solely on send time to determine ordering.

This clearly distinguishes the stages of a user instruction having been delivered, having been understood by the model, and having had the related operations carried out.

Of course, if a generation request has already been submitted upstream, a later change won't automatically alter what's being generated. The current version also can't withdraw such requests; the Worker can only decide at the next safe execution boundary whether to regenerate or apply the new constraints to subsequent steps. If the user has already stopped the task, the old generation won't restart the task's execution even if it completes afterward.

For the user, the interface also needs to reflect this state accurately, so that "message received" isn't mistaken for "the current generation has been adjusted to the new requirement."

For time-consuming generation jobs like video, audio, and 3D, a Worker also needs to release execution resources while waiting for external results, and resume precisely once the event arrives.

4.4 The execution ledger and the consistency boundaries of scheduling

So far we've discussed persisting model decisions, and using execution generations to keep an old Worker from modifying state after a task has been taken over. In real operation, though, there's another easily overlooked problem: if the database has already recorded pending work but the queue message responsible for waking a Worker gets lost, how does the task recover?

In Lomo Agent, we maintain execution state through turns (Turn) and a tool call ledger. A turn records the run state, execution generation, lease, and Checkpoint, while each tool call uses "turn + step number + call sequence number" as its stable identifier. When the model returns, the runtime saves the decision, the pending calls, and the consumed Inbox messages in a single transaction; before a tool runs, it records started, and on completion it saves the result together with the done status. That way, recovery can reuse results or safely retry based on existing records, without asking the model to guess what happened before. For calls that can't be safely retried, the result stays in an unknown state and is left to subsequent decisions.

For task scheduling, we didn't introduce a transactional Outbox for enqueuing turns. Instead, the database serves as the source of truth for execution, and the queue is mainly responsible for waking Workers. The scheduling message is sent after the turn is committed; if enqueuing fails, a background watchdog rediscovers the task from the database state. Currently the watchdog checks every 10 seconds, focusing on expired leases, jobs that have been queued for a long time, and tasks that have unprocessed messages but no active turn. Queue jobs are identified by turn ID and execution generation, which reduces duplicate enqueuing within the same generation; even if duplicate scheduling occurs, the runtime still prevents stale writes through run-state and generation checks. This means we accept a bounded recovery delay in exchange for a simpler scheduling mechanism; UI event pushes and external generation jobs each keep their own reliable delivery and compensation mechanisms.

Another class of risk comes from old and new Workers executing concurrently. A Fencing Token can stop an old Worker from overwriting internal state, but it can't undo external requests that have already been sent. To address this, we fix the idempotency key for paid generations as "turn + step number + call sequence number," without the execution generation. Even if the old and new Workers both submit the same call, the platform's generation layer maps them to the same order through a unique constraint on "workspace + idempotency key," and deduplicates the internal credit deduction. This keeps duplicate orders and billing under control inside the platform, but it doesn't mean the external upstream inherently offers an Exactly-once guarantee. By comparison, a failure before the model decision is persisted may cause that step to be re-inferred. Given our current cost structure, which is dominated by media generation, we choose to accept this relatively limited cost of repeated inference.

Harder to handle are cases where the outcome of an upstream submission is unclear. A request timeout might mean the submission failed, or it might mean the upstream created the job but the response never came back. To avoid duplicate generation, the generation service currently doesn't automatically resubmit such requests; once it successfully obtains an upstream job ID, it persists it immediately so polling can continue after an interruption. But if the process crashes exactly between obtaining the ID and saving it, there's currently no way to look that job back up. The system can only process a refund according to the rules after a timeout, with the platform absorbing any upstream cost that may already have been incurred.

These designs can't eliminate every failure window, but they make the handling boundaries clear for each situation: reuse what can be confirmed as complete, continue what can be safely retried, and avoid blindly resubmitting what can't be confirmed. For a continuously running Agent, reliability doesn't mean no step ever fails; it means that after an interruption, the system can still make controlled recovery decisions based on the facts it already has.

5. Asynchronous generation: job lifecycle and event-driven wake-ups

The durable execution described above mainly addresses how the Agent records state, handles failures, and resumes execution while running. But creative AI work has another unavoidable characteristic: a large share of tool calls are inherently asynchronous.

For example, image generation might take only a few seconds, while video and 3D jobs can run for several minutes or longer. After a job is submitted, the upstream service usually returns only a job ID; the actual result has to be obtained via polling or callbacks.

Having the Worker wait around for external jobs to finish would not only tie up execution resources but also couple model inference to uncontrollable external generation times.

So we manage the Agent's execution lifecycle and the external generation job's lifecycle separately.

5.1 Submit, wait, and resume

When a Worker calls a generation tool, the tool first submits a request to the corresponding service and returns the generation job's identifier. The runtime then saves the association between the job and the current task, including the job ID, wait state, and the information needed for subsequent execution.

Once submitted, the Worker can suspend its current execution, with no need to keep calling the model or hold open a long-running inference request.

While waiting, progress updates from the generation service can be pushed straight to the frontend as business events, with the task card showing the current status.

When the job completes, the runtime decides whether to wake the Worker again, depending on whether subsequent steps rely on the generation result.

sequenceDiagram
    participant W as Task Worker
    participant G as Generation service
    participant S as Durable runtime
    participant UI as Task card
    W->>G: Submit generation request (stable business ID)
    G-->>W: Return job ID
    W->>S: Save job association and wait state
    Note over W,S: The above is the normal path. For failures to save after submission, see 4.4
    S-->>UI: Update to generating
    Note over W: Worker suspends and releases execution resources
    G-->>S: Generation progress event
    S-->>UI: Update progress (no model inference needed)
    G-->>S: Generation completed event
    S->>S: Update job state and save outputs
    S-->>UI: Show generation result
    alt Subsequent steps depend on the result
        S->>W: Deliver resume event
        W->>W: Read latest state and continue
    else No further inference needed for now
        Note over UI: Only update result and task status
    end

Figure 6: Execution flow of an asynchronous generation job (normal path).

The direct benefit of this design is better resource utilization. During the minutes spent waiting for a video, the system only needs to maintain persistent state and handle the necessary events, without continuously occupying Agent inference resources.

At the same time, generation progress and the user interface can update independently. Users see job status in real time without waiting for the Worker to call the model again.

5.2 Asynchronous events need reliable handling too

Once generation jobs are event-driven, you also have to think about the reliability of the events themselves.

For example, an upstream service might send a completion notification twice because of network issues, or might not be able to retry immediately after a failed callback. With multiple generation jobs running concurrently, progress events may also arrive out of order.

So event handling can't simply be understood as "got a callback, keep going."

We need to update generation jobs based on their persisted state, and make sure duplicate events don't cause duplicate writes, duplicate charges, or duplicate Worker wake-ups.

For events that may arrive out of order, we also need to prevent stale progress from overwriting the final result. For example, if a job has already completed and a delayed "generating" notification arrives afterward, the state shouldn't roll back.

In addition, external callbacks aren't always reliable. For jobs that haven't produced a result for a long time, the system can reconcile status using the job ID and the upstream query API, falling back to compensating polls to recover job progress when necessary.

This works hand in hand with the durable execution described earlier: the runtime maintains trustworthy execution state, and the asynchronous job system turns changes during external generation into recoverable business events.

5.3 When should the Worker be woken up?

There's another question directly tied to model call costs: does every completed generation job require calling the model again?

The answer depends on whether there's still work later in the task that needs the Agent to make decisions.

For example, if the user just asked to "generate a picture of a cat," the system can simply display the result once generation finishes. There's no need to wake the Worker and have the model redundantly describe the image it just produced.

But if the user asked to "create a set of brand posters with a consistent style," the current result may become the input to later steps. The Agent needs to examine existing outputs, read the checklist, and decide, based on the current goal, whether to keep generating or wait for user confirmation.

So we handle generation completion events and Agent continuation decisions separately.

The generation service reports results; the runtime decides whether to resume the Worker based on the task plan, wait dependencies, authorization, and current execution state.

This also avoids feeding a flood of progress events straight to the model. A video job lasting several minutes may produce many state changes upstream, but the vast majority only need a frontend update and aren't worth triggering new inference.

This event-driven design is similar to the asynchronous waiting mechanisms found in general-purpose workflow engines and task queues. For Lomo Agent, our further focus is how to combine it with creative goals, checklists, and the user review process.

For instance, after one video shot finishes generating, if there are more shots still to produce and the user has authorized continuing, the runtime can resume the corresponding task; if the current stage requires user review, it should remain in a waiting-for-confirmation state.

A completed generation only means a particular tool call has ended. Whether the overall creative task should continue still has to be judged against the plan, the budget, and authorization.

This also leads to the autonomous progress mechanism we'll discuss later: when the Agent finishes a stage, or is even about to end the current round of inference, how does the system determine whether the task is truly done, and whether it should keep going?

6. Context engineering: maintaining useful state within a limited window

As the Agent started handling long-running, multi-stage creative tasks, context management gradually became a major factor in both execution quality and cost.

A creative task that runs for several days might accumulate hundreds of messages, dozens of tool calls, and a large amount of image, video, and workflow state. Even as model context windows keep growing, stuffing all of this into every inference simply isn't realistic.

On one hand, long context brings higher token costs and inference latency. On the other, too much history can interfere with the model's judgment about the current goal. Especially when the user has changed requirements several times, early constraints that are no longer valid can easily affect later generations if they're still mixed into the context.

So we treat context management as an engineering problem in its own right: at every inference step, how do we give the model information that is sufficient, relevant, and still valid?

6.1 Context layering and Prompt Cache

Lomo Agent organizes the model context into several layers: a relatively stable system prefix, compacted history, recent interactions, and the dynamic state needed for the current execution step.

flowchart LR
    A["Stable prefix: system rules / core tools / profile snapshot"] --> B["Segmented history summaries"]
    B --> C["Recent messages and key tool results"]
    C --> D["Dynamic tail: brief / checklist / budget / object changes"]
    D --> E["Current step input and on-demand recall"]

Figure 7: How Lomo Agent organizes context.

One factor closely tied to actual inference cost is Prompt Caching. In what follows, Prompt Cache refers to the prompt caching capability that model providers offer at the API layer; its concrete implementation and what can be reused are determined by each provider.

Model services that support it can reuse prompt prefixes that meet their caching rules, reducing the cost of redundant processing. So we try to keep system rules, core tools, and relatively stable profile information at the front of the context, and put task state that may change at every step toward the end.

If highly dynamic information is inserted at the start of the prefix, otherwise-stable content after it may no longer benefit from the cache.

That said, the actual Prompt Cache hit rate also depends on the model provider, request routing, and caching policy. Context layout can only create the conditions for cache reuse; the real gains still have to be validated against actual call data.

Moreover, a stable prefix doesn't mean the profile can go without updates for long periods. When a user changes a preference or asks the Agent to forget something, the relevant profile snapshot must be invalidated promptly, so that cache reuse doesn't keep stale information in play during inference.

6.2 Load tools on demand: the more capabilities, the more expensive the context

Lomote AI integrates many creative models and tools across different domains: image generation, video generation, 3D, asset management, document editing, and Flows workflows.

Putting the names, parameters, and usage instructions of every tool into every model request would not only increase token consumption but could also lead the model to choose wrongly among many similar tools.

So we split tools into two categories: core tools that stay visible at all times, and others that are loaded dynamically by capability domain.

For example, while the Agent is planning a storyboard, it may only need document and asset tools; once it moves into video production, video generation and workflow capabilities are loaded.

For tool domains that are no longer active, their detailed definitions can be removed at a context compaction boundary, reducing the influence of irrelevant information on subsequent inference.

This design has its costs, too. When the tool set changes, the structure of the model context may change with it, affecting the Prompt Cache hit rate; tool calls already in the history may also not fully match the tool definitions currently visible.

On this point, Manus's context engineering practices gave us a counterexample worth thinking about.

Manus prefers to keep tool definitions stable and to reduce tool-selection complexity by constraining which actions are currently available. This helps reduce prefix invalidation caused by changing tool definitions and makes it easier to keep historical tool calls consistent with the current context. The point here is how a stable context affects cache reuse; it does not equate API-level Prompt Caching with the KV Cache mechanism inside model inference.

Lomo Agent chose domain-based loading mainly because a creative platform's tool types and parameter structures are fairly large, and keeping every tool definition around permanently could incur significant context overhead.

Each strategy has its own conditions where it fits best: a stable tool set favors caching and behavioral consistency, while dynamic loading may reduce the token cost and selection noise introduced by irrelevant tools.

For our approach, the real-world impact has to be evaluated continuously against tool-switching frequency, cache hit rate, tool-call accuracy, and overall inference cost. Also, whether a tool appears in the model context only affects what the model can currently choose. Actual tool execution permissions are still enforced server-side; tool visibility in the prompt must never be relied on as a security boundary.

6.3 History compaction: beyond summaries, keep the ability to read back

Over long-running execution, history messages and tool results keep growing. Even with a large context window, compaction is needed at appropriate points.

A common approach is to have the model repeatedly summarize prior history and then continue the conversation from the latest summary. But this has a problem: through repeated compaction, summaries may gradually lose detail, and may even harden the model's own speculation into historical fact.

So we take a segmented-summary approach: older history is compacted range by range, while recent interactions are kept in full.

New summaries are appended to the existing sequence of summaries wherever possible, rather than rewriting the entire history each time.

For large tool outputs, such as asset lists, workflow node information, or complex generation results, the system can keep only the necessary summary in the current context and store the full content externally.

When the Agent later needs to check a specific parameter or historical detail, it recovers the relevant information through retrieval or paginated reads.

This has something in common with Manus's advocacy of using the file system as an external carrier of context. The model doesn't need to carry all information all the time, but it must be able to find reliable original sources again when needed.

There's an important distinction here: summaries exist to reduce the context burden of the current inference; they shouldn't become the sole source of historical truth. Especially when the user has corrected something, or a historical constraint is no longer valid, the system needs to be able to locate the relevant sources so that old summaries don't keep influencing later execution.

6.4 Keeping the current task in the model's field of view

Context compaction solves the problem of history growing too long, but there's another equally important question: halfway through execution, does the Agent still clearly know what goal it's working toward?

On long-running tasks, the model may gradually drift from the original goal after many rounds of tool calls. For example, the user asks for a complete set of brand posters, but the Agent spends most of its effort repeatedly polishing a single reference image and loses sight of the overall deliverable.

So we continuously provide the current brief, checklist, key constraints, wait state, and the latest changes to related work in the Worker's dynamic context.

This state comes from the durable runtime, not merely from the model's own summary of the earlier chat.

For example:

• The brief records the visual direction the user ultimately approved

• The checklist records which steps are done and which still need work

• Workspace state tells the model whether an asset has been replaced by the user, and which version a document is currently at.

When the user directly edits a Flows workflow or a shared document, the next round of execution can also receive an object-change notice and read the latest content when needed.

This way, even if part of the history has been compacted, the Agent can keep working based on trustworthy task state.

In this regard, Manus's practice of continuously surfacing a to-do list to keep the model focused on the goal also inspired us. Lomo Agent goes a step further by tying these checklists to runtime state, so they not only provide context for the model but also support automatic continuation and stop decisions later on. We'll cover this in detail in the section on autonomous progress.

6.5 Practical trade-offs in context engineering

Taken together, Lomo Agent's context engineering revolves around three goals:

• Reducing irrelevant information

• Controlling inference cost

• Ensuring the information behind current decisions is still valid

But these goals don't always align.

More aggressive compaction lowers token consumption but may drop important details. Dynamic tool loading narrows the tool selection space but may hurt cache hits. Keeping a stable profile long-term strengthens continuity but has to deal with preference changes and memory revocation.

So context management can hardly be solved by any single fixed strategy. It has to be tuned together with task state, the memory system, tool permissions, and actual model behavior.

For Lomo Agent, what we care about most is whether the model can find the right information when it needs it and keep moving forward based on currently valid task state. And when collaboration spans multiple tasks, multiple days, or even months, context management within a task isn't enough. The preferences a user has developed over time, past projects, and decisions made along the way need to be organized and retrieved by a separate mechanism.

7. Memory engineering: provenance, versioning, and revocability

The previous chapter discussed how to organize information within a limited context window. But for an Agent built for long-term collaboration, there's a deeper question: once a conversation has left the current context, or the related task has long since ended, what should the system remember?

In Lomo Agent's creative scenarios, users may use the platform for months, gradually forming fairly stable visual preferences, go-to creative workflows, and brand guidelines. Users shouldn't have to re-describe this information every time.

At the same time, user preferences change. A temporary style adopted for one project may not apply to all future work; a brand guideline that held in the past may become completely obsolete after a redesign.

So we want long-term memory to provide continuity across tasks and over time, while preserving the provenance and scope of validity of each fact.

7.1 From historical messages to long-term memory

The first thing long-term memory has to solve is filtering.

For example, if a user says "I usually prefer low-saturation color palettes," that may be a preference worth keeping long-term. But if they say "Let's try a highly saturated red this time," it's more likely just a local requirement for the current piece.

Both statements are about color, but they apply at different scopes. If the system writes every user message straight into a vector database and relies on semantic retrieval to backfill, it can easily confuse long-term preferences, one-off requirements, and historical information that's no longer valid.

To address this, we divide Lomo Agent's memory into several layers:

These layers are related to one another, but each has a different responsibility.

Raw messages are traceable evidence. The system can extract candidate memories from them, then decide whether to save, supplement, or update based on source, time, scope of applicability, and existing records.

Episodic summaries focus more on what happened during a stretch of collaboration. For example, last week the user made a set of brand posters, ultimately chose a warm color scheme, and decided to hold off on the follow-up video. Not all of this deserves a place in the long-term profile, but it may come in handy the next time the user brings up the project.

The user profile needs to be more restrained. It should hold a small amount of relatively stable information and avoid turning into yet another ever-growing chat history as usage accumulates.

At recall time, the system selects which memories to provide to the model based on the current task's goal and permissions, rather than loading the full history every turn.

There's another important principle here: memory must distinguish facts the user explicitly stated from the model's own inferences.

For example, if the user generates several cyberpunk images in a row, the Agent might infer a long-term preference for cyberpunk style. That can serve as a candidate judgment, but it can't be treated as equivalent to a preference the user explicitly expressed.

Likewise, web content, tool results, and summaries the Agent generates itself must not be repackaged as personal facts about the user without reliable grounding.

7.2 Lessons from existing memory architectures

For the design of long-term memory, we looked at several different technical approaches.

OpenClaw's memory practice emphasizes inspectable, maintainable storage and the relationship between memories and their original context. Its strength is high transparency, making it easy for users or developers to check what's been saved; but as memory keeps growing, retrieval, maintenance, and conflict handling still require additional mechanisms.

Mem0 focuses more on extracting valuable information from conversations and restoring relevant memories through retrieval in later interactions. This suits agents used over long periods across many topics, but memory quality depends heavily on extraction accuracy and recall strategy. If a temporary instruction is misjudged as a long-term preference, later retrieval may keep amplifying the error.

Zep / Graphiti offers another approach: explicitly maintaining entity relationships and temporal information in memory. This is especially valuable for handling facts that keep changing. For example, if a brand used blue as its primary color in the past and later switched to green, the two records don't necessarily contradict each other; they may simply correspond to different validity periods.

However, temporal relationships and knowledge graphs also bring higher maintenance complexity, including entity resolution, relationship updates, conflict handling, and retrieval cost.

Given Lomo Agent's creative scenarios, we focus on three questions:

1. Does a user preference come from a reliable source?

2. Is a historical fact still valid?

3. When a user corrects or revokes information, can the related memories be updated accordingly?

So in addition to the usual layered storage and semantic retrieval, we also maintain provenance, versions, and derivation relationships for memories.

7.3 Why "forgetting" is harder than "remembering"

There's another easily overlooked problem with long-term memory: when a user explicitly asks the system to forget something, how should it handle that?

Suppose the user first says, "Our brand's primary color is blue."

Later they add, "Blue paired with off-white feels softer overall."

The memory system might merge these two statements into a new brand preference:

"This brand prefers blue paired with off-white, with an overall soft visual style."

If the user then asks the Agent to forget the original setting about blue, deleting only the first atomic memory clearly isn't enough, because the merged version still contains that information.

As the system keeps running, this information may also find its way into episodic summaries, the user profile, or even be merged into other derived memories.

So we maintain provenance and derivation relationships for memories, describing which historical evidence and earlier versions a memory evolved from.

flowchart TD
    U["User specifies what to forget"] --> A["Revoke the memory eligibility of related evidence"]
    A --> B["Locate affected memory versions"]
    B --> C["Follow derivation links to find later versions"]
    C --> D["Disable affected content and check remaining valid evidence"]
    D --> E["Re-extract from remaining valid evidence"]
    E --> F["Update atomic memories, summaries, and user profile"]
    F --> G["Invalidate related retrieval indexes and context caches"]

Figure 8: Memory revocation flow.

In practice, we maintain how memories evolve through version chains. Taking the earlier brand primary color as an example, the original statement about blue forms valid version v1, linked to its original evidence (ORIGIN). When the user adds the requirement to pair it with off-white, the system creates a new version v2, marks the earlier version as SUPERSEDED, and records the new refining evidence (REFINE). If a later statement merely confirms an existing conclusion, it can be recorded as CONFIRM without generating a new memory.

When the user asks to forget the original blue setting, the system revokes the eligibility of the corresponding evidence and follows version relationships to locate the affected memories. Versions that depend on that evidence are deleted and their content cleared, then re-extracted from the remaining valid evidence; evidence that only served as confirmation can simply be unlinked, without affecting memories that stand on their own.

This revocation also propagates to the user profile and episodic summaries. Once the related profile entries are invalidated, the system updates the profile version so later inference no longer uses the old snapshot; affected summaries need to be rebuilt. At the same time, the system deletes the corresponding retrieval vectors and suppresses re-extraction of deleted content, lowering the risk of old information re-entering the context.

Note that model requests that have already been sent can't be revoked instantly. So we also track which messages and memories each reply and tool result was based on, and exclude affected derived content from later inference, so that forgotten information isn't reintroduced through the Agent's own past output.

The "forgetting" discussed here mainly concerns whether long-term memory and its derived context are eligible for use. It is not the same as deleting all raw chat logs, generated assets, or transaction data. Those belong to separate data lifecycle and privacy management concerns and need to be handled independently.

7.4 Memory reliability and graceful degradation

Provenance tracking and versioning make the memory system more controllable, but they can't completely eliminate model misunderstandings.

For example, a single conversation may contain several candidate facts, one of which qualifies for saving while another is just a temporary instruction the model misread. If a whole batch of extraction results is treated as an indivisible unit, a single anomaly can affect other valid memories.

So candidate memories need to be validated and processed one by one, distinguishing between cases such as unsupported provenance, conflicting content, and malformed output.

When the memory service fails, it also shouldn't bring normal creative tasks to a complete halt.

For example, a recall timeout can temporarily skip memory enrichment; a failed background extraction can be reprocessed later; and a single malformed candidate doesn't need to take other valid records down with it.

This degradation strategy means the Agent may sometimes be unable to use the full set of historical preferences, but it can still keep working from the current brief and existing execution state.

Rather than letting one problematic memory result affect an entire task, we prefer to keep the normal execution path intact and re-supply the relevant background once memory recovers.

Ultimately, the value of long-term memory has to be proven through actual collaboration. Beyond recall accuracy, we also care about whether memories go stale, whether their provenance can be explained, and whether old information still influences later decisions after the user corrects or revokes it.

Click to enlarge

As memory and task context mature, the Agent can draw on background information across time. But knowing what the user has done in the past doesn't mean it can judge whether the current task should keep going.

A task that involves multiple creative steps also needs an explicit execution plan, stop conditions, and boundaries on user authorization. That's what the autonomous progress mechanism in the next chapter sets out to solve.

8. Autonomous progress: plans, stop conditions, and user authorization

The mechanisms so far let the Agent maintain long-running tasks, resume interrupted execution, and recover historical information when needed. But for a creative goal that needs to keep moving forward, one problem remains unsolved: how does the Agent decide whether it should keep working?

For example, a user asks for a short brand film. After finishing the script and storyboard, the Agent might give a summary and then end the current turn. The model hasn't done anything wrong; it really has completed the current stage. But there's still a long way to go before the finished film the user ultimately wants.

We ran into exactly this in early demos and scenario rehearsals. Users had to keep sending "continue" before the Agent would move on to the next stage.

On the other hand, simply telling the model "don't stop until everything is done" creates a different problem: the Agent may keep retrying operations that have already failed, keep generating unnecessary assets, or expand the creative scope on its own without user confirmation. So we need task goals and execution progress to be independent of any single model reply, with the runtime controlling the boundaries of automatic progress.

8.1 Turning plans into a maintainable checklist

We introduced a structured Execution Checklist to record the main steps of a creative goal from plan to delivery.

Unlike an ordinary to-do list the model generates in chat, the checklist is structured state persisted and maintained by the system. Each step can be linked to its execution status, outputs, evidence of completion, and any required user confirmation.

For example, producing a short brand film might be organized as:

Once the user approves the execution plan, the checklist becomes the basis for subsequent progress. The Worker can plan and make tool calls according to the current step, and the runtime can check progress at the end of each round of execution.

Note that the checklist is not a fixed, unchangeable plan.

Creative processes change all the time. For example, during shot generation, the user may decide to change the character design, and existing steps and some outputs may need to be reworked. The system then needs to update the plan based on the new user instruction and maintain the links between steps and outputs.

What the checklist solves is the continuity of goals and progress. It lets the Agent find where it is in the current task even after context compaction, asynchronous waits, or process recovery.

8.2 After the model stops outputting, should the task continue?

Once the plan is persisted, a more concrete question needs answering: when the model finishes the current turn, what should the runtime do?

In designing this mechanism, we drew on ideas from Codex Goals and Claude Code Hooks.

Codex Goals ties goals to ongoing execution state, letting the Agent keep pushing toward goals that aren't yet complete. Claude Code's Stop hook allows extra checks to run when the model is about to stop, and can block the stop based on the result of those checks.

What I took from these two designs is this: the model ending a reply doesn't necessarily mean the whole task is done. The runtime needs its own independent goal state and stop-check mechanism.

However, completion criteria differ markedly between coding agents and creative agents.

In coding tasks, passing tests, a successful build, or the expected files being modified usually provide fairly clear evidence of execution. Creative tasks rarely have a unified objective standard. A video being generated successfully says nothing about whether character consistency, shot pacing, and overall visual quality meet the user's expectations.

So we handle process progress and work acceptance separately.

The former can be judged from the checklist, tool call records, and output status; the latter requires multimodal understanding, the specific task requirements, and human review where necessary.

At the end of each round of execution, the runtime checks whether the current task still has steps to advance, whether it's waiting on anything external, whether user confirmation is needed, and whether it has hit an execution limit.

flowchart TD
    E["Current execution round ends"] --> A{"Is the checklist complete?"}
    A -->|Yes| END["End execution and move to results or acceptance"]
    A -->|No| B{"Is the task paused, blocked, or over its limits?"}
    B -->|Yes| STOP["Stop auto-progress and record the reason"]
    B -->|No| C{"Is it waiting on an external event?"}
    C -->|Yes| WAIT["Suspend and wait for the event"]
    C -->|No| D{"Has it reached a human checkpoint?"}
    D -->|Yes| GATE["Show a confirmation or choice card"]
    D -->|No| F{"Has there been meaningful progress?"}
    F -->|No| LIMIT["Stop auto-continuation to avoid spinning"]
    F -->|Yes| NEXT["Schedule the next Worker round"]
    NEXT --> E

Figure 9: The runtime decides whether to continue based on persisted task state, rather than simply on whether the model has stopped outputting.

Special attention is needed here for meaningful progress.

For example, adding a usable creative output, completing a checklist step, or successfully submitting a new generation job can all count as signs of progress.

But if the Agent is merely rewriting the plan description over and over, or flipping step statuses from "to do" to "in progress" without producing any new useful results, it shouldn't be allowed to continue indefinitely.

We use progress checks and execution limits to minimize the inference cost of this kind of spinning.

Of course, this judgment still has its limits. Generating a new asset only proves that the system performed an operation; it doesn't directly prove the asset meets the quality bar. More complete multimodal quality acceptance remains an area we'll keep exploring.

8.3 Human-in-the-Loop: keeping users involved in key decisions

During creative work, users often want to hand repetitive operations to the Agent while keeping control over key creative directions.

For example, the brand's visual direction, character design, storyboard, and final shot selection are usually highly subjective. If the Agent decides these on its own and keeps generating, it can rack up a lot of wasted cost.

So we introduced Human-in-the-Loop checkpoints that pause automatic progress at stages that need the user's involvement.

When a checkpoint is reached, the Agent can present the current proposal, candidate results, and suggested actions through a structured card. The user simply chooses "Confirm and continue" or "Adjust the plan," and the system updates the task state based on that explicit action.

Click to enlarge

This kind of interaction also draws on the permission prompts in products like Claude Code: for operations with obvious side effects, the system should tell the user in advance what will be executed and let the user decide whether to continue.

In creative work, however, human checkpoints involve not just permissions but also aesthetics and creative direction. A user may have authorized the Agent to generate videos yet still want to pick the approach personally once the shot design is done. So a checkpoint has to be saved as an independent execution state, rather than relying on the model saying "waiting for confirmation" in its reply.

We ran into exactly this kind of problem in drills during development: the Agent had reached a stage that required user confirmation, but only replied "Please confirm whether to continue," without providing an actionable confirmation card, so the user had to type "continue" again.

To fix this, we added a fallback in the runtime: when a task is stopped at a checkpoint and the model hasn't asked the corresponding question, the system automatically adds a confirmation card; and for generations that only run after user approval, if the checklist still has unfinished steps, the task is automatically woken up once generation finishes. This way, waiting for confirmation and resuming execution no longer depend entirely on the model's interaction behavior; the runtime coordinates them based on task state.

Of course, checkpoints don't mean the user has to step in at every step. Irreversible operations or those beyond the authorized scope always require explicit confirmation, while repetitive, low-risk steps can proceed continuously within the authorized scope, avoiding so many confirmations that they interrupt the creative process.

8.4 Autonomy level, budget, and authorization boundaries

On an AI creation platform, the Agent can call text, image, video, audio, and 3D generation capabilities from different model providers, and these operations often come with real costs. So autonomy level and budget control need to be designed separately.

A user allowing the Agent to execute automatically doesn't mean they agree to unlimited calls to paid models; likewise, a user approving a budget doesn't mean every operation can skip review.

The runtime maintains three separate dimensions:

For example, the user approves producing three videos and allows the Agent to complete them automatically within budget.

If, during execution, the Agent finds it needs to create additional character reference images, it can judge whether these operations fall within the original plan and budget. If the scope needs to expand or costs need to increase, it should ask the user for confirmation again.

This touches on an important security boundary: neither the plan the model proposes nor any authorization the model claims can, on its own, become real user authorization.

The system should decide based on the user's explicit actions in the interface and the authorization state stored on the server.

Different tools also call for different control strategies depending on their actual side effects.

• Read-only queries are usually low-risk

• Document writes need to handle versions and permissions

• Paid generations need budget checks

• Irreversible operations such as deleting assets or revoking memories should use stricter confirmation

Compared with coding agents, agents on a creative platform face a special constraint: once many external generation operations are submitted, the cost may already be incurred, and even if the user stops the task afterward, it can't necessarily be fully reversed. So budget and authorization checks need to happen before any side effect actually occurs, and be integrated with the durable execution mechanism described earlier.

From our experience so far, the hard part of autonomous execution is likewise knowing when to stop. When the balance runs low, the user pauses, an external job is pending, human review is required, or consecutive execution makes no meaningful progress, the runtime needs to preserve the current state and wait until conditions are met before resuming.

This also means a higher degree of autonomy isn't always better. Different creative tasks vary considerably in risk, cost, and verifiability. The Agent should minimize unnecessary interruptions within the scope the user has authorized, while preserving the user's control at key points.

Longer term, we also hope multimodal quality evaluation will let the system judge character consistency, shot continuity, and visual quality more accurately. As automatic acceptance matures, some steps that currently require human confirmation may be handed over to the Agent to complete autonomously.

Of course, ongoing collaboration also involves a problem beyond task execution: users may skip the conversation entirely and edit their work directly in the workspace. When the Agent and the user edit the same document or workflow at the same time, how does the system keep state consistent?

This requires a shared workspace that, beyond task execution, maintains joint access to and editing of the work by both people and the Agent.

9. Shared Workspace: letting users and the Agent edit the same work

The mechanisms discussed so far mainly address how the Agent understands tasks, maintains context, and keeps work moving. But in real creative work, collaboration between users and the Agent doesn't only happen in the chat window.

For example, while the Agent is generating video from a storyboard, the user might directly tweak a shot's prompt in the workspace on the right, swap a reference asset, or change a model node in a Flows workflow.

These operations never go through the conversation, yet they genuinely change the work. If the Agent keeps executing based on its earlier context, it might use stale parameters or even overwrite changes the user just made.

So in Lomo Agent, we treat creative outputs as persistent objects independent of the conversation and task context, and organize collaboration between people and the Agent through a Shared Workspace.

9.1 Giving the work its own lifecycle

In traditional conversational agents, generated results are usually attached to a particular chat message. Users can view them, download them, or ask follow-up questions, but as the conversation grows, managing and revising the work becomes increasingly constrained by the session structure.

Lomo's creative workflow, by contrast, involves many objects that need long-term maintenance, such as images, videos, 3D assets, storyboard documents, asset collections, and Flows visual workflows.

These objects need stable identities, versions, and access permissions; they must be directly editable by users and referenceable across tasks.

So we manage conversational context and creative outputs separately:

• Conversation and task context maintains user intent, the execution process, and the information the Agent currently has.

• The shared workspace stores the work that actually needs ongoing editing and delivery, including object identity, content, versions, and related references.

For example, the user first discusses a brand film's storyboard in Main, and a video production task later carries the work forward. Both execution spaces can reference the same storyboard document within their permissions, with no need for each to create its own copy.

Similarly, after a video generation task ends, its outputs can still be referenced by a new workflow, regardless of whether the original Worker is still running.

This design lets work outlive any single conversation turn or task, and lays the foundation for asset reuse and multi-task collaboration.

9.2 Making the Agent aware of changes in the workspace

Shared objects solve the problem of how work is stored, but another problem remains: how does the Agent know the work has been changed?

Suppose the Agent has read a storyboard document and is about to continue generating based on its shot descriptions. At that very moment, the user edits the second shot directly in the workspace.

If the system only records chat messages, the Agent's next round of inference has no way of knowing about this edit.

So we maintain version information for shared objects and record the version each related task saw the last time it read or modified the object.

When a document, asset, or workflow changes, the system can leave a persistent change notice for the related tasks. For example:

The storyboard document has been updated from v3 to v4, and the description of the second shot has changed. Please read the latest version before continuing.

Here, we chose not to have every workspace edit immediately trigger Agent inference.

The reason is simple: when a user is adjusting a workflow, they may drag nodes, change parameters, or swap assets in quick succession. If every change woke up the Worker, it would not only produce a flood of pointless model calls but could also have the Agent repeatedly barging in before the user has finished editing.

So change notices mainly serve to let the Agent discover, on its next execution, that its information is out of date. Only when the related task genuinely needs to keep moving does the runtime decide, based on current state, whether to resume execution.

For visual workflows like Flows, the Agent also doesn't need to read the full node data on every round of inference. The system can first provide a brief summary of the changes, and fetch the latest content on demand only when the model actually needs to work with the relevant nodes.

This approach also ties back to the context engineering described earlier: the workspace stores the complete state of the work, while the context provides only the information needed for the current decision.

9.3 Version conflicts and optimistic concurrency control

Being aware of changes isn't enough to make concurrent edits safe.

Consider a more concrete scenario:

The Agent reads storyboard document v3 and prepares to adjust the third shot. Meanwhile, the user edits the same document directly and saves it as v4.

If the Agent then submits its v3-based changes directly, it could overwrite what the user just did in v4.

To prevent this, we use optimistic concurrency control.

When the Agent reads an object, it gets the current version; when it submits changes, it must include the version those changes are based on. Before writing, the server checks whether the object is still at the expected version.

If the versions match, the change is committed normally; if the object has already been updated, the system rejects any operation that could overwrite the new content and requires the Agent to re-read the latest state.

That said, different types of creative objects call for different granularity when judging version conflicts.

For documents, a full rewrite must carry the version it's based on. If the user has already edited the document, the system refuses to overwrite it; partial edits are located through exact, unique text snippets and are committed only when every expected replacement holds.

For Flows workflows, we use finer-grained conflict checks. The user may be adjusting one shot node while the Agent modifies another. If any change to the workflow caused a submission to be rejected, frequent autosaves would make it hard for both sides to edit in parallel. So the system mainly checks whether the nodes the Agent is about to modify have changed: if the same node has already been modified by the user, the Agent is required to re-read it; if the changes are in other nodes, the non-conflicting modifications are allowed to apply on top of the latest workflow, and the Agent is notified of the related changes.

This isn't automatically merging both sides' different edits to the same content. Rather, by choosing the right version granularity, non-conflicting edits can proceed as independently as possible, while old state is kept from overwriting new content.

sequenceDiagram
    participant A as Agent
    participant O as Shared work
    participant U as User
    A->>O: Read storyboard document v3
    U->>O: Edit storyboard and save as v4
    O-->>A: Record version change notice
    Note over O,A: Change notice waits to be read on next execution
    A->>O: Submit changes based on v3
    O-->>A: Version conflict, overwrite rejected
    A->>O: Re-read v4
    A->>A: Adjust changes to the latest content
    A->>O: Resubmit based on v4

Figure 10: Version control for shared work.

There are also two mechanisms working together here: change notifications let the Agent pick up updates to the work promptly, while version checks provide the final safeguard at commit time. Even if a notification is delayed, the Agent can't overwrite the user's edits with stale state.

Of course, optimistic concurrency control addresses state consistency, not conflicts in creative intent. For example, the user wants a shot in cool tones, while the Agent is still following the original plan and using warm tones. Even without a version conflict, which approach wins still has to be judged against the user's latest intent and the task constraints.

9.4 From Artifacts and Canvas to a shared creative space

On the interaction side, we drew on ideas from products like Claude Artifacts and ChatGPT Canvas.

These products pull generated content out of ordinary chat messages, letting users view and edit documents or code right alongside the conversation, which cuts the cost of repeatedly describing edits in natural language.

This interaction model is highly relevant to Lomo Agent. Creators often would rather drag a node, edit a storyboard passage, or swap an image directly than re-describe every operation to the Agent.

However, Lomo Agent's shared workspace also has to handle scenarios that differ from ordinary document editing.

On one hand, the types of creative objects are more complex: beyond text, they include images, videos, 3D assets, and Flows workflows with node dependencies. On the other hand, the same piece of work may be referenced by multiple tasks, with asynchronous generation jobs and direct user edits happening at the same time. The system has to maintain not only the content of the work but also its versions, associated tasks, and each execution's understanding of the work's state.

So we connected the editable workspace to the Agent's execution state, letting users manipulate the work directly while also letting Workers re-obtain valid object state in later execution.

Click to enlarge

From our experience so far, one important change the shared workspace brings is that users can take part in the same creative process both through natural language and through direct editing. But this also places more responsibility for state synchronization and concurrency control on the system. In particular, when one asset affects several tasks at once, how to manage object dependencies, and which downstream outputs need to be regenerated after an upstream change, are still areas we're exploring.

10. Orchestrating creative capabilities: from model calls to delivering complete works

On a creative platform like Lomote AI, the Agent faces more than just choosing and calling tools; it also needs to understand how different creative capabilities relate to one another.

Generating a single image is relatively simple, but producing a complete short film may involve a script, storyboard, character design, scene references, shot generation, and post-processing. Each stage may use a different model, and the results of one stage directly affect the next. More importantly, a single generation job succeeding doesn't mean the whole creative process will come together.

For example, suppose the Agent plans a production approach: generate the character and background first, then cut out the character, and finally composite them into a specific video frame. If the platform doesn't currently offer the corresponding matting or video compositing capability, the whole plan still can't produce a final work, no matter how good the earlier assets are.

So we need the Agent to understand, already at the planning stage, what the platform can actually do and how different tools and outputs depend on one another.

10.1 Helping the Agent understand creative workflows and capability boundaries

Lomote AI already integrates a range of image, video, and 3D generation and enhancement models, along with asset management and Flows visual workflows.

Each of these tools has its own use cases, and different models vary in input requirements, output quality, cost, and output format.

For the Agent, choosing a tool means weighing the current creative goal, the assets already available, and the requirements of later production steps.

For example, when producing a short film with a recurring character, the Agent needs to think about how to establish a character reference, which models accept reference images, how to keep shots visually consistent, and whether the results can feed into later production steps. If it simply calls models based on the user's end goal, it can easily produce a pile of intermediate assets that look good but can't be used in the final work.

To address this, we've organized creative know-how and tool capability constraints into guides that can be read on demand, covering character consistency, storyboard planning, use of reference assets, workflow orchestration, and the production capabilities the platform currently supports. These guides are provided to the Agent in relevant tasks, helping it plan executable creative steps without keeping all domain knowledge in the system prompt permanently.

This part of the design also works hand in hand with the context engineering described earlier: the Agent fetches the tools and creative knowledge it needs for the current production stage, then plans and executes in light of actual capabilities.

Of course, relying on the model to understand tool descriptions still has limits. The model may propose a production plan that looks reasonable but can't actually be carried out. So for key tool capabilities, input/output constraints, and execution conditions, the system still needs to provide explicit grounds for validation.

10.2 Multi-agent video production through the lens of ViMax

While exploring how to orchestrate creative capabilities, I also looked at multi-agent systems for video generation such as ViMax.

ViMax organizes scripting, characters, storyboarding, shot generation, and video assembly into a single production pipeline, handling the interrelated tasks of video creation through collaboration among different roles or execution stages.

This approach is very instructive. Video production is inherently staged, and each stage cares about different things. Script planning focuses on narrative structure, character design has to attend to visual consistency, and shot generation has to account for the input requirements and output characteristics of specific models.

A specialized division of labor lets each stage work around a relatively clear set of responsibilities, which also helps organize complex production processes.

That said, this kind of architecture also involves trade-offs. On one hand, multi-agent collaboration increases the cost of passing context, synchronizing state, and coordinating; on the other, a predefined production pipeline may not cover users' ever-changing creative needs.

The scenarios Lomo Agent faces are more open-ended. A user might just need a poster, or want to generate a video from existing assets, or ask the Agent to modify a Flows workflow directly. Introducing a fixed multi-agent pipeline for every creative need could actually add unnecessary layers of execution.

So for now, we lean toward a unified task execution system, combined with creative guides and tool capabilities loaded on demand, letting the Worker organize production steps according to the specific goal. For complex creative tasks that need long-term maintenance, we'll gradually explore more explicit specialization and task dependency management. This trade-off is consistent with the Main / Task design discussed earlier: an extra coordination layer is only worth adding when an independent execution space delivers real benefits.

10.3 Production dependency graphs: from linear execution to dependency management

Today, the structured checklist already helps the Agent keep track of creative progress, but when a task involves many interrelated assets and generation steps, a linear checklist structure still has its limits.

For example, a short film contains multiple shots that may share the same character design but use different scenes and generation models.

Some shots can be generated independently, while others must wait for upstream reference assets to be ready. If the user changes the character's look midway, the related shots may need to be regenerated, but completely unrelated scenes shouldn't be affected.

These relationships are better expressed through a Production DAG (production directed acyclic graph).

flowchart LR
    BR["Creative brief"] --> SCRIPT["Script / storyboard"]
    SCRIPT --> CHAR["Character design"]
    SCRIPT --> SCENE["Scene design"]
    CHAR --> SHOT1["Shot 1"]
    CHAR --> SHOT2["Shot 2"]
    SCENE --> SHOT1
    SCENE --> SHOT2
    SHOT1 --> REVIEW["Acceptance and assembly"]
    SHOT2 --> REVIEW
    REVIEW --> FINAL["Final delivery"]

Figure 11: The envisioned production dependency graph.

In such a structure, each node represents a production step or intermediate output, and the edges between nodes describe their dependencies.

Once these relationships are structured, the system has the opportunity to provide several capabilities:

• Dependency scheduling: a downstream step runs only once its required upstream assets are ready.

• Parallel generation: shots with no direct dependency on each other can run in parallel when resources and budget allow.

• Change propagation: when a character or scene design changes, locate the downstream results that may be affected, rather than regenerating everything indiscriminately.

• Incremental production: reuse still-valid assets and generation results wherever possible, reducing redundant calls and unnecessary production cost.

However, a dependency graph also brings extra engineering complexity. In particular, dependencies in creative tasks aren't always fully determined: after an upstream asset changes, whether a downstream result must be regenerated still depends on the scope of the change and the specific production requirements.

Moreover, even with correct dependencies, there's no guarantee that the generated results will look as expected. Character consistency, shot continuity, and overall pacing still require an independent quality evaluation mechanism.

So for now, the Production DAG is more of an evolutionary direction we're exploring. At this stage, our priority is making sure the Agent can complete executable creative workflows within the scope of Lomote AI's existing models and tools, while keeping the cost and usefulness of intermediate outputs under control.

As task dependencies, automatic acceptance, and multimodal understanding mature, there's an opportunity to combine these mechanisms in the future, moving the Agent from executing creative tasks step by step toward managing the entire production process. But judging whether these designs truly work takes far more than one smooth demo. Complex creative tasks also involve tool failures, user edits, state recovery, and duplicate generations.

Whether these designs actually work still has to be verified through failure-recovery drills and end-to-end collaboration rehearsals.

11. Engineering validation: from failure recovery to end-to-end collaboration

Agent systems have a rather interesting property: every module working correctly on its own doesn't mean the overall collaboration is reliable.

For example, a message is routed to the right task and the tool call returns successfully, yet the Agent forgets to continue after generation finishes. Or every step behaves as expected, but the end result is several duplicate documents, leaving the user unsure which one is the latest.

Problems like these are hard to catch with traditional unit tests or a few rounds of simple conversation.

So during Lomo Agent's development, beyond verifying each module's correctness, we also tried to have the Agent run complete workflows in near-real usage scenarios and observe how the mechanisms actually behave in combination.

11.1 Validating runtime reliability

For mechanisms such as durable execution, the message inbox, asynchronous generation, and user authorization, many problems only surface during specific state transitions.

For example:

• A video generation request has been submitted, but the Worker restarts before saving the result

• An old Worker loses connectivity, is taken over by a new process, and then resumes execution

• Just as the user clicks stop, a generation-completed event arrives

These cases rarely come up in normal demos, yet they can directly affect execution state, users' work, and even real costs. So during development, we used fault injection tests to focus on verifying how the system handles these abnormal situations.

The tests cover the following key execution boundaries:

For model outputs that can be reproduced reliably, we can replay fixed results to focus on validating the runtime's state transitions. External generation services can be simulated to produce timeouts, duplicate callbacks, and out-of-order events, so we don't have to rely on real paid models every time. The focus of these tests is the determinism of system behavior. Even though the model itself is somewhat random, task state, execution rights, and billing constraints should follow clear, verifiable rules.

One fairly representative test simulates a Worker suddenly "freezing" in the middle of a tool execution. In the test, the model uses fixed output and first decides to call a query tool; the first Worker has already saved the model decision and recorded the tool call as started, and is then deliberately suspended. Once its lease expires, a second Worker takes over, bumps the execution generation to 2, and continues from the checkpoint: since the model decision for this step has already been saved, it doesn't need to re-run inference, and completes the tool call with the same idempotency key. Finally, the first Worker is resumed and tries to commit its late result.

The results matched expectations: the turn ended up completed with an execution generation of 2; the model was called only once, there was only one tool result, and the old Worker's state writes were rejected. We also used integration tests to cover redelivery after lost queue messages, idempotent recovery of generation jobs after approval, canceled turns being blocked from creating new tasks, cost authorization limits, and memory revocation.

11.2 End-to-end collaboration rehearsals

Runtime tests can verify that the system handles abnormal states correctly, but they can't fully capture the Agent's actual collaboration experience.

For example, the Agent successfully completes storyboard planning but then stops at the next stage, not starting video generation until the user types "continue" again. From the program's point of view, this may not count as a failure, but for the user, the process clearly isn't natural enough.

To address this, I tried introducing scenario rehearsals.

In an isolated environment, we use a real text model together with simulated generation services and a simulated user with preset behaviors, and have the Agent work through a relatively complete creative process.

The rehearsal scenarios cover poster series, short film production, Flows workflow edits, 3D creation, switching between multiple tasks, resuming past tasks, and memory correction and forgetting. These scenarios involve not only continuous model and tool calls, but also user confirmations, asynchronous waits, and context linking across tasks.

For example, in the short film scenario, the simulated user first states a creative goal, and the Agent has to organize the storyboard, prepare assets, and submit generation requests. During execution, the user might adjust the plan, pick shots, or slip in another task on the fly. What we want to observe isn't just whether a single step succeeds, but whether the Agent can keep pushing toward the original goal after these changes.

I focus on observing:

• Whether the Agent pauses or continues at the right points

• Whether the user has to keep sending "continue" to move the task forward

• Whether there are unnecessary confirmations or duplicate generations

• Whether new messages are correctly linked to existing tasks

• Whether the Agent picks up the latest state after the user edits the work

• Whether the same piece of work gets accidentally split into multiple copies

• Whether old information still affects later collaboration after a memory is corrected or revoked

Compared with calling real media generation services directly, a simulated environment keeps testing costs under control and makes it easier to reconstruct specific scenarios repeatedly, so we can observe problems that may arise in interactions across modules.

This approach has clear limits, though. Simulated generation results can't validate the visual quality of real work, and a simulated user with preset strategies can't fully reflect the complex behavior of real creative work. So these rehearsals are mainly used to check the continuity of task organization, execution state, and collaboration flow; they don't directly represent success rates on real user tasks. The quality of the final work and the actual user experience still need further evaluation using real generation results and user feedback.

11.3 How to measure the quality of a collaboration

Through these rehearsals, I gradually realized that whether the model's replies are fluent and whether tool calls succeed are hardly enough to fully measure a long-running Agent.

What users care about more may be questions like these:

• How many times do I have to repeat myself?

• Does the Agent remember requirements that have already been settled?

• Can it naturally pick up where we left off a few days later?

• When I edit the work, does it know what happened?

So beyond traditional execution correctness, I've also started paying attention to dimensions such as task continuity, the cost of user intervention, repeated operations, and output consistency.

The real rehearsals also exposed some problems that are hard to spot when testing modules in isolation.

For example, while creating an English poster series, Main correctly understood the user's request and created a task, but the Worker switched to Chinese in its later planning. The problem wasn't in task routing: the task had only received the goal description as summarized by the main thread, which lacked the language information in the user's original wording. We later adjusted how the user's language is passed along, so a task can maintain its reply language based on what the user actually typed, rather than relying entirely on the model to infer it from the task description.

Another problem appeared where human checkpoints meet asynchronous generation. The Agent had reached a stage that required user confirmation but didn't provide an actionable confirmation card; or, after the user approved a generation, the external job completed normally but the task didn't move forward. For these cases, we added a confirmation-card fallback and automatic wake-up after generation completes to the runtime, reducing reliance on the model's interaction behavior.

On their own, none of these problems necessarily shows up as an obvious execution failure, yet they directly affect the collaboration continuity users experience. They also made something clearer to me: the model is responsible for understanding intent and planning steps, while key states such as waiting, authorization, and resuming execution need deterministic guarantees from the runtime.

Of course, these dimensions alone don't represent how intelligent an Agent is. On a creative platform, process completion and output quality still need to be evaluated separately. An Agent can generate every shot according to plan without guaranteeing that the final video meets the user's expectations for visual style, character consistency, and narrative pacing. More reliable multimodal acceptance remains a capability we want to keep improving.

From our experience so far, end-to-end rehearsals have given us a relatively controllable way to validate the system. They not only help us find problems in cross-module collaboration, but also put some of the principles in the architecture to the test across complete execution flows. Beyond simply adding model capabilities, how to make those capabilities fit together reliably in long-term collaboration deserves just as much ongoing investment.

12. Open problems and future exploration

Lomo Agent already has foundational capabilities such as multi-task organization, durable execution, context management, and a shared workspace, but there are still many open questions between where it is now and the creative agent I envision.

The first is cross-task collaboration.

The current Router / Worker structure lets multiple tasks progress independently and access shared outputs when needed. But going further, letting Workers proactively ask each other for help, delegate tasks, or even negotiate a shared budget, would require solving more complex problems of permissions, state synchronization, and concurrency consistency.

Multi-agent collaboration looks attractive, but more Agents don't inherently mean more efficiency. Extra coordination overhead and context passing can cancel out the gains of specialization. I'd rather first find work that's genuinely worth splitting out based on real needs, then gradually widen the scope of collaboration.

The second is dependency management in creative workflows.

The Production DAG mentioned earlier is a direction I'm particularly interested in. For complex productions with multiple shots, characters, and scenes, explicit dependencies help support parallel execution, incremental updates, and reuse of results.

But dependencies in creative tasks aren't always deterministic. When a user changes a character design, which shots must be regenerated and which can still be kept often has to be judged against the actual content.

The reliability of asynchronous generation also has room for improvement.

Currently, the system can't proactively cancel generation jobs that have already been submitted upstream; and if a job has been created but a failure occurs before the upstream job ID is saved, there's no way yet to look it back up and restore the association.

These cases already have corresponding state handling and cost fallbacks, but they can still incur unnecessary generation costs. Going forward, we hope to gradually integrate upstream services' cancellation and job query capabilities, so the system can more accurately handle execution interruptions and users changing their requirements on the fly.

Multimodal quality acceptance is another hard problem.

Today, a tool returning success or a checklist step being completed only proves that the system performed the corresponding operation. True output quality also involves visual consistency, motion continuity, audio-visual sync, and the user's subjective taste.

As multimodal models advance, I hope to gradually bring these evaluation capabilities into the execution flow, so the Agent can more accurately judge when adjustments are needed, when regeneration is worthwhile, and when the decision should be handed back to the user.

In addition, the accuracy of long-term memory, the reliability of task routing, and the boundaries of proactive follow-up also need ongoing validation.

Provenance tracking and versioning make memory more controllable, but they can't fully prevent extraction errors by the model or mistaken recall of historical information. Likewise, although the current routing mechanism already combines the task index, history retrieval, and similar-task duplicate checks to reduce misjudgments, for a large number of tasks accumulated over time we still need real usage data to further measure how accurately tasks are linked, and how often users have to correct them.

As for proactive follow-up, the system may know a task still has work left, but not necessarily whether the user wants to be interrupted right now. We want future Agents to be more proactive, but that proactivity has to be built on user authorization, the value of the task, and actual usage habits.

These problems may not all have mature solutions in the short term. As Lomo Agent moves into more real creative scenarios, we'll keep re-examining the current architecture to verify which designs truly work and which need adjusting.

Closing thoughts

Looking back on Lomo Agent's journey from initial concept to launch, I've become increasingly convinced that the evolution of Agents is confronting software architecture with some genuinely interesting new problems.

In the past, designing software usually meant first defining clear features, flows, and interaction boundaries, then having users operate within those rules. Now that AI can understand intent, call tools, and make autonomous decisions, much of the organizing and coordinating work that used to fall on users can gradually be handed to the system.

But better model capabilities are only part of the story.

An Agent capable of long-term collaboration also needs to know what it's working on, what it has already done, which information is still valid, and when it should continue or stop. It has to deal with the interruptions, concurrency, costs, and permissions of real-world systems, and it has to let users change their minds at any time.

Lomo Agent's architecture is our attempt to put these questions into practice. Many of the ideas came from excellent products and engineering work such as DeepSeek, Manus, Claude Code, and Temporal; others emerged gradually from my own creative scenarios.

The current implementation is still at an early, exploratory stage. There's a long way to go before it reaches its ideal form, and many of its designs still need to be tested by real users.

Still, I've always believed that as model capabilities keep advancing, there's enormous room for imagination in how software and people will interact in the future.

Perhaps one day we won't need to learn how to use a piece of software at all; we'll simply tell it what we want to get done.

That's the direction we hope Lomo Agent keeps moving toward.

If you're building Agents too, I'd love to compare notes on the experiences and problems you've run into in architecture and engineering;

If you're a creator, you're welcome to bring your real creative needs and try out Lomo Agent.

Try Lomo Agent: lomo agent

Click to enlarge

Click to enlarge

References and further reading

The links below mainly illustrate architectural inspiration and research background; they do not imply that each project shares Lomo Agent's implementation or design goals across all capabilities.

1. DeepSeek Super-agent Prototype

2. Temporal — Design Patterns

3. OpenAI — Using Goals in Codex

4. Anthropic — Effective Harnesses for Long-Running Agents

5. Claude Code — Hooks

6. Anthropic — How We Built Our Multi-Agent Research System

7. Manus — Context Engineering for AI Agents

8. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory

9. Zep / Graphiti: A Temporal Knowledge Graph Architecture

10. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

11. OpenClaw — Memory Architecture

12. ViMax — Multi-Agent Video Generation

Note: Throughout this article, we have tried to clearly distinguish "existing mechanisms" from "future directions." The idempotency guarantees of external services, the scope of data covered by forgetting, model caching behavior, and the effectiveness of automatic acceptance are all subject to the specific implementation and its dependencies.