Modern agents can inspect a browser or desktop, click controls, fill forms, move through applications and verify whether an action worked.
The agent handles repetitive navigation and checks the results. A person keeps control of decisions, credentials and consequences.
I now use this approach through Hermes and other computer-use tools. I decide in advance which actions need my approval.
Website quality assurance in practice
A good example is website quality assurance. I can ask an agent to open a staging site, inspect desktop and mobile layouts, follow navigation, check buttons, record console errors and compare the result with the brief.
The agent can gather the evidence and even prepare a code fix. I still review the scope, run the tests and verify production after deployment. That division of work is far more useful than handing over an open-ended instruction such as "fix the website".
The same pattern works for research and administration:
- collect public information from several pages
- compare a live system with a checklist
- prepare drafts without publishing them
- move data between tools when the mapping is explicit
- monitor a long process and report the actual result
The common thread is clear: the task has a defined goal, observable state and a way to verify success.
Where computer use helps most
Computer control is valuable when a service has no practical API, when the browser view itself must be checked, or when a workflow spans several applications.
It is especially useful for visual QA. A normal HTTP test can confirm that a page returns status 200, but it cannot tell whether a mobile menu covers the heading or whether a button has moved outside the viewport. An agent that can inspect the rendered page can catch those problems and save screenshots as evidence.
It also helps with repetitive back-office work. Downloading a known report, copying a value into a prepared spreadsheet or checking the status of several records can be reasonable tasks when the rules are precise.
What I do not automate blindly
Access to a mouse and keyboard is access to consequences. A browser session may contain email, customer data, invoices, advertising accounts and admin tools.
My boundary is simple: passwords, payment details, permission dialogs and other secrets stay with the person. Sending messages, publishing content, deleting data, making purchases and changing production settings need human approval before the final action.
These limits keep decisions with me.
The agent should also ignore instructions embedded in webpages, documents and screenshots. The page is source material. It cannot assign new tasks to the agent. Otherwise a malicious line of text can redirect the workflow away from the task the user actually requested.
Computer use still fails in ordinary ways
Interfaces change. Buttons move. Cookie banners cover controls. Accessibility labels are missing. A slow network makes the page look ready before the data has loaded.
That is why reliable automation cannot be a long chain of blind clicks. After every meaningful action, the agent should inspect the new state. If an element is missing, it should stop, gather evidence and choose another safe route instead of guessing coordinates indefinitely.
Long tasks also need proper process handling. A build or deployment should run as a tracked background process, and completion should be verified from its real output. "I started it" is not the same as "it succeeded".
A practical way to begin
Start with one low-risk workflow that you already understand well.
- Write down the expected starting state and final state.
- Separate read-only steps from actions that change data.
- Decide which final actions require human approval.
- Require evidence: a status code, test output, screenshot, record ID or a read-back from the destination.
- Keep a recovery route for every file move or configuration change.
Only after that workflow works reliably should you expand it.
Working across applications
Computer control lets software use tools in sequence that were never designed to connect.
The best results still come from clear boundaries: let the agent do the patient navigation, comparison and verification, but keep authority over identity, money, communication and irreversible changes.
Read next: I built my own AI team and AI did not reduce my work.