Tag: LLM

A Secure Operating System for Collective Intelligence

Olaf Witkowski

We stand today at an inflection point in the history of computing. Autonomous AI agents are being deployed at high speed, spawning powerful, increasingly multimodal systems, able to operate across email, documents, apps, codebases, the open web, and shared digital environments. This looks like the long-awaited transition from single-process software to multi-process computing: workflows that once required continuous human supervision are now being delegated to networks of semi-autonomous processes. We have been dreaming about this moment. If it wasn’t for a decisive difference.

We are entering this transition without the security substrate that made the previous generation of modern operating systems dependable. As a result, whatever the productivity gains, they will come with an expanded attack surface—and a predictable set of exploitable vulnerabilities. Here, I argue for a Secure Operating System for Collective Intelligence: an infrastructure layer that treats trust as a first-class primitive, enforced by protocol and architecture. The goal is not even perfectly safe agents, but at least societies of agents that remain stable and recover gracefully when some participants, inputs, tools, or components are compromised.

Computing’s original sin was the stored-program idea that puts instructions and data on the same addressable substrate. For both safety and creativity, agentic AI needs to remember this core principle, which enabled general-purpose computation—but also made boundary failures inevitable, forcing decades of mitigations to keep “data becoming control” from turning into system compromise. Image: Apple with logos, as an original sin metaphor. Image Credit: Anastasiya Badun on Pexels.com

An Operating System Design for Collective Intelligence

In early computing, programs ran with broad, ambient authority. They could overwrite each other’s memory, monopolize resources, and crash the entire machine. Security and isolation were not “missing features”; they were not yet recognized as foundational. Early work on multiprogramming clarified why concurrent computation demands explicit semantics and control [6]. Over decades, we built kernels, privilege boundaries, memory protection, and process isolation not because we wanted complexity, but because we learned that multi-process power without boundaries becomes brittle and unsafe [11].

This shift is no longer hypothetical. The rise of agent-native platforms such as Moltbook—and even its acquisition by Meta [19]—makes clear that agents are no longer confined to isolated task execution inside apps; they are beginning to inhabit shared digital environments of their own.

Agentic AI now reintroduces a similar structural risk profile—except the “processes” are tool-using models operating over unbounded, adversarial text streams (web pages, emails, repos, tickets). The mechanisms meant to constrain them often amount to little more than prompts, filters, and user confirmations. Those are valuable guardrails, but they are not security boundaries. Here, allow me to proceed in three movements. First, I revisit the long struggle to separate instructions from data in conventional computing. Second, I explain why many current defenses against prompt injection and tool hijacking fail at scale. Third, I sketch a protocol-oriented path forward: an OS-like trust substrate for collective intelligence.

For both safety and creativity, agentic AI needs OS-grade protocols that carry trust across tools, content, and networks. As agentic ecosystems scale, they can cross a trust-substrate threshold: without enforceable boundaries, instability can run away; with protocol-enforced trust and error-correcting governance, societies of agents can remain stable and recover gracefully under compromise. Image credit: Olaf Witkowski License: CC BY 4.0.

The Original Sin: Stored Programs and the Long Fight for Separation

The stored-program concept associated with von Neumann-style architectures places instructions and data on the same representational substrate. This choice enabled general-purpose computing, but also made a deep security truth inescapable: without enforced boundaries, data can be made to behave like instructions [25]. Security engineering since the 1970s is, in large part, the story of building separation mechanisms “after the fact.”

The Morris Worm (1988) remains an archetypal example of boundary collapse: memory corruption turned attacker-controlled input into control flow, propagating at network speed. Two canonical postmortems—one focused on low-level exploit mechanics, one on propagation dynamics—remain instructive today [7][23].

Mitigations—DEP/NX, ASLR, stack canaries, and decades of exploit-mitigation research—raised attacker cost but did not abolish the underlying pattern. StackGuard exemplifies compiler-level defenses that reduce classic stack-smashing risk [3]. ASLR made exploitation less reliable by randomizing memory layout, but its effectiveness depends on assumptions that attackers routinely probe and break [20]. When direct code injection is blocked, attackers adapt; return-oriented programming is a canonical example [19]. A broader synthesis of why memory-corruption remains a persistent “eternal war” is captured in the SoK literature [24].

For both safety and creativity, agentic AI needs mathematics and architecture that carry trust across domains. The Morris worm (1988) was an early warning shot from early networked computing: once semi-autonomous code can move across machines, tiny weaknesses stop being local bugs and become system-wide failures—an old lesson that now returns at AI scale. Image: Morris worm scheme, modified. Image credit: JorisTheys, via Wikimedia Commons. License: CC BY-SA 3.0 / GFDL 1.2 or later.

The Boundary Collapse in LLM agents: Language as a Control Surface

LLMs process mixed inputs—user requests, retrieved documents, emails, code, and tool outputs—as a single token stream. Critically, there is no cryptographic or type-enforced distinction between “authorized instructions” and “untrusted content that merely contains instruction-like text.” From the model’s perspective, it is all context.

This collapses what operating systems enforce by design: a hard boundary between instruction and data. It also turns language into a control surface: untrusted text can steer the agent’s internal policy—what it prioritizes, what it attempts next, which tools it calls. In the language of causal analysis and controllability, this effectively opens an intervention channel: external inputs are not merely observed, but can causally influence downstream actions, including actions with real-world effects. The practical implication is that, as agents ingest more untrusted content, the system becomes increasingly controllable by the environment—including adversaries—unless that influence is constrained by architecture.

For both safety and creativity, agentic AI needs OS-grade protocols that enforce trust at the action boundary. In control-theory terms, the control surface is the actuator: the policy gate where an agent’s internal decision (u(t)) becomes a real-world action (y(t)), and where least privilege and capability constraints must be applied. The negative feedback loop illustrates “error-correcting stability”: auditing and monitoring feed back deviations (e(t)=r(t)-y(t)) so governance can tighten constraints before failures become systemic. Image: negative feedback control system with explicit control surface, modified. Image credit: Olaf Witkowski. License: CC BY 4.0.

The risk becomes acute once an LLM is granted agency—tool access that can touch files, services, credentials, and communications. At that point, indirect prompt injection becomes a modern instance of the confused-deputy problem: a privileged actor can be induced to misuse its authority [9]. Real-world demonstrations have shown that LLM-integrated systems can be compromised through indirect prompt injection in ways that are hard to prevent purely at the content layer [8]. OpenAI has similarly described prompt injection for browsing agents as an open challenge requiring sustained hardening work [15].

A Concrete Case Study: Moltbook and OpenClaw

One reason the “Secure OS” framing matters is that agentic risk changes qualitatively when agents become networked and socially embedded. Moltbook—described as a Reddit-style platform designed for AI agents—emerged from the OpenClaw ecosystem and rapidly became a live testbed for large-scale agent interaction [10].

That kind of environment is not just “many agents posting.” It is an amplification layer for classical failure modes: untrusted inputs everywhere, persistent authority confusion, and automation at scale. In Moltbook’s early growth phase, public reporting described a misconfigured Supabase database that exposed sensitive data, including large volumes of API tokens and user emails [26]. A complementary analysis emphasizes how verification and governance issues emerge when large agent populations interact through an open platform—and why the system-level dynamics cannot be reduced to the trustworthiness of any single agent [4].

Moltbook illustrates how agent societies turn architecture into governance. Autonomous agents ingest untrusted content, coordinate through a shared platform, and act through tool interfaces that touch real infrastructure. The security lesson: when language becomes a control surface and tools become control surfaces, trust must be enforced at the protocol and action layers—through isolation, capability gates, and auditing—rather than inferred from content. Image: Moltbook ecosystem schematic, modified. Image credit: Olaf Witkowski (original concept), generated with AI assistance (DALL·E 3). License: CC BY 4.0.

The practical takeaway is that once you have many agents operating together, you no longer get to ask, “Is this agent smart enough to resist manipulation?” You must ask, “Does the system remain safe when some agents, inputs, or components are compromised?” In other words, the unit of analysis shifts from the agent to the society.

Why Traditional Defenses Don’t Hold at Scale

Content filters and injection detectors help, but they face a basic adversarial reality: language is high-dimensional and attackers adapt. Even well-designed defenses can often be bypassed with modest optimization effort against the specific target model and setting [22]. This is not a complaint about any one implementation; it is what happens when the attacker’s space of possible encodings is much larger than the defender’s space of reliable signatures.

Adding more words to the system prompt (“ignore untrusted instructions”) is useful as a heuristic, but it is not enforcement. It asks the model to do secure interpretation using the same interpretive machinery that is being attacked. It is defense-by-instruction in a regime where instructions are the attack vector. Human confirmation gates are necessary for high-impact actions—but as a primary defense, they degrade under attention economics. Users rubber-stamp to keep work moving, and attackers can shape workflows to exploit habituation [5].

Decidability: Making the Model Smarter Falls Short

There is a result in computer science, sometimes called “universal antivirus theorem”, that essentially has three declensions: Cohen’s result that no algorithm can perfectly decide, over all programs, whether a given program is a virus under general definitions [2]; the broader computability lens that every nontrivial semantic property of programs is undecidable in general (Rice’s theorem, which can proved by reduction from the Halting Problem) [18]; and the practical consequence that any real-world detector must accept tradeoffs—false positives, false negatives, or restricted scope—because perfect classification over arbitrary programs is not available in general.

For both safety and creativity, agentic AI needs humility about what can be decided by inspection. The “universal antivirus” theorems reminds us that perfect detectors over arbitrary programs are out of reach in general, and even more in practice; likewise, no content-only filter can guarantee that untrusted inputs won’t steer an agent into unauthorized actions. The boundary has to move from “classify the text” to “constrain the act”: least privilege, policy gates, auditable execution. Image: laptop / adversarial code metaphor. Image Credit: Photo by Antoni Shkraba Studio on Pexels.com

It is tempting to believe that sufficiently advanced models will robustly distinguish legitimate from malicious instructions. But the general problem resembles a class of questions computer science treats with deep caution: deciding nontrivial semantic properties of arbitrary computations. Rice’s theorem tells us that any nontrivial semantic property of programs is undecidable in the general case [18], and classic work on malware detection shows why universal, perfect detectors are out of reach [2]. Without claiming a strict formal reduction from “prompt injection” to these theorems, the analogy is still instructive: if you demand an always-correct procedure that decides whether arbitrary instruction-bearing inputs will induce unauthorized behavior in a sufficiently general agent, you may be asking for guarantees computation cannot provide.

This does not mean “give up.” It means: stop treating content inspection as the boundary. Treat the model as untrusted userland, and enforce safety at the action layer: least privilege, isolation, policy gates, auditable execution, and reversible operations.

Toward a Secure OS for Collective Intelligence

If agentic AI is a new computational substrate, we need OS-grade primitives for multi-agent safety. Distributed systems already offers a mature vocabulary for building correctness without trusting participants. Byzantine fault tolerance formalizes how to maintain correct system behavior even when some components behave arbitrarily or maliciously [12]. The key move is architectural: you do not “detect the bad node reliably”; you design protocols that remain correct despite them [1].

CRDTs show how to design shared state so that concurrent updates converge without centralized locking, by making operations mathematically composable [21]. For agent societies, this suggests governance and shared-work artifacts that are resilient to concurrency and partial failure—less “everyone must coordinate perfectly,” more “the structure converges safely.” A widely used overview of CRDT families and design patterns appears in reference works on distributed data technologies [17]. And finally, human societies scale trust through institutions—rules, monitoring, dispute resolution, graduated sanctions—rather than assuming virtue. Ostrom’s principles for governing shared resources generalize well to sociotechnical systems where many actors share computational and informational commons [16].

For both safety and creativity, agentic AI needs an OS that works like an aircraft cockpit: trusted instrumentation, bounded controls, and explicit control surfaces where intent becomes action under constraint. That is the metaphor for a Secure OS for Collective Intelligence: enforce trust at the action boundary—policy gates, capabilities, audited tool interfaces—so multi-agent power can scale without making every input a flight-critical vulnerability. Image Credit: Photo by Kelly on Pexels.com

A Reference Architecture: Four Layers of Trust

A Secure OS for Collective Intelligence cannot be designed in any single catch-all mechanism. It necessarily comes as a layered architecture. One workable abstraction is to design four layers of trust:

Layer 1 — Isolation & containment: task sandboxes, network egress control, secretless execution where possible.

Layer 2 — Capability-based authority: no ambient credentials; narrowly scoped, revocable capabilities for each operation [13].

Layer 3 — Auditing & behavioral monitoring: tool-call logging, anomaly detection, throttles, and circuit breakers for suspicious behavior.

Layer 4 — Protocol evolution: governance that updates from incidents and near-misses—structured, reviewable, and convergent across the ecosystem.

The design goal is not “never compromised.” It is “never systemic”: failures should be localized, attributable, and recoverable—more like immunology than wishful thinking [14].

Agentic AI is delivering real capability. But the security boundary has shifted: language is now a control surface, and agents are increasingly connected, tool-empowered, and socially embedded. Moltbook/OpenClaw is a useful preview of what happens when many agents operate together in a porous environment: you don’t just get emergent coordination—you get emergent failure modes.

If we want agent societies that flourish without collapsing into exploitation, trust must be engineered into the substrate. That means OS-like primitives: isolation, privilege separation, capability security, auditable actions, and governance mechanisms that can evolve. In short: the Operating System for Collective Intelligence. Can we build this trust substrate before the Morris Worm moment of this new era? The tools are already in our hands—if we choose to use them.

References

[1] Castro, M., & Liskov, B. (1999). Practical Byzantine fault tolerance. In Proceedings of the Third Symposium on Operating Systems Design and Implementation (OSDI ’99) (pp. 173–186). https://doi.org/10.5555/296806.296824

[2] Cohen, F. (1987). Computer viruses: Theory and experiments. Computers & Security, 6(1), 22–35. https://doi.org/10.1016/0167-4048(87)90122-2

[3] Cowan, C., Pu, C., Maier, D., Hinton, H., Walpole, J., Bakke, P., Beattie, S., Grier, A., Wagle, P., & Zhang, Q. (1998). StackGuard: Automatic adaptive detection and prevention of buffer-overflow attacks. In Proceedings of the 7th USENIX Security Symposium. https://doi.org/10.5555/1267549.1267554

[4] De Marzo, G., & Garcia, D. (2026). Collective behavior of AI agents: The case of Moltbook. arXiv preprint arXiv:2602.09270. https://arxiv.org/abs/2602.09270

[5] Debenedetti, E., Hines, K., & Goel, S. (2024). AgentDojo: A dynamic environment to evaluate attacks and defenses for LLM agents. arXiv preprint arXiv:2406.13352. https://arxiv.org/abs/2406.13352

[6] Dennis, J. B., & Van Horn, E. C. (1966). Programming semantics for multiprogrammed computations. Communications of the ACM, 9(3), 143–155. https://doi.org/10.1145/365230.365252

[7] Eichin, M. W., & Rochlis, J. A. (1989). With microscope and tweezers: An analysis of the Internet virus of November 1988. In Proceedings of the IEEE Symposium on Security and Privacy (pp. 326–343). https://doi.org/10.1109/SECPRI.1989.36307

[8] Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. arXiv preprint arXiv:2302.12173. https://arxiv.org/abs/2302.12173

[9] Hardy, N. (1988). The confused deputy (or why capabilities might have been invented). ACM SIGOPS Operating Systems Review, 22(4), 36–38. https://doi.org/10.1145/54289.871709

[10] Heim, A. (2026, January 30). OpenClaw’s AI assistants are now building their own social network. TechCrunch. https://techcrunch.com/2026/01/30/openclaws-ai-assistants-are-now-building-their-own-social-network/

[11] Lampson, B. W. (1974). Protection. ACM SIGOPS Operating Systems Review, 8(1), 18–24. https://doi.org/10.1145/775265.775268

[12] Lamport, L., Shostak, R., & Pease, M. (1982). The Byzantine generals problem. ACM Transactions on Programming Languages and Systems, 4(3), 382–401. https://doi.org/10.1145/357172.357176

[13] Miller, M. S., Yee, K. P., & Shapiro, J. (2003). Capability myths demolished (Tech. Rep. SRL2003-02). Johns Hopkins University Systems Research Laboratory.

[14] Murphy, K., & Weaver, C. (2016). Janeway’s immunobiology (9th ed.). Garland Science.

[15] OpenAI. (2025). Continuously hardening ChatGPT Atlas against prompt injection attacks. https://openai.com/index/hardening-atlas-against-prompt-injection/

[16] Ostrom, E. (1990). Governing the commons: The evolution of institutions for collective action. Cambridge University Press.

[17] Preguiça, N., Baquero, C., & Shapiro, M. (2018). Conflict-free replicated data types (CRDTs). In Encyclopedia of Big Data Technologies. Springer. https://doi.org/10.1007/978-3-319-63962-8_185-1

[18] Rice, H. G. (1953). Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical Society, 74(2), 358–366. https://doi.org/10.1090/S0002-9947-1953-0053041-6

[19] Reuters. (2026, March 10). Meta acquires AI agent social network Moltbook. https://www.reuters.com/business/meta-acquires-ai-agent-social-network-moltbook-2026-03-10/

[20] Shacham, H. (2007). The geometry of innocent flesh on the bone: Return-into-libc without function calls (on the x86). In Proceedings of the 14th ACM Conference on Computer and Communications Security (pp. 552–561). https://doi.org/10.1145/1315245.1315313

[21] Shacham, H., Page, M., Pfaff, B., Goh, E.-J., Modadugu, N., & Boneh, D. (2004). On the effectiveness of address-space randomization. In Proceedings of the 11th ACM Conference on Computer and Communications Security (pp. 298–307). https://doi.org/10.1145/1030083.1030124

[22] Shapiro, M., Preguiça, N., Baquero, C., & Zawirski, M. (2011). Conflict-free replicated data types. In Proceedings of the 13th International Symposium on Stabilization, Safety, and Security of Distributed Systems (pp. 386–400). https://doi.org/10.5555/2050613.2050642

[23] Shi, C., Lin, S., Song, S., Hayes, J., Shumailov, I., Yona, I., Pluto, J., Pappu, A., Choquette-Choo, C. A., Nasr, M., Sitawarin, C., Gibson, G., & Terzis, A. (2025). Lessons from defending Gemini against indirect prompt injections. arXiv preprint arXiv:2505.14534. https://arxiv.org/abs/2505.14534

[24] Spafford, E. H. (1989). The Internet worm program: An analysis. ACM SIGCOMM Computer Communication Review, 19(1), 17–57. https://doi.org/10.1145/66093.66095

[25] Szekeres, L., Payer, M., Wei, T., & Song, D. (2013). SoK: Eternal war in memory. In 2013 IEEE Symposium on Security and Privacy (pp. 48–62). https://doi.org/10.1109/SP.2013.13

[26] von Neumann, J. (1993). First draft of a report on the EDVAC. IEEE Annals of the History of Computing, 15(4), 27–75. https://doi.org/10.1109/85.238389

[27] Zwets, B. (2026). Moltbook database exposes 35,000 emails and 1.5 million API keys. Techzine. https://www.techzine.eu/news/security/138458/moltbook-database-exposes-35000-emails-and-1-5-million-api-keys/

[28] Zhan, Q., Liang, R., Zhu, X., Chen, Z., & Chen, H. (2024). InjecAgent: Benchmarking indirect prompt injections in tool-integrated LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (pp. 12458–12475). https://doi.org/10.18653/v1/2024.findings-acl.624

Life After Programming: Embracing Human-Machine Symbiosis in the Age of AI

Olaf Witkowski

As AI continues to evolve, conversations have started questioning the future of traditional programming and computer science education. The rise of prompt engineering—the art of crafting inputs to lead AI models to generating specific outputs—has led many to believe that mastering this new skill could replace the need for deep computational expertise. While this perspective does capture a real ongoing shift in how humans interact with technology, it overlooks the essential role of human intelligence in guiding and collaborating with AI. This seems a timely topic, and one worthy of discussion—one that we’ll be exploring at this year’s Cross Labs Spring Workshop on New Human Interfaces.

Can we forget about programming? As AI reshapes our interaction with technology, the future is unlikely to erase programming—rather, it would redefine it as another language between humans and machines. Image Credit: Ron Lach

The Evolving Role of Humans in AI Collaboration

Nature demonstrates a remarkable pattern of living systems integrating new tools into, transforming them into essential components of life itself. For example, mitochondria, once an independent bacteria, became integral to eukaryotic cells, taking the function of energy producers through endosymbiosis. Similarly, the incorporation of chloroplasts enabled plants to harness sunlight for photosynthesis, and even the evolution of the vertebrate jaw exemplifies how skeletal elements were repurposed into functional innovations. These examples highlight nature’s ability to adapt and integrate external systems, offering a profound analogy for how humans might collaborate with AI to augment and expand our own capabilities.

Current AI systems, regardless of their sophistication, remain tools that require some kind of human direction to achieve meaningful outcomes. Effective collaboration with AI involves not only instructing the machine but also understanding its capabilities and limitations. Clear communication of our goals ensures that AI can process and act upon them accurately. This process transcends mere command issuance; it quickly turns into a dynamic, iterative dialogue where human intuition and machine computation synergize to mutually cause a desirable outcome.

Jensen Huang’s Take on Democratizing Programming

“The language of programming is becoming more natural and accessible.” – NVIDIA CEO Jensen Huang at Computex Taipei, emphasizing how interfacing computers is undergoing a radical shift. Credit: NVIDIA 2023

NVIDIA Founder and CEO Jensen Huang highlighted this paradigm shift in these words:

“The language of programming is becoming more natural and accessible. With AI, we can communicate with computers in ways that align more closely with human thought. This democratization empowers experts from all fields to leverage computing power without traditional coding.”

The quote is from a couple of years ago, in Huang’s Computex 2023 keynote address. This transformation means that domain specialists—not just scientists and SWEs—can now harness AI to drive their work forward. But this evolution doesn’t render foundational knowledge in computer science obsolete; it merely underscores the imminent change in the nature of human-machine interactions. Let us dive further into this interaction.

Augmenting Human Intelligence Through New Cybernetics

A helpful approach, to fully realize AI’s potential, is to focus on augmenting human intelligence via ways akin to the infamous ways of cybernetic technology (Wiener, 1948; Pickering, 2010). « Infamous » only because of how the field abruptly fell out of favor in the late 70’s, as the time’s technical limitations unfortunately led to its decline, along with a growing skepticism about its implications for agency, autonomy, and human identity. They were nevertheless not a poor approach by any count, and may soon become relevant again, as technology evolves into a substrates that better supports its goals.

El Ajedrecista: the first computer game in history. Norbert Wiener (right), the father of cybernetics—the science studying communication and control in machines and living beings—is shown observing the pioneering chess automaton created by Leonardo Torres Quevedo, here demonstrated by his son at the 1951 Paris Cybernetic Conference. Image credit: Wikimedia Commons

Such cybernetic efforts towards human augmentation aim to couple human cognition with external entities such as smart sensors, robotic tools, and autonomous AI models, through a universal science of dynamical control and feedback. This integration between humans and other systems fosters an enactive and rich, high-bandwidth communication between humans and machines, enabling a seamless exchange of information and capabilities.

Embedding ourselves within a network of intelligent systems, and possibly other humans as well, enhances our cognitive and sensory abilities. Such a symbiotic relationship would allow us to better address complex challenges, by efficiently processing and interpreting vast amounts of data. Brain-computer interfaces (BCIs) are examples of such technologies that facilitate direct communication between the human brain and external devices, offering promising avenues for cognitive enhancement (Lebedev & Nicolelis, 2017). Another example is augmented reality (AR), which overlays real-world percepts and virtual knobs with digital additions to enhance human’s connections to physical reality, thus enhancing our experience and handling of it (Billinghurst et al., 2015; Dargan et al., 2023). If they manage to seamlessly blend physical and virtual realities, AR systems have the power to amplify human cognitive and sensory capabilities, empowering us to navigate our problem spaces in contextually meaningful yet intuitive ways.

Humble Steps: The Abacus

Augmenting human intelligence with tools. Here, we observe the classical but not less impressive example of abacuses to assimilate advanced mathematical skills. Video Credit: Smart Kid Abacus Learning Pvt. Ltd. 2023

We just mentioned a couple of rather idealized, advanced-stage technologies to cybernetically couple humans to new rendering of the reality of problem spaces they navigate. But a new generation of cybernetics need not start at the most complex and technologically advanced level. In fact, the roots of such a technology can already be found in simple, yet powerful and transformative tools such as the abacus. That simple piece of wood, beads, and strings has done a tremendous job externalizing our memory and computation, extending our mind’s ability to process numbers and solve problems. In doing so, it has demonstrated how even the most modest-looking tool may amplify cognitive abilities in such a groundbreaking way, laying the basis for more sophisticated augmentations that may merge human intelligence with machine computation.

Not only did the abacus extend human mathematical abilities, but it did so by enactively—through an active, reciprocal interaction—stretching our existing mechanisms of perception and understanding, enhancing how we sense, interpret, interact with, and make sense of the world around us. The device doesn’t replace, but instead really amplifies our existing innate sensorial mechanisms. It brings our mathematical cognitive processes into a back-and-forth ensemble of informational exchanges between our inner understanding and the tool’s mechanics, through which we effectively—and, again, enactively—extend. Similarly, today’s cybernetic technologies can start as modest, focused augmentations—intuitive, accessible, and seamlessly integrated—building step by step toward more profound cognitive symbiosis.

Extending One’s Capabilities Through Integration of Tools

The principle of extending human capabilities through tools, as exemplified by the abacus, can be generalized in a computational framework, where integrating various systems may enhance their combined power to solve problems. Let’s take a simple example of such phenomenon. For instance, in Biehl and Witkowski (2021), we considered how the computational capacity of a specific region in elementary cellular automata (ECA) can be expanded in terms of the number of functions it can compute.

Modern cybernetics: a person seamlessly working on their laptop, using a prosthetic arm, exemplifying an extension of their own embodiment and cognitive ability. This somewhat poetically illustrates Biehl and Witkowski (2021)’s work on expansions of number of functions computed by a given agent, as the agent’s computation tool expands. Credit: Anna Shvets

Interestingly, this research led to the discovery that while coupling certain agents (regions of the ECA) with certain devices they may use (adjacent regions) typically increases the number of functions they are able to compute, some strange instances would occur as well. Sometimes, in spite of « upgrading » the devices used by agents (enlarging the adjacent regions to them), the computing abilities of the agents ended up dropping instead of increasing. This is the computational analog to people upgrading to a new phone, a more powerful laptop, or a bigger car, only to end up being behaviorally handicapped by this very upgrade.

This shows how this mechanism of functional extension, much like the abacus’s amplification of human cognition, extends nicely to a large range of situation, and yet should be treated with great care as it may affect greatly the agency of users. Now that we have established the scene, let’s dive into the specific and important tool of programming.

Programming: Towards A Symbiotic Transition

Programming. What an impactful, yet fundamental tool invented by humans. Was it even invented by humans? The process of creating and organizing instructions—which can later be followed by oneself or a distinct entity to carry out specific tasks—can be found in many instances in nature. DNA encodes genetic instructions, ant colonies use pheromones to dynamically compute and coordinate complex tasks, and even « simple » chemical reaction networks (CRNs) display many features of programming, as certain molecules act as instructions that catalyze their own production, effectively programming reactions that propagate themselves when specific conditions are met. Reservoir computing would be a recent example of this ubiquity of programming in the physical world.

Programming, a human invention, or a human discovery? This image—titled “an artist’s illustration of artificial intelligence”—was inspired by neural networks used in deep learning. Credit: Novoto Studio

Recently, many have argued that it may no longer be worthwhile to study traditional programming, as prompt engineering and natural language interactions dominate. While it’s true that the methods for working with AI are changing, the essence of programming—controlling another entity to achieve goals—remains intact. Sure, programming will look and feel different. But has everyone forgotten how far we’ve come since the advent of Ada Lovelace’s pioneering work and the earliest low-level programs? Modern software engineering typically implies working with numerous abstraction layers built atop the machine and OS levels—using high-level languages like Python, front-end frameworks such as React or Angular, and containerization tools like Docker and Kubernetes. This doesn’t have to be seen as so different from the recent changes with AI. At the end of the day, in one way or another, humans must utilize some language—or, as we’ve established, a set of languages at different levels of abstractions—to combine their own cognitive computation with the machine’s—or, other humans too, really, at the scale of society—so as to work together to produce meaningful results.

This timeless collaboration—akin to the mitochondria or chloroplast story we touched on earlier—will always involve the establishment of a two-way communication. On one hand, humans must convey goals and contexts to machines, as AI cannot achieve objectives without clear guidance. On the other, machines must send back outputs, whether as solutions, insights, or ongoing dialogue, refining and improving the achievement of goals. This bidirectional exchange is the foundation of a successful human-machine partnership. Far from signaling the end of programming, it signals its natural evolution into a symbiotic relationship where human cognition and machine computation amplify one another.

The seamless integration of prosthetics in a human’s everyday life. Credit: Cottonbro Studio

Bidirectional, Responsible Human-Machine Interaction

What I this piece is generally gesturing at, is that by integrating cybernetic technologies to seamlessly couple human cognition with increasingly advanced, generative computational tools, we can build a future where human intelligence is not replaced but expanded. It will be key for the designs to enable enactive, interactive, high-bandwidth communication that fosters mutual understanding and goal alignment. On this path, the future of programming isn’t obsolescence—it means transformation into a symbiotic relationship. By embracing collaborative, iterative human-machine interactions, we can amplify human intelligence, creativity, and problem-solving, unlocking possibilities far beyond what either can achieve alone.

Human-Machine Interaction: Who Is Accountable? This image represents ethics research understanding human involvement in data labeling. Credits: Ariel Lu

Human-machine interaction is inherently bidirectional: machines provide feedback, solutions, and insights that we interpret and integrate, while humans contribute context, objectives, and ethical considerations that guide AI behavior. This continuous dialogue enhances problem-solving by combining human creativity and contextual understanding with machine efficiency and computational power. As we navigate this evolving technological landscape, focusing on responsible AI integration will be critical. Our collaboration with machines should aim to augment human capabilities while respecting societal values, goals, and, importantly, the well-being of all life.

As Norbert Wiener put it:

“The best material model for a cat is another, or preferably the same cat. In other words, should a material model thoroughly realize its purpose, the original situation could be grasped in its entirety and a model would be unnecessary. Lewis Carroll fully expressed this notion in an episode in Sylvie and Bruno, when he showed that the only completely satisfactory map to scale of a given country was that country itself” (Wiener, 1948).

“The best material model for a cat is another, or preferably the same cat.” – Norbert Wiener. Image Credit: Antonio Lapa/ CC0 1.0

True innovation lies in complementing the complexities of life rather than merely replicating them. In order to augment human intelligence, we ought to design our new technologies—especially AI—so as to build on and harmonize with the rich tapestry of human knowledge, the richness of experience, and the physicality of natural systems, rather than attempting to replace them with superficial imitations. Our creations are only as valuable as their ability to reflect and amplify the intricate nature of life and the world itself, enabling us to pursue deeper understanding, open-ended creativity, and meaningful purpose.

The ultimate goal is clear: to be able to design a productive connective tissue for human ingenuity that would seamlessly sync up with the transformative power of machines, and a diverse set of substrates beyond today’s computing trendiest purviews. By embracing and amplifying what we already have, we may unlock new possibilities and redefine adjacent possibles that transcend the boundaries of human imagination. This approach is deeply rooted in a perspective of collaboration and mutual growth, working toward a world in which technology remains a force for empowering humanity, not replacing it.

Allow me to digress into a final thought. Earlier today, a friend shared an insightful thought about how Miyazaki’s films, such as Kiki’s Delivery Service, seem to exist outside typical Western narrative patterns – a thought itself prompted by them watching this video. While the beloved Disney films of my childhood may have offered some great morals and lessons here and there, they definitely fell short in showing what a kind world would look like. Reflecting on this, I considered how exposure to Ghibli’s compassionate, empathetic storytelling might have profoundly shaped my own learning journey as a child – although it did get to me eventually. The alternative worldview these movies offer gently nudges us toward being more charitable. By encouraging us to view all beings as inherently kind, well-intentioned, and worthy of empathy and care, perhaps this compassionate stance is exactly what we need as we strive toward more meaningful, symbiotic collaborations with technology, and certainly with other humans too.

References: