Basic assumptions and framework
基本假设和框架

To make this whole essay more precise and grounded, it’s helpful to specify clearly what we mean by powerful AI (i.e. the threshold at which the 5-10 year clock starts counting), as well as laying out a framework for thinking about the effects of such AI once it’s present.
为了使整篇文章更加精确和有根据，有必要明确说明我们所说的强大的人工智能是什么意思（即 5 到 10 年时钟开始计时的阈值），并制定一个框架来思考这种人工智能出现后的影响。

What powerful AI (I dislike the term AGI)3 will look like, and when (or if) it will arrive, is a huge topic in itself. It’s one I’ve discussed publicly and could write a completely separate essay on (I probably will at some point). Obviously, many people are skeptical that powerful AI will be built soon and some are skeptical that it will ever be built at all. I think it could come as early as 2026, though there are also ways it could take much longer. But for the purposes of this essay, I’d like to put these issues aside, assume it will come reasonably soon, and focus on what happens in the 5-10 years after that. I also want to assume a definition of what such a system will look like, what its capabilities are and how it interacts, even though there is room for disagreement on this.
强大的人工智能（我不喜欢 AGI 这个词） 3 将会是什么样子，以及它何时（或是否）会到来，这本身就是一个巨大的话题。这是我公开讨论过的问题，可以就此写一篇完全独立的论文（我可能在某个时候会写）。显然，许多人对强大的人工智能能否很快建成持怀疑态度，有些人甚至怀疑它是否真的能建成。我认为它最早可能在 2026 年出现，尽管也有可能要花更长时间。但就本文而言，我想先把这些问题放在一边，假设它会很快到来，并关注此后 5-10 年会发生什么。我还想假设这样一个系统将会是什么样子、 它的功能是什么以及它如何交互的定义，尽管对此存在分歧。

By powerful AI, I have in mind an AI model—likely similar to today’s LLM’s in form, though it might be based on a different architecture, might involve several interacting models, and might be trained differently—with the following properties:
我心目中的强大人工智能是一种人工智能模型，其形式可能与今天的人工智能模型类似，尽管它可能基于不同的架构，可能涉及多个交互模型，并且可能以不同的方式进行训练，但它具有以下属性：

In terms of pure intelligence4, it is smarter than a Nobel Prize winner across most relevant fields – biology, programming, math, engineering, writing, etc. This means it can prove unsolved mathematical theorems, write extremely good novels, write difficult codebases from scratch, etc.
就纯粹的智能 4 而言，它比大多数相关领域的诺贝尔奖获得者还要聪明——生物学、编程、数学、工程学、写作等。这意味着它可以证明尚未解决的数学定理、写出极其优秀的小说、从头开始编写困难的代码库等。

In addition to just being a “smart thing you talk to”, it has all the “interfaces” available to a human working virtually, including text, audio, video, mouse and keyboard control, and internet access. It can engage in any actions, communications, or remote operations enabled by this interface, including taking actions on the internet, taking or giving directions to humans, ordering materials, directing experiments, watching videos, making videos, and so on. It does all of these tasks with, again, a skill exceeding that of the most capable humans in the world.
除了是“可以与之对话的智能设备”之外，它还拥有人类虚拟工作所需的所有“界面”，包括文本、音频、视频、鼠标和键盘控制以及互联网访问。它可以参与此界面启用的任何操作、通信或远程操作，包括在互联网上采取行动、接受或向人类发出指示、订购材料、指导实验、观看视频、制作视频等。它完成所有这些任务的技能也超过了世界上最有能力的人类。

It does not just passively answer questions; instead, it can be given tasks that take hours, days, or weeks to complete, and then goes off and does those tasks autonomously, in the way a smart employee would, asking for clarification as necessary.
它并不是被动地回答问题，相反，它可以被赋予需要几个小时、几天或几周才能完成的任务，然后像一个聪明的员工一样自主地完成这些任务，并在必要时要求澄清。

It does not have a physical embodiment (other than living on a computer screen), but it can control existing physical tools, robots, or laboratory equipment through a computer; in theory it could even design robots or equipment for itself to use.
它没有物理实体（除了存在于计算机屏幕上），但它可以通过计算机控制现有的物理工具、机器人或实验室设备；理论上它甚至可以设计机器人或设备供自己使用。

The resources used to train the model can be repurposed to run millions of instances of it (this matches projected cluster sizes by ~2027), and the model can absorb information and generate actions at roughly 10x-100x human speed5. It may however be limited by the response time of the physical world or of software it interacts with.
用于训练模型的资源可以重新用于运行数百万个实例（这与预计到 2027 年的集群大小相匹配），并且该模型可以吸收信息并以大约 10 倍至 100 倍的人类速度生成动作 5 。然而，它可能受到物理世界或与其交互的软件的响应时间的限制。

Each of these million copies can act independently on unrelated tasks, or if needed can all work together in the same way humans would collaborate, perhaps with different subpopulations fine-tuned to be especially good at particular tasks.
这百万个副本中的每一个都可以独立执行不相关的任务，或者如果需要，它们都可以像人类协作一样协同工作，也许不同的亚群经过微调，可以特别擅长执行某些特定任务。

We could summarize this as a “country of geniuses in a datacenter”.
我们可以将其概括为“数据中心的天才之国”。

Clearly such an entity would be capable of solving very difficult problems, very fast, but it is not trivial to figure out how fast. Two “extreme” positions both seem false to me. First, you might think that the world would be instantly transformed on the scale of seconds or days (“the Singularity”), as superior intelligence builds on itself and solves every possible scientific, engineering, and operational task almost immediately. The problem with this is that there are real physical and practical limits, for example around building hardware or conducting biological experiments. Even a new country of geniuses would hit up against these limits. Intelligence may be very powerful, but it isn’t magic fairy dust.
显然，这样的实体能够非常快速地解决非常困难的问题，但要弄清楚速度有多快并非易事。在我看来，两种“极端”观点都是错误的。首先，你可能会认为世界将在几秒钟或几天的时间内瞬间改变（“ 奇点 ”），因为高级智能会自我构建，几乎立即解决所有可能的科学、工程和操作任务。问题在于，存在真正的物理和实际限制，例如在制造硬件或进行生物实验方面。即使是一个天才云集的新国家也会遇到这些限制。智能可能非常强大，但它不是神奇的仙尘。

Second, and conversely, you might believe that technological progress is saturated or rate-limited by real world data or by social factors, and that better-than-human intelligence will add very little6. This seems equally implausible to me—I can think of hundreds of scientific or even social problems where a large group of really smart people would drastically speed up progress, especially if they aren’t limited to analysis and can make things happen in the real world (which our postulated country of geniuses can, including by directing or assisting teams of humans).
其次，反过来，你可能会认为技术进步已经饱和，或者受到现实世界数据或社会因素的限制，而比人类更优秀的智能对此的贡献非常小 6 。这在我看来同样难以置信——我能想到数百个科学甚至社会问题，其中一大群真正聪明的人会大大加快进步速度，特别是如果他们不局限于分析，并能在现实世界中实现目标（我们假设的天才国家可以做到这一点，包括通过指导或协助人类团队）。

I think the truth is likely to be some messy admixture of these two extreme pictures, something that varies by task and field and is very subtle in its details. I believe we need new frameworks to think about these details in a productive way.
我认为事实可能是这两种极端情况的杂糅，因任务和领域而异，细节非常微妙。我认为我们需要新的框架来以富有成效的方式思考这些细节。

Economists often talk about “factors of production”: things like labor, land, and capital. The phrase “marginal returns to labor/land/capital” captures the idea that in a given situation, a given factor may or may not be the limiting one – for example, an air force needs both planes and pilots, and hiring more pilots doesn’t help much if you’re out of planes. I believe that in the AI age, we should be talking about the marginal returns to intelligence7, and trying to figure out what the other factors are that are complementary to intelligence and that become limiting factors when intelligence is very high. We are not used to thinking in this way—to asking “how much does being smarter help with this task, and on what timescale?”—but it seems like the right way to conceptualize a world with very powerful AI.
经济学家经常谈论“生产要素”：劳动力、土地和资本等。“劳动力/土地/资本的边际收益”这一短语体现了这样一种思想，即在特定情况下，某个特定因素可能是也可能不是限制因素——例如，空军需要飞机和飞行员，如果没有飞机，雇佣更多的飞行员也无济于事。我认为，在人工智能时代，我们应该讨论智能的边际收益 7 ，并试图找出与智能互补的其他因素是什么，以及当智能非常高时哪些因素会成为限制因素。我们不习惯这样思考——问“更聪明对这项任务有多大帮助，在什么时间范围内？”——但这似乎是概念化一个拥有非常强大的人工智能的世界的正确方法。

My guess at a list of factors that limit or are complementary to intelligence includes:
我猜测限制或补充智力的因素包括：

Speed of the outside world. Intelligent agents need to operate interactively in the world in order to accomplish things and also to learn8. But the world only moves so fast. Cells and animals run at a fixed speed so experiments on them take a certain amount of time which may be irreducible. The same is true of hardware, materials science, anything involving communicating with people, and even our existing software infrastructure. Furthermore, in science many experiments are often needed in sequence, each learning from or building on the last. All of this means that the speed at which a major project—for example developing a cancer cure—can be completed may have an irreducible minimum that cannot be decreased further even as intelligence continues to increase.
外部世界的速度 。智能代理需要与外界进行交互才能完成任务并进行学习 8 。但世界变化的速度是有限的。细胞和动物以固定的速度运行，因此对它们进行实验需要一定的时间，而这段时间可能是无法缩短的。硬件、材料科学、任何涉及与人交流的事物，甚至我们现有的软件基础设施也是如此。此外，在科学研究中，许多实验通常需要按顺序进行，每个实验都从上一个实验中学习或构建。所有这些都意味着，完成一个重大项目（例如开发癌症治疗方法）的速度可能有一个不可缩短的最低限度，即使智能不断提升，这个最低限度也无法进一步降低。

Need for data. Sometimes raw data is lacking and in its absence more intelligence does not help. Today’s particle physicists are very ingenious and have developed a wide range of theories, but lack the data to choose between them because particle accelerator data is so limited. It is not clear that they would do drastically better if they were superintelligent—other than perhaps by speeding up the construction of a bigger accelerator.
需要数据 。有时缺乏原始数据，在缺乏原始数据的情况下，更多的智慧也无济于事。当今的粒子物理学家非常聪明，已经开发出各种各样的理论，但由于粒子加速器数据非常有限， 他们缺乏在各种理论之间进行选择的数据。目前尚不清楚，如果他们 是超级智能的——除非加快建造更大的加速器。

Intrinsic complexity. Some things are inherently unpredictable or chaotic and even the most powerful AI cannot predict or untangle them substantially better than a human or a computer today. For example, even incredibly powerful AI could predict only marginally further ahead in a chaotic system (such as the three-body problem) in the general case,9 as compared to today’s humans and computers.
内在复杂性 。有些事物天生就不可预测或混乱，即使是最强大的人工智能也无法比当今的人类或计算机更好地预测或解开它们。例如，与当今的人类和计算机相比，即使是非常强大的人工智能在一般情况下也只能对混沌系统（如三体问题 ）进行略微更进一步的预测 。9

Constraints from humans. Many things cannot be done without breaking laws, harming humans, or messing up society. An aligned AI would not want to do these things (and if we have an unaligned AI, we’re back to talking about risks). Many human societal structures are inefficient or even actively harmful, but are hard to change while respecting constraints like legal requirements on clinical trials, people’s willingness to change their habits, or the behavior of governments. Examples of advances that work well in a technical sense, but whose impact has been substantially reduced by regulations or misplaced fears, include nuclear power, supersonic flight, and even elevators.
来自人类的限制 。许多事情如果不触犯法律、伤害人类或扰乱社会，就无法完成。一个有秩序的人工智能不会想做这些事情（如果我们有一个没有秩序的人工智能，我们又要回到谈论风险的问题）。许多人类社会结构效率低下，甚至有害，但很难在尊重临床试验的法律要求、人们改变习惯的意愿或政府行为等约束的情况下改变。从技术角度来看，运行良好的进步的例子包括核能、 超音速飞行 ， 甚至电梯 ，但其影响力却因法规或错误的恐惧而大幅降低。

Physical laws. This is a starker version of the first point. There are certain physical laws that appear to be unbreakable. It’s not possible to travel faster than light. Pudding does not unstir. Chips can only have so many transistors per square centimeter before they become unreliable. Computation requires a certain minimum energy per bit erased, limiting the density of computation in the world.
物理定律 。这是第一点的更鲜明的版本。某些物理定律似乎是牢不可破的。速度不可能超过光速。 布丁不会动 。芯片每平方厘米只能容纳一定数量的晶体管 ，否则它们会变得不可靠 。计算需要每擦除一个比特的最低能量 ，这限制了世界上计算的密度。

There is a further distinction based on timescales. Things that are hard constraints in the short run may become more malleable to intelligence in the long run. For example, intelligence might be used to develop a new experimental paradigm that allows us to learn in vitro what used to require live animal experiments, or to build the tools needed to collect new data (e.g. the bigger particle accelerator), or to (within ethical limits) find ways around human-based constraints (e.g. helping to improve the clinical trial system, helping to create new jurisdictions where clinical trials have less bureaucracy, or improving the science itself to make human clinical trials less necessary or cheaper).
还有一种基于时间尺度的进一步区分。短期内是硬性限制的事物在长期内可能变得更容易被智能所利用。例如，智能可能被用来开发一种新的实验范式，使我们能够在体外学习过去需要活体学习的东西 动物实验，或构建收集新数据所需的工具（例如更大的粒子 加速器），或者（在道德限制内）找到绕过人为限制的方法（例如帮助 改善临床试验体系，帮助建立临床试验较少的新管辖区 官僚主义，或者改进科学本身，使人体临床试验变得不那么必要或更便宜）。

Thus, we should imagine a picture where intelligence is initially heavily bottlenecked by the other factors of production, but over time intelligence itself increasingly routes around the other factors, even if they never fully dissolve (and some things like physical laws are absolute)10. The key question is how fast it all happens and in what order.
因此，我们应该想象这样的画面：智能最初受到其他生产要素的严重限制，但随着时间的推移，智能本身越来越多地绕过其他要素，即使它们从未完全消失（有些东西，如物理定律，是绝对的） 10 。关键问题是这一切发生的速度有多快，以及以什么顺序发生。

With the above framework in mind, I’ll try to answer that question for the five areas mentioned in the introduction.
基于上述框架，我将尝试针对介绍中提到的五个领域来回答该问题。

参考
https://darioamodei.com/machines-of-loving-grace