In response to : What strange things might we find beneath the ice

Improving Autonomy might be too big of a step. Aiming for small groups being self-sufficient in technology and culture may just be unrealistic, with no way to get to there from here. I’m going to talk a bit about another possible way we can get better.

We are entering a time where we might be producing new cultures. Whether it be sea steading or trying colonize mars. So we might have a blank slate, to try again. What should we expect might work or be a good culture, what are the elements that would make it thrive?

There has also been criticism of capitalism, that it is destructive. We might want to improve on it,  but no strong culture has risen that defeats the free rider problem in general. Can we do better? If so how?

In this blog post I will outline the first sketch of an proto-idea of a culture. A culture I am currently calling being Sufficient.

A group is Sufficient if that group produces or encourages the production of everything it needs to thrive. It doesn’t free ride, it pays it’s way. The contrast is a parasitic group. A parasitic group is a group that relies on the produce of some other group or resource but puts little to nothing into the maintenance of that group or resource.

We are all probably parasitic on one thing or another (even if it is second level parasitism, which I shall get on to later). It is hard to stop.

Trying to be Sufficient

There are lots of movements that try to be Sufficient in one particular thing. An example being the people worried about climate change buying carbon offsets for their flights. They also would prefer companies that are carbon neutral and may pay more for their services.

However there are lots of different things you can free ride on. If you aren’t donating to the ACLU and are in America, are you free riding on them protecting your freedom of speech? Are you supporting the free software movement, wikipedia or the bloggers you read? Trying to create more public resources than you consume helps, but it is not a  targeted intervention, the public goods you produce might be consumed by parasites.

We would need a general movement that tries to cover every public good and common pool resource  (if I refer to public good from now on, I am also referring to common pool resources) that is important to their existence and encourages people to think mindfully about what they are consuming and making sure they are helping produce and protect them. And also help people monitor the companies they rely on, so that people don’t become second level parasites by proxy.

Second level parasitism

You pay all your taxes. You carbon offset your flights. You donate to all the causes that are important to you (free speech, gender equality, gun rights). You make sure the artists you love get money, even if they give their music away for free. You donate to the linux foundation, for your android phone. You fund science because it’s funding has been cut by your government but you don’t want to free ride on the funding from other governments. You pay all the bloggers you read, that don’t read anything of yours.

You still may be parasitical by proxy. The on-line services you use may not donate anything to the free software they rely on. The manufacturers of goods you buy may use tax havens to avoid paying taxes in a region, but rely on the law enforcement and justice system of that region to catch criminals that plague them.

Parasites can charge a lesser price, they win out in the market. You are going to get lots of them by default. They can buy up the non-parasitical services and make more profit by turning them into parasites. They have to be fought against or the public goods you care about will wither.

We need a way to make parasitism not pay. Or being Sufficient pay better.

Parasitism doesn’t need to Pay

To make it not pay, you have to pick a different provider of your good or service. To do this in a principled way that rewards the sufficient provider over the parasitic, you need to be able to estimate the relative parasitical and Sufficient behaviour of the two.

So lets say good a produced by company A costs 1 currency. Company A contributes 0.1 currency per a to support the public goods it needs (although it might consume 0.4 worth of public good per a).

So lets say good b produced by company B costs 1.2 currency. Company A contributes 0.3 currency per b to support the public goods it needs (although it might consume 0.4 worth of public good per b).

You would also want to look at the goods they consume and look at their parasitism levels, if possible.

From someone interested in being Sufficient b is better. They would only need to offset an extra 0.1 currency to avoid parasitism and be Sufficient.

There are two problems with this, verifying sufficient behaviour and calculating the consumption of public goods for the good or service you are wanting to use.

Calculating the consumption of public goods

This in itself is a public good! So you need a charity or other body to estimate this in some way. You might start with very rough estimates based on amount of free software usage, legal cases, police call outs, etc. It would be important to keep them dis-aggregated so that people could improve on them.

Someone who is trying to be Sufficient might estimate their own and publish them, a long with the things they did to offset them. Someone who is being Sufficient might also pay a premium for Sufficient companies to publish their own calculations. If AI gets better and reasoning gets cheaper then this should get cheaper too.

Verification of Sufficient behaviour

If charities posted all donations and governments posted all tax income you could look through them all and see whether people and companies were being Sufficient and nurturing the public goods.

You could do this in a privacy preserving way, by the beneficiary of the Sufficient behaviour posting a cryptographic hash of the amount, the donor’s name and a secret phrase chosen by the donor (common to all donations for a time period). Then the donor could then choose to verifiably reveal the amount they donated, by publicly posting the amount, name and the secret. Although you might have some problems with common names and people claiming others donations (you might need pki to solve the problem…). Then people interested in verifying the behaviour could hash those details and find it in the lists of hashes published by the beneficiaries.

Mutual reinforcement

Making Sufficient companies more competitive with parasitic ones fixes some of the problems. However there are also natural monopolies where you cannot pick sufficient behaviour and a few people acting Sufficiently might not be enough to make Sufficient behaviour competitive.

In order to help Sufficient people and companies you may chose to pay them above the market rate. Especially if they are paying above the market rate for the Sufficient services they use. In that way we might be able to form a mutually reinforcing circle of good behaviour.

These things are easier to form in periods of rapid change or new beginnings. But if we don’t have that we might want to start slowly.

Starting slowly

Paying for all the public services that you use straight away, might price you out of markets you care about. It might make it too hard to compete on the housing market as a person for example, leading you to have to rent. Or it might make your business go out of business as all the non-Sufficiency interested people go elsewhere.

So maybe you could agree to slowly increase the amount you pay to help Sufficient things, based on the percentage of people or companies that are acting somewhat Sufficiently in the economy in general. Quite what the growth rate should be and how you calculate it across everyone I think I will leave to another post.


Collective Aligned systems

I mainly talk about a system alignment in depth being aligned to one person. This is because I find collective alignment a lot more problematic. Problems include things like founder effects, edit wars and monopolies.

Collective alignment is in some ways easier from the “agent alignment” philosophy, you just have to find a goal that represents a fair aggregate of everyone’s many preferences. From the program alignment point of view you have more problems, because there are many ways that alignment changes can be caused and so many ways for things to go wrong or be problematic. One of the most likely forms of alignment that might go wrong in the collective scenario is the “Instruct” phase. The instruct phase is when a human sees the system is doing something wrong and instructs it to change it’s programming, to change things up to and including its morals and/or goals.

Let us construct a scenario where there is a moral disagreement between different members of the human population. For the sake of not too much argument, let us say that, “Wearing white after labour day” is not just a fashion faux pas but a moral one.

So Anti-white-after-labour-day people may seek to modify the AIs behaviour to do the following.

  • Prevent outfit suggestions that included white being made to themselves after labour day
  • Prevent outfit suggestions that included white being made to others after labour day (lest people get tempted into sin)
  • Cease to ship all white clothing/accessories after labour day.
  • Seek to prevent Pro-white-after-labour-day people being able to instruct the AI to modify the above rules.

If we have a scenario like this, where the Anti-white group was dominant to start with, or just had earlier access to the collective AGI, then you would expect founder effects as the agi would also have an impact on the culture of the people interacting with it. So the initial group who can influence the system have an out-sized impact on the behaviour of the system and of the culture that is interacting with it.

If there is a cultural injunction against modifications to stop instruction from other people, you might get into the equivalent of edit wars. Where instruction from each group continually overrides the programming of the system on whether the wearing of white is acceptable after labour day.

This is not ideal! Rapidly changing moral mores could make the world very hard to navigate for people. We may want a slower process  more akin to how laws are currently made. And we probably want people augmented with AGI so that they can have some hope of getting their preferences integrated with the laws.

The last potential problem is monopolies. Literal ones. Each program can be seen as occupying an ecological niche in the larger system. A niche that is monopolised by a good enough program will be unlikely to have new programs introduced into it and do well. In a mature system new programs will  be unlikely to be able to compete against the experienced programs with established relationships to other programs.

However if new AGIs are being made (and are associated with new people, to avoid stale thought patterns), then programs that represent entirely novel solutions to problems can gain experience and evolve in their own ways. As there is benefit to sharing programs around, the new programs might be able to spread to the older systems, once they have established themselves in a niche and proven their worth.


Different philosophies of alignment imply different difficulties for collective AGIs. It is worth examining the nuts and bolts of your preferred philosophy of alignment so that you can see the practicalities of collective AGIs in that philosophy. They may suggest further work necessary, above and beyond, work on individual AGIs or a preference for individually focused AGIs and seeking to solve co-ordination problems through non-AGI means.

For program alignment, Collective AIs might spawn large scale value conflicts as different values seek to become dominant inside the collective as the mechanics of program alignment is path dependent and somewhat zero sum.  Giving individuals AIs or having small collectives may alleviate this problem some what, but require other co-ordination mechanisms as mentioned above.

Collective AIs are not a panacea. We should think more at this time, about what we are likely to want so we can prepare properly. We should not settle too quickly on any particular solution.

New blog plans

It feels like society is not ready for AGI or general intelligence augmentation yet. So I am not going to try to explore the automatic code generation portions of my ideas, at this time.

However I think the economy by itself has value and we can increase our understanding of those sorts of systems and make use of them without getting near general intelligence.

Continue reading “New blog plans”

First Experiment

I have something which shows the economy working. It is very rough around the edges, so it could do with more polish. But it is something worth sharing.

Brief intro

I’m working on market based resource allocation (I/O control, memory, processing power) in computers because I believe that it is necessary for general intelligence in computers.

Very briefly again, current AI has been working on solving tasks, general AI will need the ability to move resources between tasks and attempt to get better at tasks over time. Attempting to get better at tasks needs resource allocation as well, because you need to stop failed attempts at getting better at a task taking up too many resources.

And getting this allocation right is needed for safe development of more general AI and intelligence augmentation. I’m demoing this because I think we need more people working on safety of the right things and thinking about how we want to use more general machine intelligence.

Continue reading “First Experiment”

Building a reliable feedback mechanism

Lets say there is a simple device that you can press or manipulate to give the Intelligence Augmentation system feedback. I wouldn’t want just anyone able to give my system feedback, I’d want to lock it down to just me. As giving feedback only leads to change in the long term, it is most important to notice when it has been compromised so that it can be fixed.

An obvious solution to this problem is something like bluetooth pairing, where you have to have physical access to be able to set up an encrypted wireless link. However if this is the only layer it is susceptible to physical tampering and might go unnoticed. One thing you can do to combat this problem is to pass on the encrypted payloads of the feedback, and the public key used to decrypt the payload to the programs inside the IA system. The programs inside the system can then notice when the private key changes (or feedback comes without an encrypted payload) and raise the alarm in some way. This would be harder to subvert as the programming inside the system is not fixed or at a known location, compared to a section of hardware.

This behavior would need to be tested periodically to make sure that it can survive within the system. You would also want to test similar behavior to protect against the device being stolen. So that if the system thought it had been stolen it would refuse to co-operate.

These mitigations are only short term, depending upon the exact nature of the programs in the system it might be possible to avoid the feedback changing it quickly. If the system is carrying very important information, you probably want to think about some form of self-destruct mechanism that the system can trigger.


Convincing the world to build something

I didn’t think I would write this post. It came out of trying to plan what to do next. Probably needs more thought.

If you want to convince the world to build something, you can go about it a number of ways. Or at least there is an ordering of who you try and convince when.

  1. Yourself: Do you really want to build it? Is the current idea about the thing the right idea? You may or may not convince yourself totally before moving on to other people. As you will probably learn a lot while you try and convince other people.
  2. The gatekeepers: If there is a regulatory body you need to convince them, from the get go, that they should not interfere with what you are trying to do. It is worth spending a fair bit of time with them as you need to understand them. But you needn’t convince them fully, just enough that you know your position with them and have an idea of how to communicate with them and what might be expected from you. It is good if they have some warning that something is coming down the pipe so they can get used to the idea.
  3. The builders: To convince the builders to work on something for a bit your need two things. A stable stress free environment and an interesting demo. Heading off the gatekeepers is needed for the first condition. If you want builders to put lots of effort into something, the thing they are building must coincide with their core values. Money helps, but throwing money at the problem won’t get you the right team. You want to get people who care about the thing not the money.
  4. The organizers: Highly valuable, but you need some building process for them to organize. They are also useful on the boundary between the world and the process of building.  Builders mistrust organizers a bit, so you don’t want to be organizer heavy.
  5. The act of a few people building a thing will attract thinkers and money people, looking for insight and profit respectively. Make what use of them that you can to keep on the right track.
  6. Ordinary People: Do what you can to convert them into builders, so that they can own the thing being built as well.

I am thinking about 2 and 3 currently (at least for the architecture).

This is a simplified process, I/we need to convince people to work on N things to improve autonomy. Where thing 0 is the architecture and thing 1 is the intelligence augmentation itself. For intelligence augmentation I am further behind than the architecture, I am on the steps of convincing myself and thinking about regulatory forces (1 and 2).

Plan for this week

I hope to do some of the following:

  • Write something about my view on secrecy
  • Write a more formal paper titled, “Architecture for Intelligence” or something a long those lines. It would argue that we need to bake resource allocation in at the computer architecture level for intelligence (be it AGI or general intelligence augmentation – GIA). Time to bust out the old latex TM skills.
  • Work more on the the test of the architecture. The first test is two agents, one better than the other. They compete for control of the system, which wins out?  I might write this up as part of the above paper.
  • Try and get an idea about how much it would cost to get someone to analyse the research on autonomy and happiness and see how reliable it is.
  • Put up an old zine that I made to explain the economic system on this blog
  • Do a followup to the zine on the AI alignment ideas I’ve been talking about recently
  • Write something on my approach to designing things using information flows.

I think the “Architecture for Intelligence” paper and testing the architecture are the most important, if I am to get other people thinking a long the same lines as me. The others will get done as my time and inclination allow.

Safe-coding practices inside Agorint

There are some interesting aspects of agorint that I had not thought about much before with regards to remote code injection.

Avoiding persistent code injection.

One of the ideas of agorint is to avoid malicious code by allowing you to get rid bad programs (by bad programs getting out competed by good programs). But stopping bad programs getting a foothold into the system seems important too. There a number of things I can think of for you to do (or the actors inside the system, when creating programs themselves), to avoid persistent malicious code injection.

Persistent malicious code injection is where a program manages to inject code into actor that gets stuck in the code. While code is immutable between auction periods, there is a copy of the code that can be mutated. That copy then becomes the new code after an auction. As the new code is write-able with the SetCodePoint instruction it is possible for the actor to be persistently exploited. However there are things you can do to mitigate this,

One aspect of the system is that code is not writeable in the moment and that code is also not craftable. If you don’t have a specific instructions within an EndPoints code then you can be pretty sure that that section of code won’t alter the code of the future. So those instructions are.


To avoid your actor being exploitable you should avoid having SetCodePoint in an externally addressable end point. The code that alters end points should be done in something that is called by a timer or some other internal system. It should also carefully validate its inputs.

This isn’t sufficient by itself you also need to avoid having the next code point in your externally addressable code.


This operation should also be avoided in externally adressable end points. It allows programs to pop code off the stack and execute them. In this way they could sneak in SetCodePoints and then apply them.

If you avoid these two you are safer. However there is a class of persistent threat that is harder to combat, but also hard to pull off (I think). It is possible to do Return Oriented programming in the architecture. This is where you craft a stack and use that to execute the existing code in an order that the programmer didn’t intend. In agorint it is harder than in x86, as the call_stack is a logically different stack to the one a caller to a program has access to. But it is theoretically possible. To make the Return Oriented programming persistent you would have to put the exploit into the EndPointMem (this memory persists between function calls and currently between auctions).

Code layout randomization might help combat this (you could have multiple functionally different copies of the code with different layouts or change layout frequently).

There are undoubtedly many other ways of exploiting things. I look forward to fuzzing the system thoroughly and seeing what I find.