Machine Reasoning and System 2 Thinking

Speaker 1 [00:08]

So, yeah, thanks. I'll do a super quick introduction. I am a Kaka technologist I was a founder and the first CTO of a company called Cortical Labs You may have heard of it. We did like neurons play pong Does anyone did anyone see that like articles and like motherboard if you have like yeah, that was I wrote the brain computer interface So either I'm going I'm probably going to be one of the first up against the wall when the uprising comes. We'll see So things are moving so fast in this area It's like, if I had even written this talk two months ago, it would have been pointless. I would have just rewritten the whole thing. That's my excuse for not writing the talk until the day before. So this is not going to be a review. This is not going to be a tutorial. It's a rambling conceptual overview. So yeah, if you're advanced in this field, you're going to be annoyed. And if you're just starting out, you're also going to be annoyed. I'm sorry. So what is system two? System two thinking was an idea proposed by I guess like a cognitive psychologist named Daniel Kahneman From his like super super famous and in fact, I think overrated book thinking fast and slow But anyway, his concept is that we have kind of two systems system one, which is fast automatic. It's pattern matching So when we see something we can immediately say oh like if we see the numeral three We don't even think I have to read that as a three We just see it and then we've got system two, which is like fast effortful calculating So what allows us to do deliberation, it's kind of like a more slower, rational, considered way of thinking. And overall, there has been sort of excitement and energy around trying to bring that sort of system to artificial intelligence. So in general, we want artificial intelligence is probably that are smart and reason, maybe. And so for some reason, when I was like pitching this talk, I filled in the form and I said no prior experience necessary so if you have no prior experience thank you for coming to this talk I will start from the very beginning so tokens why is everyone talking about these freaking token things the tokens words or aren't they the kind of words you break up your text into these chunks these like bite-sized chunks and they're called tokens why it's just what you do it helps in some ways there's some ways to avoid it but it just kind of became what everybody does so tokens think when people say tokens you read tokens on X or something, just think words. It's the same. Okay, so all language, large language models are spicy autocomplete. That's it. You're predicting the next word from the previous words. And this seems incredibly stupid, and that's because it is. But a lot of AI is like ideas that sound stupid at first that work surprisingly well. So it turns out that if you predict the next word, and then you predict, you pretend that that was really what was said, and then you predict the next word after that. You just keep predicting the next word from the previous words, you actually get something that's kind of useful. And why is that? Well, it's because if you ask a question like, what is the capital of France? And then you try and fill it out here, it's the most likely statistical answer from just looking at documents on the Internet or books is Paris. So it turns out for doing a bunch of things, just answering questions, whatever it is, spicy autocomplete works surprisingly well. And so clearly what people are talking about. All the time today is this thing, this AGI thing. And I'm not even going to try and argue about what AGI exactly means, other than the G is for good. That's what it means. So there is, I guess, sort of a test and a benchmark created by this researcher named François Chollet called the Abstract Reasoning Corpus, along with the ARC Prize. And the idea of it is to create a benchmark which really tests something like fluid intelligence. So a lot of what previous sort of AI benchmarks looked at were a kind of memorization. They were trying to say, learn patterns from the past and apply them again. This is what a dog looks like. This is what a house looks like. Whatever it is. The whole point of AGI, of, sorry, ARC, and ARC, AGI, ARC, man. The G, it's extra good. That's how you know. That should be a C. So what you can kind of see is you really only have two training examples in this. So it's sort of like a meta task problem in the sense that each example in Arc is what would be normally thought of as like a new benchmark task in the old way of doing things. So each one is kind of a new problem with new patterns. And you often only get maybe like two examples, three examples from which you need to generalize. And it turns out humans are pretty good at this. And I think they even have the motto like, the only unbeaten benchmark that's easy for humans but hard for AI. So there's some other complicated maths benchmarks and whatnot that some of the newest models don't do well at. But this is the kind of thing where you can get people off the street, Uber drivers, whatever it is, and they do really, really well at it. So the obvious solution here is you copy the purple things, and then you fill in the corner. It's easy. You can almost see it immediately. And it turns out that this is trying to exercise a sort of fluid intelligence. And models so far have done extremely poorly at it. So you can see here the GPT 4.5 was topping out at like, I don't know, like 3%, 4%. This was Arc AGI 1. And here's Francois Chalet looking very unimpressed at the fact we spent 50,000 times more compute to get like 2% more on this benchmark, because AGI is almost here, guys. That's Francois Chalet looking very unimpressed. So then something happened, and that was this fucking 03, like 01 and 03 models. And this is really annoying because we don't exactly know how these models work. OpenAI has played these things very close to their chest. There's some speculation later in the talk about exactly what they're actually doing. But either way, there was a step change. Qualitatively, something different happened. And so a lot of this talk will be like, what happened? And I would say overall it hasn't been completely reproduced in the open source world yet But it probably will be because I don't think the formula is like super I think what they've done is really put together a bunch of stuff that worked in a really nice way kind of similar to AlphaGo But we'll sort of get into that So Francois Chalet, of course is not okay with anybody beating his benchmarks. So he created Arc AGI 2 Which O3 still gets 3% on so here's is François Chalet looking very pleased with himself. If you think he's being pleased with himself face and his annoyed face are the same, that's because he's French. Too easy. So what makes this test difficult, right? Part of it is that they've really moved from kind of something you can do almost immediately by just pattern recognition to having to put a few things together. So one thing that's really interesting is relying on symbolic interpretation. So if you try and solve this problem, you're like, oh, I've got this. I can see, you know, what's the meme with all the little, like, you know, mats appearing? I can see it in front of your face. So, like, and, you know, this is fascinating. This is relatively easy for people. But it's basically, okay, count the number of holes, and then you need to copy this, and you need to make the color match up with the number of holes in the little key on the right, you know. And so this is a thing that I think right now models seem to struggle with probably because, again, it requires creating a sort of, we're primed as people who sort of look at diagrams, read, have language, to understand these are symbols that mean something. And we can kind of see intuitively and spatially that there's something, you know, like what do these things have, what do the blue things have in common? Well, they have two holes in them, right? So that means blue means, two holes means blue. You have compositional reasoning, compositional reasoning is also kind of pretty easy for human beings but again quite hard for these models because it's not just applying one rule once, it's applying the same simple rule but many times in a chain. So are we doing audience participation? I guess we are, I mean, all right, who wants to say what the rule is here? Come on, give one for humans. No, people oh yes we've got one yes exactly exactly and in fact they make it easy for you because there's only because the spacing is like unique right so there's only one way we'll actually connect but you have to connect it that way but the point is you have to connect them like one and then once that's placed you need to connect them the second one after that so there's sort of chaining these rules which are like that's kind of easy for people once you explain the rule you go okay, do it again. I'm good. And then finally, I found this one a little bit more, what's the word, subtle. But basically, I mean, I think this is the least coherent of the two. But I think the claim here is simply that you're basically going to push the red squares as far as they will go to the right, the blue squares as far as they will to the left, and they can move over each other. But they do collide so the blue things only collide with the other blue things when they stack and the red things only collide With the other red things that's basically the idea So you kind of have to apply a rule which is move things as far as they'll go to the right till they like crash into something but with the exception that blue things and red things don't collide and that seems to trick and Sort of trip up these language models even oh three So, okay We have to cover reinforcement learning real quick, and I put one slide in reinforcement learning. So if you went and studied reinforcement learning for years and did a PhD, well, you should have just gone to this one slide. Joking, joking. But reinforcement learning from the highest point, the most zoomed-out perspective, is just when a model does something you like, you make it do it more. So you don't tell it exactly what to do, but when it happens to do something well, you say do that more. And if it happens to do something badly, you say, do that less. And there are all these acronyms. For some reason, people love just putting more letters at the end of RL. That's what they do. That's what researchers do. That's where they get the big bucks. So RLHF is reinforcement learning from human feedback. And that's what a lot of the early frontier and foundation models used. And so that basically means you show a bunch of people a bunch of answers. And they say which ones they like better. and you use that as training material to try and make the model better or answer questions more nicely, more productively, whatever it is. That's expensive and time-consuming, so it turns out you can just make AIs critique themselves. So you write down some criteria. There was work by Anthropic, which is constitutional AI, and later deliberative alignment from OpenAI. But the simple concept is you write down a list of rules, like don't encourage people to break the law, don't be rude, whatever it is, whatever corporate bullshit you're into these days. and then you say okay here's some output which one do you the AI prefer and you use that as training data so that's RLAIF and now everyone is into RLVR which is learning from verifiable rewards and verifiable rewards just means it's a question that you know the right answer to and if you may if you ask isn't that what reinforcement learning is like isn't RLVR are just regular RL. That's true. But again, are you inventing acronyms at OpenAI? No. So then we'll cover a thing called chain of thought. And we're getting somewhere, but we have to kind of put the puzzle pieces together. So chain of thought was basically, again, a lot of AI these days have been stupid tricks that work surprisingly well. So this stupid trick is you say, let's think step by step. That's all you do. In the prompt, you just ask it to be like, yeah, just think harder or something. And what will tend to happen is that it will imitate what people do in books and online and do a thing like it's somebody's maths homework or a tutorial in a book or something, and will start writing out some steps to solve the problem. And this was really exciting to people. This is the canonical paper. This was really exciting to people because it made LLMs a lot smarter, and you didn't really have to do much. It's a good thing being in AI research nowadays. So these, what is it, like, I don't know, like half a dozen tokens was all you needed to greatly increase performance, right? But what people quickly realized is that you can go further than that. And so what do people say when they mean reasoning? And this is an annoying thing, which AI and ML researchers do. They take a kind of, like, wide sense, broad concept and then kind of colonize it to mean a very specific thing. So reasoning in general, obviously, for humans, just means going through step-by-step and thinking about things and trying to make sure everything is consistent, fits together, the premises follow conclusions, all those sorts of things. Conclusions follow from premises. But basically, if you put chain of thought and reinforcement learning together, you get a really potent formula, because what do you do? So how do you do it? What you do is you get this sort of chain of thought thing, and you say, okay, think step-by-step to solve this mathematical problem, and it'll write out some stuff and then come up with an answer. And then if it gets the right answer, you say, do that more. And if it gets the wrong answer, you say, do that less. And that's how you get artificial good intelligence. Be more good. So DeepSeek came in January this year. Stuff is moving quickly. But basically, they did exactly that. They said, okay, we're going to have a prompt, and we're going to have reasoning, we're going to have an answer. When you get the right answer, we will reinforce, we will encourage you to generate the thinking tokens more generate more thinking tokens like that and what they saw was like a natural increase in the length of time that or sort of the number of tokens that appeared in this reasoning section of the prompt before the answer appeared and they also saw this thing which got people really excited was with the aha moment so the thing that was exciting to a lot of the researchers who were reading these but look to people inside deep seek and also researchers reading the paper was this sort of like it's very easy to answer more anthropomorphize but what was actually happening was you were just saying think and all that was happening is if you got the right answer then you would reinforce whatever was in those thinking tokens but but but but it started saying things like this like oh I have an aha moment this is what this is how you get the answer which is very like exciting and surprising to people and to be a bit deflationary this is kind of what you'd expect because imagine later on you have to produce an If there is some kind of important thing to pay attention to earlier in your sort of reasoning trace, they call it, then it makes sense to put an anchor on it. So if you imagine the LLM is generating a bunch more tokens near the end, then it needs to pay attention to stuff it's output before. This is a good flag to say, pay attention to this, or even in subsequent reasoning steps. So it emergently learned to have these sort of aha moments, which is really nice. So, I am going to maybe go through this a little quickly, but, yeah, there's a couple of methods which people were using to solve Arc AGI 1. So transduction is absolutely the simplest. You just take an LLM and you fully bang in, like, the question. You just have a little ASCII grid and they're like, give me the answer. So transduction is just, I don't know, why do they use big words for it? It means just put the question in. So you go and then says, hey, here's your input, give me an output. So you're just using the LLM alone in its sort of, I guess, problem-solving ability. And importantly, though, if you were doing this with a regular LLM that didn't do any reasoning, then there would be sort of no thinking tokens. So as soon as you saw this input, the next token would have to have something to do with the answer. So one of the reasons these reasoning models can do better is because they can do a bunch of stuff between getting asked the question and having to produce the answer. So you can also do induction, which is output Python programs. It's a big word, but it means output Python programs. And that turned out to work really well as well. So the prompts kind of look like this. You say, okay, you're really great and you're smart and you do Python programming and you're going to do a Python solution. And then, yeah, you say, okay, I'm going to give you a grid. And then, of course, you're going to say write a Python function transform that can convert any given input grid to its corresponding output grid. If you recall, this is the whole Arc AGI thing. You have the little grids and you have to fill in the blank. It's kind of like an IQ test. So instead of solving the problem, you write the program to solve the problem. And it turns out you can put these things together. So some of the best, kind of most winningest entries in Arc AGI 1 and in 2 are induction plus transduction. And some of the teams started off doing the simplest thing that could work, which is, you basically created the Python program, and if that didn't work very well, like you tested it on the few examples you did have, the one or three examples you had, and if it didn't work, then you would use transduction. So it's like, write a program, and if that didn't work, just throw into the LLM and see what happens. And that worked like pretty well. I mean, well enough, right? So you can see that's the sort of canonical paper on it. So a little aside, tools. People started adding tools and tool-based training into large language models. So what that basically meant was in the token stream, you could kind of generate something like calculator, blah, and then what would happen is that outside of the model, you would go and do something like look at a web page or evaluate some Python code, and then you would throw in the answer into the token stream and then keep completing. So the idea of tools was to allow a large language model to sort of reach out from the model itself and do something in the world. Coupling this with agency, Everyone's talking about agency on X. What is it? Basically, if you read it as being good at making and executing plans, that's kind of what it is. But it means often you see agentive models will come up with like step one, step two, step three. Okay, I'm going to do step one now. Okay, I'm going to do step two now. So if, but I think agency also does mean more. Like right now, that's often practically what it means. But I think people have grand ideas because the concept of agency is kind of being able to choose what you do and how you do it in a very autonomous way as opposed to like solving kind of very small problems. So, okay, to the O3 speculation. Why does O3 work so well? Is there some kind of test time search? Maybe Francois Chalet thinks there is. A lot of writers on the internet think there is. I think probably not. I think there isn't test time search, But either way, that's a thing for sort of speculation. What test time search means is, are you actually kind of like something like AlphaZero or AlphaGo? Are you actually creating a kind of tree of possibilities in your reasoning space? Instead of having one linear reasoning trace, you have a branching reasoning trace, and you have some models which select among them. Maybe, maybe not. That's a really hot area of speculation. But I would say what's really not up for speculation is that the O3 model combined four things into one in a very, very useful and powerful way. So O3 has tools, it can reach out and do things, it has agency, and then it's been trained to create plans and follow them, that it can do induction, but importantly, it can do induction within the reasoning traces. So it can say, all right, I'm going to write a program now, and I'm going to evaluate that program with my tools, and that's part of my plan, and then I'm going to look at the output, and maybe I will notice that my program needs to be fixed or maybe I think this is the right answer and I can output it. So what you get is tools, agency induction, and transduction all in the one reasoning trace, and then that's unified so you can end-to-end train it with RL. So when it gets the right answer on thousands or millions of problems, then you can make it do all these things together better. And it seems like unifying those four things really allowed you to have this sort of step change in the ability to do well on this particular arc challenge, and I think, in general, to solve problems. So here's just a reasoning trace from the chat GPT interface. As you can see, in O3, it will often do some NumPy stuff and write some Python code, but then it will reflect on the output of that Python code. It doesn't just have to produce a program, and that program runs. It can run the program, think about it, look at the output. So a couple of days before I had to present this talk, This blockbuster essay came out by David Silver and Richard Sutton, and it's called The Era of Experience. I highly recommend reading it. I will summarize it in one minute, which is simply that we're entering a new age where instead of learning from people or the sort of residual outputs that have been collected specifically, in this next era, we need these models to learn from their own experience. That means acting with agency in the world collecting data from doing that and then learning from it So essentially the next stage is models collect their own training data The era of experience as do we I guess so It's been now almost a tradition to end every Everything with a poem start with a joke end of the poem. So there we go And what rough beast it's our come round at last slouches towards Bethlehem to be born on. That's the second coming by Yates. So, thank you for attending my talk. Are we doing questions? Yeah. Wow, I'm actually on time. That's crazy. Yeah, you're actually in advance. Thank you again for this amazing talk. And we already have the first question. Do you think models are just retrieving reasoning traces, or do Do you think there is something novel? Could you say that one more time? I'm really sorry. Yeah. Do you think models are just retrieving the reasoning traces, or do you think there is something novel? Yeah. Yeah. That's a super, super good question. So how much is reasoning actually sort of being learned, or how much is this some kind of silly statistical trick? And the evidence, I think, right now is mixed. So one thing that I actually have in my slide graveyard is this S1, simple test time scaling. And it's so funny. So basically, R1, they did this whole thing with complicated reinforcement learning, and it was amazing, and they had the aha moment. And then what these researchers did was that basically they told the model to think, and if it didn't think long enough, they just said they added the token wait, like, wait! And then they made it generate more. It turns out that that works really well as well. There's actually like no tuning to actually do reasoning more. They call it like, I think, compute budget forcing or something is like the jargon term they used for it. So that's pretty funny. And so it kind of goes to the maybe models aren't learning to reason, they're just sort of kind of parrots making this reasoning trace and somehow having more reasoning trace just sort of like helps to answer the question, but not in the way you think. And this is the problem with a lot of these things. It's like half alchemy at the moment. It's like really some stuff works and sometimes it works for reasons you don't think that wouldn't expect I think I saw another paper which I don't have a slide for but it was like essentially showing that the training itself wasn't necessarily improving reasoning per se and they had some way to show that so I'll have to dig up that paper if whoever asked that question I can try and dig up that paper for you as well cool okay next question oh thank you and what is your take on on Lekun's opinion that hierarchical joint embedding predictive architectures and MPC should be favored over LLMs and reinforcement learning. I mean, my take on that is that, I mean, Jan Lekun has said the thing that he's doing right now is actually a thing you have to do to get to AGI for the last 30 years. So sometimes he's right, sometimes he's been wrong. So I mean, yeah, that's Jan Lekun. But I would say, that's what I have to say about that. Is there any questions in the auditory? Thanks a lot for the talk. It was really interesting so far. I'm not sure if I can express it right, the question. So a lot of the material that you showed and that I read is a little bit like engineering tricks, if you know what I mean. So we're taking these capabilities and we do one more trick and it starts to behave more of what we like to see. I wonder that in other sciences we have more like limitations we think about like computational things that I'm possible and I see little things of this in in all of the LM and reasoning stuff right now so are you aware of works that it go more towards that yeah I mean it's extraordinarily like good and I think broad-ranging question as well so like Wow I will try and sort of summarize I think like a lot of what is almost verging on the philosophical which is that we have these very specific hard combinatorial problems in the world and you know that could be like for example proving some mathematical formula and in principle we already have a way to prove that mathematical formula. You enumerate every possible proof and you check if it's correct and if it's not correct you go on to the next one. The problem is it just takes too long. So you have these systems like alpha proof, which are kind of doing this very interesting thing where you're saying, well, okay, we're going to have this backbone that's extremely rational and symbolic, which is this theorem prover. But each individual step that we generate in the proof is going to be generated heuristically by these models that have been trained on like thousands of previous mathematical proofs. And this seems to be an extraordinarily like productive way of creating these algorithms of kind of like having this sort of algorithmic backbone, this theoretical symbolic backbone, but with this kind of learnable heuristic guider that kind of guides the search so that you don't end up trying every possible proof, which would take forever. And I mean, that was basically the core of AlphaZero as well and AlphaGo. They had a Monte Carlo tree search, which is a very sort of standard way of searching all the tree of possibilities, but then they boosted that search by using neural networks. So they basically said, well, okay, instead of exploring all of the tree all the time or as much of the tree as we can, we'll sort of zero in on the most productive, interesting parts of the search tree using these heuristics. So, yeah, overall I would say that that has been sort of the formula for success And like a lot of what's sort of interesting about the specific development. Where's my four things thing? It was interesting that specific development is in some sense All of these are tricks except when you put them all together and train them and then it starts to look a bit more unified And the reason it starts to look a bit more unified is again your tools and your induction this symbolic backbone like a Python program has to a degree of approximation some real semantics which are specific and then your sort of agency and transduction are kind of that like heuristical learning element and if you can do them together and they can feed back into each other like the search into the game tree you can end up with something which is very good so I would say that that's the sort of general pattern I see hope that answered your question okay damn okay so okay yeah we give that one more yeah so the next question is about comparison the model that are based only on text and multimodal models how what is your take on their efficiency for solving the problems yeah so that's an extraordinarily good question is wasn't covered in this talk is clearly like in some sense Arc AGI is sort of multimodal like a lot of the problems are like pseudo-spatial in that they need a kind of like a lot of reasoning about like topology and objectness so in that sense like these are already this is already a multi a kind of multimodal problem but a very specific corner of multimodal and I would actually argue that probably one of the other reasons why O3 does well on this, you can have 0.5 here, is that it has some relatively strong spatial priors from training multimodally. I was honestly trying to find some of these early articles. The era of GPT-3, they had this really funny thing because it wasn't a multimodal model. If you ask it to describe the letter Q, it just say the most bizarre stuff. Q is a horizontal line with a vertical shaft and then a tail that extends up to the left. What? Have you ever seen a Q? Oh, no, you haven't ever seen a Q. Early GPT models just could not describe the actual shape of, say, a symbol. Whereas, by GPT-4, it didn't have that problem anymore because it was multimodal. It had some kind of relatively well-developed spatial information inside it so i think that o3 kind of 0.5 is some degree of like spatial and topological priors it's like objectness topology connectivity priors from multimodal training and then you're seeing that have effects in performance on arc so yeah i hope that answers your question slider person or at least at least added some interest to it. Unfortunately, we're running out of the time. Thank you again for your amazing talk and discussions.