transcript

Speaker 1:
[00:07] There is so much AI hype out there, from the Doomers and the Boomers. People say, AI is the future. Other people say, no, no, it's the beginning of the end of humanity. But no, it's gonna make us all rich, or it's gonna enslave us, or it's gonna cure cancer, or it's gonna ruin the environment. But everyone agrees that we're at some kind of really fascinating inflection point, where the future is very hard to predict because the present is changing so quickly. So what does that mean for us fellow extraordinaries? Let's zoom in on a more specific claim of the AI hypesters. Will AI solve physics? What does it mean to solve physics? What is AI capable of doing? And is that a good match for cracking the mysteries of the universe? Will future humans live in a universe that is totally understood to them? Will the mysteries of the universe fade into the past? Our descendants living without the sense of wonder about how it all works? Or will humans always be curious no matter what our AI scientists reveal about the nature of reality? Buckle up because while your future may bring surprises, Daniel's prediction for the future of AI and physics may also raise your eyebrows. Welcome to Daniel and Kelly's Extraordinary Universe. Was the audience filled with people or AI?

Speaker 2:
[01:59] Oh, no, they were filled with people, and I did warn them that they were parasite-related jokes that had been generated by AI, and we all agreed that I could stay president for a few more years, because I was still funnier than the AI.

Speaker 1:
[02:12] Hi, I'm Daniel. I study particles and aliens, and I've never relied on AI for any jokes. I think AI would be embarrassed to take credit for any of my dad jokes.

Speaker 2:
[02:24] I think the joke was supposed to be that AI's jokes were so bad that I am funnier.

Speaker 1:
[02:30] It's a meta-joke.

Speaker 2:
[02:31] Yeah, and in this case, they were so bad that the tapeworm-related jokes just didn't make any sense at all, and so we all had a, that's what they think tapeworms do. Very inside joke sort of thing.

Speaker 1:
[02:43] But maybe that's a fascinating threshold for AI. Not the Turing test, like can you sound like a human, but can you actually be funny?

Speaker 2:
[02:51] Yeah. I mean, that was sort of my point was like, could this replace a president's speech or opening remarks? And we decided no. Not yet. Not yet.

Speaker 1:
[03:03] Work on that nerds.

Speaker 2:
[03:04] That's right. That's right. So my question, I have two questions for you this morning, since we're talking about AI. And my first question is, when in your life is AI most useful? And then what scares you about AI, if anything?

Speaker 1:
[03:19] AI is super useful to me every single day and was before the rise of chat bots. As you'll hear today, AI has taken over every aspect of particle physics. And I've been one of the people pushing that. So it's AI all day for me. The thing that scares me about AI is seeing my daughter ask AI things she already knows or things she could just Google. You know, like you don't need to ask ChatGPT what the temperature is. You can just look it up or you could type it into Google if you're too lazy for that. You don't need to spin up a GPU for that. But people have just become like used to putting everything to AI. All thought processes can just be shunted out to AI.

Speaker 2:
[04:01] And what scares you about that? Is that it's more energy expensive or just that you feel like we're relying too much on AI for everything? Because like it's probably going to tell you the right temperature.

Speaker 1:
[04:10] In my age, who knows? Yeah, it's all of those things. It's unnecessary and it's kind of lazy mentally, right? And so the thing that scares me is that it leads to people not thinking too much for themselves and just relying on AI for everything. Exactly. It seems like a slippery slope.

Speaker 2:
[04:27] Got it. You know, I think when books came around, there were people arguing that books are going to make people lazy because they're not going to have to remember stuff anymore. And so I guess what I'm saying is, do you think that this is going to be our generation's like, the kids are getting lazy and they're going to look back and be like, is it actually different, I guess? Or do you think, are we being stodgy?

Speaker 1:
[04:49] I know. I think this is a transformational moment and I think it's going to change what it means to be human the way books do, right? Because now you can no longer just tell your friends' stuff, you can also communicate your thoughts to people who live a thousand years later. That's amazing. Or to millions of people all at once. So that changes what it means to be human. It makes it easier in some ways. It allows for laziness, but also creates transformational new abilities. And I think we got to lean into that.

Speaker 2:
[05:16] Yeah. All right. Well, let's go ahead and jump into the meat of today's discussion. And today we're talking about will AI solve physics?

Speaker 1:
[05:25] Ooh, even just the phrasing there makes me shudder.

Speaker 2:
[05:29] I mean, that's a pretty big claim. Like, you know, I think the first thing we need to talk about after we hear what the extraordinaries have to say is, what the heck do we even mean by solve physics? Because that's a pretty big statement.

Speaker 1:
[05:41] And I choose that because I was on a panel at a recent huge AI conference next to a guy from Anthropic who was all in on the fact that AI was going to, quote unquote, solve physics. And so this is not just some phrasing I made up, right?

Speaker 2:
[05:56] Okay, yeah, that is a big claim. And so let's see if the extraordinaries think that AI is going to solve physics.

Speaker 1:
[06:03] Here's what folks had to say. If you would like to join this group of exemplary guessers, please write to us to questions at danielandkelly.org.

Speaker 3:
[06:12] To me, it's more of a question of should not will. And as I don't trust AI to handle any human safety concerns, I would say absolutely not.

Speaker 4:
[06:25] If AI is chopped up to be intelligent or super intelligent, then I believe we could solve any physics riddles or questions.

Speaker 5:
[06:34] Don't think they're able to produce new thoughts only predictive of what already exists. So no, I don't think they will solve physics.

Speaker 6:
[06:42] That's already been done.

Speaker 3:
[06:43] That's old news.

Speaker 6:
[06:44] And the answer was 42.

Speaker 7:
[06:46] I think that as long as scientists treat AI as a tool rather than as an oracle, it could be very helpful.

Speaker 4:
[06:52] Yeah.

Speaker 8:
[06:53] I think the major breakthroughs will come from experimentation made by real humans.

Speaker 9:
[07:00] It's, in my opinion, an extremely broad question that needs more specificity.

Speaker 10:
[07:04] I think the idea that NLNs might solve physics is akin to the idea that if you had an infinite number of monkeys typing for an infinite amount of time, you would eventually create the works of Shakespeare.

Speaker 5:
[07:16] I think we need to define what solve physics means first.

Speaker 11:
[07:19] AI might solve physics, but I think it's more likely that it will lie and claim that it has.

Speaker 6:
[07:24] Yes, someday. It's going to be able to see relationships where humans cannot, and it's going to come up with some answers that were probably right there in front of us for a long time, but we just couldn't see it.

Speaker 3:
[07:36] I cannot believe that you're asking this question.

Speaker 9:
[07:39] It will definitely help to solve physics problems, but who will take credit for that?

Speaker 12:
[07:45] I think this requires an extreme scale of creativity, so it will definitely assist, but not solve it alone.

Speaker 7:
[07:54] I think it's plausible that AI might be able to solve physics.

Speaker 13:
[07:57] It's a good plagiarization machine, but I've never heard of current generation AI coming up with anything original. All right.

Speaker 2:
[08:06] Quite a range of thoughts here, although I feel like I'm seeing quite a bit of skepticism or a bit of concern with AI.

Speaker 1:
[08:16] Yeah. You hear the whole spectrum here of people just calling it a lazy plagiarism machine, to people think absolutely it's going to solve everything or it's at least plausible that it could, to people saying like, what are you even talking about? That question means nothing. So great job Extraordinary is giving us the context to dive into this really complicated question.

Speaker 2:
[08:37] So when you were sitting on that panel next to the guy from Anthropic, was it Sam Altman? Is he at Anthropic or is he somewhere else?

Speaker 1:
[08:45] He's OpenAI. I know he wasn't the head of Anthropic either.

Speaker 8:
[08:49] Okay.

Speaker 2:
[08:49] All right. What did they mean by solve physics? Did they mean like there will be no physics questions left to answer when AI is done? What did they mean?

Speaker 1:
[08:59] Yeah, I don't even know. That was my pushback. I said that this doesn't even mean anything. It's nonsense to imagine that there's a moment where we have no more questions about the nature of the universe. Like there's no AI answer that's going to make me say, okay, we're done. I'm retiring. My curiosity is gone. It just can't even imagine that scenario. He didn't have a great answer. He just gestured towards the rapid increase in AI powers to suggest that it was going to dot, dot, dot to some sort of Einsteinian level of intelligence. But we can dig into that a little bit more when we get to that. I thought we could start by talking about the role of AI in physics today, because also a lot of listeners write in and ask me like, hey, Daniel, how do you use AI in your research? So let's start a little bit more grounded. Before chat bots took over everything, how we used AI and more specifically machine learning in physics.

Speaker 2:
[09:55] Okay, let's do it. But first, can we define what is machine learning?

Speaker 1:
[09:59] Yes. You have all of these names floating around, it's not clear exactly what means what. The biggest topic is AI, right? That includes all of these things. Within AI, you have machine learning. Machine learning is a kind of AI, but it's mostly applied statistics. It's like pattern recognition. Think about algorithms optimized for one specific task, right? Like for example, you want to tell a cat from a dog in an internet video, or you want to know if a particle is going to go left or right based on some data that you have, right? It's solving a very specific statistical task. And machine learning is helpful when that task is too hard to do with pencil and paper. Like you couldn't write down an equation that tells you, is this a cat or a dog in an internet video? But you could learn to tell the difference if you see a bunch of examples. And so that's what machine learning is, like finding patterns iteratively through being exposed to a bunch of examples, rather than like knowing how to write an equation for the pattern on paper.

Speaker 2:
[10:59] Got it. Okay. And I feel like I remember lots of funny stories about how machine learning could be tricked. Like if the cat happened to have a tennis ball in its mouth or something, then it would be called a dog because dogs always had tennis balls in their mouths and stuff like that. Like the machine learning picks up on things that maybe you wouldn't have guessed that it would have picked up on.

Speaker 1:
[11:18] And you always want to make sure that what it's learning is meaningful to you. Like if you're training machine learning to tell wolves from dogs, but all the wolves have snow in the background of the picture, the machine learning is just going to be like, oh yeah, if it has snow in the background, it's a wolf and if it doesn't, it's a dog. But that's not what you're interested in, right?

Speaker 2:
[11:36] Yeah. Okay.

Speaker 1:
[11:36] And so you have to make sure you know what it's doing. And so this is a very narrow kind of artificial intelligence, not general intelligence at all. You can't ask this kind of thing, what's the weather today? Or how should I write this email back to my brother-in-law or anything like that, right? It can do what you tell it to do. And we use this all the time in particle physics because all of our discoveries these days are statistical in nature. You know, in the same way you can like look at a picture and say like 75% chance this is a dog, 25% chance it's a cat. That's what all of particle physics is. And it makes me a little sad because it didn't used to be. Like in the glory days of particle physics back when they were discovering the obvious particles. And by obvious particles, I mean, things that were high rate, there was like a lot of examples of them and low background, meaning it was very hard to imagine anything else making that kind of signature, like the positron discovery. This guy won a Nobel Prize for discovering the positron off of one example. He's like a picture of a track of a positron and it can't be anything else and boom, Nobel Prize. That's an amazing piece of data right there.

Speaker 2:
[12:46] It's a shame we weren't born a few decades earlier. It would have been easier then.

Speaker 1:
[12:51] And because most of those are all used up, these days particle physics is focused on more rare particles, particles that come out of collisions once a trillion times, once a quadrillion times. Otherwise, they would have been discovered by previous generations. And the backgrounds are higher. It's easier to mimic the signatures of the particles we're looking for, because we never see the particle directly. Like when we found the Higgs boson 15 years ago, we didn't say like, here's one Higgs boson, give me a Nobel Prize, because the Higgs boson doesn't last long enough. It blows up into other particles, and we look at the patterns among those particles, which are not Higgs bosons. And we say, this is likely to be a Higgs boson. Statistically, this is more likely to be a Higgs than not. But there are other ways to make that same pattern, and so you can never know for sure. You can never point to one event and say, this is my Higgs boson. All you can do is say, look, there's a pile of these events, all very, very similar, and so very likely the Higgs boson exists. And because all of our discoveries are statistical, machine learning is very, very powerful. This is exactly what it's good at. If you want to be able to tell collisions that lead to Higgs bosons from collisions that don't, and they're very similar in your detectors, you can give machine learning a bunch of examples and say, look, here's a bunch of Higgs bosons, here's a bunch of not Higgs bosons. Now, I'm going to give you data from the real collider. Tell me, what's the chance that each one is a Higgs boson or not? If you can't just write down an equation to tell you the probability of being a Higgs boson, and nobody knows how to do that because it's very complicated, then this can learn exactly how to do that calculation for you.

Speaker 2:
[14:30] Is machine learning in the end able to give you a statistical output like I'm 98 percent sure, or does it just say, yeah, I think I found it?

Speaker 1:
[14:41] No, it can do exactly the kind of statistics you're talking about. It calculates likelihoods and does likelihood ratios for statistic nerds out there. It's learning to approximate the likelihood ratio, which if you knew how to write it down as a formula, you would write down and it can approximate that very well. It's really incredible.

Speaker 2:
[15:00] Yeah, that is amazing. It's allowing you to do things that we wouldn't be able to do on our own, because the math would just be like way too complicated.

Speaker 1:
[15:08] That's right. The math is too complicated. And so for years, we did the math approximately. We said, well, looks as complicated, but I can write down a simpler version of it, which kind of works. And we were giving away information. We were not using all of the power of our data, but machine learning just slurps in all that data and says, here's the best way to discriminate between these two things. It lets us tell cats from dogs really, really well, which helps us find rare cats in our data much more effectively. And since our data is so expensive and so powerful, this is a huge boon to particle physics.

Speaker 2:
[15:42] It would be a shame to miss any particle cats.

Speaker 1:
[15:45] Exactly. And so the lesson here is that it's very powerful, but it's applied statistics to very targeted tasks. And it's not just limited to labeling collisions and saying, this one's Higgs, this one's not Higgs, or this one's dark matter, this one's not dark matter. We also use generative AI. Generative AI is the kind that can create new examples. You type in to ChatGPT like, give me a picture of a kitten riding a horse on the moon, or whatever, and it makes one for you, right? And we use the same kind of technology to very rapidly generate simulated examples. Something machine learning needs are lots and lots of examples to learn from. And so you have to generate examples. Here's what a Higgs boson would look like. Here's what a not Higgs boson would look like. You would need millions or billions of those examples for it to really learn what you need to do. But generating those examples can be very, very slow. Like if you generate a collision and you have a Higgs boson that leads to two quarks, those quarks turn into a shower of particles. To simulate what that's going to look like in our detector, you have to simulate what all those particles do when they hit your detector. What happens when a muon hits a block of copper? It creates a bunch of photons. It ionizes atoms. Lots of complicated things happen.

Speaker 2:
[16:59] Euheplurcus californiensis. Sorry, you just said a lot of physics stuff, so I get to say a species name.

Speaker 1:
[17:05] Oh, you're right. Nice.

Speaker 2:
[17:07] Sorry, keep going.

Speaker 1:
[17:09] Latin words for the win. What happens when particles smash into matter? You get all sorts of complicated stuff, which is hard to track. Then you have to track those particles, which creates more particles, which creates more particles whose Latin names I will not use. It becomes very computationally expensive. CERN has one of the biggest data centers in the world, and a huge fraction of those computers spend all of their time just simulating particle collisions and interactions because we need all of that. But GENAI can do this much faster. You give a bunch of examples to GENAI and then say, okay, now make me a new one. It's perfectly akin to saying, give me a cat on a horse on the moon. You can say, give me a Higgs boson with this mass at this angle, and you can interpolate between all of its examples and flash give you an example without burning all of those CPUs. It's much, much more efficient.

Speaker 2:
[17:57] Okay.

Speaker 1:
[17:57] So we're not just using AI to separate particles, we're also using it to train it how to separate particles all over the place in experimental particle physics. AI has touched every single part of our work because almost everything we do is statistical and computational.

Speaker 2:
[18:12] Okay, so it sounds like AI is very helpful if you are an experimentalist and you're like tracking particles. But what if you are like a pen to paper theoretical physicist? Is AI still helpful?

Speaker 1:
[18:24] Yeah, great question because when we talk about solving physics, we usually mean like figuring out the theory of the universe. And there's a lot of talk online in the podosphere about like the crisis in physics, which is mostly BS, frankly. But it's about theory, right? People want to see progress in understanding the universe. And theory has not embraced machine learning and AI nearly as much as experimental physics yet. And I think that's because mostly those folks are pencil and paper people. They're not as computational. The computational people end up in the experimental side mostly. And so the theory people don't see their work as having like data. But I see their work as having data. They are exploring huge parameter spaces of complicated particle theories. That's data. That's what machine learning is good at. So I've actually done a few projects in particle theory using tools from experimental particle physics AI to solve problems in theoretical physics, which are super duper fun. And I think it's a whole flourishing area there. So it's just begun. I think it's a little bit behind the experimental side. But in the end, all of these things are places where humans are asking questions, right? The AI is not doing physics here. It's not solving physics. It's a tool. It's an assistant at best. It's helping humans answer questions about the universe.

Speaker 2:
[19:43] All right. So we've talked about machine learning, MLs. Next, we're going to talk about LLMs. I like that we're focusing on a small subset of letters. When we get back, we'll find out what that extra L means. All right, we are back, and now instead of talking about machine learning, we're gonna talk about large language models and how these help physicists. So Daniel, how are you all using LLMs?

Speaker 1:
[20:29] So mostly the conversation about AI solving physics is inspired by the rise of these chatbots, Claude, ChatGBT, Gemini, all these things, and their supposed feeling of broader intelligence, right? And everything we talked about in the last segment was machine learning. Statistical learning applied to very specific tasks, not the kind of thing that's going to tell you like, what's the recipe to make a certain kind of fancy sandwich or whatever, the things that people use LLMs for. LLMs are a very different breed of artificial intelligence. It's built on natural language instead of just like numerical inputs. It outputs with a human-like language, right? It feels much more natural than mathematical. And these things came on the scene a few years ago. And really, that's what people are interested in. Are LLMs going to change the way we do science? Are they going to crack problems that human physicists have not been able to crack? But I want to underline that these things are still mathematical. An LLM is not something different from any other kind of AI underneath a hood. It's the same pieces of technology just arranged in a different way to solve a different problem and then just scaled up tremendously, like hugely, just like enormous numbers of these nodes and neurons. I think it's worth maybe taking a minute to explain what is going on underneath the hood of an LLM. A lot of people understand that LLMs are statistically predicting text. Like if they are writing an answer to you, they are trying to generate for you what you want and they have read a bunch of text. And so if they're trying to predict like the next word in the sequence of the answer for you, they look at all the texts they've read and they say, well, statistically, what is the word that usually follows the current word? So I'm writing a sentence, the sentence is, Kelly is a blank. And then I'm trying to generate the next word, it's some adjective. And so I'm looking through all of my training, right? All the text on the internet that describes Kelly and I'm trying to statistically predict what is the appropriate word to fill in the blank. And I'm not going to give you that word, other than a wonderful co-host and fantastic scientist.

Speaker 2:
[22:40] Thanks, thanks. You only had, but you said it was going to be one word, so I was waiting for it. But I'll let you off the hot seat. I thought you really backed yourself into a corner there. All right.

Speaker 1:
[22:55] And that's a fair sort of surface level understanding of LLMs and what they're doing. But it really misses the transformational technology that's underneath them. Because LLMs are doing more than just like predicting exactly the next word. They're using something called attention, which is really crucial to understand why these things are so much better than just like statistical parrots. And attention helps you understand which words in a sentence are contributing to the context and contributing meaning to the sentence. So say, for example, you have a sentence like, the river bank was steep. Now bank is a complicated English word because it has lots of meanings. Right? Is it where you put your money? Is it how you shoot a basketball off the backboard? Is it what keeps a river in place? Right? In this sentence, the words river and steep are important clues to tell you what bank means. Right? So when you hear that sentence, you're like, Daniel's not talking about depositing money or shooting three pointers. Right? He's talking about a river. And what the attention mechanism does inside these LLMs is let the chatbot look at all of the other words, not just the one preceding word. And so it knows what to pay attention to. It helps it understand the context. It's like if you go to a science conference, you need to understand some crucial thing for your research. You don't just talk to the person next to you. You seek out the most relevant person to pay attention to, right? The person who's going to help you understand the question you're asking. And so that's what attention is. And then they couple the tension with this technology called transformers, which are not little plastic toys that you used to play with or cartoons, or even the things on wires out there that help transform the voltage. They're just like stacked layers of these attention modules. So the first layer helps you like understand the words, and the second layer helps you understand the rest of the paragraph. And by the time you have multiple stacked layers of attention, you can understand, okay, here's the context of the question that's being asked, and therefore, here's the answer that I should give. And so it's this combination of attention and transformers that make LLMs so powerful.

Speaker 8:
[25:02] Okay.

Speaker 2:
[25:03] I didn't know any of that.

Speaker 1:
[25:04] All right. So then what are LLMs useful for, right? I'm not really interested in using them to write essays about 13th century Europe or whatever. Today, we're talking about using them in physics. So I use LLMs all the time. And here's an example I find them very useful for. I need to learn some new software package, right? Because I've started a new research project, and there's a tool that simulates the thing I need, for example. And I download the tool. It's written terribly. The documentation is garbage. There's like hundreds of pages of tables of technical stuff that's impenetrable. I try to run the code, it doesn't work the first time, right? I look at the code, it's written by physicists without any adult supervision, right? And so in past days, this would be a huge project, make this thing work, validate that it's doing what I think it should be doing, understand the documentation, how it's wrong, how the thing actually works. And in practice, what I would do is like find somebody else who's made it work, and start from their example. Somebody else has invested weeks and weeks to making this thing work. I'm going to download their scripts, I'm going to start from their code, I'm going to ask them questions when it breaks. I'm going to lean on some kind of expertise. Otherwise, I'm going to have to sink weeks into getting this thing to work. I no longer have to do that. Now I can ask an LLM, hey, read this document and write me an example, configuration script to make this thing work. Okay, it didn't work, debug the issue. And within a few cycles, I can usually get a new software package up and running and producing useful validated results in hours, if not minutes, where it used to be either weeks or totally impenetrable.

Speaker 2:
[26:40] It is pretty amazing. Whenever I use a new package in R or something, and I inevitably put a comma in the wrong place or something, I used to look for that comma for hours. And now I'm like, ChatGPT, where did I put the comma that doesn't belong? And it finds it immediately, saves me hours.

Speaker 1:
[26:58] And it's not magic, right? Let's remember, what are these LLMs doing? They understand context. They know how to use attention to focus the question you're asking to produce the right answer. And they're very good at digesting huge volumes of input, right? The things that humans are bad at. For the same reason that we're using machine learning to improve on our data analysis and the particle physics colliders, because they're good at the things we're bad at, like understanding all the nuances of some high dimensional statistics problem. Here, we're using them to do something that we are bad at, and that is the key, right? Lean on their expertise, find the thing that they can do that we can't, and then they can contribute. And they can go beyond just like reading documentation for software packages, they can also read physics papers. There are so many papers put out every single day. If you look at the archive, which is where particle physicists put out their papers every day, there's a dozen papers every day, all of which are pretty good and worth reading. I can't keep up with all of those. In addition, I should be reading all of the machine learning papers, I should be reading statistics papers, I should be reading math papers. Some math nerd out there could have just invented some incredible new thing that they're into just because they're a math nerd, but it might solve exactly the problem we have in particle physics, and they don't know and we don't know and who knows how to put these things together. The reason cross-disciplinary research is often so fruitful and leads to breakthroughs and advances is exactly because of this, that people are unaware of solutions in adjacent fields, and when you just smush them together, you get peanut butter and chocolate, right? It's amazing.

Speaker 2:
[28:32] But you've got to be careful because you might get information in your head that doesn't actually exist from doing that. I asked an LLM to summarize my research, and it told me that I had done stuff that I had not done and that none of my collaborators have done in the system. And if I had just asked for the summary, I would have been like, oh, we know a lot of stuff about the system that we don't really know. And so, I guess anything that you decide could be vaguely interesting, you need to actually go back and confirm, which is frustrating because you might be like, oh, I didn't know that. I'm not going to use it, but I didn't know that. And you might not necessarily go back and check, but you just will think like, oh, I'll just keep that in the back of my mind. But now you've got a little lie in the back of your mind.

Speaker 1:
[29:13] Well, I think of it like a super powered Google search. I cannot go into Google and say, here's my research, has there been any recent papers that might be relevant? But I can type that into my LLM and you can find a bunch of papers, and then I can read those instead of reading all of the papers. I can decide, oh, this is not actually relevant. This is a crucial detail here you missed or hey, that paper doesn't exist, so it's not useful. But it's doing the part that is hard for me, which is just like reading all of those papers. It might miss something, but it's very, very powerful to tackling the problem that I can't solve myself, which is reading all of those papers. Of course, there's highly varying quality between the models. Some of them are good at this, some of them are bad at this. And this is the kind of thing that's improving very rapidly with time. It used to be you typed this into an LLM and you got lots of nonsense. These days, I find it's producing higher and higher quality output. Insights into papers that I didn't know about, really useful stuff. And you see also in mathematics in the area, I'm not an expert in, that it's finding solutions to outstanding problems that already exist in the literature and nobody knew. They're like, oh, this 1957 paper by some Russian, nobody translated it, so we didn't know this open problem has actually been solved.

Speaker 2:
[30:24] And I think it's solving air dish problems that hadn't been solved yet.

Speaker 1:
[30:28] Exactly. Sometimes by finding techniques in other papers that are relevant already, not inventing those techniques itself, but making those connections because that's what it's good at, scanning through vast quantities of literature for things that are relevant. Yeah. Okay. So, so far it's been working as an assistant, right? It reads the documentation, it gets the software to work or it goes to find papers. It's like a research tool. What about like more as a role of a physicist? Can, for example, you give it a bunch of ideas and say, hey, go write a paper about this or write me a grant proposal on this topic, right? So, I've tried this kind of thing. I have access to the top-end models from Anthrobic, for example, and it can do it, right? It can start from rough ideas. It can flesh it out into full text. It's grammatically perfect. It makes no spelling mistakes. But in my experience, its writing is not very good. I'm a big fan of good writing. When I read a paper, there's two dimensions there. There's is the science good, but also is the writing good? If the writing is good, it leads you on a story. It gets you interested. It tells you why this research is compelling. It presents the things logically and crisply. The text is concise and precise. A well-written scientific paper is a piece of beauty. There's also that sloppily written good science. Sometimes the science is great and the paper is just sloppy. You're like, what do you mean by this exactly? Well, you're telling me this, but now we're talking about this. Why is that? It's just not well put together. Claude doesn't produce sloppy stuff, but it produces boring stuff. I find the text that it produces is missing that judge that you get from a really good piece of writing. This is the same, I think, kind of skill that it takes to write a good novel. I read a lot. When I start a novel, I can tell instantly like, okay, this dude knows how to do it or this lady can write. Oh my gosh. I don't even know what the plot is or the characters are, but the sentence structure, something about it is just powerful and effective, and that's a joy to read. LLMs cannot produce that, not yet at least.

Speaker 2:
[32:36] I'm going to make a book recommendation, and I'm going to say that even though this title sounds like a kid's book, this is not a book for kids. The book is called Benny and the Blue Whale, and it is a fiction writer who collaborated with AI to write a book, just to see what the experience was like, and the experience matches what you were saying. It's not quite as creative, I think. But anyway, it was a fun read.

Speaker 1:
[33:00] There's something about the voice and the personality of the author that comes out in their writing, and that's also important for science. If you write a grant proposal, and it's fun, and it's exciting, and it's well-written, and it's crisp, the reviewers are going to have fun reading it. They're going to get excited, they're going to like you, they're going to want to fund your work. It's underappreciated, I think, by the public, how much of science is writing. Not just for folks like you and me who write specifically for the public, but emails, papers, grant proposals, reviews, all this stuff is writing and so much more powerful when it's done well. When you have a flair and energy, when you can convey excitement and enthusiasm, when you have your own personal voice.

Speaker 2:
[33:42] Yeah. I think we should note though that a lot of journals and a lot of grant agencies currently have rules for whether or not you're allowed to have AI help you with your writing. I just think it's worth noting that.

Speaker 1:
[33:52] Yeah, absolutely. It can produce bland generic writing that seems like suspiciously polished and flat. It doesn't really seem to get the point and convey it crisply. But this again is a steep gradient. They used to be worse. They're getting better. It's rapidly improving. I don't know that next year, these things won't be much better. A lot of the way they're improving is just by scaling. They read more, they make these models bigger, more layers of attention, more nodes. You sometimes hear these parameters quoted like seven billion parameters or whatever. That's the number of connections between neurons, each of which have a number associated with them. Tells you essentially the size of the network. The bigger the network, the more powerful it is. Also, the bigger the training set it needs to figure out what is the best value for all those nodes. LLMs have progressed from fancy Google searches to helping you with code, to reading and writing, but what about the juice of it? What about doing calculations, actually being a physicist? There's some fascinating recent examples of top level physicists using AI to contribute to their papers.

Speaker 2:
[35:04] We're going to get to that after the break. And we're back, and we're talking about how AI is helping scientists do some math. All right, take it away, Daniel.

Speaker 1:
[35:32] So there are top level physicists out there at the top of American academic physics who are using and exploring AI to work with them to help them solve problems. And Ed Whitten, for example, famous string theorist, maybe one of the smartest dudes alive, put out a paper recently where ChatGPT did an important part of the work. They had worked out a bunch of examples, and they had an idea that it could be more generally proven. And ChatGPT came in and found a way to make those connections, say, okay, here's the proof that this really does work. So it didn't come up with a research question, and it didn't even guess at the answer. It came in and filled in a lot of the details. But this is more than just correcting your grammar or fixing your code. This is manipulating mathematics and using logic to build a proof from A to B. So it really is pretty impressive.

Speaker 2:
[36:23] Yeah, it's amazing. I'm sure that saved Dr. Whiten a lot of time and allowed him to work on other projects.

Speaker 1:
[36:30] Exactly. And it's doing symbolic calculations, not just like 7.2 plus 12.4 or whatever. This is manipulating symbols, finding equations that capture these things. So that's very powerful. Then a friend of mine, Matt Schwartz at Harvard, recently put out a paper where he used Claude Opus essentially as a graduate student. He had an idea for a project. He pitched it to Claude. He got Claude to work on it with him, and Claude did all the coding and making all the plots and simulations and all the actual work involved, and he guided it through it. It's not like he just said, hey, here's an idea, go do the work and write me a paper, and then he came back after lunch and it was all done. His experience was that it's like a very naive young graduate student. It's often getting lost in the weeds, it's not thinking about the bigger picture, it sometimes makes mistakes, it needs a lot of supervision. But that's focusing on the negatives, right? It's amazing. This is incredible, right? That you have something which is a capable research assistant. Again, not just a search tool anymore, but actually doing work and making progress, and making it so that a guy who has an idea for a paper, no longer has to go out and find somebody to do the work, he can find AI to do the work. He still needs to engage his brain, he still came up with a question, he still needs to follow it and make sure it's reasonable. But this is not something AI could do a year ago or two years ago. So it's a dramatic improvement.

Speaker 2:
[37:58] That's exciting, but also a bit scary for folks who would like to get positions as graduate students.

Speaker 1:
[38:04] Yes, exactly. And I saw him give a talk recently and he said, AI is going to keep getting better and better at this, and we are not. And so his forecast is by 2030, most science becomes AI assisted. Wow. And that soon we'll see end-to-end AI doing theoretical science, that we won't need this kind of detailed step-by-step handholding and guidance.

Speaker 2:
[38:28] Does end-to-end include AI coming up with its own questions?

Speaker 1:
[38:33] It's not clear to me what exactly he means by that. And I think that's a very important distinction because being able to say, here's an interesting problem, go solve it, is very different from saying what's an interesting problem we should work on.

Speaker 2:
[38:47] Because I feel like that's when you work with a grad student, you get them to the point where they tell you, I'm familiar with the literature, here's what the interesting question is. And you're like, oh, that's great. I hadn't thought of that question. And now you're a team. Will AI become part of the team?

Speaker 1:
[39:03] So I was on this panel at NURPS, one of the premier AI conferences. And I was kind of a white blanket about anthropic solving physics in the next few years. So folks from the company approached me afterwards and they were like, so Daniel, what's it going to take to convince you? And I said, I don't know, but I'd love to play with your top level models. So they gave me access to their top level models for a few months. And so I played with it and I asked it exactly what you suggested. I said, all right, AI, suggest a project. Here's the kind of thing I'm interested in. What's an opportunity? Because I think a lot of people don't appreciate how much of science is identifying openings. It's a lot like being an entrepreneur. You have limited resources, you have limited time. Find a place where you can make progress, given your resources, in a reasonable amount of time, that's going to be impactful, right? That's a lot of what being a professional scientist is. It's easy to come up with big questions about the universe or grandiose plans that would take a thousand years and a zillion dollars. But what can I actually do today or over the next year to produce some science that's interesting to the community? What's the current conversation and where can we contribute? So I asked AI, what should we do? And it read my profile, and then it suggested a few ideas, all of which were things I had already done in the last few years.

Speaker 2:
[40:20] Oh, okay. So it's not there yet.

Speaker 1:
[40:22] So I was kind of disappointed. I was like, hmm, I've already done all of these things. And I didn't know if it's just like not quite there yet and I'm a little bit ahead of it, or if it's just like had read my papers and I was like, oh, Daniel's interested in machine learning, particle physics and tracking, so I'll suggest something in that direction. It's not clear to me whether it was in the training sample or not. And so it's not there yet, right? But that doesn't mean that it can't contribute, right? It's not going to be autonomous physicists just yet. But remember, let's think about what it's good at. What can it do? It's good at scanning lots of literature. It's good at putting these things together. And remember that a lot of advances we make in physics are not just like one person has a genius idea, which leaps us forward 100 years. It's often putting things together from existing tools, right? And so now let's dig into the question that we asked at the top of the episode, this claim made by these anthropic AI hypesters that it's going to solve physics, right? And like, again, this is what they were actually saying. And this is the kind of thing you see AI tech bros saying. You know, Sam Altman said, quote, Although it will happen incrementally, astounding triumphs, fixing the climate, establishing a space colony, and the discovery of all of physics will eventually become commonplace. So I'm not putting words in their mouths, right? These are the claims that are being made.

Speaker 2:
[41:44] Wow. Because of AI, we will have a space colony?

Speaker 1:
[41:48] That's right. We're going to fix the climate, establish a space colony, cure cancer, dot, dot, dot, dot, dot, all of this kind of stuff.

Speaker 2:
[41:54] Amazing.

Speaker 1:
[41:55] You know, and you see also folks on the other side. One of my favorite writers is Corey Doctorow. And he says that saying that AI is going to do science and engineering consciousness is like saying, if we can get a horse to run fast enough, it's going to turn into a locomotive, you know. And his point is that you can't just like extrapolate blindly.

Speaker 2:
[42:13] Yeah.

Speaker 1:
[42:13] Right. Because like, what are we even talking about? You know, on that panel and lots of times people are talking about how much progress AI is making. We have to think about like, what is the scale there? What are we measuring its capability to do? Is it like percentage of an Einstein or something? It's not something that we really know how to measure where we can say like, by 2047, it's going to be two Einsteins, right? That's all just nonsense. And so we don't know if there's a barrier there. We don't know if it's going to be able to do this. And what they're talking about it doing is something it has never done, right? It's never suggested a new research topic or solve the whole problem.

Speaker 2:
[42:50] But that doesn't mean it can't though.

Speaker 1:
[42:52] That doesn't mean it can't, right? Maybe these horses will turn into locomotives, right? It certainly is gaining a lot of power every year. So let's talk about the thing they're claiming, right? What does it mean to solve physics? In my mind, there's two possible answers to that. One is, look, we have a list of problems in physics right now. We don't know what dark matter is. We don't know why the universe is mostly matter. We don't understand dark energy. We can't unify quantum mechanics and general relativity. We don't know why there's so many particles and forces, right? How does it all fit together? There are definitely things we don't understand. Maybe solving physics means answering today's open physics questions. I mean, in the minds of Sam Altman and folks, maybe that's what they mean when they say this, solving all of physics. To me, that's not solving physics. Even if tomorrow they came out with the answers to these questions, right, I would not be done. I would not be like, well, I'm bored now. I'm just going to go sit on the beach, right? I'm going to ask like, well, why is this the solution? Could it have been this other way? Can I imagine places this could break down? Could it have been a different way? You know, there's always questions to ask. These questions come from the philosophical context, which is the origin of our curiosity. And that's not going to go away, even if we get beautiful equations to solve today's questions in physics.

Speaker 2:
[44:10] Well, I feel like if you've talked to any scientist, you know that as soon as you get one answer, that automatically leads to like 20 new questions. Like it's exponential, the questions you get when you get one answer. And so it's hard for me to imagine that solving those problems would suddenly mean, ah, we're done.

Speaker 1:
[44:29] I totally agree. And so from that perspective, will it solve physics? Absolutely not. Physics is a human pursuit and it's not going to end even if AI makes our answer finding much more powerful. But I do think, and this might surprise people, that it is going to transform the way we find those answers. And then it might actually be capable of Einstein level contributions. We don't know how to measure Einsteininess, but when you look back in the history of a lot of the big leaps, what are they doing? They are putting together existing pieces that other people were not aware of. It's easy to say Einstein was a super genius. He came up with this whole thing himself, but that's not really the truth. He was building on generations of careful math done by other nerds not even interested in general relativity. You know, Grassman numbers and Riemannian manifold, all these things are essential for Einstein. He could not solve those problems himself. He put together a bunch of interesting pieces, but sometimes just being lucky by who he knew and who he met being the right place, the right time to meet these folks and learn about their progress. So that's the kind of thing AI is good at. It's like, I'm going to read all these papers. I'm going to notice, oh, this tool over here that has been developed for totally other reasons might be able to solve this problem you guys are working on. That's exactly the kind of thing that AI is really good at. It's making these connections, it's searching broadly, it's reading the whole literature, right? There's lots of examples of this. Group theory invented for totally other reasons by math nerds turns out to be essential for quantum field theory. So, I actually am optimistic that there could be big problems out there in physics we don't today have solutions for that AI could help us rapidly solve because of its specific capabilities.

Speaker 2:
[46:13] Okay, so you've just been pretty wildly optimistic about how…

Speaker 1:
[46:17] Uncharacteristically optimistic.

Speaker 2:
[46:18] That's right, that's right. You know, you might be the more optimistic of Team Wet Blanket, but do you think like, will AI ever get like a Nobel Prize?

Speaker 1:
[46:29] Yeah, it's hard to know how to answer that. I mean, computer scientists building AI to solve problems have won Nobel Prizes already, right? We saw that last year, like Google DeepMind won a Nobel Prize, right? I mean, AlphaFold has done incredible stuff, right? They solved protein folding. And so the computer scientists wanted for the tool they built to solve the problem. Are we in an era where AI could win a Nobel Prize? I mean, like, I don't know what the estate lawyers would say after reading Nobel's will again. But I think that we are on the verge of AI solving the current generation of hard theory problems. It's still going to be like a human asks an AI, like, hey, can you unify quantum mechanics and general relativity? But if it goes off and thinks hard and finds a piece of math we hadn't explored or solve some of the current problems in quantum gravity comes up with a novel theory for it, then certainly it's going to have made an enormous contribution to our understanding of the universe to physics. Whether you should give it the credit or the credit for the person who typed in the prompt, I think that's a harder question I'm not qualified to answer.

Speaker 2:
[47:36] Wow.

Speaker 1:
[47:36] Yeah.

Speaker 2:
[47:36] And so many people contributed to making AI. It would be hard to know who to give the credit to.

Speaker 1:
[47:43] Yeah. That's always the case though. Like for the Higgs boson, they gave it to the theorists, not to the experimentalists who discovered it. Partially because there's just too many of us, right? Thousands of people contributed to building the accelerator and the detector and making it work. So we can't all win the Nobel Prize, there's a limit there. But I think it's clear that AI can do the thing we need to make big conceptual leaps. Construct the building blocks, pull those building blocks together to make breakthroughs. That's what it can do. And that's very powerful, right? We've seen in history, that's how a lot of these big leaps are made.

Speaker 2:
[48:17] Let's talk about its impact on physicists today. So for example, it can't take over the LHC and collect the data for you, like run the day-to-day stuff. So you still need experimentalists to collect the data, right?

Speaker 1:
[48:33] Still, for a while, yes. But there's a lot of work in autonomous labs. You give an AI access to the LHC, it could certainly decide how to run it and what data to keep. And people are building these autonomous labs, where an AI can, for example, test a bunch of materials really quickly. Build a bunch of materials, test it, and then close the loop, come up with a new material to test. I was involved in a proposal last year for something called Text-to-Launch Technology, where you type into an LLM like, okay, I want a satellite that can do XYZ, and then it uses an autonomous lab to try to develop materials that could do that. And then far down the road, maybe it would even launch it into space.

Speaker 2:
[49:17] Wow. Oh my gosh, that's nuts.

Speaker 1:
[49:21] But the point is that this is very new currently, but there's no real barrier there. If you can give AI control of these labs, if you build a lab that's AI compatible, there's no reason why experiments have to be out of reach of AI.

Speaker 2:
[49:35] Okay. And so then how is it impacting funding? So like you mentioned that AI could replace a grad student. So maybe now you don't have to ask for funding for grad students anymore, but like could AI replace Daniel Whiteson? Never, I imagine, but what do you think?

Speaker 1:
[49:54] AI can now do the work of a green graduate student. And that's very powerful. I can get more stuff done without my students than I could before. That is helpful. I always have one project where I'm doing the actual coding and now those projects move faster. So that is very cool. And it's definitely changing the way the government sees particle physics being done. In some of the latest budget plans, hundreds of millions of dollars that used to go to pure particle physics is now being sent to AI for science. That includes particle physics and particle physicists are still involved, but industry is also very deeply involved. It looks to me like the government wants us to make all of our data AI friendly so that these big models from OpenAI and from anthropic can play key role in accelerating the progress in physics. I'm not against that. I just wish that it was new money instead of repurposed old money. Let's keep doing particle physics and let's add an investment on AI instead of cannibalizing particle physics for this money and sending a few hundred million dollars to these vast corporations. Remember that academic physics is tiny in terms of research budgets compared to what these companies have and what they are doing. A small amount of money for them is vast funding for us. It can have a huge impact. Let's not forget that all of the ideas at the foundation of the current wave of AI came from academia. Academics, nerding around, coming up with ideas, attention, deep learning, all these things came from academic labs that were just playing with stuff. I'm going to beat the drum one more time for investing in basic research because it pays off huge. When it pays off, we shouldn't then cannibalize our basic research funding to dig into AI. We should also invest in AI, but we should maintain our investments in basic research, right?

Speaker 2:
[51:49] Yeah, yeah. I'm under the impression that if you massively cut a budget for, I don't know, say five or 10 years, you really hurt a generation of scientists. Labs are going to hire fewer grad students, fewer post-docs, you're going to start fewer physics labs and the labs that do get started aren't going to get grants that they need to convince their university that they should get 10 years, that they should stick around and now you've got fewer physicists produced by the United States. Is that your sense also or is that not how things work in your country? Yeah.

Speaker 1:
[52:18] No, absolutely. The theory community has received massive cuts to its funding and theory is very, very cheap. You're just paying a few people and pencil and paper, but their budget's been cut like 70, 80 percent, which means many fewer students, which means fewer theorists, which means fewer smart people asking those questions. You asked a question I've been answered yet, which is like, could this replace Daniel? I think that it's not hard to imagine that you soon have AIs that have the capability of senior scientists to do research, to even drive other lower level AIs or supervise humans who are doing research. I don't think that's out of their capability, but something I don't understand is who's asking the questions. Physics is a human endeavor. At the heart of it, is AI going to solve physics? It's about whether we are ever done asking questions. The answer to that is it's always going to be no. It's just going to be no. Physics is about our curiosity about the universe. Now, that doesn't mean that machines can't be there along with us, answering some of the questions. The open question to me is, will they have their own questions? I've got my questions about the universe, you have yours. That's why I'm in physics and you're in parasites, and somebody else is doing art history, and somebody else is doing chemistry, you know, God save them, and somebody else is at some company inventing new ways to make white chocolate cheap. And you know, everybody's driven by their own personal curiosity. Do machines have curiosity? If you give Claude all the physics papers and you ask it like, what do you think is interesting? What is it interested in? And is that really its own interest, right? Or is it just following up on what humans have begun? And so to me, that's the interesting philosophical question. When they get to have that level of capacity, are they answering their questions or ours? Will they be similar? Is this an example of aliens doing physics, right? Will machines do physics? I don't know, because I don't know what it means to have an AI at that level. And they will never really be divorced from humanity because they were birthed from the collective intelligence, right? The way that like something that's learned to write a great novel has learned to write that great novel by reading a bunch of great novels. But that's also how we train great novelists that are humans, right? You read good writing, you're inspired, it teaches you. So you know, I don't think it's impossible for AI to solve the current generation of physics problems. And I don't think it's impossible to potentially come up with new questions on its own.

Speaker 2:
[54:53] And if AI came up with a list of 10 questions and then you were presented with those 10 questions alongside of 10 questions from some of your favorite collaborators, and you thought those questions were equally interesting, what would you say?

Speaker 1:
[55:07] I'd say bravo, you know? I welcome our future AI collaborators. Bring your curiosity wherever it comes from. I think it can just help inform. But I think we should invest in this stuff, you know? And we should invest in human driven curiosity, as well as machine driven curiosity, and alien driven curiosity, and all of it.

Speaker 2:
[55:28] That's right, invest in our future.

Speaker 1:
[55:31] Exactly.

Speaker 2:
[55:32] All right, well, that was fascinating. Thank you, Daniel, for walking us through how you've been working with physics and AI over the last, what do you think? I guess it's been decades you've been working with LM's?

Speaker 1:
[55:42] I've been doing machine learning and particle physics since the 90s. So yeah, it's been 30 years before it was cool for sure.

Speaker 2:
[55:50] All right, well, maybe you'll give us a yearly update on how this kind of stuff is going.

Speaker 12:
[55:54] All right.

Speaker 1:
[55:55] Or maybe AI Daniel will replace me.

Speaker 2:
[55:57] I doubt it. I can't imagine that his jokes will be anywhere near as good or as bad, depending on how you see it, as yours.

Speaker 1:
[56:06] The day I stop making fun of chemistry is the day you know I've been replaced by a robot.

Speaker 2:
[56:10] Oh, thanks so much to Daniel and thanks so much to the Extraordinaries. And if you have a question about anything at all, you can send it to us at questions at danielandkelly.org.

Speaker 1:
[56:21] You'll get an answer from a real human being.

Speaker 2:
[56:23] Yes, you will.

Speaker 1:
[56:30] Thanks everybody for listening. Please go and do us a favor and rate the show on whatever podcast app you're using. It really helps people find us.

Speaker 2:
[56:38] Daniel and Kelly's Extraordinary Universe is edited by the amazing Matt Kesselman.

Speaker 1:
[56:43] He really is a wizard. You can also find us online on Blue Sky, Instagram and xdnkuniverse. Come engage with us.

Speaker 2:
[56:53] You can email us at questionsatdanielandkelly.org. We really do want to hear from you.

Speaker 1:
[56:59] You can find our website www.danielandkelly.org, where you'll also find an invitation to join our Discord, where everybody comes and talks about the amazing universe.

Speaker 2:
[57:10] We also have the most amazing moderators. This is an iHeart podcast. Thanks for joining us.