Streamlining the Cosmos: Pythonic Workflow Management for Astronomical Analysis

Speaker 1 [00:07]

RAFAEL WIEDING. I'm Rafael Wieding. I'm a postdoctoral researcher at the Max Planck Institute for Astronomy down in Heidelberg. And I'm an astronomer. And really what that means is I'm a data scientist, but the data that I work on tend to be from complex sources that come from, well, outside the Earth, usually, well, what we hope. And so it's the last talk of the day. We'll look at some pretty pictures. We'll try and keep things light. But I really want to talk about some things that have actually really changed the way that I structure my data science workflows over the past year that have made a big difference in the way that I do science, that I try and go from data that we get from observatories, both on the ground and in space, to scientific, actionable results, things that we can learn about our universe, conclusions about the way that stars and galaxies behave. And if you're interested in that, which, let's be honest, who isn't a little bit, you can talk to me. I love to talk about this kind of stuff. And so we're really starting to, I think in most areas, but astronomy especially, we're starting to really ramp up into the era of big data. So I'm showing some things from about 10 years ago. And these were our foremost surveys 10 years ago that were doing decade-long studies of large patches of the sky, collecting results on millions of sources. And we could get up to data volumes in the hundreds of terabytes. OK, yeah, big whoop. And really, since the start of this decade, We've started to see, now we've really started to scale up, and we're introducing multiple projects that, over their lifetimes, are expected to generate petabytes of raw data. And in fact, the Rubin Observatory just got first light a couple of days ago, which is, over its estimated lifetime, it's estimated to produce 500 petabytes of raw data. Now, this is raw, off-the-sensor data. This isn't secondary results. This isn't analyses. This is just data from the cameras, or what we think of as cameras, coming to our hard drives. And the story is basically true in space. So if we look at telescopes that launched or, well, tried to launch in the 2010s, we look at estimated lifetimes of the petabyte regime over their entire lifetimes. And now, as of this decade, we've launched telescopes, which, again, raw data are expected to generate many tens of petabytes of information that then have to be processed, that then have to be scientifically diagnosed, and then results generated from that. Yeah. And in fact, it's pretty impressive that we can actually download this much data from space. These things are far away. But that's a whole other thing to talk about. It's cool. We also haven't made anything easier for ourselves as astronomer, because we've decided to become more computationally complex as a field. Going from raw data to scientific data, something that actually looks like the images that we show in the news, is really complicated. that requires many steps of validation, cleaning, taking care of background noise, cosmic rays, which are basically high-energy particles that smash into our detectors and look like stars, or they just kind of look like weird things. Because these images are so large, we're really basically required to have massive memory overheads just to process individual frames from these telescopes, which basically means that we need to push us to actually high-performance computing environments. So in general, I work a lot more with HPCs rather than something like the cloud. But this sort of the idea is the same, that we are forced to go into these environments that can actually support this. These are data that I can't process on my laptop. And then, of course, we actually have made ourselves sort of double the computational problem. Because once we actually have the really cool data, we've decided that we want to use the latest and greatest Bayesian analyses that take forever to run and produce massive outputs that we then need to store and then merge and collate, et cetera. We basically love putting in the most complex physical simulations that we can think of that, on their own, take a supercomputer to run. And then we want to use those as inputs. And so getting all this together is incredibly hard to coordinate. And then finally, we want to run machine and deep learning models on all of these objects to classify, what's a galaxy? What's a star? Are these things just artifacts in our data? Because we're now producing the number of images, or we're getting to the point, where no one will look through them over the time that they exist, or at least not in the next 100 years. That's the amount of data volume that we're talking about. OK. I'm going to tell you, I'll take a breath. I'll drink a little water. So maybe you're a scientist like me, or maybe you kind of think of me, you're kind of empathizing with what I'm saying. And you're like, yeah, are there good ways of doing testing on small environments that I actually have a little bit more familiarity and control on, and then actually being able to scale this up? So I'll give an example of what a workflow for myself might look like, and the tools that I've been using at a high level to actually orchestrate these in an efficient way that can scale to large supercomputing environments. So I might take some SQL query that gives me a metadata about the images that I want to look at, where in the sky the telescope was pointing at, how long it was pointing for, et cetera, et cetera. And I might use that to generate or to download thousands or tens of thousands of images that essentially can be split into like, we have 100 images pointing that way. We have 100 images pointing that way, that I'll call data sets. And that we actually want to take this data and actually process them. Then, for example, we want to go from a couple of data sets, which are in different directions on the sky, and actually break those back out into all the stars or all the galaxies that we can see in those images. And then we want to process and figure out all the properties of all of those objects and all of these things. And then we want to synthesize that information back down into catalogs. And then we want to combine those catalogs back into results and plots. And so we have this huge pipeline of basically how we actually orchestrate all of these things, which turns into a mess. It takes a lot of time to actually build the infrastructure to orchestrate this, which I don't want to do. I want to focus on the things that I'm interested in, which is measuring properties from galaxies. I really like taking an image and measuring, what do we actually know about this galaxy? How many stars are there? Does it have a black hole in it, et cetera? So I don't want to have to worry about coordinating this workflow or building the code to do that. And then especially what I deal with a lot is the fact that all of these different steps, every arrow here requires different conflicting dependencies within Python or different languages. And then how do I get this to run easily without too much hassle on a supercomputing environment which actually is able to handle the load that I want to put onto it? So the answer that I have found that works extremely well for me is SnakeMake. Has anyone here used or heard of SnakeMake before? That's great. OK. rest of you, I hope you use it. It's actually a package designed for, do you guys like it? Do you think good things about SnakeMake? Ooh, OK, maybe I'll change your mind by the end of this. But it really changes the way I work with things. But OK, that's good. That's good. Let's get to differing opinions on things. But OK. SnakeMake is a workflow management tool designed for building reproducible and scalable workflows, which is exactly in the title of the slide there. You've probably already read it by now. And I'll repeat it one more time in the first bullet point. And it was actually originally designed for bioinformatics. I don't know that much about what it was actually designed for. I know that on the GitHub issues, I sometimes run into people doing cancer research. But in essence, we have the same issues. I just showed a workflow that maybe many of you can empathize with, and we have the same issues. And so the idea is that SnakeMake, which is very similar to CMake in the way that it orchestrates, the goal is to build and define your entire workflow in a series of simple rules that go from inputs to outputs. And so by essentially reworking your thinking into, or sometimes it doesn't even take that much reworking, but by putting pen to paper of, well, what are my inputs? And what are my intermediary and then final outputs? SnakeMate can essentially take each of those steps in your pipeline and automagically, I'll say automagically, figure out exactly how to combine them, which parts are dependent on each other, which parts are parallelizable. and then immediately scale them up to supercomputers. So before SnakeMake, I had to write Slurm scripts. Who here has written a Slurm script for submitting to a supercomputer? Okay, there are a couple of scientists, I think. This is how we submit to a job submission queue on a Slurm script. Who likes writing Slurm scripts? Let me see another show of hands. Okay, there's a sort of a maybe, there's sort of a maybe. No one else, because Slurm scripts are a lot of boilerplate and they're kind of annoying. You don't have to, the moment I start using SnakeMake, I've never had to write a Slurm script again, because it automates doing all of this for me, which is great, because I want to spend time on doing what I actually care about, which is science. So for example, we might take this complex. I thought I built these in, but OK, I'll show them all at once. Ignore the left, this part for now. So we take this really, really complex workflow, or relatively complex. In reality, this thing can branch a lot more. But the synthesized version of the workflow that I showed before, and essentially what we do is all we write code for is the very simple, hopefully, again, they're not simple, but that's why I want to spend time on them, is writing the digestible code that goes from just a single galaxy to a single set of its properties, going from a data set to detecting the galaxies. I'm sorry, I'm moving too much. Going from all the properties and synthesizing back into catalogs. So how can we just synthesize into exactly what we want to do and not have to worry about writing the Slurm scripts to automate taking all of these galaxies and writing MPI scripts to submit them and use memory efficiently and make sure we're not exceeding storage quotas, et cetera. We want to find a better way of actually getting the computer to do this work for us. And so what's great about SnakeMake is that it essentially does all of this for us, and it's open source. You essentially provided the constraints of, here are the resources I have access to, and here's what I want to get done. And its scheduler solves the graph problem and essentially uses those resources as optimally as possible. If I say that I have a supercomputing node with a terabyte of memory and 256 processors, and I tell it, here are all the galaxies. Each one's going to take x amount of memory, and I need x processors to process it, it knows how to fill those slots up and then submit the Slurm scripts for me to actually group those things together. I see some people nodding like, oh, that sounds useful. Yeah, it really is. What's great is I can define per rule environments or containers. So for each of these rules, which normally have conflicting dependencies, I can actually specify conda environments for each. And we'll talk about in a little bit why I don't like using conda. And we're not going to talk about it. Or if you really want ultra reproducibility, actually containers. And then you really guarantee that all of your rules are isolated and can run properly. You can also prioritize critical workflows. This has less application to astronomy, because every galaxy is just as special as every other. But for my friends in the medical fields, they sometimes like finding patients that have, especially for cancer research, they're usually taking samples from patients, processing. I'm talking too much about something I don't know about. But process it, and then give them a diagnosis. And they want to be able to say, this patient needs the diagnosis sooner. And so you can bump them up in the workflow. So it actually supports doing things like that, which is really, really, really, really cool. And you never have to write a Slurm script. And if we, for example, add a new galaxy or add a new patient into the mix, SnakeMake can take a look at the entire swath of everything that we've done and say, well, I've already run on every single other 1,000,000 minus 1 galaxy. I'm only going to run the processing I need to to produce this last galaxy. And so it only runs when it has to. So I've hopefully made the case for snake make. And then what I'm going to talk about next is where this has been really useful in my own research by giving you, unfortunately, a little bit of an astronomy lesson. Because if you came to an astronomy talk, you're going to have to learn a little bit about the cool science for doing astronomy. So this is the James Webb Space Telescope. It was launched a couple of years ago. It's NASA's flagship mission. It's the biggest telescope we have in space that we know of. Talk to me about that afterwards. There's some interesting stories about what the US military gets up to, but anyway. This is a person for scale. Oh, sorry. This is a person for scale. Teeny tiny. It was so big you had to fold it up and fit it in the rocket and then launch it. It is not a data monster like some of our other telescopes that are coming up, but it sort of trades that for precision. It is an ultra-precise machine that can do really, really amazing science. And actually, it was in the news pretty recently about detecting life signatures on other planets. You may have seen that. That result is probably bogus. Don't want to . But, well, not bogus, but maybe requires more evidence before we say we found aliens. But that's the kind of work that James Webb does. OK. What's really cool about the James Webb, what really gets me excited, is that it's a very efficient machine. James Webb has four cameras on it that all look at slightly different areas of the sky. So one's kind of looking over here, and one's kind of looking over there. And what we do is that if someone wants to use this camera to look at that object, we just go point at it. But there's no rule that says you can't use the other camera while it's doing that. And so if you don't care about what you're looking at, if you want to do really sort of unbiased surveys of the universe by pointing your telescope in a random direction and just looking with the most sensitive, most advanced instrument that we have in space that we know of, then you can do what's called pure parallel, where you, Some other folk who's doing amazing research will be looking at this really interesting galaxy that has a supermassive black hole inside of it that's driving out flows. They're doing really cool science. And we're going to look at this patch of sky that's never really been looked at before in any great detail. And we're going to see what happens. And just to give you an idea, this whole thing is maybe 20 times smaller than the full moon. So it's a tiny little patch of sky, but that's just for fun. So what you get is that this is real. This is a patch of the sky that we looked at, and approximately this angle. And you can see that there's not that much in it. You can get really close to the screen if you like and see maybe you can count how many points are in it. But then when we actually look at it with the most advanced telescope in the world is we get an image that looks like this, where every single point of light in this image, except for three, and I don't have anything to give you, but if you can find them, maybe I'll give you a sticker or something, are galaxies. There are three stars in here that I can count, and the rest are all galaxies. And so this is hundreds, if not thousands, of galaxies. And we have pure parallel pointings like this all over the sky. We have hundreds of them. Each one of these contains dozens. This is a composite of many, many images, which is really cool. I mean, it's really exciting. But what I want to hone in on, and we're actually going to have a talk about Euclid tomorrow. I think it's the keynote talk. So wake up at 9 to make sure you can get to it. I guess earlier than 9 so you can get to it. They're doing a type of, they have a type of camera that we have on James Webb that is fundamentally different than the way we normally think about images. Now, I'll make the point to you that the light that arrives from space is fundamentally three-dimensional. Who can name the three dimensions of light that comes from space? What do we think? Anyone? Time, phase. Okay, you chose the ones that I wanted to ignore, but you're right. Time, phase, and coherence are technically true, But I'm going to think of x, y, two spatial dimensions, and one more dimension. I think you might. Redshift or just wavelength, right? Color, right? We have x, y, and then, for example, this image is an RGB image. We have three slices in color space that tell us about these galaxies, right? Are they red? Are they blue? Et cetera, et cetera. And we can learn something about them, right? Like redshift. But fundamentally, this image has done a disservice to that third dimension. Because we have about 2,000 pixels, even more, in this dimension. We've got 2,000 pixels in this dimension. And then how many pixels do we get in that dimension for a typical image? Three. We get three. That's horrible. I mean, for one of the fundamental dimensions of the universe, the fact that we only reserve three pixels for it, and sometimes we do a little more, maybe 20, it's a travesty. So what we've started to do is to sacrifice some of the spatial information and spread the color information, the wavelength information in one of those axes, and you get something that looks like this. So every galaxy that you see here, I know I can't move too much. Every galaxy you see here, we smear out along the Y axis to make a trace, and these colors are kind of fake, but they're more or less accurate to kind of where they trace in a shifted version of the optical spectrum. And we smear them out, each one of them, in this image. And so I'll show a little movie just to give you an idea. Choose a bright galaxy, maybe that one right in the middle. See everyone see that little spiral, kind of yellowish galaxy in the middle? Look at that, and I'm going to fade in the image that's right on top of it. And we see that it smears down in that direction. I'll do that one more time. Choose another galaxy. These images don't quite overlap, so it's not perfect, but that's the way pure parallel goes sometimes. I'll do it one more time, maybe with some oohs and aahs. Ooh, ah, great, OK, awesome. So this kind of data analysis, by blending the second and third dimension, is incredibly computationally intensive. And in fact, we don't even really have good algorithms to deal with it yet. And we've built an entire 100 petabyte telescope designed to do only, well, not only, but to do primarily this kind of information. It was sold to be launched on the promise of doing this over about a third of the sky and generating 100 petabytes of information. And it is incredibly computationally intensive to, one, figure out what stripe goes with what and what that tells you about the galaxy overall. Oh, my gosh. And so I'll show you even one more thing. This is a relatively sparse field, right? You know, it's like a sparse matrix. Only a couple of points in this frame have a galaxy in them. It's pretty much empty. Space is empty. But what we've been starting to think about more and more is, well, can we point this at a really dense part of space, like where there are a lot of stars, our own Milky Way, like the center of our own Milky Way, or a galaxy nearby where we can actually see stars, and then you get something that looks like this. We currently cannot process this. This is something that we aren't able to do yet. And you'll notice that this has a lot of little dots in it. Those are all cosmic rays. They're really hard to get rid of. So all I have to do, rather than thinking about, well, I have 200 of these fields all over the sky, that each have, you know, hundreds of images, then I have to process each one of them individually, and then I have to make sure, okay, how can I process them as efficiently as possible? How do I worry about my high-priority fields and make sure those get done first, right? If I have a field that has 200 images, it should be able to share some space with a field that only has 20 images, right? Just shunt those 20 images onto the end so we can add up to 256, right? How can we make most of our, make most use of our high-performance computing systems as possible? You know, compute isn't free. It kind of is for us. But for the most part, compute isn't free, right? Time isn't free. So all I have to do is define a few simple rules. Download the data, do stage one processing, pre-process the data, combine the data into mosaics, do the contamination modeling, which is figuring out which stripes go to what, and then extract the data and learn some things from it. And then I'm only showing you one small little slice of the diagram that it builds, but it builds the whole diagram for me, coordinates the entire workflow, submits it to the supercomputer. I come back two days later, and it's all done. And then I upload it to AWS and share it with all my friends, which includes you guys now. So anyway. Cool. So I really hope I've made the case for SnakeMake. I only have five minutes left, so I want to shift to talking very quickly about what about rule dependencies, right? Especially when we talk about reproducibility in science, which is a core tenet of the scientific method. Nowadays, it's really important that we make sure our software is reproducible just as much as our methods at a high level. So how can we best ensure reproducibility for our Python ecosystem management? Conda is relatively limited. It's namespace scoped. And what I mean by that is that maybe you make an environment to look at a galaxy, and you build this really wonderful Conda environment, and you're like, it works really well for this. And then you get a new data set. Tell me if this has ever happened to you. You get a new data set. You use your same pipeline. You update a couple packages. You modify things. And then a year later, you come back to your old project with the conda environment that you've updated a couple of times. And lo and behold, nothing runs because everything's been updated, and you have no record of what it actually looked like. Who's had kind of a similar experience with conda before? Yes. Great. So we want reusable environments, environments that can actually be stable and be tied, for example, with something like versioning. And conda is actually relatively slow. But that's not the worst thing in the world. Docker is a great way to do this. It also has some slow startup time, though that can be tweaked. It usually requires root, though, again, and for HPC systems, it's a problem, but you can get around that. And it's kind of cumbersome for testing on local systems. And in general, Docker, it has some modularization, but it can be pretty large. It's hard for me to rag on Docker, because if you really want ultra reproducibility, it's kind of hard to get around, but we're lazy. So is there a solution that currently maintains strict reproducibility, supports multi-language projects like C and Rust and Python, and can use existing kind of PyPi infrastructure. And the answer is Pixie. So who here has used Pixie before? Give it a try. It's the bee's knees. You get isolated, fully reproducible environments by declaring specifically what goes in. And these are project scope, which means each environment lives in the directory that you're running the code in. So for example, it gets tracked by things like Git. It's extraordinarily fast, because it's all written in Rust. And it uses UV on the back end for some of the things. And it uses Conda as well. And it can handle really complex environments for these complex pipeline by doing different features. So for example, I can have all of my rules here. I know I got to rush a little bit. Where for example, I need different packages sort of as a combination in all three of these. And all I have to do is I can essentially define my environments by what they need, and Pixie will figure out exactly what they need, combine them, and deploy the environments as necessary in these rules. Anyway, so I don't have too much time to talk about Pixie. I would say go see the Pixie talk. It's the same time as this one. But really, if you're tired of Conda, talk to me about Pixie. This is awesome. So takeaways. This is my final slide, just a lot of bullet points. Not a lot. I suggest that if you're starting a project, think about what your deliverables are. Which, you know, as a scientist sometimes it's a little hard for us to do. Maybe it's easier for some other people in the room because we get very focused on the process. But think about the deliverables. Is it plots? Is it tables? Et cetera, et cetera. And how can we define very simple rules that aren't necessarily parallel? Again, it's still good to write parallel codes for some of these specific rules. But for where we don't have to worry about, for example, pleasingly parallel situations, how can I build these workflows? And so SnakeMate takes care of all of that. And it's modular, smart, and introducing new data and changing steps is really easy. So for example, if I change one of the rules, it knows to rerun that rule and every rule that depends on it. So it knows exactly what's changed and what comes next. It's great. Pixie, Project Soap environments keep things really reproducible. Awesome. It's really fast for local environments. I mean, Pixie environments take seconds for me to install for complex environments where conda could take minutes. It's really, really, really nice. And I don't have to deal with Docker. It's all command line. There's no root access necessary. It's great. But I will mention it's still pretty. It's on version 0.4, which sometimes doesn't mean a lot. But things change. Things get added. It's cool. And if you're like, well, this is great. I know you talked a lot. How could I actually do this? I built a couple simple, very simple examples for people to try out that are on my GitHub. So you can get started with Pixie. I mean, read the Pixie and SnakeMake documentation. I think the Pixie documentation is great. SnakeMake lacks in examples, but have fun. Play around. And oh, yeah, sorry. Basic examples of some different types of procedures. And that's it for me. So thanks for coming, guys.

Speaker 2 [25:24]

Thank you as well. So I love the talk, to be honest. Very nice energy, especially for the time of the day. And maybe some other people felt the same. Yeah. So we already have two questions in the Q&A. And the first one is, what is your topic of study you are doing research on?

Speaker 1 [25:42]

Oh, great. Great question. I'll talk about this for another 20 minutes. So no, very, very briefly, I love studying galaxies outside of our own Milky Way. So maybe you're familiar with the Milky Way. It's a collection of about 100 billion stars. We know that there are about a trillion galaxies in the universe. I want to know about the life cycles of galaxies. How do they form? And what changes them over their lifetimes? And specifically, I try to study supermassive black holes, which are basically a type of black hole, which again, it's not supermassive. It's very scary. that scary. But they are the most luminous sustained energy emitters in the universe. And they can really punch holes in galaxies. And they've punched a couple holes, maybe a couple holes in our galaxy, too. And we think that can change how a galaxy evolves over time. So I'm really interested in galaxies and supermassive black holes. Talk to me more. I have a lot to say. Or read my papers, I guess. But that's a lot to ask. Okay. And second question.

Speaker 2 [26:32]

very nice second question why does the jw not use the four cameras for the same target at the same time were they meant as backups why is pure parallel in the pov and not zoomed out

Speaker 1 [26:45]

Can you say the beginning part again?

Speaker 2 [26:49]

Yes. The first question is, why does the JW not use the four cameras for the same target at the same time?

Speaker 1 [26:54]

Yeah, so the cameras aren't movable. The cameras are fixed in place. They don't look at the same place in the sky, and that's by design. Mostly because building a space telescope is really, really hard, because you're weight constrained and everything. And the less moving parts you have, the better. So you just can't move the little pieces of the telescope. In fact, the first thing we expect to fail on the Webb telescope is nothing like it runs out of fuel or anything like this. We expect one of the moving parts to fail. And so you usually want to build a telescope with as few moving parts as fail as possible, because this thing is four times further than the moon. So if something breaks, we can't fix it. It's more expensive to fix it than just launch another one. And launching the first one was really expensive. By science standards, it was only $12 billion, which, like, you know. But anyway.

Speaker 2 [27:40]

You answered the second part and the third one is why is Pure Parallel in the POV and not zoomed out?

Speaker 1 [27:48]

Yeah, I mean, again, the idea is, like, we build the cameras the way they look. We can't just point them somewhere else. We can't really go that much further. Again, these telescopes are designed to look at incredibly tiny things. We're talking, like, there's statistics of, like, you know, it could see a dime on the person in Munich or whatever. You know, it could see a dime in Munich or something, right, crazy like that. We're looking at very, very, very, very, very small regions of the sky. And so all the cameras are really laser-focused, and we're moving them by tiny amounts just so they don't kind of occupy the same space. Let's do this again.

Speaker 2 [28:20]

Next one is, have you tried Nix instead of Pixie?

Speaker 1 [28:24]

So I like Nix. I use it to maintain stuff for my personal projects. Nix, again, requires root, so it's hard to scale it to HPC environments, I think. I mean, you essentially have to have these mounted storage solutions to... Nix is cool. I think it's great for building... I deploy a home server on Nix. I think that's really, really fun. But for these kind of workflows where we want to be changing things all the time, and you need root for that, Nix can be a little space inefficient. And Nix is kind of closer to the Docker end, where it's a little bit too much for the kind of things that we want to do. But there are some people here that have experimented with Nix, or in my profession that have experimented with Nix. And it's an idea, but for the kind of Python-centered workflows we're doing, I think Pixie kind of fills the niche without scaling up to something like Docker, Aptainer, Nix.

Speaker 2 [29:16]

Very nice any other questions in between as you have one can you upload it also to You cannot find the link okay, maybe then yeah, I just send you the microphone Hi, I don't come from your field, but I come from Aerospace and we we do lots of studies in our aircrafts in the largest ones we have right now in the moment and We run very intensive computational Yeah, processes and we use LSF. Do you also use LSF?

Speaker 1 [29:49]

I'm not as familiar with LSF, remind me what that stands for again?

Speaker 2 [29:56]

that's basically the interface to your HPC. Then you use the VSUB command to schedule, to send your jobs, and then there's a schedule running in the background.

Speaker 1 [30:06]

No, so I think we usually use a different scheduler. I think different HPC systems will use different schedulers. The idea is that SnakeMake does all the scheduling for you. So it's integrated with whatever scheduler you want to use. They have plug-ins for different schedulers. So instead of you worrying about how to orchestrate your job, so if I have a million galaxies, I don't want to submit a million jobs. I want to submit 10 jobs where a million galaxies are batched together. And so the idea is it's, for the most part, queue system agnostic. Whatever queue system you have, you just have to write one, not simple, but you can spend your time writing the script that goes from galaxy to properties and not worry about how you multiprocess all of that and let SnakeMake do that for you and batch your jobs together and submit them. And again, batch jobs exist. There's lots of ways of using existing implementations. But especially when you have a really heterogeneous data set, like we get here, where we have 1,000 images in that direction, 10 in this direction, different memory requirements. I mean, that might be looking at 1,000 galaxies. Here, we might only be looking at 10 galaxies. And combining those together, it's really helpful for us, especially in a heterogeneous data set.

Speaker 2 [31:19]

We have one more online, which is, can the public get access to these images?

Speaker 1 [31:26]

JOHN MCCUTCHAN- Yes, absolutely. So the pure parallel, so the way web works briefly is that if you, the way web works is that you ask to use time on it. And 90% of people, they say no. And the last 10% get their data, and they usually get to keep it for about 12 months before it becomes public immediately. So any data that's older than 12 months from the James Webb Space Telescope, you can download it right now. It's got an API and an interface to download whatever you want. They look, most of the time, like garbage because data coming from a space telescope that hasn't been calibrated or cleaned doesn't look good. But it's all there. The data that I showed you, Pure Parallel, becomes public immediately. And usually programs over a certain size become public immediately because they're considered in the public interest. So yeah, I mean, this data, I can show you exactly where to download it. But the secret sauce for turning it into this beautiful RGB image, well, I also put that on GitHub, so that's not necessarily secret either. Because there's no point in keeping it secret, at least for me. But yes, all the data from all NASA missions, Hubble, et cetera, is all available. Some of it's under a slight proprietary period of time. But I mean, it's all public information. It's a public good.

Speaker 2 [32:36]

very nice very nice so there's the last online question but i think we had one more in the audience and that will be the wrap-up yeah thank you for the presentation so one of one of your first like astronomic images there was a galaxy on the left

Speaker 1 [32:53]

There's a Galaxy on the left.

Speaker 2 [32:54]

and there was something

Speaker 1 [32:55]

There's something black in the middle. Yeah, this one. Oh, this? Yeah, on the left. I'm just wondering, is it a black hole, or like an effect? Ah, I should have, sorry. This is a star. The center of the star is way too bright, and so it just burns out. It's like pointing your smartphone at the sun. You shouldn't do this. And I mean, the sun. I mean, this is fine. But if you point something at, eventually your sensor just has like an overflow, right? And so all we're seeing here is we recognize what's an overflow, and then we just mask that part. There's no usable data there. And so it's just a bright star. And the center of the star is obviously the brightest part. And the rest is what you're seeing from around the star is just the star is not actually this size. It's just the light, like lens flare, right, essentially. That's the, yeah. But I mean, there's lots of data in here. For example, we go back to my other example.

Speaker 2 [33:47]

I don't know.

Speaker 1 [33:51]

I don't really show it very well here. But there's a lot of data in here that's bad. That's not good. Yeah, I mean, again, we launched these telescopes into space. And it's flooded with cosmic radiation. And so a good amount of these images gets destroyed. And that's why we take multiple exposures. We do long exposure. We have lots of strategies for mitigating this. But it's all under the hood. I mean, and especially a lot of the data that they should, like all the really pretty pictures, they've done a lot of Photoshop on those. I'm not saying they're not good for the purpose that they are for. It is an art to turn scientific data into human visualizations. It's incredibly important work, and it's the reason why astronomy gets so much attention. But you have to sacrifice some scientific quality to make it look pretty. And I am not good at that. That is not in my skill set. And so I show you as it is, right?

Speaker 2 [34:42]

Yeah, thank you very much. Maybe also from the audience another applause for you