Sentinel Values in Python: Semantics, Double Dispatch, and the Limits of Typing
Python's `None` keyword is frequently used to represent missing values, defaults, or errors, but this versatility creates ambiguity when `None` is also a valid value within a specific domain. For example, in a cache dictionary, `None` cannot distinguish between a missing key and a key explicitly mapped to `None`. Sentinel values solve this by providing a unique object outside the valid domain to signal a specific state. Built-in examples include the `Ellipsis` object used in NumPy for slicing and the `NotImplemented` sentinel used in double dispatch to signal that a binary operation should be attempted by the other operand.
Implementing custom sentinels using `missing = object()` is a common pattern in the Python standard library, but it introduces three primary issues: uninformative string representations in stack traces, failure to maintain identity after pickling or serialization, and a lack of support for type narrowing in static analysis tools like Pyright. While `TypeGuard` functions can assist with narrowing, a more robust approach involves using a `Literal` combined with an `Enum`. By defining a sentinel as a member of an `Enum` and assigning its type as a `Literal`, developers can achieve precise type narrowing and ensure the sentinel remains a singleton after serialization.
PEP 661 proposed introducing first-class sentinel objects to the language to resolve these inconsistencies, though the proposal remains deferred due to the significant interpreter changes required. Currently, the `typing_extensions` library provides an experimental `sentinel` implementation. Until such features are standardized, the `Literal` Enum pattern is the most effective method for creating type-safe, serializable sentinels.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track Programming & Software Engineering & Testing and was classified suitable for intermediate domain / intermediate python by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
Sentinel values are a fundamental but under-documented part of Python’s design. They are used to represent absence, unsupported operations, incomplete state, and to coordinate control flow between objects. Yet, they are often treated as ad-hoc implementation details.
This talk starts by clarifying what sentinel values are and why None is frequently semantically overloaded and incorrect for modelling “missing” or “unset” values. We then examine built-in sentinels such as NotImplemented, Ellipsis, and dataclasses.MISSING, with a detailed look at how NotImplemented enables double dispatch in equality and ordering operations.
The second half of the talk focuses on typing, where sentinel values expose fundamental tensions between Python’s dynamic semantics and static type systems. We will discuss:
- why Optional[T] does not mean “unset”
- why Literal appears attractive for sentinels but rarely works in practice
- what limited type narrowing is possible today and under which assumptions
- why a fully reliable, user-defined sentinel with correct narrowing is currently not achievable in a portable way
To ground this in practice, we will look at real-world patterns used in production code, including Pydantic’s experimental missing concept, and explain the trade-offs these designs make.
Finally, we will examine PEP 661, the proposal to standardize sentinel values and their typing semantics. We will explain what it would solve, why it was deferred, and what that deferral means for library and API authors today.
The talk concludes with concrete, honest guidelines: when sentinel values are the right tool, how to design APIs around them, and how to communicate absence clearly in typed Python code without pretending the type system can do more than it currently can.
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:01]
Hello everyone. My name is Amin. I'm going to be your session chair for the next two hours. First talk is going to be about Sentinel and about things that are used in multiple places in Python that we saw in multiple times that we never paid attention to. And today we have someone who hopefully would explain this to us. I'm going to introduce him. He doesn't know what I wrote, so let's see his reaction. Florian, he's a long-term friend and supporter of PyCon.de and he's a passionate Pythonista since the start he's the head of data science and mathematical modeling at InnoVex, KMBH he has a PhD in mathematics and he has more than 10 years of experience in predictive and prescriptive analytics he's a big fan of running and cycling as far as I know I personally worked with him at the program committee and it was a delightful experience let's say So now, for you.
Speaker 2 [01:00]
Yeah, thank you. Thanks for the kind introduction Yeah, welcome to the talk sentinel values and Python semantics double dispatch and the limits of typing So since we are a little bit short of time Yeah, you heard the introduction and I just skip over so I work for InnoVex should definitely mention my employer who is kind enough to let me go here and Yeah, we're an IT project center focusing on innovation excellence and there's more than 20 of my colleagues here on site so if you see us talk to us we have six talks and of course we also still also are hiring so if you're interested just come and find us so to jump right in I mean surely all of you who are programming Python have used the none key word, right? In many different places. And none actually comes in Python with a lot of ambiguity, depending on where it's used for what. So it serves as a kind of Swiss army knife, actually, in Python. So sometimes its meaning is more like to express, hey, there's something missing. Sometimes, hey, it's unsaid. There's a default value somewhere, or even as an error. So let's first talk about those various ambiguities. To start with a simple example, let's assume you have some kind of cache, like some dictionary, and there's keys and values in there, and you're caching, of course, the values given some key, and now if you use the get method and you get some missing key, you will get a none because it was not found. And if you get now some input, and some input also has a none, then you also get a none. So you don't see the difference if it was a missing key or it's an actual value. And of course, you can also provide with the get method a default you get if there's a miss. And you could, for instance, don't do it, use some magic string. And by checking for this magic string, you would then see if you actually had a cache miss or you use the in operator to check before if the key is in there. But yeah, the point here is that none in this case is a valid value and at the same time it's returned by default in case of a cache miss. Similar problems are in things like iterators. There it's solved if you call next on some iterator and there's no more values in it, you get the stop iteration exception is raised, or also in queues and worker pools, so we often have situations like this. Another example, something I found also in real code, is this example where none is more used like an unset value but it's strange in its meaning what does it say I update the config and the timeout is none so if I don't provide anything I get some default timeout or do I set it to there's no timeout at all so this also shows there's a certain ambiguity in in the context another example you might have also seen something similar is let's say you get a user profile you have this function and what happens within the function is that there's some call to some web service and if anything goes wrong like a timeout error so the function just returns none to show you as the caller that there was some kind of error and you don't get nothing. So those examples show us that in a given context we always have to think of a domain of valid values in this context like integers, floats, the booleans like some string dictionaries and so on but quite often like in the case of cache and so on none is part of this of this valid domain in our given context so what's a sentinel a sentinel is something outside this valid context that gives us a special meaning and why is it called sentinel by the way if you're interested in how did they come up with a name like this with a kind of guardian because in C it was already used for instance if you have strings in C and there's a special the null character which says okay here's the end of the string and it kind of guards you to run over the end of the string and land in some wrong memory places so this is why It got its name So there are many built-in Sentinels in Python you might use quite often, but I'm not aware or I have never seen it in this light for instance NumPy users will know there's the ellipsis and you can save a lot of writing with the ellipsis For instance in this case if you want to set for all dimensions except of the last one a value of of 255, then this is easier to do it like this than using columns here. And this is not just some syntactic sugar. It's not like the interpreter is seeing this and interpreting it in a different way. It's really implemented that this is a value, and the value given to this Dunder method here knows then how to interpret it. so you can also come up with your own logic. Also talking here about different semantics, if you've ever watched or looked into one of those Python interface files, then here the ellipsis is used for something completely different. So those interface files, they do the type hinting, and there the ellipsis is just you need to pass something, and this gives us a semantics, okay, the actual definition is somewhere else. This is only for type hints. You could also write something like pass or even just any kind of statement. And yeah, this also shows it can be used at different places and semantics is important. You could, of course, also use it as a default value instead of none. again you shouldn't but it's just again another value and let's say in here none would be part of this valid domain I was talking about before and I needed another value to show okay I really want to have a missing value for instance then I could use the ellipsis here. Talking about double dispatch because you surely have also seen this not implemented which is in a way also a sentinel so let's assume we have some price class and we want to be able to say one price minus the other price and there we define this thunder sub method and say yeah well if the other operand is a price we just do the subtraction and get a new price object and in any other case the return and not implemented and here again a sentinel because it could be for other methods like sub here it's pretty clear that it's going to be a float or an integer or any kind of number but for other ones it could be anything right it could also be none so we need some kind of special value here to show you how this can be be used for instance if you want to be able to say hey I have a price and minus a discount minus some coupon where I have to do the percentage math and so on we could of course make another if instance other discount or yeah coupon in here but this would be pretty bad right because we kind of mix the implementation of the two classes. So here the double dispatch helps because if the other right operand, it's always left to right in Python, is not a price, we get not implemented. So what Python does is it knows, okay, I'm going to look for the dunder R submethod and find the implementation how I calculate a new price and given some percentage coupon another built-in Sentinel that I find really useful or was at least lacking for quite some some while is in pidentic because if you have let's say a web service some rest interface and you build the models for it and let's assume this interface takes bio equals none if you want to delete the biography of a user but if you don't want to change it at all you you just remove the whole key and without this it was not really possible to to express it in a in a nice way so to say okay for this use I really want to clear the biography and if I don't pass anything I have this missing and then pidentic knows I not when I serialize this there's no bio key at all so I don't know this anyone ever came about a problem that's that there's nothing missing in pidentic yeah it's quite useful so the question is of course how do I create now my own ones in case there's no built-in one and if you looked into the default library and the center library of Python you will often see one pattern that you just say missing equals object and then again another function let's say update user this time the email and then I say okay if no email is passed if it is missing the object I defined then we just do nothing if it is none we delete it by calling some SQL on some database or otherwise we normalize it and update the email address and this works because object gets us a new unique object and everything is fine so we could do it like this and this could be even the end of the talk but so far we have seen like no modern Python at all right in modern Python we write of course type hints and here is where it gets really problematic because if we add the types here use ID int email string of course an object since yeah this is a specific object then we check for this and we let's let's say my pi run tie or pi write or whatever and it tells us well here at that point object has no attribute lower so the the type never ring so that after this branch it realizes okay now it's a nun or string doesn't work because how should it if I say I accept an object any kind of object then I could also pass another one and this branch cannot narrow it down so this is why those like easy way of doing sentinels doesn't work now you could think okay then I just define my own more specific type I call it missing type even give it a nice representation and to let it be quite lightweight I even overwrite under and I say missing is like missing type and I do a is instance check and this I could instead of repeating is instance the whole time I could define a function a generic function which uses type is which is the newer type guard in Python and tells more or less the type hinder okay if instance is true then I know it's a missing type in the other case it's no missing type and with this the type narrowing that we have just seen before will be fine. I just need to remember every time that I need to use the function is missing I can I just say if email is missing like in case of none so this would be in way but it's still not as elegant you could also think of using a literal but literals in Python yeah also don't really work sometimes as expected for instance if you define missing is magic strings and magic strings are like never a really good idea and then it will first complain because you would have need to repeat I would need to repeat this string here because literal and some variable doesn't work even if I made it like this and kept repeating this over and over again it's problematic that string and a literal string is still a string so I end up having a string again so this does not work what works is having a literal enum as to define some some sentinel value so So what do we do here? We have this enum with a value missing. We define our actual sentinel missing as the value of our defined enum. And now we define the type. So in your Python version, so I can just say type missing type equals literal. And this looks strange, but in the end, it works that I can now say update user is the email is of type string or missing type and I can say if email is missing and now the the type checker can make the narrowing and kind of in fear at that point that email is either either none or at that point a string so email dot lower will work and everything is fine so this would be one way to do it in a yeah in a type safe way so far we have seen this that it's possible it's still kind of cumbersome to write right and there's still one difference because Python gives us this syntactic sugar in a way that we can say none in the parameter type and none as value, but we have to remember that we need to type missing type in the parameters and missing as value. So we're still not 100% there to have something as compared to none. Also one more thing about pickling, serialization and clean representation, because this is also something you want to have, if you use really this object trick and it's still used in a lot of places in the Python standard library, you should be aware that if you serialize it, if you pickle missing, if you load it again, then this check that we actually want to have it for breaks, because in the end an object is serialized, a new object is created, so it's the same as copying, and the is operator is actually checking if it's an identical object. So this doesn't work. In case of this, if you use enums for sentinels, then this works because Python knows, okay, this is something unique, and when I reconstruct it, I reconstruct it as the same object, and That is why it works. One more thing, of course, if you have a stack trace, and in a stack trace you also might not want to see this strange representation of an enum, or in case of an object and some memory address, so you could also just add a Dunder representation method here to have it like this. And this gets you to almost all you want to have about a sentinel, about your custom sentinel. Of course, I'm not the first one and only one who ever thought of this. So in PEP 661, this was created like five years ago. Someone raised those points why it's bad to do sentinels with objects. and exactly the three points we just discussed. First of all, error messages are uninformative and overly verbose. The typing problems we just mentioned and that comparison fails in case of copying or unpickling. And, yeah, the person suggested to have sentinel objects as like a first-level supported object. And unfortunately, this was deferred like one year ago. It was not rejected, so maybe it will come later. But at that point, they are still thinking about this. Luckily, the typing extensions library, which is really nice to backport newer typing features from newer Python versions back to old ones, if you're stuck with an old Python version, implemented it. So how would it actually look like in a world where we just have sentinels then we could just import it we could say missing equals sentinel missing and then we can use missing here so not missing type just missing we can check for missing and it's actually also kind of easy to understand what's going on here instead of writing all this literal enum stuff that I wrote before but yeah this as I said it's not official so it only works with pyrite right now if you have the right experimental settings turned on so I would not use it in productive code but it's more to show how something like this could look like and with this with this I want to come to the conclusion so if you take anything from this talk then please be aware of the ambiguity of when working with none that depending on your context none has different meanings and also that none could be a valid value maybe also that sentinels help you to resolve ambiguity if you define your own ones of course always prefer built-in ones if you yeah if there's yeah if those are provided by the library you're using like in case of pydendic and if you then want to write your own one and the best way currently in python is to use this literal enum trick that i just showed you and with this i want to say thank Thank you, and I don't know if we have any time left for questions.
Speaker 1 [22:11]
So we have four minutes for questions. I already collected three questions. First one is, why was the PEP deferred? Do you have a background? Like, why did PEP defer the decision? Is it like, unfortunately it was deferred and that's why it's in the experimental now? Yeah.
Speaker 2 [22:27]
So there was still not clear how you do this in a in a really nice way. I think they were a little bit Especially That you have I mean that none is here Like a value and you don't write none type right if you type type none then actually none type So why would you why would you not type here none type? And this is actually a strong, so this would need to be changed in the Python interpreter. So it's not just a small, let's change something in the library, in the standard library. It's some huge changes and they are really careful to not change so much. And yeah, maybe they wait until more people complain about this. Yeah.
Speaker 1 [23:23]
So you say the solution is not optimal yet, to some extent.
Speaker 2 [23:28]
It would be a large change, and as far as I read it, they're still not sure if it really is that important to make that large change. Okay. In Python 4, yeah.
Speaker 1 [23:43]
In Python 4, okay.
Speaker 2 [23:43]
In Python 4.
Speaker 1 [23:46]
The last question, because we're really out of time, so I take the big one. So catching an error and returning none just feels like an anti-pattern. What is the advantage of a new structure here? You could just keep it as is, possibly custom exception here, and catch that outside the function. To me, it feels like a deficiency of type hints regarding exceptions.
Speaker 2 [24:08]
Yeah definitely, so in this case either you have actual result types or you throw an exception. So the example I showed you, so definitely don't do this, but you sometimes see it in the code, it was more like to show that none is sometimes used to express like there was an error before. But definitely in this example that I showed, I would also either not even catch the exception, but just let it fall through, or define some own one if it's really necessary. But yeah, use exceptions where suitable.
Speaker 1 [24:51]
Cool. Thank you, Sean.
Speaker 2 [24:52]
Thank you.
Speaker 1 [24:52]
Thank you.
