# How to replace \`nan\` without external libraries when strings are present?

**URL:** <https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190>\
**Category:** Python Help\
**Tags:** help\
**Created:** [July 2, 2024, 1:37pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190 "2024-07-02T13:37:20Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![ryan-duve](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ryan-duve/32/28362_2.png) [@ryan-duve](https://discuss.python.org/u/ryan-duve)\
**Post date:** [July 2, 2024, 1:37pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/1 "2024-07-02T13:37:20Z")

</div>

Given a dictionary whose keys are strings and values are a combination of strings, floats, integers and `math.nan`

```python
import math

d = {"name": "Ryan", "value": 123.456, "other_value": math.nan}

```

I am trying to convert the `nan` values to an empty string like this:

```python
{'name': 'Ryan', 'value': 123.456, 'other_value': ''}

```

The following gives an error:

```python
{k: "" if math.isnan(v) else v for k, v in d.items()}
TypeError: must be real number, not str

```

What is the best way to check for `nan` that doesn’t break on non-numeric types? The only options I could come up with are:

1. Using an `if` statement that checks if each value is a float:  
`{k: v if not (isinstance(v, float) and math.isnan(v)) else '' for k,v in d.items()}`
2. Utilizing the `json` module:  
`json.loads(json.dumps(d), parse_constant={"NaN":""}.get)`
3. Relying on every other value being equal to itself:  
`{k: v if v==v else "" for k,v in d.items()}`

There are ways to do this with dependencies like Numpy/Pandas, but they are not available at the time this code is executed. I am wondering whether there is a standard, idiomatic way to do this with just Python and the standard library before I pick from one of the above.

---

<div class="post-metadata">

**Author:** ![franklinvp](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/franklinvp/32/11142_2.png) [@franklinvp](https://discuss.python.org/u/franklinvp)\
**Post date:** [July 2, 2024, 1:46pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/2 "2024-07-02T13:46:28Z")

</div>

Is it ensured that it is only going to be `math.nan` and never, for example, a `float('nan')` or a NaN created in some other way?

---

<div class="post-metadata">

**Author:** ![JamesParrott](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jamesparrott/32/10534_2.png) [@JamesParrott](https://discuss.python.org/u/JamesParrott)\
**Post date:** [July 2, 2024, 1:47pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/3 "2024-07-02T13:47:56Z")

</div>

```python
{k: "" if isinstance(v, float) and math.isnan(v) else v 
 for k, v in d.items()
}

```

---

<div class="post-metadata">

**Author:** ![ryan-duve](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ryan-duve/32/28362_2.png) [@ryan-duve](https://discuss.python.org/u/ryan-duve)\
**Post date:** [July 2, 2024, 1:52pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/4 "2024-07-02T13:52:17Z")

</div>

> [@franklinvp](#):
>
> Is it ensured that it is only going to be `math.nan` and never, for example, a `float('nan')` or a NaN created in some other way?

@franklinvp – It is currently a mixture. They are made with

- `float('nan')`
- `json.loads('{"key": NaN}')`
- `math.nan`

However, we control this and can standardize to make it simpler if necessary.

---

<div class="post-metadata">

**Author:** ![JamesParrott](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jamesparrott/32/10534_2.png) [@JamesParrott](https://discuss.python.org/u/JamesParrott)\
**Post date:** [July 2, 2024, 1:58pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/5 "2024-07-02T13:58:13Z")

</div>

Any nan that is not handled correctly by `math.isnan` is a bug.

```python
>>> math.isnan(math.nan)
True
>>> math.isnan(float('nan'))
True
>>> import json
>>> math.isnan(json.loads('{"key": NaN}')['key'])
True

```

---

<div class="post-metadata">

**Author:** ![franklinvp](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/franklinvp/32/11142_2.png) [@franklinvp](https://discuss.python.org/u/franklinvp)\
**Post date:** [July 2, 2024, 2:04pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/7 "2024-07-02T14:04:17Z")

</div>

The set of NaNs is larger than the one value of `math.nan`.

It is a different question if they want to replace all NaNs than if they one to replace only the `math.nan`.

---

<div class="post-metadata">

**Author:** ![JamesParrott](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jamesparrott/32/10534_2.png) [@JamesParrott](https://discuss.python.org/u/JamesParrott)\
**Post date:** [July 2, 2024, 2:22pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/8 "2024-07-02T14:22:04Z")

</div>

Sure. If only specific NaNs are wanted, then a more specific test is needed.

I was reading around the lines and took an educated guess about what OP meant to say they wanted, not what they actually said that they wanted. I.e. general data cleaning, and fewer bugs

Why would anyone want to keep one type of NaN, and not another (we already know they’re being replaced with an empty string, not a particular NaN option)?

---

<div class="post-metadata">

**Author:** ![litlighilit](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/litlighilit/32/20849_2.png) [@litlighilit](https://discuss.python.org/u/litlighilit)\
**Post date:** [July 2, 2024, 3:55pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/9 "2024-07-02T15:55:10Z")

</div>

> [@ryan-duve](#):
>
> Relying on every other value being equal to itself:  
> `{k: v if v==v else "" for k,v in d.items()}`

Using self-equality-check like `if v != v` over `if math.isnan(v)`  
is fine.

According to IEEE-754: nan != nan.

---

<div class="post-metadata">

**Author:** ![litlighilit](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/litlighilit/32/20849_2.png) [@litlighilit](https://discuss.python.org/u/litlighilit)\
**Post date:** [July 2, 2024, 4:04pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/10 "2024-07-02T16:04:22Z")

</div>

> [@franklinvp](#):
>
> The set of NaNs is larger than the one value of `math.nan`.

> [@litlighilit](#):
>
> According to IEEE-754: nan != nan.

I test for `numpy.float*`, they all follows `nan != nan`, and scipy.nan is just alias for float(‘nan’)

The only exception is `sympy.nan`, where you have to check via `not sympy.Eq(v, v)` to check if v`` is of `sympy.nan`.

* * *

> [@ryan-duve](#):
>
> I am wondering whether there is a standard, idiomatic way to do this with just Python and the standard library before I pick from one of the above.

So if you do not use sympy, then `v != v` just works fine!

---

<div class="post-metadata">

**Author:** ![franklinvp](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/franklinvp/32/11142_2.png) [@franklinvp](https://discuss.python.org/u/franklinvp)\
**Post date:** [July 2, 2024, 4:59pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/11 "2024-07-02T16:59:23Z")

</div>

You have the implication “`isnan(x)` implies `x != x`”.

The reverse implication is not true.

```python
class X:
  def __eq__ (s, o):
    return False

x = X()
x != x # is False.

```

Therefore, the two conditions are not equivalent.

You could use `x != x`, as long as you can ensure you don’t have a case in which they are not testing the same thing.

---

<div class="post-metadata">

**Author:** ![litlighilit](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/litlighilit/32/20849_2.png) [@litlighilit](https://discuss.python.org/u/litlighilit)\
**Post date:** [July 2, 2024, 5:02pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/12 "2024-07-02T17:02:43Z")

</div>

I assume ` __eq__ ` conforms to reflexivity for any other normal types 😂

> [@ryan-duve](#):
>
> Given a dictionary whose keys are strings and values are a combination of strings, floats, integers and `math.nan`

In such a dict, each element’s ` __eq__ ` conforms to it except nan.

---

<div class="post-metadata">

**Author:** ![kknechtel](https://avatars.discourse-cdn.com/v4/letter/k/e47c2d/32.png) [@kknechtel](https://discuss.python.org/u/kknechtel)\
**Post date:** [July 2, 2024, 6:00pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/13 "2024-07-02T18:00:10Z")

</div>

> [@ryan-duve](#):
>
> Using an `if` statement that checks if each value is a float:  
> `{k: v if not (isinstance(v, float) and math.isnan(v)) else '' for k,v in d.items()}`

I agree with @JamesParrott - this is the way.

---

<div class="post-metadata">

**Author:** ![Rosuav](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rosuav/32/3429_2.png) [@Rosuav](https://discuss.python.org/u/Rosuav)\
**Post date:** [July 2, 2024, 6:04pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/14 "2024-07-02T18:04:09Z")

</div>

> [@ryan-duve](#):
>
> Relying on every other value being equal to itself:  
> `{k: v if v==v else "" for k,v in d.items()}`

Yep. That’s the standard idiom for nan testing. If there are any other values that aren’t equal to themselves, deal with it when you find it; you’ll probably find that those would be considered to be nan by various other tools too. There’s really no reason to force a float check first.

---

<div class="post-metadata">

**Author:** ![micky388](https://avatars.discourse-cdn.com/v4/letter/m/cab0a1/32.png) [@micky388](https://discuss.python.org/u/micky388)\
**Post date:** [July 8, 2024, 1:48pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/15 "2024-07-08T13:48:04Z")

</div>

To replace nan without external libraries when strings are present, you can iterate through your data and check each element. Use a conditional statement to replace ‘nan’ strings with a desired value, ensuring data integrity and consistency throughout your dataset.

---

<div class="post-metadata">

**Author:** ![ryan-duve](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ryan-duve/32/28362_2.png) [@ryan-duve](https://discuss.python.org/u/ryan-duve)\
**Post date:** [July 8, 2024, 2:52pm UTC](https://discuss.python.org/t/how-to-replace-nan-without-external-libraries-when-strings-are-present/57190/16 "2024-07-08T14:52:01Z")

</div>

Hey @micky388 and welcome to Python discussions! I’m actually really new here myself, but you seem to be newer so I thought I’d offer the welcome.

I appreciate you trying to help me, but I don’t see how I can take your comment and put it to action. The original post lists three ways to do what you suggest, but you don’t seem to identify which of them (if any) to go with. Let me know if I’m missing something from your post.

* * *

For everyone else, I went with

```python
{k: "" if isinstance(v, float) and math.isnan(v) else v for k, v in d.items()}

```

because I think it’s more clear for code readers. I like how succinct `if v==v else ""` is, but I think the explicit type check is more clear to someone newer to Python (which is expected in the business context of this module). Thanks to everyone who answered!
