# UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-14: character maps to \<undefined\>

**URL:** <https://discuss.python.org/t/unicodeencodeerror-charmap-codec-cant-encode-characters-in-position-0-14-character-maps-to-undefined/12814>\
**Category:** Python Help\
**Tags:** help\
**Created:** [December 31, 2021, 2:04pm UTC](https://discuss.python.org/t/unicodeencodeerror-charmap-codec-cant-encode-characters-in-position-0-14-character-maps-to-undefined/12814 "2021-12-31T14:04:14Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![morningsashimi](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@morningsashimi](https://discuss.python.org/u/morningsashimi)\
**Post date:** [December 31, 2021, 2:04pm UTC](https://discuss.python.org/t/unicodeencodeerror-charmap-codec-cant-encode-characters-in-position-0-14-character-maps-to-undefined/12814/1 "2021-12-31T14:04:14Z")

</div>

Hi! How to solve "UnicodeEncodeError: ‘charmap’ codec can’t encode characters in position 0-14: character maps to "

for this code:  
stockList = ts.get\_stock\_basics()

---

<div class="post-metadata">

**Author:** ![steven.daprano](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steven.daprano/32/1083_2.png) [@steven.daprano](https://discuss.python.org/u/steven.daprano)\
**Post date:** [January 1, 2022, 12:30am UTC](https://discuss.python.org/t/unicodeencodeerror-charmap-codec-cant-encode-characters-in-position-0-14-character-maps-to-undefined/12814/2 "2022-01-01T00:30:16Z")

</div>

Hi Jonathan,

Sorry you have not provided enough information to solve the problem. I’m  
going to have to do a lot of guessing.

The _absolute minimum_ we would need to see would be the entire  
traceback, showing the full error starting with the line “Traceback” and  
ending with the error message. But even that may not be enough.

What is “ts”? Where did it come from? What does it do?

Why is it using the ‘charmap’ codec? Can you tell it to use a different  
encoding instead of ‘charmap’?

Are you reading data from a website? What encoding is the website using?  
If you’re unsure, it is probably using UTF-8. Can you tell the mystery  
ts object to use encoding=‘utf-8’?

You could try running your script on an Apple Mac or Linux instead of  
Windows, that could fix the problem. (The ‘charmap’ codec is only used  
on Windows.)

---

<div class="post-metadata">

**Author:** ![morningsashimi](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@morningsashimi](https://discuss.python.org/u/morningsashimi)\
**Post date:** [January 1, 2022, 6:30am UTC](https://discuss.python.org/t/unicodeencodeerror-charmap-codec-cant-encode-characters-in-position-0-14-character-maps-to-undefined/12814/3 "2022-01-01T06:30:56Z")

</div>

I tried utf-8 in the code like this before: stockList = ts.get\_stock\_basics(encoding= “utf-8”) . but gave me other error message. I don’t know how to put the encoding.

import pymysql  
import sys  
import tushare as ts

try:  
db = pymysql.connect(host=“localhost”,user=“root”,password=“cixlerui83466”,database=‘pythonstock’,port=3306)  
except:  
print(‘Error whaen Connecting to DB.’)  
sys.exit()  
cursor = db.cursor()  
stockList = ts.get\_stock\_basics()

#for code in stockList.index:

```
#try:
    #createSql='CREATE TABLE stock_'+code+'( date varchar(255) , open float, close float, high float, low float, vol int(11))'
    #cursor.execute(createSql)
#except:
    #print('Error when Creating table for:' + code)

```

db.commit()  
cursor.close()  
db.close()

Traceback (most recent call last):  
File “D:\Users\jon\_j\eclipse-workspace\HelloP8\src\CreateTable.py”, line 11, in   
stockList = ts.get\_stock\_basics()  
File “D:\Python310\lib\site-packages\tushare\stock\fundamental.py”, line 52, in get\_stock\_basics  
print("\u672c\u63a5\u53e3\u5373\u5c06\u505c\u6b62\u66f4\u65b0\uff0c\u8bf7\u5c3d\u5feb\u4f7f\u7528Pro\u7248\u63a5\u53e3\uff1ahttps://waditu.com/document/2")  
File “D:\Python310\lib\encodings\cp1252.py”, line 19, in encode  
return codecs.charmap\_encode(input,self.errors,encoding\_table)[0]  
UnicodeEncodeError: ‘charmap’ codec can’t encode characters in position 0-14: character maps to

---

<div class="post-metadata">

**Author:** ![eryksun](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/eryksun/32/697_2.png) [@eryksun](https://discuss.python.org/u/eryksun)\
**Post date:** [January 1, 2022, 7:20am UTC](https://discuss.python.org/t/unicodeencodeerror-charmap-codec-cant-encode-characters-in-position-0-14-character-maps-to-undefined/12814/4 "2022-01-01T07:20:03Z")

</div>

> [@morningsashimi](#):
>
> print(“\u672c\u63a5\u53e3\u5373\u5c06\u505c\u6b62\u66f4\u65b0\uff0c\u8bf7\u5c3d\u5feb\u4f7f\u7528Pro\u7248\u63a5\u53e3\uff1ahttps://waditu.com/document/2”)  
> File “D:\Python310\lib\encodings\cp1252.py”, line 19, in encode  
> return codecs.charmap\_encode(input,self.errors,encoding\_table)[0]  
> UnicodeEncodeError: ‘charmap’ codec can’t encode characters in position 0-14: character maps to

In Python 3.6+, output to a Windows console file uses Unicode. All other files, however, default to using the process ANSI code page, which is typically a legacy encoding. The latter includes the standard files (i.e. stdin, stdout, stderr) when they’re redirected to something other than a console, such as a disk file or pipe. In your case the encoding of `sys.stdout` is code page 1252, maybe because the output is piped to Eclipse. There are two environment variables that you can set to override the encoding of `sys.stdout`. To force just the standard files to use UTF-8 , set `PYTHONIOENCODING=utf-8`. To use UTF-8 as the default in all cases, enable UTF-8 mode via `PYTHONUTF8=1`. To configure the latter for a single running instance of Python, use the command-line option `-X utf8` instead of setting `PYTHONUTF8=1`.

---

<div class="post-metadata">

**Author:** ![steven.daprano](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steven.daprano/32/1083_2.png) [@steven.daprano](https://discuss.python.org/u/steven.daprano)\
**Post date:** [January 1, 2022, 7:39am UTC](https://discuss.python.org/t/unicodeencodeerror-charmap-codec-cant-encode-characters-in-position-0-14-character-maps-to-undefined/12814/5 "2022-01-01T07:39:08Z")

</div>

Jonathan said:

“I tried utf-8 in the code like this before: stockList =  
ts.get\_stock\_basics(encoding= “utf-8”) . but gave me other error  
message.”

Don’t be shy. Copy and paste the error message so we can see it.

> I don’t know how to put the encoding.

What does the tushare documentation say? I can only find these, which  
are all in Chinese:

> **[Tushare数据](https://tushare.pro/)**
>
> Tushare为金融数据分析提供便捷、快速的接口，与投研和量化策略无缝对接

> **[Tushare -财经数据接口包](http://tushare.org/)**
>
> Tushare为金融数据分析提供便捷、快速的接口，与pandas无缝对接

You could try Google Translate.

---

<div class="post-metadata">

**Author:** ![steven.daprano](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steven.daprano/32/1083_2.png) [@steven.daprano](https://discuss.python.org/u/steven.daprano)\
**Post date:** [January 1, 2022, 7:45am UTC](https://discuss.python.org/t/unicodeencodeerror-charmap-codec-cant-encode-characters-in-position-0-14-character-maps-to-undefined/12814/6 "2022-01-01T07:45:19Z")

</div>

Looking at the source code here:

> <https://github.com/waditu/tushare/blob/093856995af0811d3ebbe8c179b8febf4ae706f0/tushare/stock/fundamental.py#L22>

the get\_stock\_basics() function is hard-coded to use the GBK encoding:

```
text = text.decode('GBK')

```

so at this point I would say it is a bug in the tushare library.

---

<div class="post-metadata">

**Author:** ![eryksun](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/eryksun/32/697_2.png) [@eryksun](https://discuss.python.org/u/eryksun)\
**Post date:** [January 1, 2022, 9:55am UTC](https://discuss.python.org/t/unicodeencodeerror-charmap-codec-cant-encode-characters-in-position-0-14-character-maps-to-undefined/12814/7 "2022-01-01T09:55:16Z")

</div>

The `get_stock_basics()` call prints “本接口即将停止更新，请尽快使用Pro版接口：https​://waditu.com/document/2”, which translates as “This interface will stop updating, please use the Pro version interface as soon as possible: https​://waditu.com/document/2”. Here’s the source from [PyPI](https://files.pythonhosted.org/packages/4d/49/15484f4f36860dc10aadb41461488c1f9a60d6fb7ea0535572b9ac8264ca/tushare-1.2.78.tar.gz):

```auto
def get_stock_basics(date=None):
    """
        获取沪深上市公司基本情况
    Parameters
    date:日期YYYY-MM-DD，默认为上一个交易日，目前只能提供2016-08-09之后的历史数据

    Return
    --------
    DataFrame
               code,代码
               name,名称
               industry,细分行业
               area,地区
               pe,市盈率
               outstanding,流通股本
               totals,总股本(万)
               totalAssets,总资产(万)
               liquidAssets,流动资产
               fixedAssets,固定资产
               reserved,公积金
               reservedPerShare,每股公积金
               eps,每股收益
               bvps,每股净资
               pb,市净率
               timeToMarket,上市日期
    """
    print("本接口即将停止更新，请尽快使用Pro版接口：https://tushare.pro/document/2")
    wdate = du.last_tddate() if date is None else date
    wdate = wdate.replace('-', '')
    if wdate < '20160809':
        return None
    datepre = '' if date is None else wdate[0:4] + wdate[4:6] + '/'
    request = Request(ct.ALL_STOCK_BASICS_FILE%(datepre, '' if date is None else wdate))
    text = urlopen(request, timeout=10).read()
    text = text.decode('GBK')
    text = text.replace('--', '')
    df = pd.read_csv(StringIO(text), dtype={'code':'object'})
    df = df.set_index('code')
    return df

```

The hard-coded GBK `decode()` in the source is unrelated to the encoding error. One can reasonably assume that this is a known encoding for the text on the website that’s being accessed via `urlopen()`.

The problem is just the `print()` call and the encoding of `sys.stdout`. The Chinese text that it tries to print can’t be encoded with code page 1252, which is an 8-bit encoding for Western European alphabets. Refer to my previous post for a few simple ways to force `sys.stdout` to use UTF-8.

---

<div class="post-metadata">

**Author:** ![steven.daprano](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steven.daprano/32/1083_2.png) [@steven.daprano](https://discuss.python.org/u/steven.daprano)\
**Post date:** [January 1, 2022, 2:33pm UTC](https://discuss.python.org/t/unicodeencodeerror-charmap-codec-cant-encode-characters-in-position-0-14-character-maps-to-undefined/12814/8 "2022-01-01T14:33:07Z")

</div>

Nicely spotted. Thanks Eryk Sun.