python 3.2 UnicodeEncodeError: 'charmap' codec can't encode character '\u2013' in position 9629: character maps to <undefined>

Mattias :

I'm trying to make a script that gets data out from an sqlite3 database, but I have run in to a problem.

The field in the database is of type text and the contains a html formated text. see the text below

<html>
<head>
<title>Yahoo!</title>
</head>
<body>
<style type="text/css">
html {}
.yshortcuts {border-bottom:none !important;}
.ReadMsgBody {width:100%;}
.ExternalClass{width:100%;}
</style>
<table cellpadding="0" cellspacing="0" bgcolor="#ffffff">    
<tr>
<td width="550" valign="top" align="left">

    <table cellpadding="0" cellspacing="0" width="500">
        <tr>
            <td colspan="3"><img        src="http://mail.yimg.com/nq/assets/sharedmessages/v1/us/logo.gif" width="292" height="51" style="display:block;" border="0" alt="Yahoo! Mail"></td>
        </tr>
        <tr>
            <td rowspan="3" width="1" bgcolor="#c7c4ca"></td>
            <td width="498" height="1" bgcolor="#c7c4ca"></td>
            <td rowspan="3" width="1" bgcolor="#c7c4ca"></td>
        </tr>
        <tr>
            <td width="498" valign="top" align="left">
            <table cellpadding="0" cellspacing="0">
                <tr>
                    <td width="498" bgcolor="#61399d" align="left" valign="top">
                    <table cellspacing="0" cellpadding="0"><tr><td height="24"></td></tr></table>
                    <div style="font-family:Arial, Helvetica, sans-serif;font-size:23px;line-height:27px;margin-bottom:10px;color:#ffffff;margin-left:15px;"><span style="color:#ffffff;text-decoration:none;font-weight:bold;line-height:27px;">Välkommen till Yahoo! Mail.</span></div>
                    <div style="font-family:Arial, Helvetica, sans-serif;font-size:22px;line-height:26px;margin-bottom:1px;color:#ffffff;margin-left:15px;margin-bottom:7px;margin-right:15px;">Ansluta och dela går snabbt och enkelt och är tillgängligt överallt.</div>
                    </td>
                </tr>
                <tr>
                    <td><img src="http://mail.yimg.com/nq/assets/sharedmessages/v1/all/b1.gif" width="498" height="18" style="display:block;" border="0"></td>
                </tr>
            </table>
            <table cellpadding="0" cellspacing="0" width="498">
                <tr>
                    <td width="292" valign="top">
                    <table cellpadding="0" cellspacing="0">
                        <tr>
                            <td><img src="http://mail.yimg.com/nq/assets/sharedmessages/v1/all/grad.gif" width="292" height="9" style="display:block;"></td>
                        </tr>
                        <tr>
                            <td width="292" bgcolor="#ffffff" align="left" valign="top">
                            <table cellspacing="0" cellpadding="0"><tr><td height="11"></td></tr></table>
                            <div style="margin-left:15px;">                  
                                <div style="font-family:Arial, Helvetica, sans-serif;font-size:14px;line-height:18px;color:#333333;margin-bottom:11px;font-weight:bold;">Det är lätt som en plätt att komma igång.</div>
                                <table cellpadding="0" cellspacing="0" width="267">
                                    <tr>
                                        <td width="16" align="left" valign="top"><div style="font-family:Arial, Helvetica, sans-serif;font-size:14px;line-height:16px;color:#61399d;margin-bottom:9px;font-weight:bold;">1. </div></td>
                                        <td align="left" valign="top"><div style="font-family:Arial, Helvetica, sans-serif;font-size:13px;line-height:16px;color:#61399d;margin-bottom:9px;"><a rel="nofollow" target="_blank" href="http://us-mg999.mail.yahoo.com/neo/launch?action=contacts" style="text-decoration:underline;color:#61399d;"><span>Lägg till alla dina kontakter på en plats</span></a>.</div></td>
                                    </tr>
                                    <tr>
                                        <td align="left" valign="top"><div style="font-family:Arial, Helvetica, sans-serif;font-size:14px;line-height:16px;color:#61399d;margin-bottom:9px;font-weight:bold;">2. </div></td>
                                        <td align="left" valign="top"><div style="font-family:Arial, Helvetica, sans-serif;font-size:13px;line-height:16px;color:#61399d;margin-bottom:9px;"><a rel="nofollow" target="_blank" href="http://mrd.mail.yahoo.com/themes" style="text-decoration:underline;color:#61399d;"><span>Anpassa din nya inkorg</span></a>.</div></td>
                                    </tr>
                                    <tr>
                                        <td align="left" valign="top"><div style="font-family:Arial, Helvetica, sans-serif;font-size:14px;line-height:16px;color:#61399d;margin-bottom:9px;font-weight:bold;">3. </div></td>
                                        <td align="left" valign="top"><div style="font-family:Arial, Helvetica, sans-serif;font-size:13px;line-height:16px;color:#61399d;"><a rel="nofollow" target="_blank" href="http://se.overview.mail.yahoo.com/mobile" style="text-decoration:underline;color:#61399d;"><span>Anslut överallt på dina mobila enheter</span></a>.</div></td>
                                    </tr>
                                </table>

                            </div>
                            </td>
                        </tr>
                        <tr><td height="13"></td></tr>
                    </table>
                    </td>
                    <td width="196" valign="top">
                    <table cellpadding="0" cellspacing="0">
                        <tr>
                            <td width="1" bgcolor="#fbfbfd" valign="top"><img src="http://mail.yimg.com/nq/assets/sharedmessages/v1/all/g1.gif" width="1" height="21" style="display:block;"></td>
                            <td width="1" bgcolor="#f5f6fa" valign="top"><img src="http://mail.yimg.com/nq/assets/sharedmessages/v1/all/g2.gif" width="1" height="21" style="display:block;"></td>
                            <td width="1" bgcolor="#e8eaf1" valign="top"><img src="http://mail.yimg.com/nq/assets/sharedmessages/v1/all/g3.gif" width="1" height="21" style="display:block;"></td>
                            <td width="1" bgcolor="#d4d4d4"></td>
                            <td width="186" bgcolor="#f0f0f0" align="left" valign="top">  
                            <table cellspacing="0" cellpadding="0"><tr><td height="3">   </td></tr></table>
                            <div style="margin-left:11px;">
                            <div style="font-family:Arial, Helvetica, sans-serif;font-size:13px;line-height:16px;color:#333333;margin-bottom:9px;"><b>Info för dig:</b></div>
                            <div style="font-family:Arial, Helvetica, sans-serif;font-size:12px;color:#43494e;line-height:18px;margin-bottom:10px;">
                            Yahoo!-ID och e-postadress:<br />
                            <div style="font-family:Arial, Helvetica, sans-serif;font-size:12px;color:#43494e;line-height:18px;">
                            Håll ditt konto och inställningar aktuella. <br><a rel="nofollow" target="_blank" href="https://edit.yahoo.com/config/eval_profile" style="text-decoration:underline;color:#61399d;"><span>Mitt konto</span></a> 
                            </div>
                            </div>
                            <table cellspacing="0" cellpadding="0"><tr><td height="20"></td></tr></table>
                            </td>
                            <td width="1" bgcolor="#dbdbdb"></td>
                            <td width="1" bgcolor="#ced2de"></td>
                            <td width="1" bgcolor="#dbdfed"></td>
                            <td width="1" bgcolor="#e8ebf3"></td>
                            <td width="1" bgcolor="#f3f4f9"></td>
                            <td width="1" bgcolor="#fafbfc"></td>
                        </tr>
                        <tr>
                            <td colspan="11"><img src="http://mail.yimg.com/nq/assets/sharedmessages/v1/all/b2.gif" width="196" height="8" style="display:block;" border="0"></td>
                        </tr>
                        <tr><td height="13"></td></tr>
                    </table>
                    </td>
                    <td width="10"></td>
                </tr>
            </table>
            </td>
        </tr>
        <tr>
            <td width="498" height="1" bgcolor="#c7c4ca"></td>
        </tr>
    </table>
    <table cellpadding="0" cellspacing="0" width="500">
        <tr>
            <td align="center" valign="top">
            <table cellspacing="0" cellpadding="0"><tr><td height="10"></td></tr></table>
                <div style="font-family:Arial, Helvetica, sans-serif;font-size:11px;line-height:18px;margin-bottom:10px;">
                <a rel="nofollow" target="_blank" href="http://info.yahoo.com/legal/se/yahoo/utos.html" style="text-decoration:underline;color:#61399d;">Yahoo! Villkor för användning</a>&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;<a rel="nofollow" target="_blank" href="http://info.yahoo.com/legal/se/yahoo/mail/atos.html" style="text-decoration:underline;color:#61399d;">Yahoo! Mail –Villkor för användning</a>&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;<a rel="nofollow" target="_blank" href="http://info.yahoo.com/privacy/se/yahoo/details.html" style="text-decoration:underline;color:#61399d;">Yahoo! Sekretesspolicy</a>
                </div>
            </td>
        </tr>
        <tr>
            <td align="left" valign="top">
                <div style="font-family:Arial, Helvetica, sans-serif;font-size:11px;line-height:14px;color:#545454;margin-left:16px;margin-right:14px;">Var god svara inte på detta meddelande. Detta är ett servicemeddelande som rör din användning av Yahoo! Mail. Om du vill veta mer om Yahoo!s användning av personlig information, inklusive användning av webb-beacons i HTML-baserad e-post, kan du läsa vår Yahoo! Sekretesspolicy. Yahoo!s adress är 701 First Avenue, Sunnyvale, CA 94089, USA.<br /><br />RefID: lp-1037111</div>
            </td>
        </tr>
    </table>





    </td>
</tr>
</table>
<img width="1" height="1" src="http://pclick.internal.yahoo.com/p/s=2143684696">
</body>
</html>`

and the python code that try to extract the data is as follows.

>>> import sqlite3
>>> conn = sqlite3.connect('C:/temp/Mobils/export/com.yahoo.mobile.client.android.mail/databases/mail.db')
>>> c = conn.cursor()
>>> conn.row_factory=sqlite3.Row
>>> c.execute('select body from messages_1 where _id=7')
<sqlite3.Cursor object at 0x0000000001FB78F0>
>>> r = c.fetchone()
>>> r.keys()
['body']
>>> print(r['body'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python32\lib\encodings\cp850.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2013' in position 9629: character maps to <undefined>
>>>

Does anybody have any idea of how to print/write this to a file. Yes I know that this is printed to stdout, but I get the same UnicodeEncodeError when I try to write to a file. I tried both write method of a file object and print(r['body'], file=f).

Mark Ransom :

When you open the file you want to write to, open it with a specific encoding that can handle all the characters.

with open('filename', 'w', encoding='utf-8') as f:
    print(r['body'], file=f)

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

python 3.2 UnicodeEncodeError: 'charmap' codec can't encode character '\u2013' in position 9629: character maps to <undefined>

UnicodeEncodeError: 'charmap' codec can't encode character '\u2010': character maps to <undefined>

Python : UnicodeEncodeError: 'charmap' codec can't encode character '\u2190' in position 2936

python charmap codec can't decode byte X in position Y character maps to <undefined>

Pandas.to_csv gives error 'ascii' codec can't encode character u'\u2013' in position 8: ordinal not in range(128)

Python - UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 44: character maps to <undefined>

python3 UnicodeEncodeError: 'charmap' codec can't encode characters in position 95-98: character maps to <undefined>

Python - UnicodeEncodeError: 'charmap' codec can't encode characters in position 85-89: character maps to <undefined>

Python requests: UnicodeEncodeError: 'charmap' codec can't encode character

UnicodeEncodeError: 'charmap' codec can't encode character '\x97' in position 206: character maps to <undefined>

Python - UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1070: character maps to <undefined>

UnicodeEncodeError: 'charmap' codec can't encode character '\ufeff' in position 0: character maps to <undefined>

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 49: character maps to <undefined>

UnicodeEncodeError: 'latin-1' codec can't encode character '\u2013' (writing to PDF)

UnicodeEncodeError: 'charmap' codec can't encode character '\u010d'

'charmap' codec can't encode character '\ufb01' in position 108: character maps to <undefined>

Why I'm getting "UnicodeEncodeError: 'charmap' codec can't encode character '\u25b2' in position 84811: character maps to <undefined>" error?

UnicodeEncodeError: 'charmap' codec can't encode character (in Python 3.3)

Python eyed3 UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 17: ordinal not in range(128)

UnicodeEncodeError: 'charmap' codec can't encode character : character maps to <undefined>

UnicodeEncodeError: 'charmap' codec can't encode character '\u2080' in position 28: character maps to <undefined>

UnicodeEncodeError: 'charmap' codec can't encode character '\u2264'

Unicode Encode Error 'latin-1' codec can't encode character u'\u2013' in position 4939: ordinal not in range(256)

python 3 - UnicodeEncodeError: 'charmap' codec can't encode character (Encode so it's in a file)

Python 2 to 3 migration - UnicodeEncodeError: 'ascii' codec can't encode character ...: ordinal not in range

UnicodeEncodeError: 'latin-1' codec can't encode character '\u2013' in position 88: ordinal not in range(256)

UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f937' in position 0: character maps to <undefined>

'charmap' codec can't encode character '\u010c' in position 0: character maps to <undefined> MariaDB and SQLAlchemy

python - UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 131198: character maps to <undefined>

TOP Ranking

  1. 1

    Failed to listen on localhost:8000 (reason: Cannot assign requested address)

  2. 2

    Loopback Error: connect ECONNREFUSED 127.0.0.1:3306 (MAMP)

  3. 3

    How to import an asset in swift using Bundle.main.path() in a react-native native module

  4. 4

    pump.io port in URL

  5. 5

    Compiler error CS0246 (type or namespace not found) on using Ninject in ASP.NET vNext

  6. 6

    BigQuery - concatenate ignoring NULL

  7. 7

    ngClass error (Can't bind ngClass since it isn't a known property of div) in Angular 11.0.3

  8. 8

    ggplotly no applicable method for 'plotly_build' applied to an object of class "NULL" if statements

  9. 9

    Spring Boot JPA PostgreSQL Web App - Internal Authentication Error

  10. 10

    How to remove the extra space from right in a webview?

  11. 11

    java.lang.NullPointerException: Cannot read the array length because "<local3>" is null

  12. 12

    Jquery different data trapped from direct mousedown event and simulation via $(this).trigger('mousedown');

  13. 13

    flutter: dropdown item programmatically unselect problem

  14. 14

    How to use merge windows unallocated space into Ubuntu using GParted?

  15. 15

    Change dd-mm-yyyy date format of dataframe date column to yyyy-mm-dd

  16. 16

    Nuget add packages gives access denied errors

  17. 17

    Svchost high CPU from Microsoft.BingWeather app errors

  18. 18

    Can't pre-populate phone number and message body in SMS link on iPhones when SMS app is not running in the background

  19. 19

    12.04.3--- Dconf Editor won't show com>canonical>unity option

  20. 20

    Any way to remove trailing whitespace *FOR EDITED* lines in Eclipse [for Java]?

  21. 21

    maven-jaxb2-plugin cannot generate classes due to two declarations cause a collision in ObjectFactory class

HotTag

Archive