Python Forum
Python Forum

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 562: ord
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Thread Modes
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 562: ord
ctrldan
Programmer named Tim
Posts: 17
Threads: 3
Joined: Apr 2023
Reputation: 5
#1
Question  Apr-21-2023, 03:58 PM
I admit that I'm not an expert of python, I'm working in a python version 2.6.6 (for compatibility with command like unicode). I explain why I use this version: I'm working with a program created several years ago (find the project here https://github.com/danthedeckie/OpenLP-T...-Converter). In short, the program takes a sqlite database and converts it into a proprietary database of the ProPresenter program. The problem is that I get a Unicode error that I can't resolve: [Image: braYo.png]. I attach the file, if you solve it a huge thank you.
There is also a db test if you want to try.. sqlite database trial
I was forgetting.. the original code was written 9 years ago for OSX (I tried to fit everything I could, for example I tried another expression for reload)
This program is a free license, thank you for all the replys

I think the error is on line 351 
Error:
for v in ParseLyric(song['lyrics'])])
#!/usr/bin/python
# -*- coding: utf-8 -*-

#############################################################################
#                                                                           #
#                      OpenLP to ProPresentor Converter                     #
#                    (C) 2012 Daniel Fairhead, OMNIvision                   #
#                                                                           #
#---------------------------------------------------------------------------#
#                                                                           #
# This program is free to use, change, edit, do what you want with.         #
# Please consider a donation to OMNIvision if you find it useful. :-)       #
#                                                                           #
# http://www.omnivision.om.org/                                             #
#                                                                           #
# OMNIvision is the media & events team of Operation Mobilisation (OM)      #
#                                                                           #
# http://www.om.org/                                                        #
#                                                                           #
#############################################################################

# Configuration Options:

OPENLP_DATABASE = r"C:\Users\Daniele Uni\Downloads\Telegram Desktop\songs.sqlite"
OUTPUT_DIRECTORY = "C:/Users/Daniele/Downloads"


DEFAULT_FONT = "Helvetica"
DEFAULT_COLOR = ('255', '255', '255')  # RGB, white.
BACKGROUND_COLOR = '0 0 0 1'     # RGBA black.

MAX_LINES = 4

OPEN_DIR_ON_EXIT = True  # If you want to open the dir of new files...

# Options you probably don't need to edit, but can easily enough:
MAX_VERSE_COLORS = 14               # Number of defined colors for verses
VERSE_COLORS = ['0 0 1 1',    # 0 = Blue
                '0 1 0 1',    # 1 = Green
                '1 0.5 0 1',  # 2 = Orange
                '1 0 1 1',    # 3 = Purple
                '0 1 1 1',    # 4 = Yellow
                '0.5 1 0 1',  # ...
                '1 1 0.5 1',
                '0 0 1 1',
                '0 1 0 1',
                '1 0.5 0 1',
                '1 0 1 1',
                '0 1 1 1',
                '0.5 1 0 1',
                '1 1 0.5 1']

CHORUS_COLOR = '1 0 0 1'

VERBOSE_NAMES = {'c': 'Chorus',
                 'v': 'Verse',
                 'b': 'Bridge',
                 'e': 'Ending',
                 'p': 'Pre-Chorus'}

# Cramming everything into this one file as opposed to separating it out
# into modules may be rather messy and bad practice.  I do apologise.
# My intention is to have this as a stand alone python script, to make
# it as easy as possible to distribute.

import sqlite3 as sql            # To import from OpenLP
import os                        # File loading stuff.
import sys                       # For UTF-8 settings.
try:
    # Python 2: "reload" is built-in
    reload
except NameError:
    from importlib import reload
                                 #    wasn't loaded originally.
sys.stdout.encoding
'UTF-8'  #    Oh for Py3k everywhere.

import re                        # For regexy stuff.
import xml.parsers.expat         # For Parsing OpenLP Lyrics Data

from base64 import b64encode     # For ProPresenter RTF Data blobs
from uuid import uuid1           # For ProPresenter Slide UUIDs
from datetime import datetime    # Guess.

# Apparently it's faster / better to compile regex:

__re_uni_x = re.compile(r'\\x..')      # Unicode \x form
__re_uni_u = re.compile(r'\\u....')    # Unicode \u form
__re_year = re.compile(r'^\d\d\d\d')   # Year from Copyright String
__re_xml_attr = re.compile('[<>\n"]')  # Stuff to strip from XML attributes

###################################################
#
# Short generic(ish) useful functions:
#
###################################################


def xml_attr(text):
    return (re.sub(__re_xml_attr,
            ' ',
            text.replace('&', '&amp;').replace('"', '&quot;')) if text != None else '')


def make_uuid():
    return uuid1().__str__().upper()


def Verbose_names(key):
    # Could use a collections.defaultdict, but I can't be bothered right now.
    if key in VERBOSE_NAMES:
        return VERBOSE_NAMES[key]
    else:
        return key


# Stupid BLASTED RTF.  This took me a *very* long time to figure out,
# and I'm not sure it's in any way anything like bullet proof now anyway.
# At least it works for our database, currently.
# I *never* want to work with RTF again.



def AntiUnicode(text):

    def escape_u(t):
#        """ turns a '\u####' type hexadecimal unicode escape char into
#            it's RTF '\uxxxx' decimal.
#           For use in a re.sub function as the callback. """
#        return r'\u' + unicode(int(t.group()[2:], 16)) + ' '
        return r'\u' + unicode(int(t.group()[2:], 16)) + ' '

    return re.sub(__re_uni_x,   escape_u,
               re.sub(__re_uni_u, escape_u,
                   text.encode('unicode-escape')))\
           .replace(r'\n','\\\n')


def MakeRTFBlob(text):
    return b64encode(
             '{\\rtf1\\ansi\\ansicpg1252\\cocoartf1038\\cocoasubrtf360\n{\\fonttbl\\f0\\fswiss\\fcharset0 '+DEFAULT_FONT+';}\n' 
           + '{\\colortbl;\\red'+DEFAULT_COLOR[0]+'\\green'+DEFAULT_COLOR[1]+'\\blue'+DEFAULT_COLOR[2]+';}\n'
           + '\\pard\\tx560\\tx1120\\tx1680\\tx2240\\tx2800\\tx3360\\tx3920\\tx4480\\tx5040\\tx5600\\tx6160\\tx6720\\qc\\pardirnatural\n\n'
           + '\\f0\\fs102\\fsmilli51200 \\cf1 \\expnd0\\expndtw0\\kerning0\n\\outl0\\strokewidth-20 \\strokec0 \\uc0 ' + AntiUnicode(text) + '}')


########################
#
# XML sections. (To be written to ProPresenter files)
#
########################

# This is messy, and there is no way to help it,
# besides having a separate template file.


def VerseBlock(block_name, block_type, text_sections, color='0 0 0 0'):

    def list_split_substrings_by_lines(max_lines, oldlist):
        # Very imperative, I know.  There's probably a better
        # (functional/pythonic) way to do this...
        new_list = []
        for raw_item in oldlist:
            x = 1
            new_item = ''
            for line in raw_item.splitlines():
                if x < max_lines:
                    new_item += line + '\n'
                    x += 1
                else:
                    new_item += line
                    new_list.append(new_item)
                    new_item = ''
                    x = 1
            if new_item != '':
                new_list.append(new_item)

        new_list.reverse()  # Not sure why reverse is needed...
        return new_list

    def list_split_substrings(split_by, oldlist):
        newlist = []
        for item in oldlist:
            newlist += item.split(split_by)
        return newlist

    all_sections = map(unicode.strip(
                       list_split_substrings_by_lines(MAX_LINES,  
                           list_split_substrings('\n\n', 
                               list_split_substrings ('[---]', 
                                   text_sections)))))

    return ('<RVSlideGrouping name="' + block_name + '" uuid="' + make_uuid()
           + '" color="' + color + '" serialization-array-index="0"><slides containerClass="NSMutableArray">'
           + ''.join(map(SlideBlock, all_sections))
           + '</slides></RVSlideGrouping>')

def SlideBlock(text):
    return '<RVDisplaySlide backgroundColor="' + BACKGROUND_COLOR + '" enabled="1" highlightColor="0 0 0 0" hotKey="" label="" notes="" slideType="1" sort_index="0" UUID="' + make_uuid() + '" drawingBackgroundColor="0" chordChartPath="" serialization-array-index="0"><cues containerClass="NSMutableArray"></cues><displayElements containerClass="NSMutableArray"><RVTextElement displayDelay="0" displayName="Default" locked="0" persistent="0" typeID="0" fromTemplate="1" bezelRadius="0" drawingFill="0" drawingShadow="1" drawingStroke="0" fillColor="0 0 0 0" rotation="0" source="" adjustsHeightToFit="0" verticalAlignment="0" RTFData="' + MakeRTFBlob(text) + '" revealType="0" serialization-array-index="0"><_-RVRect3D-_position x="30" y="30" z="0" width="964" height="708"></_-RVRect3D-_position><_-D-_serializedShadow containerClass="NSMutableDictionary"><NSNumber serialization-native-value="4" serialization-dictionary-key="shadowBlurRadius"></NSNumber><NSColor serialization-native-value="0 0 0 1" serialization-dictionary-key="shadowColor"></NSColor><NSMutableString serialization-native-value="{2.82843, -2.82843}" serialization-dictionary-key="shadowOffset"></NSMutableString></_-D-_serializedShadow><stroke containerClass="NSMutableDictionary"><NSColor serialization-native-value="0 0 0 0" serialization-dictionary-key="RVShapeElementStrokeColorKey"></NSColor><NSNumber serialization-native-value="0" serialization-dictionary-key="RVShapeElementStrokeWidthKey"></NSNumber></stroke></RVTextElement></displayElements><_-RVProTransitionObject-_transitionObject transitionType="-1" transitionDuration="1" motionEnabled="0" motionDuration="20" motionSpeed="100"></_-RVProTransitionObject-_transitionObject></RVDisplaySlide>'



def HeaderBlock(Name='New Song',
                Authors='',
                Artist='',
                CCLICopyRightInfo='',
                CCLILicenceNumber='',
                Publisher='',
                Notes=''):

    return '<RVPresentationDocument height="768" width="1024" versionNumber="500" docType="0" creatorCode="1349676880" lastDateUsed="' + datetime.now().strftime('%Y-%m-%dT%H:%M:%S') + '" usedCount="0" category="Song" resourcesDirectory="" backgroundColor="0 0 0 1" drawingBackgroundColor="0" notes="' + Notes + '" artist="' + Artist + '" author="' + Authors + '" album="" CCLIDisplay="0" CCLIArtistCredits="" CCLISongTitle="' + Name + '" CCLIPublisher="' + Publisher + '" CCLICopyrightInfo="' + CCLICopyRightInfo + '" CCLILicenseNumber="' + CCLILicenceNumber + '" chordChartPath=""><timeline timeOffSet="0" selectedMediaTrackIndex="0" unitOfMeasure="60" duration="0" loop="0"><timeCues containerClass="NSMutableArray"></timeCues><mediaTracks containerClass="NSMutableArray"></mediaTracks></timeline><bibleReference containerClass="NSMutableDictionary"></bibleReference><_-RVProTransitionObject-_transitionObject transitionType="-1" transitionDuration="1" motionEnabled="0" motionDuration="20" motionSpeed="100"></_-RVProTransitionObject-_transitionObject><groups containerClass="NSMutableArray">'


#def FooterBlock():
FooterBlock = '</groups><arrangements containerClass="NSMutableArray"></arrangements></RVPresentationDocument>'



###########
#
# (OpenLP) Lyrics XML Parsing
#
###########

def ParseLyric(text):
    current_verses = []

    def _element(name, attrs):
        if name == 'verse':
            current_verses.append(attrs)
            current_verses[-1]['text'] = []

    def _chardata(data):
        current_verses[-1]['text'].append(data)
        
    parser = xml.parsers.expat.ParserCreate()
    parser.StartElementHandler = _element
    parser.CharacterDataHandler = _chardata
    parser.buffer_text = True

    parser.Parse(text,True)

    current_verses.reverse() #Not really sure why we need this...

    return current_verses

#######
#
# Database functions:
#
#######

def filterbyfield(id,table,field='id'):
    return [row for row in table if row[field] == id]


###################################################
#
# Actually do stuff:
#
###################################################


def main():
    # First load the data from the OpenLP database:

    try:
        con = None

        # Fetch all the data first. Gets it in memory to use, rather than
        # loads of SQLlite queries. This seems to be faster, with a little 
        # profiling.  If re-writing it as loads of sqlite queries
        # works better for you, I'm cool with that too.

        print ("OpenLP to Pro-Presenter 5 converter.\n")
        print ("Loading Database:\n  "
                + os.path.expanduser(OPENLP_DATABASE) + "\n")

        con = sql.connect(os.path.expanduser(OPENLP_DATABASE))
        con.row_factory = sql.Row

        cur = con.cursor()
        cur.execute('SELECT id, title, ccli_number, copyright, comments, lyrics FROM songs')
        songs = cur.fetchall()
        
        cur.execute('SELECT id, display_name FROM authors')
        authors = dict()
        for author in cur.fetchall():
            authors[author['id']] = author['display_name']

        cur.execute('SELECT song_id, author_id FROM authors_songs')
        authors_songs = cur.fetchall()

        print ('  Cool.  ' + str(len(songs)) + ' songs loaded.\n' );

        # Database helper functions:
        
        def get_song_authornames(song_id):
            return ' &amp; '.join(
                [authors[id] for id in 
                    [row['author_id'] for row in authors_songs 
                        if row['song_id'] == song_id]])

    except:

        print ("Sorry - There was a problem loading the OpenLP Database.\n" +
              "(" + OPENLP_DATABASE + ")\n" +
              "Maybe OpenLP isn't set up on this user?\n\n" +
              "If you know where the database is, you can edit the path at \n" +
              "the top of this script and set it manually.")
        exit()


    # Now go through songs and output the files.

    print ("And writing the new files to\n  " + OUTPUT_DIRECTORY + "\n")

    for song in songs:

        song_authors = get_song_authornames(song['id'])

        # Find the copyright year (this would be briefer in perl...)

        get_year = re.match(__re_year, xml_attr(song['copyright']))

        if get_year != None:
            copyright_year = get_year.group()
            copyright = song['copyright'][4:].strip()
        else:
            copyright_year = ''
            copyright = ''


        # Prepare Header Block to write:

        to_write = ( HeaderBlock(Name          = xml_attr(song['title']),
                             CCLILicenceNumber = xml_attr(song['ccli_number']),
                             Notes             = xml_attr(song['comments']),
                             CCLICopyRightInfo = xml_attr(copyright_year),
                             Publisher         = xml_attr(copyright),
                             Authors           = xml_attr(song_authors)) )
        
        # Prepare Verses to write: (funny python syntax...)

        to_write += ''.join(
            [VerseBlock(Verbose_names(v['type']) + ' ' + v['label'],
                        v['type'],
                        v['text'],
                        color = CHORUS_COLOR if v['type'] == 'c'\
                                             else VERSE_COLORS[min(MAX_VERSE_COLORS-1,int(v['label']))])
            for v in ParseLyric(song['lyrics'])])

        try:
            # Now actually write the thing.
            f = open(OUTPUT_DIRECTORY + song['title'].replace('/','') + '.pro5','w')
            f.write ( to_write + FooterBlock )
            f.close()
        except:
            print ('Oh dear. Something went wrong with writing "' + song['title'] + '".\nSorry.\n\n' +
                   'Maybe it\'s a file-write permissions issue?\n' +
                   'You could try changing where this script is writing to, it\'s the line\n' +
                   '  OUTPUT_DIRECTORY="' + OUTPUT_DIRECTORY + '"\n' +
                   'that will need to be changed.' )

            try:
                f.close()
            except:
                pass

            exit();

    print ("Finished.")
    if OPEN_DIR_ON_EXIT:
        os.system('open "' + OUTPUT_DIRECTORY + '"')

if __name__ == "__main__":
    main()
Find
Reply
Larz60+
aetate et sapientia
Super Moderators
Posts: 12,137
Threads: 496
Joined: Sep 2016
Reputation: 452
#2
Apr-21-2023, 08:37 PM
That code is from 2012, so written for a legacy 2.
Version 3.6.o was not released until Dec. 23, 2016.
This was a major update, and there were many changes,
see: https://docs.python.org/3/howto/pyportin...o-python-3 for the Unicode differences.
Find
Reply
ctrldan
Programmer named Tim
Posts: 17
Threads: 3
Joined: Apr 2023
Reputation: 5
#3
Apr-22-2023, 08:53 AM
(Apr-21-2023, 08:37 PM)Larz60+ Wrote: That code is from 2012, so written for a legacy 2.
Version 3.6.o was not released until Dec. 23, 2016.
This was a major update, and there were many changes,
see: https://docs.python.org/3/howto/pyportin...o-python-3 for the Unicode differences.

Infact I'm working in vers. 2.6.6 that was released on 2010, like the creator of the code.
Find
Reply
snippsat

Administrators
Posts: 7,431
Threads: 125
Joined: Sep 2016
Reputation: 511
#4
Apr-22-2023, 04:17 PM (This post was last modified: Apr-22-2023, 04:18 PM by snippsat.)
(Apr-22-2023, 08:53 AM)ctrldan Wrote: Infact I'm working in vers. 2.6.6 that was released on 2010, like the creator of the code.
Could try a newer version like eg Python 3.10 or 11.
I took a quick look at did some changes to make it work it Python 3.10,but will not open songs.sqlite.
Error:
    con = sql.connect(OPENLP_DATABASE)
sqlite3.OperationalError: unable to open database file
So not not sure why i did not work,maybe need old sqlite version.

For you error that you have now so is it using ASCII encoding and that will not work.
The old trix used for Python 2 dos not help,example from code.
# -*- coding: utf-8 -*-
import sys                       # For UTF-8 settings.
reload(sys)                      # Stupid hack to re-apply UTF-8 if it
                                 # wasn't loaded originally.
sys.setdefaultencoding('utf-8')  # Oh for Py3k everywhere.
The problem is in this line:
parser.Parse(text,True)
So the parser xml.parsers.expat is using wrong encoding .
Try change to.
parser = xml.parsers.expat.ParserCreate(encoding='utf-8')
The code in whole is written not so good at all,just saying Undecided
Find
Reply
ctrldan
Programmer named Tim
Posts: 17
Threads: 3
Joined: Apr 2023
Reputation: 5
#5
Apr-22-2023, 07:29 PM
(Apr-22-2023, 04:17 PM)snippsat Wrote:
(Apr-22-2023, 08:53 AM)ctrldan Wrote: Infact I'm working in vers. 2.6.6 that was released on 2010, like the creator of the code.
Could try a newer version like eg Python 3.10 or 11.
I took a quick look at did some changes to make it work it Python 3.10,but will not open songs.sqlite.
Error:
    con = sql.connect(OPENLP_DATABASE)
sqlite3.OperationalError: unable to open database file
So not not sure why i did not work,maybe need old sqlite version.

For you error that you have now so is it using ASCII encoding and that will not work.
The old trix used for Python 2 dos not help,example from code.
# -*- coding: utf-8 -*-
import sys                       # For UTF-8 settings.
reload(sys)                      # Stupid hack to re-apply UTF-8 if it
                                 # wasn't loaded originally.
sys.setdefaultencoding('utf-8')  # Oh for Py3k everywhere.
The problem is in this line:
parser.Parse(text,True)
So the parser xml.parsers.expat is using wrong encoding .
Try change to.
parser = xml.parsers.expat.ParserCreate(encoding='utf-8')
The code in whole is written not so good at all,just saying Undecided

First of all, thank for you reply, I tried to edit parser and the error was the same, I repeat that I'm a begginer on python and for this I will do beginner considerations Big Grin , I tried to remove 
parser.Parse(text,True)
and now the program give antoher problem, python ask me to kill the program and without kill there is this message:

image with error

How could I change 
parser.Parse(text,True)
? Which advantage could I obtain if I use python3.10 and how I can overpass unicode problem for map(unicode.strip... line 187?

now the code is: 
#!/usr/bin/python
# -*- coding: utf-8 -*-

#############################################################################
#                                                                           #
#                      OpenLP to ProPresentor Converter                     #
#                    (C) 2012 Daniel Fairhead, OMNIvision                   #
#                                                                           #
#---------------------------------------------------------------------------#
#                                                                           #
# This program is free to use, change, edit, do what you want with.         #
# Please consider a donation to OMNIvision if you find it useful. :-)       #
#                                                                           #
# http://www.omnivision.om.org/                                             #
#                                                                           #
# OMNIvision is the media & events team of Operation Mobilisation (OM)      #
#                                                                           #
# http://www.om.org/                                                        #
#                                                                           #
#############################################################################

# Configuration Options:

OPENLP_DATABASE = r"C:\Users\Daniele Uni\Downloads\Telegram Desktop\songs.sqlite"
OUTPUT_DIRECTORY = "C:/Users/Daniele/Downloads"


DEFAULT_FONT = "Helvetica"
DEFAULT_COLOR = ('255', '255', '255')  # RGB, white.
BACKGROUND_COLOR = '0 0 0 1'     # RGBA black.

MAX_LINES = 4

OPEN_DIR_ON_EXIT = True  # If you want to open the dir of new files...

# Options you probably don't need to edit, but can easily enough:
MAX_VERSE_COLORS = 14               # Number of defined colors for verses
VERSE_COLORS = ['0 0 1 1',    # 0 = Blue
                '0 1 0 1',    # 1 = Green
                '1 0.5 0 1',  # 2 = Orange
                '1 0 1 1',    # 3 = Purple
                '0 1 1 1',    # 4 = Yellow
                '0.5 1 0 1',  # ...
                '1 1 0.5 1',
                '0 0 1 1',
                '0 1 0 1',
                '1 0.5 0 1',
                '1 0 1 1',
                '0 1 1 1',
                '0.5 1 0 1',
                '1 1 0.5 1']

CHORUS_COLOR = '1 0 0 1'

VERBOSE_NAMES = {'c': 'Chorus',
                 'v': 'Verse',
                 'b': 'Bridge',
                 'e': 'Ending',
                 'p': 'Pre-Chorus'}

# Cramming everything into this one file as opposed to separating it out
# into modules may be rather messy and bad practice.  I do apologise.
# My intention is to have this as a stand alone python script, to make
# it as easy as possible to distribute.

import sqlite3 as sql            # To import from OpenLP
import os                        # File loading stuff.
import sys                       # For UTF-8 settings.
try:
    # Python 2: "reload" is built-in
    reload
except NameError:
    from importlib import reload
                                 #    wasn't loaded originally.
sys.stdout.encoding
'UTF-8'  #    Oh for Py3k everywhere.

import re                        # For regexy stuff.
import xml.parsers.expat         # For Parsing OpenLP Lyrics Data

from base64 import b64encode     # For ProPresenter RTF Data blobs
from uuid import uuid1           # For ProPresenter Slide UUIDs
from datetime import datetime    # Guess.

# Apparently it's faster / better to compile regex:

__re_uni_x = re.compile(r'\\x..')      # Unicode \x form
__re_uni_u = re.compile(r'\\u....')    # Unicode \u form
__re_year = re.compile(r'^\d\d\d\d')   # Year from Copyright String
__re_xml_attr = re.compile('[<>\n"]')  # Stuff to strip from XML attributes

###################################################
#
# Short generic(ish) useful functions:
#
###################################################


def xml_attr(text):
    return (re.sub(__re_xml_attr,
            ' ',
            text.replace('&', '&amp;').replace('"', '&quot;')) if text != None else '')


def make_uuid():
    return uuid1().__str__().upper()


def Verbose_names(key):
    # Could use a collections.defaultdict, but I can't be bothered right now.
    if key in VERBOSE_NAMES:
        return VERBOSE_NAMES[key]
    else:
        return key


# Stupid BLASTED RTF.  This took me a *very* long time to figure out,
# and I'm not sure it's in any way anything like bullet proof now anyway.
# At least it works for our database, currently.
# I *never* want to work with RTF again.



def AntiUnicode(text):

    def escape_u(t):
#        """ turns a '\u####' type hexadecimal unicode escape char into
#            it's RTF '\uxxxx' decimal.
#           For use in a re.sub function as the callback. """
#        return r'\u' + unicode(int(t.group()[2:], 16)) + ' '
        return r'\u' + unicode(int(t.group()[2:], 16)) + ' '

    return re.sub(__re_uni_x,   escape_u,
               re.sub(__re_uni_u, escape_u,
                   text.encode('unicode-escape')))\
           .replace(r'\n','\\\n')


def MakeRTFBlob(text):
    return b64encode(
             '{\\rtf1\\ansi\\ansicpg1252\\cocoartf1038\\cocoasubrtf360\n{\\fonttbl\\f0\\fswiss\\fcharset0 '+DEFAULT_FONT+';}\n' 
           + '{\\colortbl;\\red'+DEFAULT_COLOR[0]+'\\green'+DEFAULT_COLOR[1]+'\\blue'+DEFAULT_COLOR[2]+';}\n'
           + '\\pard\\tx560\\tx1120\\tx1680\\tx2240\\tx2800\\tx3360\\tx3920\\tx4480\\tx5040\\tx5600\\tx6160\\tx6720\\qc\\pardirnatural\n\n'
           + '\\f0\\fs102\\fsmilli51200 \\cf1 \\expnd0\\expndtw0\\kerning0\n\\outl0\\strokewidth-20 \\strokec0 \\uc0 ' + AntiUnicode(text) + '}')


########################
#
# XML sections. (To be written to ProPresenter files)
#
########################

# This is messy, and there is no way to help it,
# besides having a separate template file.


def VerseBlock(block_name, block_type, text_sections, color='0 0 0 0'):

    def list_split_substrings_by_lines(max_lines, oldlist):
        # Very imperative, I know.  There's probably a better
        # (functional/pythonic) way to do this...
        new_list = []
        for raw_item in oldlist:
            x = 1
            new_item = ''
            for line in raw_item.splitlines():
                if x < max_lines:
                    new_item += line + '\n'
                    x += 1
                else:
                    new_item += line
                    new_list.append(new_item)
                    new_item = ''
                    x = 1
            if new_item != '':
                new_list.append(new_item)

        new_list.reverse()  # Not sure why reverse is needed...
        return new_list

    def list_split_substrings(split_by, oldlist):
        newlist = []
        for item in oldlist:
            newlist += item.split(split_by)
        return newlist

    all_sections = map(unicode.strip(
                       list_split_substrings_by_lines(MAX_LINES,  
                           list_split_substrings('\n\n', 
                               list_split_substrings ('[---]', 
                                   text_sections)))))

    return ('<RVSlideGrouping name="' + block_name + '" uuid="' + make_uuid()
           + '" color="' + color + '" serialization-array-index="0"><slides containerClass="NSMutableArray">'
           + ''.join(map(SlideBlock, all_sections))
           + '</slides></RVSlideGrouping>')

def SlideBlock(text):
    return '<RVDisplaySlide backgroundColor="' + BACKGROUND_COLOR + '" enabled="1" highlightColor="0 0 0 0" hotKey="" label="" notes="" slideType="1" sort_index="0" UUID="' + make_uuid() + '" drawingBackgroundColor="0" chordChartPath="" serialization-array-index="0"><cues containerClass="NSMutableArray"></cues><displayElements containerClass="NSMutableArray"><RVTextElement displayDelay="0" displayName="Default" locked="0" persistent="0" typeID="0" fromTemplate="1" bezelRadius="0" drawingFill="0" drawingShadow="1" drawingStroke="0" fillColor="0 0 0 0" rotation="0" source="" adjustsHeightToFit="0" verticalAlignment="0" RTFData="' + MakeRTFBlob(text) + '" revealType="0" serialization-array-index="0"><_-RVRect3D-_position x="30" y="30" z="0" width="964" height="708"></_-RVRect3D-_position><_-D-_serializedShadow containerClass="NSMutableDictionary"><NSNumber serialization-native-value="4" serialization-dictionary-key="shadowBlurRadius"></NSNumber><NSColor serialization-native-value="0 0 0 1" serialization-dictionary-key="shadowColor"></NSColor><NSMutableString serialization-native-value="{2.82843, -2.82843}" serialization-dictionary-key="shadowOffset"></NSMutableString></_-D-_serializedShadow><stroke containerClass="NSMutableDictionary"><NSColor serialization-native-value="0 0 0 0" serialization-dictionary-key="RVShapeElementStrokeColorKey"></NSColor><NSNumber serialization-native-value="0" serialization-dictionary-key="RVShapeElementStrokeWidthKey"></NSNumber></stroke></RVTextElement></displayElements><_-RVProTransitionObject-_transitionObject transitionType="-1" transitionDuration="1" motionEnabled="0" motionDuration="20" motionSpeed="100"></_-RVProTransitionObject-_transitionObject></RVDisplaySlide>'



def HeaderBlock(Name='New Song',
                Authors='',
                Artist='',
                CCLICopyRightInfo='',
                CCLILicenceNumber='',
                Publisher='',
                Notes=''):

    return '<RVPresentationDocument height="768" width="1024" versionNumber="500" docType="0" creatorCode="1349676880" lastDateUsed="' + datetime.now().strftime('%Y-%m-%dT%H:%M:%S') + '" usedCount="0" category="Song" resourcesDirectory="" backgroundColor="0 0 0 1" drawingBackgroundColor="0" notes="' + Notes + '" artist="' + Artist + '" author="' + Authors + '" album="" CCLIDisplay="0" CCLIArtistCredits="" CCLISongTitle="' + Name + '" CCLIPublisher="' + Publisher + '" CCLICopyrightInfo="' + CCLICopyRightInfo + '" CCLILicenseNumber="' + CCLILicenceNumber + '" chordChartPath=""><timeline timeOffSet="0" selectedMediaTrackIndex="0" unitOfMeasure="60" duration="0" loop="0"><timeCues containerClass="NSMutableArray"></timeCues><mediaTracks containerClass="NSMutableArray"></mediaTracks></timeline><bibleReference containerClass="NSMutableDictionary"></bibleReference><_-RVProTransitionObject-_transitionObject transitionType="-1" transitionDuration="1" motionEnabled="0" motionDuration="20" motionSpeed="100"></_-RVProTransitionObject-_transitionObject><groups containerClass="NSMutableArray">'


#def FooterBlock():
FooterBlock = '</groups><arrangements containerClass="NSMutableArray"></arrangements></RVPresentationDocument>'



###########
#
# (OpenLP) Lyrics XML Parsing
#
###########

def ParseLyric(text):
    current_verses = []

    def _element(name, attrs):
        if name == 'vrse':
            current_verses.append(attrs)
            current_verses[-1]['text'] = []

    def _chardata(data):
        current_verses[-1]['text'].append(data)
        
    parser = xml.parsers.expat.ParserCreate('utf-8')
    parser.StartElementHandler = _element
    parser.CharacterDataHandler = _chardata
    parser.buffer_text = True



    current_verses.reverse() #Not really sure why we need this...

    return current_verses

#######
#
# Database functions:
#
#######

def filterbyfield(id,table,field='id'):
    return [row for row in table if row[field] == id]


###################################################
#
# Actually do stuff:
#
###################################################


def main():
    # First load the data from the OpenLP database:

    try:
        con = None

        # Fetch all the data first. Gets it in memory to use, rather than
        # loads of SQLlite queries. This seems to be faster, with a little 
        # profiling.  If re-writing it as loads of sqlite queries
        # works better for you, I'm cool with that too.

        print ("OpenLP to Pro-Presenter 5 converter.\n")
        print ("Loading Database:\n  "
                + os.path.expanduser(OPENLP_DATABASE) + "\n")

        con = sql.connect(os.path.expanduser(OPENLP_DATABASE))
        con.row_factory = sql.Row

        cur = con.cursor()
        cur.execute('SELECT id, title, ccli_number, copyright, comments, lyrics FROM songs')
        songs = cur.fetchall()
        
        cur.execute('SELECT id, display_name FROM authors')
        authors = dict()
        for author in cur.fetchall():
            authors[author['id']] = author['display_name']

        cur.execute('SELECT song_id, author_id FROM authors_songs')
        authors_songs = cur.fetchall()

        print ('  Cool.  ' + str(len(songs)) + ' songs loaded.\n' );

        # Database helper functions:
        
        def get_song_authornames(song_id):
            return ' &amp; '.join(
                [authors[id] for id in 
                    [row['author_id'] for row in authors_songs 
                        if row['song_id'] == song_id]])

    except:

        print ("Sorry - There was a problem loading the OpenLP Database.\n" +
              "(" + OPENLP_DATABASE + ")\n" +
              "Maybe OpenLP isn't set up on this user?\n\n" +
              "If you know where the database is, you can edit the path at \n" +
              "the top of this script and set it manually.")
        exit()


    # Now go through songs and output the files.

    print ("And writing the new files to\n  " + OUTPUT_DIRECTORY + "\n")

    for song in songs:

        song_authors = get_song_authornames(song['id'])

        # Find the copyright year (this would be briefer in perl...)

        get_year = re.match(__re_year, xml_attr(song['copyright']))

        if get_year != None:
            copyright_year = get_year.group()
            copyright = song['copyright'][4:].strip()
        else:
            copyright_year = ''
            copyright = ''


        # Prepare Header Block to write:

        to_write = ( HeaderBlock(Name          = xml_attr(song['title']),
                             CCLILicenceNumber = xml_attr(song['ccli_number']),
                             Notes             = xml_attr(song['comments']),
                             CCLICopyRightInfo = xml_attr(copyright_year),
                             Publisher         = xml_attr(copyright),
                             Authors           = xml_attr(song_authors)) )
        
        # Prepare Verses to write: (funny python syntax...)

        to_write += ''.join(
            [VerseBlock(Verbose_names(v['type']) + ' ' + v['label'],
                        v['type'],
                        v['text'],
                        color = CHORUS_COLOR if v['type'] == 'c'\
                                             else VERSE_COLORS[min(MAX_VERSE_COLORS-1,int(v['label']))])
            for v in ParseLyric(song['lyrics'])])

        try:
            # Now actually write the thing.
            f = open(OUTPUT_DIRECTORY + song['title'].replace('/','') + '.pro5','w')
            f.write ( to_write + FooterBlock )
            f.close()
        except:
            print ('Oh dear. Something went wrong with writing "' + song['title'] + '".\nSorry.\n\n' +
                   'Maybe it\'s a file-write permissions issue?\n' +
                   'You could try changing where this script is writing to, it\'s the line\n' +
                   '  OUTPUT_DIRECTORY="' + OUTPUT_DIRECTORY + '"\n' +
                   'that will need to be changed.' )

            try:
                f.close()
            except:
                pass

            exit();

    print ("Finished.")
    if OPEN_DIR_ON_EXIT:
        os.system('open "' + OUTPUT_DIRECTORY + '"')

if __name__ == "__main__":
    main()
Find
Reply
snippsat

Administrators
Posts: 7,431
Threads: 125
Joined: Sep 2016
Reputation: 511
#6
Apr-22-2023, 08:20 PM (This post was last modified: Apr-22-2023, 08:20 PM by snippsat.)
I did a test with Python 2.7,and then could i open songs.sqlite.
So fixed the code,the run.
C:\Python27
ฮป python song.py
OpenLP to Pro-Presenter 5 converter.

Loading Database:
  songs.sqlite

  Cool.  1 songs loaded.

And writing the new files to
  C:/p/
fixed code
Run formatter as it's one line output,to get .xml file that can view.
output
Find
Reply
ctrldan
Programmer named Tim
Posts: 17
Threads: 3
Joined: Apr 2023
Reputation: 5
#7
Apr-22-2023, 09:19 PM
sorry I don't understand this affermation Confused the real songs.sqlite is with 1600 elements.
(Apr-22-2023, 08:20 PM)snippsat Wrote: Run formatter as it's one line output,to get .xml file that can view.


I tried to open the fixed code code in my python and I have the same output on IDLE of the past screenshot[quote='ctrldan' pid='168799' dateline='1682191765']
image with error , in your fixed code the program finished the task?
I appreciate your help, thank you
Find
Reply
snippsat

Administrators
Posts: 7,431
Threads: 125
Joined: Sep 2016
Reputation: 511
#8
Apr-22-2023, 09:43 PM
Try make a own folder eg C:\songs.
Then in code.
OUTPUT_DIRECTORY = "C:/songs/"
Find
Reply
ctrldan
Programmer named Tim
Posts: 17
Threads: 3
Joined: Apr 2023
Reputation: 5
#9
Apr-22-2023, 10:08 PM
I tried and there was the same situation, I tried to force the code and I changed (in line 353) 
try:
            # Now actually write the thing.
            f = open(OUTPUT_DIRECTORY + song['title'].replace('/','') + '.pro5','w')
            f.write ( to_write + FooterBlock )
            f.close()
        except:
            print ('Oh dear. Something went wrong with writing "' + song['title'] + '".\nSorry.\n\n' +
                   'Maybe it\'s a file-write permissions issue?\n' +
                   'You could try changing where this script is writing to, it\'s the line\n' +
                   '  OUTPUT_DIRECTORY="' + OUTPUT_DIRECTORY + '"\n' +
                   'that will need to be changed.' )
 
            try:
                f.close()
            except:
                pass
with
try:
            # Now actually write the thing.
            f = open(OUTPUT_DIRECTORY + song['title'].replace('/','') + '.pro5','w')
            f.write ( to_write + FooterBlock )
            f.close()
        except:
            # Now actually write the thing.
            f = open(OUTPUT_DIRECTORY + song['title'].replace('/','') + '.pro5','w')
            f.write ( to_write + FooterBlock )
            f.close()
 
            try:
                f.close()
            except:
                pass
and the hidden error when I runned the code was:
Error:
Traceback (most recent call last):
  File "C:\Users\Daniele Uni\Downloads\Telegram Desktop\testt", line 376, in <module>
    main()
  File "C:\Users\Daniele Uni\Downloads\Telegram Desktop\testt", line 361, in main
    f.write ( to_write + FooterBlock )
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 361: ordinal not in range(128)
Find
Reply
snippsat

Administrators
Posts: 7,431
Threads: 125
Joined: Sep 2016
Reputation: 511
#10
Apr-22-2023, 10:57 PM (This post was last modified: Apr-22-2023, 10:57 PM by snippsat.)
(Apr-22-2023, 09:19 PM)ctrldan Wrote: the real songs.sqlite is with 1600 elements.
If you can post the file i can take a look.
I can be problem with one or more songs as error point to song Egile รจ Degno.
Look where the song is in database with eg DB Browser for SQLite.
Find
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  ASCII-Codec in Python3 [SOLVED] AlphaInc 6 12,926 Jul-19-2025, 08:53 AM
Last Post: Gribouillis
  encode/decode to show correct country letters in a CTk combobox janeik 2 2,543 Sep-02-2023, 09:46 AM
Last Post: janeik
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd2 in position 16: invalid cont Melcu54 3 14,269 Mar-26-2023, 12:12 PM
Last Post: Gribouillis
  [SOLVED] [Debian] UnicodeEncodeError: 'ascii' codec Winfried 1 2,339 Nov-16-2022, 11:41 AM
Last Post: Winfried
  UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 34: character Melcu54 7 35,292 Sep-26-2022, 10:09 AM
Last Post: Melcu54
  UnicodeEncodeError - Dealing with Japanese Characters fioranosnake 2 5,705 Jul-07-2022, 08:43 PM
Last Post: fioranosnake
  UnicodeEncodeError: 'ascii' codec can't encode character '\xfd' in position 14: ordin Armandito 6 5,536 Apr-29-2022, 12:36 PM
Last Post: Armandito
  UnicodeEncodeError caused by print when program runs from Popen SheeppOSU 5 6,606 Jan-13-2022, 08:11 AM
Last Post: SheeppOSU
  [UnicodeEncodeError from smtplib] yoohooos 0 5,369 Sep-25-2021, 04:27 AM
Last Post: yoohooos
  [solved] unexpected character after line continuation character paul18fr 4 10,609 Jun-22-2021, 03:22 PM
Last Post: deanhystad

Forum Jump:

User Panel Messages

Log Out
My Profile
Pay your profile a visit
User Control Panel
Do some changes on your profile
My Messages
View private messages unread
Avatar
Change avatar
Signature
Change signature
Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020