Error without reason

[Azdaghost]

Hello and I'm trying to make a custom language (again) and I came across this problem
I got an error but without an reason:

Error:
>>> 1
Traceback (most recent call last):
  File "main.py", line 6, in <module>
    print(lexer.get_tokens())
          ^^^^^^^^^^^^^^^^^^
  File "reader.py", line 22, in get_tokens
    elif isinstance(eval(save), int) and not isinstance(eval(self.text[char + 1]), int):
                    ^^^^^^^^^^

Full Code:

class Token:
    def __init__(self, type, value=None):
        self.type = type
        self.value = value
    def __repr__(self):
        return f'{self.type}:{self.value}' if self.value else f'{self.type}'

class Lexer:
    def __init__(self, text):
        self.text = text

    def get_tokens(self):
        tokens = []
        save = ''
        
        for char in self.text:
            if char == ' ' or char == '\t':
                pass
            elif char == ',':
                tokens += save
                save = ''
            elif isinstance(eval(save), int) and not isinstance(eval(self.text[char + 1]), int):
                tokens.append(Token('TT_INT', eval(save)))
            elif isinstance(eval(save), float) and (not isinstance(eval(self.text[char + 1]), int) or self.text[char + 1] == '.'):
                tokens.append(Token('TT_FLOAT', eval(save)))
            elif isinstance(eval(save), bool):
                tokens.append(Token('TT_BOOL', eval(save)))
            elif char == '+':
                tokens.append(Token('TT_PLUS'))
            elif char == '-':
                tokens.append(Token('TT_MINUS'))
            elif char == '*':
                tokens.append(Token('TT_MULTI'))
            elif char == '/':
                tokens.append(Token('TT_DIVIDE'))
            elif char == '$':
                tokens.append(Token('TT_VARIABLE'))
            else:
                save += char
        return tokens

[buran]

There is no such thing "error without reason".
Right now you just show definition of two classes and not the code where you use them

[Azdaghost]

There is no such thing "error without reason".
Right now you just show definition of two classes and not the code where you use them

Here is the code where I'm using them:

import reader
lexer = reader.Lexer(None)

while True:
    lexer.text = input('>>> ')
    lexer.get_tokens()

[deanhystad]

You need to find out why your python interpreter (trinket I believe) is not reporting errors. You post a lot of code that you say is "without errors" but is full of errors that should've been reported.

When I run, python reports a syntax error here. The code is correct, but the data is not. save =="", but "" is not a valid python statement

elif isinstance(eval(save), int) and not isinstance(eval(self.text[char + 1]), int):

If you fix that error your code still won't parse int or float tokens using your approach. For example, you cannot look ahead using self.text[char+1] because char is not an index into text.

Parsing is difficult. You should read about it before writing code. This is a nice, short discussion about parsing.

https://tomassetti.me/parsing-in-python/

I wrote a simple calculator using the python ast module (abstract syntax tree).

https://python-forum.io/thread-44062-pos...#pid184020

As written, Lexar should be a function, not a class. It only has one instance variable (text), and that should be passed as an argument to get_tokens().

[RogereK]

Ah, I found the issue.

Here’s the main error:

elif isinstance(eval(save), int) and not isinstance(eval(self.text[char + 1]), int):

You’re using eval(save) and also self.text[char + 1]. But char in your loop is the character itself, not the index of the character. So self.text[char + 1] is incorrect — you’re treating char as an index, but it’s a character like '1' or 'a'.

This line is causing a runtime error because self.text[char + 1] is trying to access the string at position '1' + 1, which makes no sense.

So, here's your fixed get_tokens():

def get_tokens(self):
    tokens = []
    save = ''
    
    for i in range(len(self.text)):
        char = self.text[i]
        next_char = self.text[i + 1] if i + 1 < len(self.text) else None
        
        if char in ' \t':
            pass
        elif char == ',':
            if save:
                try:
                    tokens.append(Token('TT_INT', int(save)))
                except ValueError:
                    try:
                        tokens.append(Token('TT_FLOAT', float(save)))
                    except ValueError:
                        tokens.append(Token('TT_VAR', save))
                save = ''
        elif char.isdigit() or char == '.':
            save += char
            if next_char is None or (not next_char.isdigit() and next_char != '.'):
                if '.' in save:
                    tokens.append(Token('TT_FLOAT', float(save)))
                else:
                    tokens.append(Token('TT_INT', int(save)))
                save = ''
        elif char in '+-*/$':
            if save:
                tokens.append(Token('TT_VAR', save))
                save = ''
            if char == '+':
                tokens.append(Token('TT_PLUS'))
            elif char == '-':
                tokens.append(Token('TT_MINUS'))
            elif char == '*':
                tokens.append(Token('TT_MULTI'))
            elif char == '/':
                tokens.append(Token('TT_DIVIDE'))
            elif char == '$':
                tokens.append(Token('TT_VARIABLE'))
        else:
            save += char

    if save:
        tokens.append(Token('TT_VAR', save))
    return tokens