Genetic Algorithm Tetris Python not improving

[Fanto88]

I'm trying to create an AI using a Neural Network a Genetic Algorithm to learn how to play tetris, but it looks like something is wrong because, even after 20 generations, i can't see any improvement.

The code of the neural network is this:

import numpy
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'  # or any {'0', '1', '2'}
from keras.models import Sequential
from keras.layers import Dense
import configuration


class NeuralNetwork(object):
    def __init__(self, weights):
        # define the keras model
        self.model = Sequential()
        self.model.add(Dense(configuration.NEURONS_HIDDEN_1, input_dim=configuration.INPUT, activation='relu', use_bias=False))
        self.model.add(Dense(configuration.OUTPUT, activation='softmax', use_bias=False))

        x = weights[0:configuration.INPUT * configuration.NEURONS_HIDDEN_1].reshape(configuration.INPUT,configuration.NEURONS_HIDDEN_1)
        self.model.layers[0].set_weights([x,])

        x = weights[configuration.INPUT * configuration.NEURONS_HIDDEN_1 :].reshape(configuration.NEURONS_HIDDEN_1,
                                                                                    configuration.OUTPUT)
        self.model.layers[1].set_weights([x,])

    def update_parameters(self, vision):
        """Update all the input values of the Neural Network."""
        self.__input_values = vision.reshape(-1, configuration.INPUT)

    def get_action(self):
        predictions = self.model.predict(self.__input_values)
        return numpy.argmax(numpy.array(predictions))

Where the configuration for those parameters are those one:

    # NEURAL NETWORK
    INPUT = 234
    NEURONS_HIDDEN_1 = 144
    OUTPUT = 5
    NUMBER_WEIGHTS = INPUT * NEURONS_HIDDEN_1 + NEURONS_HIDDEN_1 * OUTPUT

Input are:

All the board where 0 means block free and 1 means the block is already occupied. The board is 22x10 so those are 220 inputs already.
7 input are for the piece the AI is actually using since you can have 7 different pieces.
4 are for the rotation of the piece
2 are for the X coordinate and Y coordinate.

The output are 5

    action = self.get_action_from_nn()
            if action == 0:
                self.shape.move_piece(1)
            if action == 1:
                self.shape.move_piece(-1)
            if action == 2:
                self.shape.drop()
                fast_piece_multiplier = configuration.points['fast_piece_multiplier']
            if action == 3:
                self.shape.rotate()

The last output is "doing nothing"

The genetic algorithm parameters are like this:

   # GENETIC ALGORITHM
    NUMBER_OF_POPULATION = 1000
    NUMBER_OF_GENERATION = 200
    NUMBER_PARENTS_CROSSOVER = 50
    MUTATION_PERCENTAGE = 0.05

The fitness function is this one:

  alfa = -0.510066
    beta = 0.760666
    charlie = -0.35663
    delta = -0.184483

    score = alfa * (self.aggregate_height()) + beta * self.total_cleared + charlie * self.holes() + delta * self.bumpiness()
    return score

Where

`self.aggregate_height()`

calculate the total height of each column inside the board,

`self.total_cleared` 

are how many rows the AI cleared,

`self.holes()` 

calculate how many holes are inside the board and

`self.bumpiness()` 

calculate the difference in height between each pair of columns.

This is not my fitness function, i tried with a lot of different ones and found this one inside a blog where another tetris project was discussed and that was the Fitness function the guy used.

This is the genetic algorithm implementation:

import multiprocessing
import random
from multiprocessing import Process
import numpy
import configuration
from game import Game


def thread_function(procnum, tetris_ai, return_dict_scores, return_dict_shapes):
    game = Game(tetris_ai)
    return_dict_scores[procnum] = game.end_game_score()
    #return_dict_shapes[procnum] = game.end_game_shapes()


def calculate_fitness(population):
    """Calculate the fitness value for the entire population of the generation."""
    # First we create all_fit, an empty array, at the start. Then we proceed to start the chromosome x and we will
    # calculate his fit_value. Then we will insert, inside the all_fit array, all the fit_values for each chromosome
    # of the population and return the array. max_points is used to return all the apple positions of the game with
    # the best snake so that we can watch again the game later
    all_fit = []
    pieces_backup = []
    max_points = 0

    manager = multiprocessing.Manager()
    return_dict_scores = manager.dict()
    return_dict_shapes = manager.dict()

    index = -1
    for j in range(1):
        processes = []
        for i in range(configuration.NUMBER_OF_POPULATION):
            index += 1
            p = Process(target=thread_function, args=(index, population[index], return_dict_scores, return_dict_shapes,))
            p.start()
            processes.append(p)

        for p in processes:
            p.join()

    for value in return_dict_scores.values():
        if value > max_points:
            pieces_backup = None
            max_points = value
        all_fit.append(value)
    return all_fit, pieces_backup


def select_best_individuals(population, fitness):
    """Select X number of best parents based on their fitness score."""
    # Create an empty array of the size of number_parents_crossover and the shape of the weights
    # after that we need to create an array with x number of the best parents, where x is NUMBER_PARENTS_CROSSOVER
    # inside config file. Then we search for the fittest parents inside the fitness array created by the
    # calculate_fitness function. Numpy.where return (array([], dtype=int64),) that satisfy the query, so we
    # take only the first element of the array and then it's value (the index inside fitness array). After we have
    # the index of the element we just need to take all the weights of that chromosome and insert them as a new
    # parent. Finally we change the fitness value of the fitness value of that chromosome inside the fitness
    # array in order to have all different parents and not only the fittest
    parents = numpy.empty((configuration.NUMBER_PARENTS_CROSSOVER, population.shape[1]))
    for parent_num in range(configuration.NUMBER_PARENTS_CROSSOVER):
        index_fittest = numpy.where(fitness == numpy.max(fitness))
        index_fittest = index_fittest[0][0]
        parents[parent_num, :] = population[index_fittest, :]
        fitness[index_fittest] = -999999999999
    return parents


def crossover(parents, offspring_size):
    """Create a crossover of the best parents."""
    # First we start by creating and empty array with the size equal to offspring_size we want. The type of the
    # array is [ [Index, Weights[]] ]. We select 2 random parents and then mix their weights based on a probability
    offspring = numpy.empty(offspring_size)
    for offspring_index in range(offspring_size[0]):
        while True:
            index_parent_1 = random.randint(0, parents.shape[0] - 1)
            index_parent_2 = random.randint(0, parents.shape[0] - 1)
            if index_parent_1 != index_parent_2:
                for weight_index in range(offspring_size[1]):
                    if numpy.random.uniform(0, 1) < 0.5:
                        offspring[offspring_index, weight_index] = parents[index_parent_1, weight_index]
                    else:
                        offspring[offspring_index, weight_index] = parents[index_parent_2, weight_index]
                break
    return offspring


def mutation(offspring_crossover):
    """Mutating the offsprings generated from crossover to maintain variation in the population."""
    # We mutate each genes of a chromosome based on a probability, but the range of the weights must be -1 and 1
    for offspring_index in range(offspring_crossover.shape[0]):
        for index in range(offspring_crossover.shape[1]):
            if numpy.random.random() < configuration.MUTATION_PERCENTAGE:
                value = numpy.random.choice(numpy.arange(-1, 1, step=0.01), size=1)
                offspring_crossover[offspring_index, index] += value
                if offspring_crossover[offspring_index, index] < -1:
                    offspring_crossover[offspring_index, index] = -1
                elif offspring_crossover[offspring_index, index] > 1:
                    offspring_crossover[offspring_index, index] = -1

    return offspring_crossover

Credit: [Tetris Project][1]


[1]: https://codemyroad.wordpress.com/2013/04...ct-player/