Keras.Predict into Dataframe

[Finpyth]

Hallo.

i'm working with multiple machine learning models.
I want to have all my predictions into a dataframe. The following is working for my other ml scripts(sklearn), expect for this one where i use Kera library.
It return the error: 'Exception: Data must be 1-dimensional'
My DataFrame for all my features is called 'feature_list' and all my targets is 'target_list'

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import statsmodels.api as sm
import keras 
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense
from sklearn.metrics import r2_score

period = df['2011-01-01':'2015-01-01']
X = np.array(period[feature_list])
predictions = pd.DataFrame()
for i in target_list:
    y = np.array(period[i])
    split = sm.add_constant(X)
    train_size = int(0.75*y.shape[0])

    train_features = X[:train_size]
    train_targets = y[:train_size]
    test_features = X[train_size:]
    test_targets = y[train_size:]

    model = Sequential()
    model.add(Dense(5,input_dim=train_features.shape[1],activation='relu'))
    model.add(Dense(5,activation='relu'))
    model.add(Dense(1,activation='linear'))
    model.compile(optimizer='adam', loss='mse')
    model.fit(X, y, epochs=5, verbose=0)
    print(i)
    predictions[i] = model.predict(X[train_size:])

if i remove '[i]' in 'predictions[i] = model.predict(X[train_size:])' in return some values, so i think the mistake is in the compiling in some way.

So how do i compile all my predictions into a dataframe for all my targets?

Best regards Fynpyth from Denmark.

[jefsummers]

Code is incomplete so cannot test, but I think line 32 should be

    predictions = model.predict(train_features)

---

and when you are ready to test change to test_features

---

and when you are ready to test change to test_features

[Finpyth]

Code is incomplete so cannot test, but I think line 32 should be

    predictions = model.predict(train_features)

---

and when you are ready to test change to test_features

---

and when you are ready to test change to test_features

Thanks for answer.
The "train_features" match the "X[train_size:]" and the code is running for both if i remove '[i]' in the predictions[i], but it's only return one columns of predictions, and i need for all my 40 targets for a trading strategy.

I'm sorry i can't upload all my code, but it is for my thesis and my thesis can be flaged as plagiarism if i upload it 1 to 1.

sincerly.

[jefsummers]

The colon is on the opposite side in train_features vs X[train_size:], so those are not the same thing.
After correcting that, do

predictions.shape()

and post

[Finpyth]

The colon is on the opposite side in train_features vs X[train_size:], so those are not the same thing.
After correcting that, do

predictions.shape()

and post

You are right about X[train_size:] and train_features is not the same. My bad.

I tried your solution but it did not work, but i kinda found a strange 2-step solution.
maybe you know why it works?

predictions = pd.DataFrame()
for i in target_list:
   y = np.array(period[i])
   split = sm.add_constant(X)
   train_size = int(0.75*y.shape[0])

   train_features = X[:train_size]
   train_targets = y[:train_size]
   test_features = X[train_size:]
   test_targets = y[train_size:]

   model = Sequential()
   model.add(Dense(50,input_dim=train_features.shape[1],activation='relu'))
   model.add(Dense(50,activation='relu'))
   model.add(Dense(1,activation='linear'))
   model.compile(optimizer='adam', loss='mse')
   model.fit(X, y, epochs=1, verbose=1)
   print(i)
   predictions = pd.DataFrame(model.predict(X[train_size:]), columns=[i])

then i remove "predictions = pd.DataFrame()" and add [i] in the last line and get this:

for i in target_list:
   y = np.array(period[i])
   split = sm.add_constant(X)
   train_size = int(0.75*y.shape[0])

   train_features = X[:train_size]
   train_targets = y[:train_size]
   test_features = X[train_size:]
   test_targets = y[train_size:]

   model = Sequential()
   model.add(Dense(50,input_dim=train_features.shape[1],activation='relu'))
   model.add(Dense(50,activation='relu'))
   model.add(Dense(1,activation='linear'))
   model.compile(optimizer='adam', loss='mse')
   model.fit(X, y, epochs=1, verbose=1)
   print(i)
   predictions[i] = pd.DataFrame(model.predict(X[train_size:]), columns=[i])

and now it returns the dataframe, "predictions" with all the predictions for all my targets. If i don't edit the script, it doesn't run

best ragards

[jefsummers]

The line you remove actually should not do anything - you assign predictions to a blank dataframe and then reassign it in the for loop. Adding the [i] then makes predictions a list of dataframes. I'd be interested if you used

type(predictions)

after your loop to verify that...

[Finpyth]

The line you remove actually should not do anything - you assign predictions to a blank dataframe and then reassign it in the for loop. Adding the [i] then makes predictions a list of dataframes. I'd be interested if you used

type(predictions)

after your loop to verify that...

That was also what I was thinking.

It return nothing for the "type(predictions)"
In the "variable explorer" in spyder, it say it is a DataFrame.

[jefsummers]

May be running into 2 variables - predictions and predictions[].

Idea - go back and in line 1 put

predictions[]

Then, in your prediction line use

predictions.append(pd.DataFrame(model.predict(X[train_size:]), columns=[i]))

Then you should have an array of dataframes. You can examine each one, and you can combine them into a single frame later.

[Finpyth]

May be running into 2 variables - predictions and predictions[].

Idea - go back and in line 1 put

predictions[]

Then, in your prediction line use

predictions.append(pd.DataFrame(model.predict(X[train_size:]), columns=[i]))

Then you should have an array of dataframes. You can examine each one, and you can combine them into a single frame later.

When i run

predictions[]

in the first line, it return an "invalid syntax".

[jefsummers]

predictions = []. Sorry, forgot the equal sign
It's just to define predictions as a list so you can use append