Developing larger Neural Networks

[Chriskelm]

 def train():
    #random init of weights
    w1 = np.random.randn()
    w2 = np.random.randn()
    b = np.random.randn()
    
    iterations = 1000000
    learning_rate = 0.01
    costs = [] # keep costs during training, see if they go down
    
    for i in range(iterations):
        # get a random point
        ri = np.random.randint(len(data))
        point = data[ri]
        
        z = point[0] * w1 + point[1] * w2 + b
        pred = sigmoid(z) # networks prediction
        
        target = point[2]
        
        # cost for current random point
        cost = np.square(pred - target)
        
        # print the cost over all data points every 1k iters
        if i % 100 == 0:
            c = 0
            for j in range(len(data)):
                p = data[j]
                p_pred = sigmoid(w1 * p[0] + w2 * p[1] + b)
                c += np.square(p_pred - p[2])
            costs.append(c)
        
        dcost_dpred = 2 * (pred - target)
        dpred_dz = sigmoid_p(z)
        
        dz_dw1 = point[0]
        dz_dw2 = point[1]
        dz_db = 1
        
        dcost_dz = dcost_dpred * dpred_dz
        
        dcost_dw1 = dcost_dz * dz_dw1
        dcost_dw2 = dcost_dz * dz_dw2
        dcost_db = dcost_dz * dz_db
        
        w1 = w1 - learning_rate * dcost_dw1
        w2 = w2 - learning_rate * dcost_dw2
        b = b - learning_rate * dcost_db
        
    return costs, w1, w2, b
        
costs, w1, w2, b = train()

Hello All,

I am a new user to Python and have been attempting to develop logistic regressions and Neural Networks in Python over the past few weeks. I've hit a stumbling block with neural networks whereas when trying to handle a large dataset with many inputs i need many weightings, will i have to write them each individually out with the equations as done above for a 2 input model?

Above is for a 2 vraibles, about 8 inputs and 1 output
I want to be able to work with multiple variables, very larger datasets but still aim for 1 output.

Will i need to type our a all the individual weighting equations and such?

Sorry if its not clear.

Cheers,
Chris

[ichabod801]

It looks like you have one vector of weights for the connections between the one output node and the various input nodes. Let's say you added a layer of nodes between the output node and the input nodes. Each node will need a vector of weights that is as long as the layer before it. If we have eight input nodes and four middle layer nodes, each middle layer node will need eight weights and the output node will need four.

You will then need to loop through the layers (middle then output), and then loop through the nodes in that layer, and then do the calculation you have for each node. That calculation just needs to be there once in the inner most loop. This can then be generalized for any number of layers with any number of nodes on each layer.

So, to work that backwards. You have the calculation for a single node's weights. You need to make a loop around that each node in a layer with more than one node. Then you need to make a loop around that for each layer in the neural net.

[brighteningeyes]

hi,
why do you want to write the network from scratch yourself?
you can use tensorflow, torch, mxnet etc to do the heavy work for you.
the benefit of using them is, they are fast in terms of computation (they support GPU to do the computation).
also, something like gluon (which i'm using), keras allow you to add the network layers with a single call.
if you want to design by yourself and learn from it, then that's another thing.