Scheduler runs but then fails

[marciokoko]

I have this scheduler code:

# class to hold read and write
#!/usr/bin/env python

import serial
import logging
import time, datetime
from firebase import firebase
from firebase_token_generator import create_token
from apscheduler.schedulers.blocking import BlockingScheduler

class AIHome:

	def __init__(self, onTime, offTime):
	
		self.onTime=onTime
		self.offTime=offTime
		self.updateInterval = 6
		self.webPush = False
		self.relayStatesA = []
		self.relayStatesD = {}
		logging.basicConfig()
		#Call fetchUpdate every 1 hours
		
		print('initting AIHome...scheduling job')
		sched = BlockingScheduler()
		@sched.scheduled_job('interval', hours=1)
		def timed_job():
			print('This job runs every 1 hrs. timed_job gets called or something else')
			#call fetchUpdate()
			self.fetchUpdate();
		
		sched.configure()
		#options_from_ini_file
		sched.start()
		 
	def fetchUpdate(self):
	    #Must separate credentials
		AIHome.authentication = firebase.FirebaseAuthentication('key', 'email', extra={'id': 123})
		AIHome.firebase = firebase.FirebaseApplication('https://app.firebaseio.com/', AIHome.authentication)
		print AIHome.authentication.extra

		AIHome.user = AIHome.authentication.get_user()
		print AIHome.user.firebase_auth_token

		AIHome.results = AIHome.firebase.get('/Relays', None)#, {'print': 'pretty'})
		print AIHome.results
		
		#Commented out for debugging purposes
		print AIHome.results['Relay1ON']
		print AIHome.results['Relay1OFF']
		
		#Call time comparator method
		self.relayToggle()
		

		
	def relayToggle(self):
	
		timestring = AIHome.results['Relay1ON']
		print timestring
		hours,minutes = timestring.split(":")
		print hours
		print minutes
		print(datetime.datetime.now())
		ref_time = datetime.datetime.combine(datetime.datetime.now(), datetime.time(int(hours), int(minutes)))
		if ref_time > datetime.datetime.now():
			print("ref_time>relay should be OFF")
			self.write(1,0)
		else:
			print("now>relay should be ON")
			self.write(1,1)
	
	
	def write(self, relay, state):
		ser = serial.Serial(
		port='/dev/serial0',
		baudrate=9600,
		parity=serial.PARITY_NONE,
		stopbits=serial.STOPBITS_ONE,
		bytesize=serial.EIGHTBITS,
		timeout=1
		)
		print "Serial is open: " + str(ser.isOpen())
		print "Now Writing"
		#ser.write("AT+CONNL")
		print relay
		print state
		self.read()
		
		#switch case
		while True:
			# Relay1ON 1
			if relay == 1 & state == 0:
				ser.write("o")
				print ('Relay toggled off')
				break

			# case 2
			if relay == 1 & state == 1:
				ser.write("e")
				print ('Relay toggled on')
				break

			# else case 3
			print ('something else')
			ser.write("o")
			x = ser.readline()
			print "Something else '" + x + "'"
			break
		
		
		print "Did write, now read"
		x = ser.readline()
		print "Writing & Reading got '" + x + "'"
		self.read()
		ser.close()
	

	def convert_hex_to_int(hexChars):
		#convert string of hex chars to a list of ints
		try:
			ints = [ord(char) for char in hexChars]
			return ints
		except TypeError:
			pass
		return []
	
	def convert_hex_to_bin_str(hexChars):
		#convert hex char into byte string
		response = convert_hex_to_int(hexChars)[0]
		# convert int to binary string
		responseBinary = bin(response)
		# first 2 chars of binary string are '0b' so ignore these
		return responseBinary[2:]
	
	# To be used later when fetching relay states from app or web
	def read(self):
		ser = serial.Serial(
			port='/dev/serial0',
			baudrate=9600,
			parity=serial.PARITY_NONE,
			stopbits=serial.STOPBITS_ONE,
			bytesize=serial.EIGHTBITS,
			timeout=1
		)

		print "Read:Serial is open: " + str(ser.isOpen())
		print "Read:Now Reading"
		ser.write("[")
		print "Did read"
		x = ser.readline()
		print "Read:Reading got '" + x + "'"
		#***if ser.close() at self.read then response will be null/false, so code around it or dont close***
		responseBits = convert_hex_to_bin_str (x)
		responseBits = responseBits.zfill(4)
		responseBits = list(responseBits)
		responseBits.reverse()
		relayStates = {}
		relay = 1
		for bit in responseBits:
			relayStates[relay] = int(bit)
			relay += 1
		self.relayStatesD = relayStates
		#report states to someone?
		print relayStatesD

x = AIHome(1,2)

Its running on a RPi2 to which I ssh into from my mac.  Its been running for about a day and it logs every hour, but last night at 22:00hrs it stopped logging and then later today the connection between the mac and RPi2 broke as can be seen in the logs.  The script stopped logging after 22:00hrs but even though the connection pipe broke sometime after 7am (because I checked and the broken pipe log wasnt there at 7am that I left the house), the script stopped running before 7am, probably at 22:00hrs because the script is supposed to check and fire some relays and log results before and after firing the relay, and there is no logging after 22:00hrs and the lights are still on (which should turn off at 06:00hrs).  

So Im wondering 2 things:

1.  What makes the scheduler fail, is it common that it fails, and how can I make it start up again automatically from my py script?

2.  The problem shouldnt be that the connection pipe between mac and RPi2 broke, right?  I mean I should be able to ssh into the pi, run the python script (which should run indefinitely) and then logout and the pi should continue running the script indefinitely even though I manually ssh'd out, right?

Thanks.  here is a screenshot of terminal logs

[wavic]

It's hard to fallow the code without indentation. Where is the full error traceback?

[marciokoko]

I had to manually indent because for some reason it didnt work from TW this time. I fixed it. How do I get a full error traceback?