TypeError: a bytes-like object is required, not 'str' - Help Please.

[IanJ]

Hi,

I have inherited a number of python scripts for AWS Lambda that need upgrading from python 2.7 to v3. The following script has been through the 2to3 python script. However I'm getting ' TypeError: a bytes-like object is required, not 'str' '.

Sorry but I'm new to python and no idea how I should fix this.

Thanks in advance.

This is my code -

import os
import json
import urllib.request, urllib.parse, urllib.error
import boto3
import botocore
import datetime
import json
import csv

print('Loading function')

region_list = {'eu-west-1','eu-west-2','eu-central-1'}

date = datetime.date.today().strftime("%Y_%m_%d")
#file path
filepath ='/tmp/'
#filename

filename = os.environ['filename'] + date + '.csv'
tempfile = filepath + filename

mybucket = os.environ['bucket']
account = os.environ['account']


#print ('filename "%s"' % str(filename))

session = boto3.session.Session()
s3_client = session.client('s3', config= boto3.session.Config(signature_version='s3v4'))

def lambda_handler(event, context):
        with open(tempfile, 'wb+') as out:

                dict_writer = csv.DictWriter(out, fieldnames=['Region','InstanceId','Name','Status','Instance_Size','Launch_Time','Service','CPU.Average','CPU.Max','CPUCreditBalance.Max','CPUCreditBalance.Min'])
                dict_writer.writeheader()

                for region in region_list:
                        client = session.client('ec2', region_name=region)
                        cwclient = session.client('cloudwatch', region_name=region)
                        #filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
                        #response = client.describe_instances(Filters=filters)
                        response = client.describe_instances()

                        instance_list = []
                        for instances in response['Reservations']:
                                for instance in instances['Instances']:
                                        inst = {}
                                        inst['Region'] = region
                                        inst['InstanceId'] = instance['InstanceId']
                                        inst['Name'] = instance['InstanceId']
                                        inst['Instance_Size'] = instance['InstanceType']
                                        state = instance['State']
                                        inst['Status'] = state['Name']
                                        launch = instance['LaunchTime']
                                        inst['Launch_Time'] = launch.strftime("%Y_%m_%d %H:%M")
                                        inst['Service'] = ''
                                        inst['Name'] = ''
                                        inst['CPU.Average'] = ''
                                        inst['CPU.Max'] = ''
                                        try:
                                                        if len(instance['Tags']) > 0:
                                                                for tag in instance['Tags']:
                                                                        if tag['Key'] == 'Service':
                                                                                        inst['Service'] = tag['Value']
                                                                        if tag['Key'] == 'Name':
                                                                                        inst['Name'] = tag['Value']
                                        except KeyError as e:
                                                        print(('I got a KeyError - reason "%s"' % str(e)))

                                        cwresponse=cwclient.get_metric_statistics(
                                                        Period = 900,
                                                        StartTime = datetime.datetime.utcnow() - datetime.timedelta(days=4),
                                                                EndTime = datetime.datetime.utcnow(),
                                                        Namespace = 'AWS/EC2',
                                                        MetricName = 'CPUUtilization',
                                                        Statistics = ['Average','Maximum'],
                                                        Dimensions = [{'Name' : 'InstanceId', 'Value' : instance['InstanceId']}]
                                        )
                                        average = 0
                                        maxcpu = 0
                                        if cwresponse['Datapoints']:
                                                        for data in cwresponse['Datapoints']:
                                                                        cpu = data['Maximum']
                                                                        ave = data['Average']
                                                                        if cpu > maxcpu:
                                                                                        maxcpu = cpu
                                                                        if ave > average:
                                                                                        average = ave
                                                                        inst['CPU.Max'] = maxcpu
                                                                        inst['CPU.Average'] = average

                                        cwresponse2=cwclient.get_metric_statistics(
                                                        Period = 900,
                                                        StartTime = datetime.datetime.utcnow() - datetime.timedelta(days=4),
                                                                        EndTime = datetime.datetime.utcnow(),
                                                        Namespace = 'AWS/EC2',
                                                        MetricName = 'CPUCreditBalance',
                                                        Statistics = ['Minimum','Maximum'],
                                                        Dimensions = [{'Name' : 'InstanceId', 'Value' : instance['InstanceId']}]
                                        )
                                        minbal = 999999
                                        maxbal = 0
                                        if cwresponse2['Datapoints']:
                                                        for data in cwresponse2['Datapoints']:
                                                                        minimum = data['Minimum']
                                                                        maximum = data['Maximum']
                                                                        if minimum < minbal:
                                                                                        minbal = minimum
                                                                        if maximum > maxbal:
                                                                                        maxbal = maximum

                                                                        inst['CPUCreditBalance.Max'] = maxbal
                                                                        inst['CPUCreditBalance.Min'] = minbal

                                        dict_writer.writerow(inst)

                                        instance_list.append(inst)

        out.close()

        # Upload the file to S3
        s3_client.upload_file(tempfile, mybucket, account + filename, ExtraArgs={'ACL': 'public-read'})

Error:
[ERROR] TypeError: a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 35, in lambda_handler
    dict_writer.writeheader()
  File "/var/lang/lib/python3.8/csv.py", line 143, in writeheader
    return self.writerow(header)
  File "/var/lang/lib/python3.8/csv.py", line 154, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))

[Larz60+]

The error occurs in writerow, probably caused by line 35,
but you don't show writerow, so hard to say

[malcomjarr]

Python makes a clear distinction between bytes and strings . Bytes objects contain raw data — a sequence of octets — whereas strings are Unicode sequences . Conversion between these two types is explicit: you encode a string to get bytes, specifying an encoding (which defaults to UTF-8); and you decode bytes to get a string. Clients of these functions should be aware that such conversions may fail, and should consider how failures are handled.

We can convert bytes to string using bytes class decode() instance method, So you need to decode the bytes object to produce a string. In Python 3 , the default encoding is "utf-8" , so you can use directly:

b"python byte to string".decode("utf-8")

[deanhystad]

The problem is that you open the file as binary

with open(tempfile, 'wb+') as out:

And then you try to write str objects which use unicode characters, not bytes.

You can either convert all the str to bytes, or not open the file as binary. Since this is a csv file I don't know why you would want it to be binary.