Skip to content

feat: Feast AWS Athena offline store (again) #3044

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 11 commits into from
Aug 10, 2022
Merged

feat: Feast AWS Athena offline store (again) #3044

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
693e9df
fixed bugs, cleaned code, added AthenaDataSourceCreator
Jul 24, 2022
7cbd232
fixed bugs, cleaned code, added some methods. test_universal_historic…
Jul 31, 2022
9011036
fixed bugs to pass test_validation
Aug 7, 2022
23905a1
changed boolean data type mapping
Aug 8, 2022
5fc9e6e
1.added test-python-universal-athena in Makefile 2.replaced database…
Aug 8, 2022
e8db748
format,run lint
Aug 9, 2022
228555c
revert merge changes
adchia Aug 9, 2022
e9b6180
add entity_key_serialization
adchia Aug 9, 2022
c7e59a3
restore deleted file
adchia Aug 9, 2022
327289e
modified confusing environment variable names, added how to use Athena
younggyu-oh Aug 9, 2022
ce1710c
enforce AthenaSource to have a name
younggyu-oh Aug 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixed bugs to pass test_validation 
Signed-off-by: Youngkyu OH <toping4445@gmail.com>
  • Loading branch information
@younggyu-oh
toping4445 authored and younggyu-oh committed Aug 9, 2022
commit 901103633b69e1d6b48b9c52e1ad76ff08702ad6
4 changes: 2 additions & 2 deletions sdk/python/feast/infra/offline_stores/contrib/athena_offline_store/athena.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,8 +149,8 @@ def pull_all_from_table_or_query(

date_partition_column = data_source.date_partition_column

start_date = start_date.astimezone(tz=utc)
end_date = end_date.astimezone(tz=utc)
start_date = start_date.astimezone(tz=utc).strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]
end_date = end_date.astimezone(tz=utc).strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]

query = f"""
SELECT {field_string}
Expand Down
2 changes: 1 addition & 1 deletion sdk/python/feast/infra/utils/aws_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -971,7 +971,7 @@ def upload_arrow_table_to_athena(
bucket, key = get_bucket_and_key(s3_path)

column_query_list = ", ".join(
[f"{field.name} {pa_to_athena_value_type(field.type)}" for field in schema]
[f"`{field.name}` {pa_to_athena_value_type(field.type)}" for field in schema]
)

with tempfile.TemporaryFile(suffix=".parquet") as parquet_temp_file:
Expand Down