Skip to content

Commit bb1875f

Browse files
ahtaht
committed
Merge remote-tracking branch 'upstream/master' into dynamodb-improvements
* upstream/master: (103 commits) refactor: Move create cluster script ci: Add validation for upper case PR titles Switch order of views to keep/delete to fix apply logging ci: Fix broken Prow plugin which ignores config.yaml ci: Add PR linting to ensure conventional commits are followed for PR titles Add owner field to FeatureService (feast-dev#2321) Disable Redis cluster tests (feast-dev#2327) Use PR title for commit message Add redis cluster initialziation to master github workflow (feast-dev#2317) Fix materialize bug with RedisCluster (feast-dev#2311) Add support for Dask during historical retrieval (feast-dev#1954) downgrade datatypes to 8 Update pom.xml to 0.18.2 SNAPSHOT Update changelog for 0.18.1 and update pom.xml Update helm chart version Don't require `snowflake` to always be installed (feast-dev#2309) Fixing the Java helm charts and adding a demo tutorial on how to use them (feast-dev#2298) Include infra objects in registry dump and fix Infra's from_proto (feast-dev#2295) Remove old flag warning with the python feature server (feast-dev#2300) podAnnotations Values in the feature-server chart (feast-dev#2304) ...
2 parents b4a011a + 9221624 commit bb1875f

339 files changed

Lines changed: 10913 additions & 9117 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/ISSUE_TEMPLATE/bug_report.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
name: Bug report
33
about: Create a report to help us improve
44
title: ''
5-
labels: ''
5+
labels: 'kind/bug, priority/p2'
66
assignees: ''
77

88
---

‎.github/ISSUE_TEMPLATE/feature_request.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
name: Feature request
33
about: Suggest an idea for this project
44
title: ''
5-
labels: ''
5+
labels: 'kind/feature'
66
assignees: ''
77

88
---

‎.github/workflows/lint_pr.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: lint-pr
2+
3+
on:
4+
pull_request_target:
5+
types:
6+
- opened
7+
- edited
8+
- synchronize
9+
10+
jobs:
11+
validate-title:
12+
name: Validate PR title
13+
runs-on: ubuntu-latest
14+
steps:
15+
- uses: amannn/action-semantic-pull-request@v4
16+
with:
17+
# Must use uppercase
18+
subjectPattern: ^(?=[A-Z]).+$
19+
subjectPatternError: |
20+
The subject "{subject}" found in the pull request title "{title}"
21+
didn't match the configured pattern. Please ensure that the subject
22+
starts with an uppercase character.
23+
env:
24+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎.github/workflows/master_only.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,11 +120,20 @@ jobs:
120120
run: pip install pip-tools
121121
- name: Install dependencies
122122
run: make install-python-ci-dependencies
123+
- name: Setup Redis Cluster
124+
run: |
125+
docker pull vishnunair/docker-redis-cluster:latest
126+
docker run -d -p 6001:6379 -p 6002:6380 -p 6003:6381 -p 6004:6382 -p 6005:6383 -p 6006:6384 --name redis-cluster vishnunair/docker-redis-cluster
123127
- name: Test python
124128
env:
125129
FEAST_SERVER_DOCKER_IMAGE_TAG: ${{ needs.build-lambda-docker-image.outputs.DOCKER_IMAGE_TAG }}
126130
FEAST_USAGE: "False"
127131
IS_TEST: "True"
132+
SNOWFLAKE_CI_DEPLOYMENT: ${{ secrets.SNOWFLAKE_CI_DEPLOYMENT }}
133+
SNOWFLAKE_CI_USER: ${{ secrets.SNOWFLAKE_CI_USER }}
134+
SNOWFLAKE_CI_PASSWORD: ${{ secrets.SNOWFLAKE_CI_PASSWORD }}
135+
SNOWFLAKE_CI_ROLE: ${{ secrets.SNOWFLAKE_CI_ROLE }}
136+
SNOWFLAKE_CI_WAREHOUSE: ${{ secrets.SNOWFLAKE_CI_WAREHOUSE }}
128137
run: pytest -n 8 --cov=./ --cov-report=xml --verbose --color=yes sdk/python/tests --integration --durations=5
129138
- name: Upload coverage to Codecov
130139
uses: codecov/codecov-action@v1
@@ -140,6 +149,11 @@ jobs:
140149
FEAST_SERVER_DOCKER_IMAGE_TAG: ${{ needs.build-lambda-docker-image.outputs.DOCKER_IMAGE_TAG }}
141150
FEAST_USAGE: "False"
142151
IS_TEST: "True"
152+
SNOWFLAKE_CI_DEPLOYMENT: ${{ secrets.SNOWFLAKE_CI_DEPLOYMENT }}
153+
SNOWFLAKE_CI_USER: ${{ secrets.SNOWFLAKE_CI_USER }}
154+
SNOWFLAKE_CI_PASSWORD: ${{ secrets.SNOWFLAKE_CI_PASSWORD }}
155+
SNOWFLAKE_CI_ROLE: ${{ secrets.SNOWFLAKE_CI_ROLE }}
156+
SNOWFLAKE_CI_WAREHOUSE: ${{ secrets.SNOWFLAKE_CI_WAREHOUSE }}
143157
run: pytest --verbose --color=yes sdk/python/tests --integration --benchmark --benchmark-autosave --benchmark-save-data --durations=5
144158
- name: Upload Benchmark Artifact to S3
145159
run: aws s3 cp --recursive .benchmarks s3://feast-ci-pytest-benchmarks

‎.github/workflows/pr_integration_tests.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,12 +145,21 @@ jobs:
145145
run: pip install pip-tools
146146
- name: Install dependencies
147147
run: make install-python-ci-dependencies
148+
- name: Setup Redis Cluster
149+
run: |
150+
docker pull vishnunair/docker-redis-cluster:latest
151+
docker run -d -p 6001:6379 -p 6002:6380 -p 6003:6381 -p 6004:6382 -p 6005:6383 -p 6006:6384 --name redis-cluster vishnunair/docker-redis-cluster
148152
- name: Test python
149153
if: ${{ always() }} # this will guarantee that step won't be canceled and resources won't leak
150154
env:
151155
FEAST_SERVER_DOCKER_IMAGE_TAG: ${{ needs.build-docker-image.outputs.DOCKER_IMAGE_TAG }}
152156
FEAST_USAGE: "False"
153157
IS_TEST: "True"
158+
SNOWFLAKE_CI_DEPLOYMENT: ${{ secrets.SNOWFLAKE_CI_DEPLOYMENT }}
159+
SNOWFLAKE_CI_USER: ${{ secrets.SNOWFLAKE_CI_USER }}
160+
SNOWFLAKE_CI_PASSWORD: ${{ secrets.SNOWFLAKE_CI_PASSWORD }}
161+
SNOWFLAKE_CI_ROLE: ${{ secrets.SNOWFLAKE_CI_ROLE }}
162+
SNOWFLAKE_CI_WAREHOUSE: ${{ secrets.SNOWFLAKE_CI_WAREHOUSE }}
154163
run: pytest -n 8 --cov=./ --cov-report=xml --verbose --color=yes sdk/python/tests --integration --durations=5
155164
- name: Upload coverage to Codecov
156165
uses: codecov/codecov-action@v1

‎.github/workflows/release.yml‎

Lines changed: 61 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ jobs:
103103
runs-on: ubuntu-latest
104104
needs: get-version
105105
env:
106-
HELM_VERSION: v2.17.0
106+
HELM_VERSION: v3.8.0
107107
VERSION_WITHOUT_PREFIX: ${{ needs.get-version.outputs.version_without_prefix }}
108108
steps:
109109
- uses: actions/checkout@v2
@@ -144,5 +144,63 @@ jobs:
144144
python3 setup.py sdist bdist_wheel
145145
python3 -m twine upload --verbose dist/*
146146
147-
# TODO(adchia): publish java sdk once maven repo is updated
148-
# See https://github.com/feast-dev/feast-java/blob/master/.github/workflows/release.yml#L104
147+
publish-python-sdk-no-telemetry:
148+
runs-on: ubuntu-latest
149+
needs: get-version
150+
env:
151+
TWINE_USERNAME: __token__
152+
TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
153+
container: python:3.7
154+
steps:
155+
- uses: actions/checkout@v2
156+
- name: Install pip-tools
157+
run: pip install pip-tools
158+
- name: Install dependencies
159+
run: make install-python-ci-dependencies PYTHON=3.7
160+
- name: Publish Python Package
161+
env:
162+
SETUPTOOLS_SCM_PRETEND_VERSION: ${{ needs.get-version.outputs.version_without_prefix }}
163+
run: |
164+
cd sdk/python
165+
sed -i 's/DEFAULT_FEAST_USAGE_VALUE = "True"/DEFAULT_FEAST_USAGE_VALUE = "False"/g' feast/constants.py
166+
sed -i 's/NAME = "feast"/NAME = "feast-no-telemetry"/g' setup.py
167+
python3 -m pip install --user --upgrade setuptools wheel twine
168+
python3 setup.py sdist bdist_wheel
169+
python3 -m twine upload --verbose dist/*
170+
171+
publish-java-sdk:
172+
container: maven:3.6-jdk-11
173+
runs-on: ubuntu-latest
174+
needs: get-version
175+
steps:
176+
- uses: actions/checkout@v2
177+
with:
178+
submodules: 'true'
179+
- name: Set up JDK 11
180+
uses: actions/setup-java@v1
181+
with:
182+
java-version: '11'
183+
java-package: jdk
184+
architecture: x64
185+
- uses: actions/setup-python@v2
186+
with:
187+
python-version: '3.7'
188+
architecture: 'x64'
189+
- uses: actions/cache@v2
190+
with:
191+
path: ~/.m2/repository
192+
key: ${{ runner.os }}-it-maven-${{ hashFiles('**/pom.xml') }}
193+
restore-keys: |
194+
${{ runner.os }}-it-maven-
195+
- name: Publish java sdk
196+
env:
197+
VERSION_WITHOUT_PREFIX: ${{ needs.get-version.outputs.version_without_prefix }}
198+
GPG_PUBLIC_KEY: ${{ secrets.GPG_PUBLIC_KEY }}
199+
GPG_PRIVATE_KEY: ${{ secrets.GPG_PRIVATE_KEY }}
200+
MAVEN_SETTINGS: ${{ secrets.MAVEN_SETTINGS }}
201+
run: |
202+
echo -n "$GPG_PUBLIC_KEY" > /root/public-key
203+
echo -n "$GPG_PRIVATE_KEY" > /root/private-key
204+
mkdir -p /root/.m2/
205+
echo -n "$MAVEN_SETTINGS" > /root/.m2/settings.xml
206+
infra/scripts/publish-java-sdk.sh --revision ${VERSION_WITHOUT_PREFIX} --gpg-key-import-dir /root

‎.github/workflows/unit_tests.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,12 @@ jobs:
4242
- name: Install dependencies
4343
run: make install-python-ci-dependencies
4444
- name: Test Python
45+
env:
46+
SNOWFLAKE_CI_DEPLOYMENT: ${{ secrets.SNOWFLAKE_CI_DEPLOYMENT }}
47+
SNOWFLAKE_CI_USER: ${{ secrets.SNOWFLAKE_CI_USER }}
48+
SNOWFLAKE_CI_PASSWORD: ${{ secrets.SNOWFLAKE_CI_PASSWORD }}
49+
SNOWFLAKE_CI_ROLE: ${{ secrets.SNOWFLAKE_CI_ROLE }}
50+
SNOWFLAKE_CI_WAREHOUSE: ${{ secrets.SNOWFLAKE_CI_WAREHOUSE }}
4551
run: FEAST_USAGE=False pytest -n 8 --cov=./ --cov-report=xml --verbose --color=yes sdk/python/tests
4652
- name: Upload coverage to Codecov
4753
uses: codecov/codecov-action@v1

‎.prow.yaml‎

Lines changed: 0 additions & 126 deletions
Original file line numberDiff line numberDiff line change
@@ -1,102 +1,4 @@
1-
presubmits:
2-
- name: test-core-and-ingestion
3-
decorate: true
4-
spec:
5-
containers:
6-
- image: maven:3.6-jdk-11
7-
command: ["infra/scripts/test-java-core-ingestion.sh"]
8-
resources:
9-
requests:
10-
cpu: "2000m"
11-
memory: "1536Mi"
12-
skip_branches:
13-
- ^v0\.(3|4)-branch$
14-
15-
- name: test-core-and-ingestion-java-8
16-
decorate: true
17-
always_run: true
18-
spec:
19-
containers:
20-
- image: maven:3.6-jdk-8
21-
command: ["infra/scripts/test-java-core-ingestion.sh"]
22-
resources:
23-
requests:
24-
cpu: "2000m"
25-
memory: "1536Mi"
26-
branches:
27-
- ^v0\.(3|4)-branch$
28-
29-
- name: test-serving
30-
decorate: true
31-
spec:
32-
containers:
33-
- image: maven:3.6-jdk-11
34-
command: ["infra/scripts/test-java-serving.sh"]
35-
skip_branches:
36-
- ^v0\.(3|4)-branch$
37-
38-
- name: test-serving-java-8
39-
decorate: true
40-
always_run: true
41-
spec:
42-
containers:
43-
- image: maven:3.6-jdk-8
44-
command: ["infra/scripts/test-java-serving.sh"]
45-
branches:
46-
- ^v0\.(3|4)-branch$
47-
48-
- name: test-java-sdk
49-
decorate: true
50-
spec:
51-
containers:
52-
- image: maven:3.6-jdk-11
53-
command: ["infra/scripts/test-java-sdk.sh"]
54-
skip_branches:
55-
- ^v0\.(3|4)-branch$
56-
57-
- name: test-java-sdk-java-8
58-
decorate: true
59-
always_run: true
60-
spec:
61-
containers:
62-
- image: maven:3.6-jdk-8
63-
command: ["infra/scripts/test-java-sdk.sh"]
64-
branches:
65-
- ^v0\.(3|4)-branch$
66-
67-
- name: test-golang-sdk
68-
decorate: true
69-
spec:
70-
containers:
71-
- image: golang:1.13
72-
command: ["infra/scripts/test-golang-sdk.sh"]
73-
741
postsubmits:
75-
- name: publish-python-sdk
76-
decorate: true
77-
spec:
78-
containers:
79-
- image: python:3
80-
command:
81-
- sh
82-
- -c
83-
- |
84-
make package-protos && make compile-protos-python && infra/scripts/publish-python-sdk.sh \
85-
--directory-path sdk/python --repository pypi
86-
volumeMounts:
87-
- name: pypirc
88-
mountPath: /root/.pypirc
89-
subPath: .pypirc
90-
readOnly: true
91-
volumes:
92-
- name: pypirc
93-
secret:
94-
secretName: pypirc
95-
branches:
96-
# Filter on tags with semantic versioning, prefixed with "v"
97-
# https://github.com/semver/semver/issues/232
98-
- ^v(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(-(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(\.(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*)?(\+[0-9a-zA-Z-]+(\.[0-9a-zA-Z-]+)*)?$
99-
1002
- name: publish-java-sdk
1013
decorate: true
1024
spec:
@@ -128,31 +30,3 @@ postsubmits:
12830
branches:
12931
# Filter on tags with semantic versioning, prefixed with "v".
13032
- ^v(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(-(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(\.(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*)?(\+[0-9a-zA-Z-]+(\.[0-9a-zA-Z-]+)*)?$
131-
132-
- name: publish-java-8-sdk
133-
decorate: true
134-
spec:
135-
containers:
136-
- image: maven:3.6-jdk-8
137-
command:
138-
- bash
139-
- -c
140-
- infra/scripts/publish-java-sdk.sh --revision ${PULL_BASE_REF:1}
141-
volumeMounts:
142-
- name: gpg-keys
143-
mountPath: /etc/gpg
144-
readOnly: true
145-
- name: maven-settings
146-
mountPath: /root/.m2/settings.xml
147-
subPath: settings.xml
148-
readOnly: true
149-
volumes:
150-
- name: gpg-keys
151-
secret:
152-
secretName: gpg-keys
153-
- name: maven-settings
154-
secret:
155-
secretName: maven-settings
156-
branches:
157-
# Filter on tags with semantic versioning, prefixed with "v". v0.3 and v0.4 only.
158-
- ^v0\.(3|4)\.(0|[1-9]\d*)(-(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(\.(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*)?(\+[0-9a-zA-Z-]+(\.[0-9a-zA-Z-]+)*)?$

‎.prow/config.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,9 @@ tide:
6464
merge_method:
6565
feast-dev/feast: squash
6666
feast-dev/feast-spark: squash
67+
merge_commit_template:
68+
feast-dev/feast:
69+
title: '{{.Title}}'
6770
blocker_label: merge-blocker
6871
squash_label: tide/squash
6972

‎.prow/plugins.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,8 @@ config_updater:
3434
maps:
3535
.prow/config.yaml:
3636
name: config
37+
.prow.yaml:
38+
name: job-config
3739

3840
external_plugins:
3941
feast-dev/feast:

0 commit comments

Comments
 (0)