Configuration
7. Maven Resources
Spring Cloud Dataflow supports referencing artifacts via Maven (maven:).
If you want to override specific Maven configuration properties (remote repositories, proxies, and others) or run the Data Flow Server behind a proxy,
you need to specify those properties as command-line arguments when you start the Data Flow Server, as shown in the following example:
$ java -jar spring-cloud-dataflow-server-2.10.3.jar --spring.config.additional-location=/home/joe/maven.ymlThe preceding command assumes a maven.yaml similar to the following:
maven:
localRepository: mylocal
remote-repositories:
repo1:
url: https://repo1
auth:
username: user1
password: pass1
snapshot-policy:
update-policy: daily
checksum-policy: warn
release-policy:
update-policy: never
checksum-policy: fail
repo2:
url: https://repo2
policy:
update-policy: always
checksum-policy: fail
proxy:
host: proxy1
port: "9010"
auth:
username: proxyuser1
password: proxypass1By default, the protocol is set to http. You can omit the auth properties if the proxy does not need a username and password. Also, by default, the maven localRepository is set to ${user.home}/.m2/repository/.
As shown in the preceding example, you can specify the remote repositories along with their authentication (if needed). If the remote repositories are behind a proxy, you can specify the proxy properties, as shown in the preceding example.
You can specify the repository policies for each remote repository configuration, as shown in the preceding example.
The key policy is applicable to both the snapshot and the release repository policies.
See the Repository Policies topic for the list of supported repository policies.
As these are Spring Boot @ConfigurationProperties you need to specify by adding them to the SPRING_APPLICATION_JSON environment variable. The following example shows how the JSON is structured:
$ SPRING_APPLICATION_JSON='
{
"maven": {
"local-repository": null,
"remote-repositories": {
"repo1": {
"url": "https://repo1",
"auth": {
"username": "repo1user",
"password": "repo1pass"
}
},
"repo2": {
"url": "https://repo2"
}
},
"proxy": {
"host": "proxyhost",
"port": 9018,
"auth": {
"username": "proxyuser",
"password": "proxypass"
}
}
}
}
'7.1. Wagon
There is a limited support for using Wagon transport with Maven. Currently, this
exists to support preemptive authentication with http-based repositories
and needs to be enabled manually.
Wagon-based http transport is enabled by setting the maven.use-wagon property
to true. Then you can enable preemptive authentication for each remote
repository. Configuration loosely follows the similar patterns found in
HttpClient HTTP Wagon.
At the time of this writing, documentation in Maven’s own site is slightly misleading
and missing most of the possible configuration options.
The maven.remote-repositories.<repo>.wagon.http namespace contains all Wagon
http related settings, and the keys directly under it map to supported http methods — namely, all, put, get and head, as in Maven’s own configuration.
Under these method configurations, you can then set various options, such as
use-preemptive. A simpl preemptive configuration to send an auth
header with all requests to a specified remote repository would look like the following example:
maven:
use-wagon: true
remote-repositories:
springRepo:
url: https://repo.example.org
wagon:
http:
all:
use-preemptive: true
auth:
username: user
password: passwordInstead of configuring all methods, you can tune settings for get
and head requests only, as follows:
maven:
use-wagon: true
remote-repositories:
springRepo:
url: https://repo.example.org
wagon:
http:
get:
use-preemptive: true
head:
use-preemptive: true
use-default-headers: true
connection-timeout: 1000
read-timeout: 1000
headers:
sample1: sample2
params:
http.socket.timeout: 1000
http.connection.stalecheck: true
auth:
username: user
password: passwordThere are settings for use-default-headers, connection-timeout,
read-timeout, request headers, and HttpClient params. For more about parameters,
see Wagon ConfigurationUtils.
8. Security
By default, the Data Flow server is unsecured and runs on an unencrypted HTTP connection. You can secure your REST endpoints as well as the Data Flow Dashboard by enabling HTTPS and requiring clients to authenticate with OAuth 2.0.
|
Appendix Azure contains more information how to setup Azure Active Directory integration. |
|
By default, the REST endpoints (administration, management, and health) as well as the Dashboard UI do not require authenticated access. |
While you can theoretically choose any OAuth provider in conjunction with Spring Cloud Data Flow, we recommend using the CloudFoundry User Account and Authentication (UAA) Server.
Not only is the UAA OpenID certified and is used by Cloud Foundry, but you can also use it in local stand-alone deployment scenarios. Furthermore, the UAA not only provides its own user store, but it also provides comprehensive LDAP integration.
8.1. Enabling HTTPS
By default, the dashboard, management, and health endpoints use HTTP as a transport.
You can switch to HTTPS by adding a certificate to your configuration in
application.yml, as shown in the following example:
server:
port: 8443 (1)
ssl:
key-alias: yourKeyAlias (2)
key-store: path/to/keystore (3)
key-store-password: yourKeyStorePassword (4)
key-password: yourKeyPassword (5)
trust-store: path/to/trust-store (6)
trust-store-password: yourTrustStorePassword (7)| 1 | As the default port is 9393, you may choose to change the port to a more common HTTPs-typical port. |
| 2 | The alias (or name) under which the key is stored in the keystore. |
| 3 | The path to the keystore file. You can also specify classpath resources, by using the classpath prefix - for example: classpath:path/to/keystore. |
| 4 | The password of the keystore. |
| 5 | The password of the key. |
| 6 | The path to the truststore file. You can also specify classpath resources, by using the classpath prefix - for example: classpath:path/to/trust-store |
| 7 | The password of the trust store. |
| If HTTPS is enabled, it completely replaces HTTP as the protocol over which the REST endpoints and the Data Flow Dashboard interact. Plain HTTP requests fail. Therefore, make sure that you configure your Shell accordingly. |
Using Self-Signed Certificates
For testing purposes or during development, it might be convenient to create self-signed certificates. To get started, execute the following command to create a certificate:
$ keytool -genkey -alias dataflow -keyalg RSA -keystore dataflow.keystore \
-validity 3650 -storetype JKS \
-dname "CN=localhost, OU=Spring, O=Pivotal, L=Kailua-Kona, ST=HI, C=US" (1)
-keypass dataflow -storepass dataflow| 1 | CN is the important parameter here. It should match the domain you are trying to access - for example, localhost. |
Then add the following lines to your application.yml file:
server:
port: 8443
ssl:
enabled: true
key-alias: dataflow
key-store: "/your/path/to/dataflow.keystore"
key-store-type: jks
key-store-password: dataflow
key-password: dataflowThis is all you need to do for the Data Flow Server. Once you start the server,
you should be able to access it at localhost:8443/.
As this is a self-signed certificate, you should hit a warning in your browser, which
you need to ignore.
| Never use self-signed certificates in production. |
Self-Signed Certificates and the Shell
By default, self-signed certificates are an issue for the shell, and additional steps are necessary to make the shell work with self-signed certificates. Two options are available:
-
Add the self-signed certificate to the JVM truststore.
-
Skip certificate validation.
Adding the Self-signed Certificate to the JVM Truststore
In order to use the JVM truststore option, you need to export the previously created certificate from the keystore, as follows:
$ keytool -export -alias dataflow -keystore dataflow.keystore -file dataflow_cert -storepass dataflowNext, you need to create a truststore that the shell can use, as follows:
$ keytool -importcert -keystore dataflow.truststore -alias dataflow -storepass dataflow -file dataflow_cert -nopromptNow you are ready to launch the Data Flow Shell with the following JVM arguments:
$ java -Djavax.net.ssl.trustStorePassword=dataflow \
-Djavax.net.ssl.trustStore=/path/to/dataflow.truststore \
-Djavax.net.ssl.trustStoreType=jks \
-jar spring-cloud-dataflow-shell-2.10.3.jar|
If you run into trouble establishing a connection over SSL, you can enable additional
logging by using and setting the |
Do not forget to target the Data Flow Server with the following command:
dataflow:> dataflow config server --uri https://localhost:8443/Skipping Certificate Validation
Alternatively, you can also bypass the certification validation by providing the
optional --dataflow.skip-ssl-validation=true command-line parameter.
If you set this command-line parameter, the shell accepts any (self-signed) SSL certificate.
|
If possible, you should avoid using this option. Disabling the trust manager defeats the purpose of SSL and makes your application vulnerable to man-in-the-middle attacks. |
8.2. Authentication by using OAuth 2.0
To support authentication and authorization, Spring Cloud Data Flow uses OAuth 2.0. It lets you integrate Spring Cloud Data Flow into Single Sign On (SSO) environments.
| As of Spring Cloud Data Flow 2.0, OAuth2 is the only mechanism for providing authentication and authorization. |
The following OAuth2 Grant Types are used:
-
Authorization Code: Used for the GUI (browser) integration. Visitors are redirected to your OAuth Service for authentication
-
Password: Used by the shell (and the REST integration), so visitors can log in with username and password
-
Client Credentials: Retrieves an access token directly from your OAuth provider and passes it to the Data Flow server by using the Authorization HTTP header
| Currently, Spring Cloud Data Flow uses opaque tokens and not transparent tokens (JWT). |
You can access the REST endpoints in two ways:
-
Basic authentication, which uses the Password Grant Type to authenticate with your OAuth2 service
-
Access token, which uses the Client Credentials Grant Type
| When you set up authentication, you really should enable HTTPS as well, especially in production environments. |
You can turn on OAuth2 authentication by adding the following to application.yml or by setting
environment variables. The following example shows the minimal setup needed for
CloudFoundry User Account and Authentication (UAA) Server:
spring:
security:
oauth2: (1)
client:
registration:
uaa: (2)
client-id: myclient
client-secret: mysecret
redirect-uri: '{baseUrl}/login/oauth2/code/{registrationId}'
authorization-grant-type: authorization_code
scope:
- openid (3)
provider:
uaa:
jwk-set-uri: http://uaa.local:8080/uaa/token_keys
token-uri: http://uaa.local:8080/uaa/oauth/token
user-info-uri: http://uaa.local:8080/uaa/userinfo (4)
user-name-attribute: user_name (5)
authorization-uri: http://uaa.local:8080/uaa/oauth/authorize
resourceserver:
opaquetoken:
introspection-uri: http://uaa.local:8080/uaa/introspect (6)
client-id: dataflow
client-secret: dataflow| 1 | Providing this property activates OAuth2 security. |
| 2 | The provider ID. You can specify more than one provider. |
| 3 | As the UAA is an OpenID provider, you must at least specify the openid scope.
If your provider also provides additional scopes to control the role assignments,
you must specify those scopes here as well. |
| 4 | OpenID endpoint. Used to retrieve user information such as the username. Mandatory. |
| 5 | The JSON property of the response that contains the username. |
| 6 | Used to introspect and validate a directly passed-in token. Mandatory. |
You can verify that basic authentication is working properly by using curl, as follows:
curl -u myusername:mypassword http://localhost:9393/ -H 'Accept: application/json'As a result, you should see a list of available REST endpoints.
When you access the Root URL with a web browser and
security enabled, you are redirected to the Dashboard UI. To see the
list of REST endpoints, specify the application/json Accept header. Also be sure
to add the Accept header by using tools such as
Postman (Chrome)
or RESTClient (Firefox).
|
Besides Basic Authentication, you can also provide an access token, to access the REST API. To do so, retrieve an OAuth2 Access Token from your OAuth2 provider and pass that access token to the REST Api by using the Authorization HTTP header, as follows:
$ curl -H "Authorization: Bearer <ACCESS_TOKEN>" http://localhost:9393/ -H 'Accept: application/json'8.3. Customizing Authorization
The preceding content mostly deals with authentication — that is, how to assess the identity of the user. In this section, we discuss the available authorization options — that is, who can do what.
The authorization rules are defined in dataflow-server-defaults.yml (part of
the Spring Cloud Data Flow Core module).
Because the determination of security roles is environment-specific,
Spring Cloud Data Flow, by default, assigns all roles to authenticated OAuth2
users. The DefaultDataflowAuthoritiesExtractor class is used for that purpose.
Alternatively, you can have Spring Cloud Data Flow map OAuth2 scopes to Data Flow roles by
setting the boolean property map-oauth-scopes for your provider to true (the default is false).
For example, if your provider’s ID is uaa, the property would be
spring.cloud.dataflow.security.authorization.provider-role-mappings.uaa.map-oauth-scopes.
For more details, see the chapter on [configuration-security-role-mapping].
You can also customize the role-mapping behavior by providing your own Spring bean definition that
extends Spring Cloud Data Flow’s AuthorityMapper interface. In that case,
the custom bean definition takes precedence over the default one provided by
Spring Cloud Data Flow.
The default scheme uses seven roles to protect the REST endpoints that Spring Cloud Data Flow exposes:
-
ROLE_CREATE: For anything that involves creating, such as creating streams or tasks
-
ROLE_DEPLOY: For deploying streams or launching tasks
-
ROLE_DESTROY: For anything that involves deleting streams, tasks, and so on.
-
ROLE_MANAGE: For Boot management endpoints
-
ROLE_MODIFY: For anything that involves mutating the state of the system
-
ROLE_SCHEDULE: For scheduling related operation (such as scheduling a task)
-
ROLE_VIEW: For anything that relates to retrieving state
As mentioned earlier in this section, all authorization-related default settings are specified
in dataflow-server-defaults.yml, which is part of the Spring Cloud Data Flow Core
Module. Nonetheless, you can override those settings, if desired — for example,
in application.yml. The configuration takes the form of a YAML list (as some
rules may have precedence over others). Consequently, you need to copy and paste
the whole list and tailor it to your needs (as there is no way to merge lists).
Always refer to your version of the application.yml file, as the following snippet may be outdated.
|
The default rules are as follows:
spring:
cloud:
dataflow:
security:
authorization:
enabled: true
loginUrl: "/"
permit-all-paths: "/authenticate,/security/info,/assets/**,/dashboard/logout-success-oauth.html,/favicon.ico"
rules:
# About
- GET /about => hasRole('ROLE_VIEW')
# Audit
- GET /audit-records => hasRole('ROLE_VIEW')
- GET /audit-records/** => hasRole('ROLE_VIEW')
# Boot Endpoints
- GET /management/** => hasRole('ROLE_MANAGE')
# Apps
- GET /apps => hasRole('ROLE_VIEW')
- GET /apps/** => hasRole('ROLE_VIEW')
- DELETE /apps/** => hasRole('ROLE_DESTROY')
- POST /apps => hasRole('ROLE_CREATE')
- POST /apps/** => hasRole('ROLE_CREATE')
- PUT /apps/** => hasRole('ROLE_MODIFY')
# Completions
- GET /completions/** => hasRole('ROLE_VIEW')
# Job Executions & Batch Job Execution Steps && Job Step Execution Progress
- GET /jobs/executions => hasRole('ROLE_VIEW')
- PUT /jobs/executions/** => hasRole('ROLE_MODIFY')
- GET /jobs/executions/** => hasRole('ROLE_VIEW')
- GET /jobs/thinexecutions => hasRole('ROLE_VIEW')
# Batch Job Instances
- GET /jobs/instances => hasRole('ROLE_VIEW')
- GET /jobs/instances/* => hasRole('ROLE_VIEW')
# Running Applications
- GET /runtime/streams => hasRole('ROLE_VIEW')
- GET /runtime/streams/** => hasRole('ROLE_VIEW')
- GET /runtime/apps => hasRole('ROLE_VIEW')
- GET /runtime/apps/** => hasRole('ROLE_VIEW')
# Stream Definitions
- GET /streams/definitions => hasRole('ROLE_VIEW')
- GET /streams/definitions/* => hasRole('ROLE_VIEW')
- GET /streams/definitions/*/related => hasRole('ROLE_VIEW')
- POST /streams/definitions => hasRole('ROLE_CREATE')
- DELETE /streams/definitions/* => hasRole('ROLE_DESTROY')
- DELETE /streams/definitions => hasRole('ROLE_DESTROY')
# Stream Deployments
- DELETE /streams/deployments/* => hasRole('ROLE_DEPLOY')
- DELETE /streams/deployments => hasRole('ROLE_DEPLOY')
- POST /streams/deployments/** => hasRole('ROLE_MODIFY')
- GET /streams/deployments/** => hasRole('ROLE_VIEW')
# Stream Validations
- GET /streams/validation/ => hasRole('ROLE_VIEW')
- GET /streams/validation/* => hasRole('ROLE_VIEW')
# Stream Logs
- GET /streams/logs/* => hasRole('ROLE_VIEW')
# Task Definitions
- POST /tasks/definitions => hasRole('ROLE_CREATE')
- DELETE /tasks/definitions/* => hasRole('ROLE_DESTROY')
- GET /tasks/definitions => hasRole('ROLE_VIEW')
- GET /tasks/definitions/* => hasRole('ROLE_VIEW')
# Task Executions
- GET /tasks/executions => hasRole('ROLE_VIEW')
- GET /tasks/executions/* => hasRole('ROLE_VIEW')
- POST /tasks/executions => hasRole('ROLE_DEPLOY')
- POST /tasks/executions/* => hasRole('ROLE_DEPLOY')
- DELETE /tasks/executions/* => hasRole('ROLE_DESTROY')
# Task Schedules
- GET /tasks/schedules => hasRole('ROLE_VIEW')
- GET /tasks/schedules/* => hasRole('ROLE_VIEW')
- GET /tasks/schedules/instances => hasRole('ROLE_VIEW')
- GET /tasks/schedules/instances/* => hasRole('ROLE_VIEW')
- POST /tasks/schedules => hasRole('ROLE_SCHEDULE')
- DELETE /tasks/schedules/* => hasRole('ROLE_SCHEDULE')
# Task Platform Account List */
- GET /tasks/platforms => hasRole('ROLE_VIEW')
# Task Validations
- GET /tasks/validation/ => hasRole('ROLE_VIEW')
- GET /tasks/validation/* => hasRole('ROLE_VIEW')
# Task Logs
- GET /tasks/logs/* => hasRole('ROLE_VIEW')
# Tools
- POST /tools/** => hasRole('ROLE_VIEW')The format of each line is the following:
HTTP_METHOD URL_PATTERN '=>' SECURITY_ATTRIBUTE
where:
-
HTTP_METHOD is one HTTP method (such as PUT or GET), capital case.
-
URL_PATTERN is an Ant-style URL pattern.
-
SECURITY_ATTRIBUTE is a SpEL expression. See Expression-Based Access Control.
-
Each of those is separated by one or whitespace characters (spaces, tabs, and so on).
Be mindful that the above is a YAML list, not a map (thus the use of '-' dashes
at the start of each line) that lives under the spring.cloud.dataflow.security.authorization.rules key.
Authorization — Shell and Dashboard Behavior
When security is enabled, the dashboard and the shell are role-aware, meaning that, depending on the assigned roles, not all functionality may be visible.
For instance, shell commands for which the user does not have the necessary roles are marked as unavailable.
|
Currently, the shell’s |
Conversely, for the Dashboard, the UI does not show pages or page elements for which the user is not authorized.
Securing the Spring Boot Management Endpoints
When security is enabled, the
Spring Boot HTTP Management Endpoints
are secured in the same way as the other REST endpoints. The management REST endpoints
are available under /management and require the MANAGEMENT role.
The default configuration in dataflow-server-defaults.yml is as follows:
management:
endpoints:
web:
base-path: /management
security:
roles: MANAGE| Currently, you should not customize the default management path. |
8.4. Setting up UAA Authentication
For local deployment scenarios, we recommend using the CloudFoundry User Account and Authentication (UAA) Server, which is OpenID certified. While the UAA is used by Cloud Foundry, it is also a fully featured stand alone OAuth2 server with enterprise features, such as LDAP integration.
Requirements
You need to check out, build and run UAA. To do so, make sure that you:
-
Use Java 8.
-
Have Git installed.
-
Have the CloudFoundry UAA Command Line Client installed.
-
Use a different host name for UAA when running on the same machine — for example,
uaa/.
If you run into issues installing uaac, you may have to set the GEM_HOME environment
variable:
export GEM_HOME="$HOME/.gem"You should also ensure that ~/.gem/gems/cf-uaac-4.2.0/bin has been added to your path.
Prepare UAA for JWT
As the UAA is an OpenID provider and uses JSON Web Tokens (JWT), it needs to have a private key for signing those JWTs:
openssl genrsa -out signingkey.pem 2048
openssl rsa -in signingkey.pem -pubout -out verificationkey.pem
export JWT_TOKEN_SIGNING_KEY=$(cat signingkey.pem)
export JWT_TOKEN_VERIFICATION_KEY=$(cat verificationkey.pem)Later, once the UAA is started, you can see the keys when you access uaa:8080/uaa/token_keys.
Here, the uaa in the URL uaa:8080/uaa/token_keys is the hostname.
|
Download and Start UAA
To download and install UAA, run the following commands:
git clone https://github.com/pivotal/uaa-bundled.git
cd uaa-bundled
./mvnw clean install
java -jar target/uaa-bundled-1.0.0.BUILD-SNAPSHOT.jarThe configuration of the UAA is driven by a YAML file uaa.yml, or you can script the configuration
using the UAA Command Line Client:
uaac target http://uaa:8080/uaa
uaac token client get admin -s adminsecret
uaac client add dataflow \
--name dataflow \
--secret dataflow \
--scope cloud_controller.read,cloud_controller.write,openid,password.write,scim.userids,sample.create,sample.view,dataflow.create,dataflow.deploy,dataflow.destroy,dataflow.manage,dataflow.modify,dataflow.schedule,dataflow.view \
--authorized_grant_types password,authorization_code,client_credentials,refresh_token \
--authorities uaa.resource,dataflow.create,dataflow.deploy,dataflow.destroy,dataflow.manage,dataflow.modify,dataflow.schedule,dataflow.view,sample.view,sample.create\
--redirect_uri http://localhost:9393/login \
--autoapprove openid
uaac group add "sample.view"
uaac group add "sample.create"
uaac group add "dataflow.view"
uaac group add "dataflow.create"
uaac user add springrocks -p mysecret --emails [email protected]
uaac user add vieweronly -p mysecret --emails [email protected]
uaac member add "sample.view" springrocks
uaac member add "sample.create" springrocks
uaac member add "dataflow.view" springrocks
uaac member add "dataflow.create" springrocks
uaac member add "sample.view" vieweronlyThe preceding script sets up the dataflow client as well as two users:
-
User springrocks has have both scopes:
sample.viewandsample.create. -
User vieweronly has only one scope:
sample.view.
Once added, you can quickly double-check that the UAA has the users created:
curl -v -d"username=springrocks&password=mysecret&client_id=dataflow&grant_type=password" -u "dataflow:dataflow" http://uaa:8080/uaa/oauth/token -d 'token_format=opaque'The preceding command should produce output similar to the following:
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to uaa (127.0.0.1) port 8080 (#0)
* Server auth using Basic with user 'dataflow'
> POST /uaa/oauth/token HTTP/1.1
> Host: uaa:8080
> Authorization: Basic ZGF0YWZsb3c6ZGF0YWZsb3c=
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 97
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 97 out of 97 bytes
< HTTP/1.1 200
< Cache-Control: no-store
< Pragma: no-cache
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Date: Thu, 31 Oct 2019 21:22:59 GMT
<
* Connection #0 to host uaa left intact
{"access_token":"0329c8ecdf594ee78c271e022138be9d","token_type":"bearer","id_token":"eyJhbGciOiJSUzI1NiIsImprdSI6Imh0dHBzOi8vbG9jYWxob3N0OjgwODAvdWFhL3Rva2VuX2tleXMiLCJraWQiOiJsZWdhY3ktdG9rZW4ta2V5IiwidHlwIjoiSldUIn0.eyJzdWIiOiJlZTg4MDg4Ny00MWM2LTRkMWQtYjcyZC1hOTQ4MmFmNGViYTQiLCJhdWQiOlsiZGF0YWZsb3ciXSwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDkwL3VhYS9vYXV0aC90b2tlbiIsImV4cCI6MTU3MjYwMDE3OSwiaWF0IjoxNTcyNTU2OTc5LCJhbXIiOlsicHdkIl0sImF6cCI6ImRhdGFmbG93Iiwic2NvcGUiOlsib3BlbmlkIl0sImVtYWlsIjoic3ByaW5ncm9ja3NAc29tZXBsYWNlLmNvbSIsInppZCI6InVhYSIsIm9yaWdpbiI6InVhYSIsImp0aSI6IjAzMjljOGVjZGY1OTRlZTc4YzI3MWUwMjIxMzhiZTlkIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsImNsaWVudF9pZCI6ImRhdGFmbG93IiwiY2lkIjoiZGF0YWZsb3ciLCJncmFudF90eXBlIjoicGFzc3dvcmQiLCJ1c2VyX25hbWUiOiJzcHJpbmdyb2NrcyIsInJldl9zaWciOiJlOTkyMDQxNSIsInVzZXJfaWQiOiJlZTg4MDg4Ny00MWM2LTRkMWQtYjcyZC1hOTQ4MmFmNGViYTQiLCJhdXRoX3RpbWUiOjE1NzI1NTY5Nzl9.bqYvicyCPB5cIIu_2HEe5_c7nSGXKw7B8-reTvyYjOQ2qXSMq7gzS4LCCQ-CMcb4IirlDaFlQtZJSDE-_UsM33-ThmtFdx--TujvTR1u2nzot4Pq5A_ThmhhcCB21x6-RNNAJl9X9uUcT3gKfKVs3gjE0tm2K1vZfOkiGhjseIbwht2vBx0MnHteJpVW6U0pyCWG_tpBjrNBSj9yLoQZcqrtxYrWvPHaa9ljxfvaIsOnCZBGT7I552O1VRHWMj1lwNmRNZy5koJFPF7SbhiTM8eLkZVNdR3GEiofpzLCfoQXrr52YbiqjkYT94t3wz5C6u1JtBtgc2vq60HmR45bvg","refresh_token":"6ee95d017ada408697f2d19b04f7aa6c-r","expires_in":43199,"scope":"scim.userids openid sample.create cloud_controller.read password.write cloud_controller.write sample.view","jti":"0329c8ecdf594ee78c271e022138be9d"}By using the token_format parameter, you can request the token to be either:
-
opaque
-
jwt
9. Configuration - Local
9.1. Feature Toggles
Spring Cloud Data Flow Server offers specific set of features that can be enabled/disabled when launching. These features include all the lifecycle operations and REST endpoints (server and client implementations, including the shell and the UI) for:
-
Streams (requires Skipper)
-
Tasks
-
Task Scheduler
One can enable and disable these features by setting the following boolean properties when launching the Data Flow server:
-
spring.cloud.dataflow.features.streams-enabled -
spring.cloud.dataflow.features.tasks-enabled -
spring.cloud.dataflow.features.schedules-enabled
By default, stream (requires Skipper), and tasks are enabled and Task Scheduler is disabled by default.
The REST /about endpoint provides information on the features that have been enabled and disabled.
9.2. Database
A relational database is used to store stream and task definitions as well as the state of executed tasks. Spring Cloud Data Flow provides schemas for MariaDB, MySQL, Oracle, PostgreSQL, Db2, SQL Server, and H2. The schema is automatically created when the server starts.
| The JDBC drivers for MariaDB, MySQL (via the MariaDB driver), PostgreSQL, SQL Server are available without additional configuration. To use any other database you need to put the corresponding JDBC driver jar on the classpath of the server as described here. |
To configure a database the following properties must be set:
-
spring.datasource.url -
spring.datasource.username -
spring.datasource.password -
spring.datasource.driver-class-name
The username and password are the same regardless of the database. However, the url and driver-class-name vary per database as follows.
| Database | spring.datasource.url | spring.datasource.driver-class-name | Driver included |
|---|---|---|---|
MariaDB 10.4+ |
jdbc:mariadb://${db-hostname}:${db-port}/${db-name} |
org.mariadb.jdbc.Driver |
Yes |
MySQL 5.7 |
jdbc:mysql://${db-hostname}:${db-port}/${db-name}?permitMysqlScheme |
org.mariadb.jdbc.Driver |
Yes |
MySQL 8.0+ |
jdbc:mysql://${db-hostname}:${db-port}/${db-name}?allowPublicKeyRetrieval=true&useSSL=false&autoReconnect=true&permitMysqlScheme[1] |
org.mariadb.jdbc.Driver |
Yes |
PostgresSQL |
jdbc:postgres://${db-hostname}:${db-port}/${db-name} |
org.postgresql.Driver |
Yes |
SQL Server |
jdbc:sqlserver://${db-hostname}:${db-port};databasename=${db-name}&encrypt=false |
com.microsoft.sqlserver.jdbc.SQLServerDriver |
Yes |
DB2 |
jdbc:db2://${db-hostname}:${db-port}/{db-name} |
com.ibm.db2.jcc.DB2Driver |
No |
Oracle |
jdbc:oracle:thin:@${db-hostname}:${db-port}/{db-name} |
oracle.jdbc.OracleDriver |
No |
9.2.1. H2
When no other database is configured then Spring Cloud Data Flow uses an embedded instance of the H2 database as the default.
| H2 is good for development purposes but is not recommended for production use nor is it supported as an external mode. |
9.2.2. Database configuration
When running locally, the database properties can be passed as environment variables or command-line arguments to the Data Flow Server. For example, to start the server with MariaDB using command line arguments execute the following command:
java -jar spring-cloud-dataflow-server/target/spring-cloud-dataflow-server-2.10.3.jar \
--spring.datasource.url=jdbc:mariadb://localhost:3306/mydb \
--spring.datasource.username=user \
--spring.datasource.password=pass \
--spring.datasource.driver-class-name=org.mariadb.jdbc.DriverLikewise, to start the server with MariaDB using environment variables execute the following command:
SPRING_DATASOURCE_URL=jdbc:mariadb://localhost:3306/mydb
SPRING_DATASOURCE_USERNAME=user
SPRING_DATASOURCE_PASSWORD=pass
SPRING_DATASOURCE_DRIVER_CLASS_NAME=org.mariadb.jdbc.Driver
java -jar spring-cloud-dataflow-server/target/spring-cloud-dataflow-server-2.10.3.jar9.2.3. Adding a Custom JDBC Driver
To add a custom driver for the database (for example, Oracle), you should rebuild the Data Flow Server and add the dependency to the Maven pom.xml file.
You need to modify the maven pom.xml of spring-cloud-dataflow-server module.
There are GA release tags in GitHub repository, so you can switch to desired GA tags to add the drivers on the production-ready codebase.
To add a custom JDBC driver dependency for the Spring Cloud Data Flow server:
-
Select the tag that corresponds to the version of the server you want to rebuild and clone the github repository.
-
Edit the spring-cloud-dataflow-server/pom.xml and, in the
dependenciessection, add the dependency for the database driver required. In the following example , an Oracle driver has been chosen:
<dependencies>
...
<dependency>
<groupId>com.oracle.jdbc</groupId>
<artifactId>ojdbc8</artifactId>
<version>12.2.0.1</version>
</dependency>
...
</dependencies>-
Build the application as described in Building Spring Cloud Data Flow
You can also provide default values when rebuilding the server by adding the necessary properties to the dataflow-server.yml file, as shown in the following example for PostgreSQL:
spring:
datasource:
url: jdbc:postgresql://localhost:5432/mydb
username: myuser
password: mypass
driver-class-name:org.postgresql.Driver9.2.4. Schema Handling
On default database schema is managed with Flyway which is convenient if it’s possible to give enough permissions to a database user.
Here’s a description what happens when Skipper server is started:
-
Flyway checks if
flyway_schema_historytable exists. -
Does a baseline(to version 1) if schema is not empty as Dataflow tables may be in place if a shared DB is used.
-
If schema is empty, flyway assumes to start from a scratch.
-
Goes through all needed schema migrations.
Here’s a description what happens when Dataflow server is started:
-
Flyway checks if
flyway_schema_history_dataflowtable exists. -
Does a baseline(to version 1) if schema is not empty as Skipper tables may be in place if a shared DB is used.
-
If schema is empty, flyway assumes to start from a scratch.
-
Goes through all needed schema migrations.
|
We have schema ddl’s in our source code
schemas
which can be used manually if Flyway is disabled by using configuration
|
9.3. Deployer Properties
You can use the following configuration properties of the Local deployer to customize how Streams and Tasks are deployed.
When deploying using the Data Flow shell, you can use the syntax deployer.<appName>.local.<deployerPropertyName>. See below for an example shell usage.
These properties are also used when configuring Local Task Platforms in the Data Flow server and local platforms in Skipper for deploying Streams.
| Deployer Property Name | Description | Default Value |
|---|---|---|
workingDirectoriesRoot |
Directory in which all created processes will run and create log files. |
java.io.tmpdir |
envVarsToInherit |
Array of regular expression patterns for environment variables that are passed to launched applications. |
<"TMP", "LANG", "LANGUAGE", "LC_.*", "PATH", "SPRING_APPLICATION_JSON"> on windows and <"TMP", "LANG", "LANGUAGE", "LC_.*", "PATH"> on Unix |
deleteFilesOnExit |
Whether to delete created files and directories on JVM exit. |
true |
javaCmd |
Command to run java |
java |
shutdownTimeout |
Max number of seconds to wait for app shutdown. |
30 |
javaOpts |
The Java Options to pass to the JVM, e.g -Dtest=foo |
<none> |
inheritLogging |
allow logging to be redirected to the output stream of the process that triggered child process. |
false |
debugPort |
Port for remote debugging |
<none> |
As an example, to set Java options for the time application in the ticktock stream, use the following stream deployment properties.
dataflow:> stream create --name ticktock --definition "time --server.port=9000 | log"
dataflow:> stream deploy --name ticktock --properties "deployer.time.local.javaOpts=-Xmx2048m -Dtest=foo"As a convenience, you can set the deployer.memory property to set the Java option -Xmx, as shown in the following example:
dataflow:> stream deploy --name ticktock --properties "deployer.time.memory=2048m"At deployment time, if you specify an -Xmx option in the deployer.<app>.local.javaOpts property in addition to a value of the deployer.<app>.local.memory option, the value in the javaOpts property has precedence. Also, the javaOpts property set when deploying the application has precedence over the Data Flow Server’s spring.cloud.deployer.local.javaOpts property.
9.4. Logging
Spring Cloud Data Flow local server is automatically configured to use RollingFileAppender for logging.
The logging configuration is located on the classpath contained in a file named logback-spring.xml.
By default, the log file is configured to use:
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/spring-cloud-dataflow-server-local.log}"/>with the logback configuration for the RollingPolicy:
<appender name="FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_FILE}</file>
<rollingPolicy
class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<!-- daily rolling -->
<fileNamePattern>${LOG_FILE}.${LOG_FILE_ROLLING_FILE_NAME_PATTERN:-%d{yyyy-MM-dd}}.%i.gz</fileNamePattern>
<maxFileSize>${LOG_FILE_MAX_SIZE:-100MB}</maxFileSize>
<maxHistory>${LOG_FILE_MAX_HISTORY:-30}</maxHistory>
<totalSizeCap>${LOG_FILE_TOTAL_SIZE_CAP:-500MB}</totalSizeCap>
</rollingPolicy>
<encoder>
<pattern>${FILE_LOG_PATTERN}</pattern>
</encoder>
</appender>
To check the java.io.tmpdir for the current Spring Cloud Data Flow Server local server,
jinfo <pid> | grep "java.io.tmpdir"If you want to change or override any of the properties LOG_FILE, LOG_PATH, LOG_TEMP, LOG_FILE_MAX_SIZE, LOG_FILE_MAX_HISTORY and LOG_FILE_TOTAL_SIZE_CAP, please set them as system properties.
9.5. Streams
Data Flow Server delegates to the Skipper server the management of the Stream’s lifecycle. Set the configuration property spring.cloud.skipper.client.serverUri to the location of Skipper, e.g.
$ java -jar spring-cloud-dataflow-server-2.10.3.jar --spring.cloud.skipper.client.serverUri=https://192.51.100.1:7577/apiThe configuration of how streams are deployed and to which platforms, is done by configuration of platform accounts on the Skipper server.
See the documentation on platforms for more information.
9.6. Tasks
The Data Flow server is responsible for deploying Tasks.
Tasks that are launched by Data Flow write their state to the same database that is used by the Data Flow server.
For Tasks which are Spring Batch Jobs, the job and step execution data is also stored in this database.
As with streams launched by Skipper, Tasks can be launched to multiple platforms.
If no platform is defined, a platform named default is created using the default values of the class LocalDeployerProperties, which is summarized in the table Local Deployer Properties
To configure new platform accounts for the local platform, provide an entry under the spring.cloud.dataflow.task.platform.local section in your application.yaml file or via another Spring Boot supported mechanism.
In the following example, two local platform accounts named localDev and localDevDebug are created.
The keys such as shutdownTimeout and javaOpts are local deployer properties.
spring:
cloud:
dataflow:
task:
platform:
local:
accounts:
localDev:
shutdownTimeout: 60
javaOpts: "-Dtest=foo -Xmx1024m"
localDevDebug:
javaOpts: "-Xdebug -Xmx2048m"
By defining one platform as default allows you to skip using platformName where its use would otherwise be required.
|
When launching a task, pass the value of the platform account name using the task launch option --platformName If you do not pass a value for platformName, the value default will be used.
| When deploying a task to multiple platforms, the configuration of the task needs to connect to the same database as the Data Flow Server. |
You can configure the Data Flow server that is running locally to deploy tasks to Cloud Foundry or Kubernetes. See the sections on Cloud Foundry Task Platform Configuration and Kubernetes Task Platform Configuration for more information.
Detailed examples for launching and scheduling tasks across multiple platforms, are available in this section Multiple Platform Support for Tasks on dataflow.spring.io.
9.7. Security Configuration
9.7.1. CloudFoundry User Account and Authentication (UAA) Server
See the CloudFoundry User Account and Authentication (UAA) Server configuration section for details how to configure for local testing and development.
9.7.2. LDAP Authentication
LDAP Authentication (Lightweight Directory Access Protocol) is indirectly provided by Spring Cloud Data Flow using the UAA. The UAA itself provides comprehensive LDAP support.
|
While you may use your own OAuth2 authentication server, the LDAP support documented here requires using the UAA as authentication server. For any other provider, please consult the documentation for that particular provider. |
The UAA supports authentication against an LDAP (Lightweight Directory Access Protocol) server using the following modes:
|
When integrating with an external identity provider such as LDAP, authentication within the UAA becomes chained. UAA first attempts to authenticate with a user’s credentials against the UAA user store before the external provider, LDAP. For more information, see Chained Authentication in the User Account and Authentication LDAP Integration GitHub documentation. |
LDAP Role Mapping
The OAuth2 authentication server (UAA), provides comprehensive support for mapping LDAP groups to OAuth scopes.
The following options exist:
-
ldap/ldap-groups-null.xmlNo groups will be mapped -
ldap/ldap-groups-as-scopes.xmlGroup names will be retrieved from an LDAP attribute. E.g.CN -
ldap/ldap-groups-map-to-scopes.xmlGroups will be mapped to UAA groups using the external_group_mapping table
These values are specified via the configuration property ldap.groups.file controls. Under the covers
these values reference a Spring XML configuration file.
|
During test and development it might be necessary to make frequent changes to LDAP groups and users and see those reflected in the UAA. However, user information is cached for the duration of the login. The following script helps to retrieve the updated information quickly: |
9.7.3. Spring Security OAuth2 Resource/Authorization Server Sample
For local testing and development, you may also use the Resource and Authorization Server support provided by Spring Security. It allows you to easily create your own OAuth2 Server by configuring the SecurityFilterChain.
Samples can be found at: Spring Security Samples
9.7.4. Data Flow Shell Authentication
When using the Shell, the credentials can either be provided via username and password or by specifying a credentials-provider command. If your OAuth2 provider supports the Password Grant Type you can start the Data Flow Shell with:
$ java -jar spring-cloud-dataflow-shell-2.10.3.jar \
--dataflow.uri=http://localhost:9393 \ (1)
--dataflow.username=my_username \ (2)
--dataflow.password=my_password \ (3)
--skip-ssl-validation \ (4)| 1 | Optional, defaults to localhost:9393. |
| 2 | Mandatory. |
| 3 | If the password is not provided, the user is prompted for it. |
| 4 | Optional, defaults to false, ignores certificate errors (when using self-signed certificates). Use cautiously! |
| Keep in mind that when authentication for Spring Cloud Data Flow is enabled, the underlying OAuth2 provider must support the Password OAuth2 Grant Type if you want to use the Shell via username/password authentication. |
From within the Data Flow Shell you can also provide credentials by using the following command:
server-unknown:>dataflow config server \
--uri http://localhost:9393 \ (1)
--username myuser \ (2)
--password mysecret \ (3)
--skip-ssl-validation \ (4)| 1 | Optional, defaults to localhost:9393. |
| 2 | Mandatory.. |
| 3 | If security is enabled, and the password is not provided, the user is prompted for it. |
| 4 | Optional, ignores certificate errors (when using self-signed certificates). Use cautiously! |
The following image shows a typical shell command to connect to and authenticate a Data Flow Server:
Once successfully targeted, you should see the following output:
dataflow:>dataflow config info
dataflow config info
╔═══════════╤═══════════════════════════════════════╗
║Credentials│[username='my_username, password=****']║
╠═══════════╪═══════════════════════════════════════╣
║Result │ ║
║Target │http://localhost:9393 ║
╚═══════════╧═══════════════════════════════════════╝Alternatively, you can specify the credentials-provider command in order to
pass-in a bearer token directly, instead of providing a username and password.
This works from within the shell or by providing the
--dataflow.credentials-provider-command command-line argument when starting the Shell.
|
When using the credentials-provider command, please be aware that your specified command must return a Bearer token (Access Token prefixed with Bearer). For instance, in Unix environments the following simplistic command can be used: |
9.8. About API Configuration
The Spring Cloud Data Flow About Restful API result contains a display name, version, and, if specified, a URL for each of the major dependencies that comprise Spring Cloud Data Flow. The result (if enabled) also contains the sha1 and or sha256 checksum values for the shell dependency. The information that is returned for each of the dependencies is configurable by setting the following properties:
| Property Name | Description |
|---|---|
spring.cloud.dataflow.version-info.spring-cloud-dataflow-core.name |
Name to be used for the core |
spring.cloud.dataflow.version-info.spring-cloud-dataflow-core.version |
Version to be used for the core |
spring.cloud.dataflow.version-info.spring-cloud-dataflow-dashboard.name |
Name to be used for the dashboard |
spring.cloud.dataflow.version-info.spring-cloud-dataflow-dashboard.version |
Version to be used for the dashboard |
spring.cloud.dataflow.version-info.spring-cloud-dataflow-implementation.name |
Name to be used for the implementation |
spring.cloud.dataflow.version-info.spring-cloud-dataflow-implementation.version |
Version to be used for the implementation |
spring.cloud.dataflow.version-info.spring-cloud-dataflow-shell.name |
Name to be used for the shell |
spring.cloud.dataflow.version-info.spring-cloud-dataflow-shell.version |
Version to be used for the shell |
spring.cloud.dataflow.version-info.spring-cloud-dataflow-shell.url |
URL to be used for downloading the shell dependency |
spring.cloud.dataflow.version-info.spring-cloud-dataflow-shell.checksum-sha1 |
Sha1 checksum value that is returned with the shell dependency info |
spring.cloud.dataflow.version-info.spring-cloud-dataflow-shell.checksum-sha256 |
Sha256 checksum value that is returned with the shell dependency info |
spring.cloud.dataflow.version-info.spring-cloud-dataflow-shell.checksum-sha1-url |
if |
spring.cloud.dataflow.version-info.spring-cloud-dataflow-shell.checksum-sha256-url |
if the |
9.8.1. Enabling Shell Checksum values
By default, checksum values are not displayed for the shell dependency. If
you need this feature enabled, set the
spring.cloud.dataflow.version-info.dependency-fetch.enabled property to true.
9.8.2. Reserved Values for URLs
There are reserved values (surrounded by curly braces) that you can insert into the URL that will make sure that the links are up to date:
-
repository: if using a build-snapshot, milestone, or release candidate of Data Flow, the repository refers to the repo-spring-io repository. Otherwise, it refers to Maven Central. -
version: Inserts the version of the jar/pom.
For example,
https://myrepository/org/springframework/cloud/spring-cloud-dataflow-shell/\{version}/spring-cloud-dataflow-shell-\{version}.jarproduces
https://myrepository/org/springframework/cloud/spring-cloud-dataflow-shell/2.1.4/spring-cloud-dataflow-shell-2.1.4.jarif you were using the 2.1.4 version of the Spring Cloud Data Flow Shell.
10. Configuration - Cloud Foundry
This section describes how to configure Spring Cloud Data Flow server’s features, such as security and which relational database to use. It also describes how to configure Spring Cloud Data Flow shell’s features.
10.1. Feature Toggles
Data Flow server offers a specific set of features that you can enable or disable when launching. These features include all the lifecycle operations and REST endpoints (server, client implementations including Shell and the UI) for:
-
Streams
-
Tasks
You can enable or disable these features by setting the following boolean properties when you launch the Data Flow server:
-
spring.cloud.dataflow.features.streams-enabled -
spring.cloud.dataflow.features.tasks-enabled
By default, all features are enabled.
The REST endpoint (/features) provides information on the enabled and disabled features.
10.2. Deployer Properties
You can use the following configuration properties of the Data Flow server’s Cloud Foundry deployer to customize how applications are deployed.
When deploying with the Data Flow shell, you can use the syntax deployer.<appName>.cloudfoundry.<deployerPropertyName>. See below for an example shell usage.
These properties are also used when configuring the Cloud Foundry Task platforms in the Data Flow server and and Kubernetes platforms in Skipper for deploying Streams.
| Deployer Property Name | Description | Default Value |
|---|---|---|
services |
The names of services to bind to the deployed application. |
<none> |
host |
The host name to use as part of the route. |
hostname derived by Cloud Foundry |
domain |
The domain to use when mapping routes for the application. |
<none> |
routes |
The list of routes that the application should be bound to. Mutually exclusive with host and domain. |
<none> |
buildpack |
The buildpack to use for deploying the application. Deprecated use buildpacks. |
|
buildpacks |
The list of buildpacks to use for deploying the application. |
|
memory |
The amount of memory to allocate. Default unit is mebibytes, 'M' and 'G" suffixes supported |
1024m |
disk |
The amount of disk space to allocate. Default unit is mebibytes, 'M' and 'G" suffixes supported. |
1024m |
healthCheck |
The type of health check to perform on deployed application. Values can be HTTP, NONE, PROCESS, and PORT |
PORT |
healthCheckHttpEndpoint |
The path that the http health check will use, |
/health |
healthCheckTimeout |
The timeout value for health checks in seconds. |
120 |
instances |
The number of instances to run. |
1 |
enableRandomAppNamePrefix |
Flag to enable prefixing the app name with a random prefix. |
true |
apiTimeout |
Timeout for blocking API calls, in seconds. |
360 |
statusTimeout |
Timeout for status API operations in milliseconds |
5000 |
useSpringApplicationJson |
Flag to indicate whether application properties are fed into |
true |
stagingTimeout |
Timeout allocated for staging the application. |
15 minutes |
startupTimeout |
Timeout allocated for starting the application. |
5 minutes |
appNamePrefix |
String to use as prefix for name of deployed application |
The Spring Boot property |
deleteRoutes |
Whether to also delete routes when un-deploying an application. |
true |
javaOpts |
The Java Options to pass to the JVM, e.g -Dtest=foo |
<none> |
pushTasksEnabled |
Whether to push task applications or assume that the application already exists when launched. |
true |
autoDeleteMavenArtifacts |
Whether to automatically delete Maven artifacts from the local repository when deployed. |
true |
env.<key> |
Defines a top level environment variable. This is useful for customizing Java build pack configuration which must be included as top level environment variables in the application manifest, as the Java build pack does not recognize |
The deployer determines if the app has Java CfEnv in its classpath. If so, it applies the required configuration. |
Here are some examples using the Cloud Foundry deployment properties:
-
You can set the buildpack that is used to deploy each application. For example, to use the Java offline buildback, set the following environment variable:
cf set-env dataflow-server SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_BUILDPACKS java_buildpack_offline-
Setting
buildpackis now deprecated in favour ofbuildpackswhich allows you to pass on more than one if needed. More about this can be found from How Buildpacks Work. -
You can customize the health check mechanism used by Cloud Foundry to assert whether apps are running by using the
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_HEALTH_CHECKenvironment variable. The current supported options arehttp(the default),port, andnone.
You can also set environment variables that specify the HTTP-based health check endpoint and timeout: SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_HEALTH_CHECK_ENDPOINT and SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_HEALTH_CHECK_TIMEOUT, respectively. These default to /health (the Spring Boot default location) and 120 seconds.
-
You can also specify deployment properties by using the DSL. For instance, if you want to set the allocated memory for the
httpapplication to 512m and also bind a postgres service to thejdbcapplication, you can run the following commands:
dataflow:> stream create --name postgresstream --definition "http | jdbc --tableName=names --columns=name"
dataflow:> stream deploy --name postgresstream --properties "deployer.http.memory=512, deployer.jdbc.cloudfoundry.services=postgres"|
You can configure these settings separately for stream and task apps. To alter settings for tasks,
substitute |
10.3. Tasks
The Data Flow server is responsible for deploying Tasks.
Tasks that are launched by Data Flow write their state to the same database that is used by the Data Flow server.
For Tasks which are Spring Batch Jobs, the job and step execution data is also stored in this database.
As with Skipper, Tasks can be launched to multiple platforms.
When Data Flow is running on Cloud Foundry, a Task platfom must be defined.
To configure new platform accounts that target Cloud Foundry, provide an entry under the spring.cloud.dataflow.task.platform.cloudfoundry section in your application.yaml file for via another Spring Boot supported mechanism.
In the following example, two Cloud Foundry platform accounts named dev and qa are created.
The keys such as memory and disk are Cloud Foundry Deployer Properties.
spring:
cloud:
dataflow:
task:
platform:
cloudfoundry:
accounts:
dev:
connection:
url: https://api.run.pivotal.io
org: myOrg
space: mySpace
domain: cfapps.io
username: [email protected]
password: drowssap
skipSslValidation: false
deployment:
memory: 512m
disk: 2048m
instances: 4
services: rabbit,postgres
appNamePrefix: dev1
qa:
connection:
url: https://api.run.pivotal.io
org: myOrgQA
space: mySpaceQA
domain: cfapps.io
username: [email protected]
password: drowssap
skipSslValidation: true
deployment:
memory: 756m
disk: 724m
instances: 2
services: rabbitQA,postgresQA
appNamePrefix: qa1
By defining one platform as default allows you to skip using platformName where its use would otherwise be required.
|
When launching a task, pass the value of the platform account name using the task launch option --platformName If you do not pass a value for platformName, the value default will be used.
| When deploying a task to multiple platforms, the configuration of the task needs to connect to the same database as the Data Flow Server. |
You can configure the Data Flow server that is on Cloud Foundry to deploy tasks to Cloud Foundry or Kubernetes. See the section on Kubernetes Task Platform Configuration for more information.
Detailed examples for launching and scheduling tasks across multiple platforms, are available in this section Multiple Platform Support for Tasks on dataflow.spring.io.
10.4. Application Names and Prefixes
To help avoid clashes with routes across spaces in Cloud Foundry, a naming strategy that provides a random prefix to a
deployed application is available and is enabled by default. You can override the default configurations
and set the respective properties by using cf set-env commands.
For instance, if you want to disable the randomization, you can override it by using the following command:
cf set-env dataflow-server SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_ENABLE_RANDOM_APP_NAME_PREFIX false10.5. Custom Routes
As an alternative to a random name or to get even more control over the hostname used by the deployed apps, you can use custom deployment properties, as the following example shows:
dataflow:>stream create foo --definition "http | log"
sdataflow:>stream deploy foo --properties "deployer.http.cloudfoundry.domain=mydomain.com,
deployer.http.cloudfoundry.host=myhost,
deployer.http.cloudfoundry.route-path=my-path"The preceding example binds the http app to the myhost.mydomain.com/my-path URL. Note that this
example shows all of the available customization options. In practice, you can use only one or two out of the three.
10.6. Docker Applications
Starting with version 1.2, it is possible to register and deploy Docker based apps as part of streams and tasks by using Data Flow for Cloud Foundry.
If you use Spring Boot and RabbitMQ-based Docker images, you can provide a common deployment property
to facilitate binding the apps to the RabbitMQ service. Assuming your RabbitMQ service is named rabbit, you can provide the following:
cf set-env dataflow-server SPRING_APPLICATION_JSON '{"spring.cloud.dataflow.applicationProperties.stream.spring.rabbitmq.addresses": "${vcap.services.rabbit.credentials.protocols.amqp.uris}"}'For Spring Cloud Task apps, you can use something similar to the following, if you use a database service instance named postgres:
cf set-env SPRING_DATASOURCE_URL '${vcap.services.postgres.credentials.jdbcUrl}'
cf set-env SPRING_DATASOURCE_USERNAME '${vcap.services.postgres.credentials.username}'
cf set-env SPRING_DATASOURCE_PASSWORD '${vcap.services.postgres.credentials.password}'
cf set-env SPRING_DATASOURCE_DRIVER_CLASS_NAME 'org.mariadb.jdbc.Driver'For non-Java or non-Boot applications, your Docker app must parse the VCAP_SERVICES variable in order to bind to any available services.
|
Passing application properties
When using non-Boot applications, chances are that you want to pass the application properties by using traditional
environment variables, as opposed to using the special |
10.7. Application-level Service Bindings
When deploying streams in Cloud Foundry, you can take advantage of application-specific service bindings, so not all services are globally configured for all the apps orchestrated by Spring Cloud Data Flow.
For instance, if you want to provide a postgres service binding only for the jdbc application in the following stream
definition, you can pass the service binding as a deployment property:
dataflow:>stream create --name httptojdbc --definition "http | jdbc"
dataflow:>stream deploy --name httptojdbc --properties "deployer.jdbc.cloudfoundry.services=postgresService"where postgresService is the name of the service specifically bound only to the jdbc application and the http
application does not get the binding by this method.
If you have more than one service to bind, they can be passed as comma-separated items
(for example: deployer.jdbc.cloudfoundry.services=postgresService,someService).
10.8. Configuring Service binding parameters
The CloudFoundry API supports providing configuration parameters when binding a service instance. Some service brokers require or recommend binding configuration. For example, binding the Google Cloud Platform service using the CF CLI looks something like:
cf bind-service my-app my-google-bigquery-example -c '{"role":"bigquery.user"}'Likewise the NFS Volume Service supports binding configuration such as:
cf bind-service my-app nfs_service_instance -c '{"uid":"1000","gid":"1000","mount":"/var/volume1","readonly":true}'Starting with version 2.0, Data Flow for Cloud Foundry allows you to provide binding configuration parameters may be provided in the app level or server level cloudfoundry.services deployment property. For example, to bind to the nfs service, as above :
dataflow:> stream deploy --name mystream --properties "deployer.<app>.cloudfoundry.services='nfs_service_instance uid:1000,gid:1000,mount:/var/volume1,readonly:true'"The format is intended to be compatible with the Data Flow DSL parser.
Generally, the cloudfoundry.services deployment property accepts a comma delimited value.
Since a comma is also used to separate configuration parameters, and to avoid white space issues, any item including configuration parameters must be enclosed in singe quotes. Valid values incude things like:
rabbitmq,'nfs_service_instance uid:1000,gid:1000,mount:/var/volume1,readonly:true',postgres,'my-google-bigquery-example role:bigquery.user'
Spaces are permitted within single quotes and = may be used instead of : to delimit key-value pairs.
|
10.9. User-provided Services
In addition to marketplace services, Cloud Foundry supports User-provided Services (UPS). Throughout this reference manual, regular services have been mentioned, but there is nothing precluding the use of User-provided Services as well, whether for use as the messaging middleware (for example, if you want to use an external Apache Kafka installation) or for use by some of the stream applications (for example, an Oracle Database).
Now we review an example of extracting and supplying the connection credentials from a UPS.
The following example shows a sample UPS setup for Apache Kafka:
cf create-user-provided-service kafkacups -p '{”brokers":"HOST:PORT","zkNodes":"HOST:PORT"}'The UPS credentials are wrapped within VCAP_SERVICES, and they can be supplied directly in the stream definition, as
the following example shows.
stream create fooz --definition "time | log"
stream deploy fooz --properties "app.time.spring.cloud.stream.kafka.binder.brokers=${vcap.services.kafkacups.credentials.brokers},app.time.spring.cloud.stream.kafka.binder.zkNodes=${vcap.services.kafkacups.credentials.zkNodes},app.log.spring.cloud.stream.kafka.binder.brokers=${vcap.services.kafkacups.credentials.brokers},app.log.spring.cloud.stream.kafka.binder.zkNodes=${vcap.services.kafkacups.credentials.zkNodes}"10.10. Database Connection Pool
As of Data Flow 2.0, the Spring Cloud Connector library is no longer used to create the DataSource. The library java-cfenv is now used which allows you to set Spring Boot properties to configure the connection pool.
10.11. Maximum Disk Quota
By default, every application in Cloud Foundry starts with 1G disk quota and this can be adjusted to a default maximum of 2G. The default maximum can also be overridden up to 10G by using Pivotal Cloud Foundry’s (PCF) Ops Manager GUI.
This configuration is relevant for Spring Cloud Data Flow because every task deployment is composed of applications (typically Spring Boot uber-jar’s), and those applications are resolved from a remote maven repository. After resolution, the application artifacts are downloaded to the local Maven Repository for caching and reuse. With this happening in the background, the default disk quota (1G) can fill up rapidly, especially when we experiment with streams that are made up of unique applications. In order to overcome this disk limitation and depending on your scaling requirements, you may want to change the default maximum from 2G to 10G. Let’s review the steps to change the default maximum disk quota allocation.
10.11.1. PCF’s Operations Manager
From PCF’s Ops Manager, select the “Pivotal Elastic Runtime” tile and navigate to the “Application Developer Controls” tab. Change the “Maximum Disk Quota per App (MB)” setting from 2048 (2G) to 10240 (10G). Save the disk quota update and click “Apply Changes” to complete the configuration override.
10.12. Scale Application
Once the disk quota change has been successfully applied and assuming you have a running application,
you can scale the application with a new disk_limit through the CF CLI, as the following example shows:
→ cf scale dataflow-server -k 10GB
Scaling app dataflow-server in org ORG / space SPACE as user...
OK
....
....
....
....
state since cpu memory disk details
#0 running 2016-10-31 03:07:23 PM 1.8% 497.9M of 1.1G 193.9M of 10GYou can then list the applications and see the new maximum disk space, as the following example shows:
→ cf apps
Getting apps in org ORG / space SPACE as user...
OK
name requested state instances memory disk urls
dataflow-server started 1/1 1.1G 10G dataflow-server.apps.io10.13. Managing Disk Use
Even when configuring the Data Flow server to use 10G of space, there is the possibility of exhausting
the available space on the local disk. To prevent this, jar artifacts downloaded from external sources, i.e., apps registered as http or maven resources, are automatically deleted whenever the application is deployed, whether or not the deployment request succeeds.
This behavior is optimal for production environments in which container runtime stability is more critical than I/O latency incurred during deployment.
In development environments deployment happens more frequently. Additionally, the jar artifact (or a lighter metadata jar) contains metadata describing application configuration properties
which is used by various operations related to application configuration, more frequently performed during pre-production activities (see Application Metadata for details).
To provide a more responsive interactive developer experience at the expense of more disk usage in pre-production environments, you can set the CloudFoundry deployer property autoDeleteMavenArtifacts to false.
If you deploy the Data Flow server by using the default port health check type, you must explicitly monitor the disk space on the server in order to avoid running out space.
If you deploy the server by using the http health check type (see the next example), the Data Flow server is restarted if there is low disk space.
This is due to Spring Boot’s Disk Space Health Indicator.
You can configure the settings of the Disk Space Health Indicator by using the properties that have the management.health.diskspace prefix.
For version 1.7, we are investigating the use of Volume Services for the Data Flow server to store .jar artifacts before pushing them to Cloud Foundry.
The following example shows how to deploy the http health check type to an endpoint called /management/health:
---
...
health-check-type: http
health-check-http-endpoint: /management/health10.14. Application Resolution Alternatives
Though we recommend using a Maven Artifactory for application Register a Stream Application, there might be situations where one of the following alternative approaches would make sense.
-
With the help of Spring Boot, we can serve static content in Cloud Foundry. A simple Spring Boot application can bundle all the required stream and task applications. By having it run on Cloud Foundry, the static application can then serve the über-jar’s. From the shell, you can, for example, register the application with the name
http-source.jarby using--uri=http://<Route-To-StaticApp>/http-source.jar. -
The über-jar’s can be hosted on any external server that’s reachable over HTTP. They can be resolved from raw GitHub URIs as well. From the shell, you can, for example, register the app with the name
http-source.jarby using--uri=http://<Raw_GitHub_URI>/http-source.jar. -
Static Buildpack support in Cloud Foundry is another option. A similar HTTP resolution works on this model, too.
-
Volume Services is another great option. The required über-jars can be hosted in an external file system. With the help of volume-services, you can, for example, register the application with the name
http-source.jarby using--uri=file://<Path-To-FileSystem>/http-source.jar.
10.15. Security
By default, the Data Flow server is unsecured and runs on an unencrypted HTTP connection. You can secure your REST endpoints
(as well as the Data Flow Dashboard) by enabling HTTPS and requiring clients to authenticate.
For more details about securing the
REST endpoints and configuring to authenticate against an OAUTH backend (UAA and SSO running on Cloud Foundry),
see the security section from the core Security Configuration. You can configure the security details in dataflow-server.yml or pass them as environment variables through cf set-env commands.
10.15.1. Authentication
Spring Cloud Data Flow can either integrate with Pivotal Single Sign-On Service (for example, on PWS) or Cloud Foundry User Account and Authentication (UAA) Server.
Pivotal Single Sign-On Service
When deploying Spring Cloud Data Flow to Cloud Foundry, you can bind the application to the Pivotal Single Sign-On Service. By doing so, Spring Cloud Data Flow takes advantage of the Java CFEnv, which provides Cloud Foundry-specific auto-configuration support for OAuth 2.0.
To do so, bind the Pivotal Single Sign-On Service to your Data Flow Server application and provide the following properties:
SPRING_CLOUD_DATAFLOW_SECURITY_CFUSEUAA: false (1)
SECURITY_OAUTH2_CLIENT_CLIENTID: "${security.oauth2.client.clientId}"
SECURITY_OAUTH2_CLIENT_CLIENTSECRET: "${security.oauth2.client.clientSecret}"
SECURITY_OAUTH2_CLIENT_ACCESSTOKENURI: "${security.oauth2.client.accessTokenUri}"
SECURITY_OAUTH2_CLIENT_USERAUTHORIZATIONURI: "${security.oauth2.client.userAuthorizationUri}"
SECURITY_OAUTH2_RESOURCE_USERINFOURI: "${security.oauth2.resource.userInfoUri}"| 1 | It is important that the property spring.cloud.dataflow.security.cf-use-uaa is set to false |
Authorization is similarly supported for non-Cloud Foundry security scenarios. See the security section from the core Data Flow Security Configuration.
As the provisioning of roles can vary widely across environments, we by default assign all Spring Cloud Data Flow roles to users.
You can customize this behavior by providing your own AuthoritiesExtractor.
The following example shows one possible approach to set the custom AuthoritiesExtractor on the UserInfoTokenServices:
public class MyUserInfoTokenServicesPostProcessor
implements BeanPostProcessor {
@Override
public Object postProcessBeforeInitialization(Object bean, String beanName) {
if (bean instanceof UserInfoTokenServices) {
final UserInfoTokenServices userInfoTokenServices == (UserInfoTokenServices) bean;
userInfoTokenServices.setAuthoritiesExtractor(ctx.getBean(AuthoritiesExtractor.class));
}
return bean;
}
@Override
public Object postProcessAfterInitialization(Object bean, String beanName) {
return bean;
}
}Then you can declare it in your configuration class as follows:
@Bean
public BeanPostProcessor myUserInfoTokenServicesPostProcessor() {
BeanPostProcessor postProcessor == new MyUserInfoTokenServicesPostProcessor();
return postProcessor;
}Cloud Foundry UAA
The availability of Cloud Foundry User Account and Authentication (UAA) depends on the Cloud Foundry environment.
In order to provide UAA integration, you have to provide the necessary
OAuth2 configuration properties (for example, by setting the SPRING_APPLICATION_JSON
property).
The following JSON example shows how to create a security configuration:
{
"security.oauth2.client.client-id": "scdf",
"security.oauth2.client.client-secret": "scdf-secret",
"security.oauth2.client.access-token-uri": "https://login.cf.myhost.com/oauth/token",
"security.oauth2.client.user-authorization-uri": "https://login.cf.myhost.com/oauth/authorize",
"security.oauth2.resource.user-info-uri": "https://login.cf.myhost.com/userinfo"
}By default, the spring.cloud.dataflow.security.cf-use-uaa property is set to true. This property activates a special
AuthoritiesExtractor called CloudFoundryDataflowAuthoritiesExtractor.
If you do not use CloudFoundry UAA, you should set spring.cloud.dataflow.security.cf-use-uaa to false.
Under the covers, this AuthoritiesExtractor calls out to the
Cloud Foundry
Apps API and ensure that users are in fact Space Developers.
If the authenticated user is verified as a Space Developer, all roles are assigned.
10.16. Configuration Reference
You must provide several pieces of configuration. These are Spring Boot @ConfigurationProperties, so you can set
them as environment variables or by any other means that Spring Boot supports. The following listing is in environment
variable format, as that is an easy way to get started configuring Boot applications in Cloud Foundry.
Note that in the future, you will be able to deploy tasks to multiple platforms, but for 2.0.0.M1 you can deploy only to a single platform and the name must be default.
# Default values appear after the equal signs.
# Example values, typical for Pivotal Web Services, are included as comments.
# URL of the CF API (used when using cf login -a for example) - for example, https://api.run.pivotal.io
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_URL=
# The name of the organization that owns the space above - for example, youruser-org
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_ORG=
# The name of the space into which modules will be deployed - for example, development
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_SPACE=
# The root domain to use when mapping routes - for example, cfapps.io
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_DOMAIN=
# The user name and password of the user to use to create applications
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_USERNAME=
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_PASSWORD
# The identity provider to be used when accessing the Cloud Foundry API (optional).
# The passed string has to be a URL-Encoded JSON Object, containing the field origin with value as origin_key of an identity provider - for example, {"origin":"uaa"}
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_LOGIN_HINT=
# Whether to allow self-signed certificates during SSL validation (you should NOT do so in production)
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_SKIP_SSL_VALIDATION
# A comma-separated set of service instance names to bind to every deployed task application.
# Among other things, this should include an RDBMS service that is used
# for Spring Cloud Task execution reporting, such as my_postgres
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_SERVICES
spring.cloud.deployer.cloudfoundry.task.services=
# Timeout, in seconds, to use when doing blocking API calls to Cloud Foundry
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_API_TIMEOUT=
# Timeout, in milliseconds, to use when querying the Cloud Foundry API to compute app status
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_STATUS_TIMEOUTNote that you can set spring.cloud.deployer.cloudfoundry.services,
spring.cloud.deployer.cloudfoundry.buildpacks, or the Spring Cloud Deployer-standard
spring.cloud.deployer.memory and spring.cloud.deployer.disk
as part of an individual deployment request by using the deployer.<app-name> shortcut, as the following example shows:
stream create --name ticktock --definition "time | log"
stream deploy --name ticktock --properties "deployer.time.memory=2g"The commands in the preceding example deploy the time source with 2048MB of memory, while the log sink uses the default 1024MB.
When you deploy a stream, you can also pass JAVA_OPTS as a deployment property, as the following example shows:
stream deploy --name ticktock --properties "deployer.time.cloudfoundry.javaOpts=-Duser.timezone=America/New_York"10.17. Debugging
If you want to get better insights into what is happening when your streams and tasks are being deployed, you may want to turn on the following features:
-
Reactor “stacktraces”, showing which operators were involved before an error occurred. This feature is helpful, as the deployer relies on project reactor and regular stacktraces may not always allow understanding the flow before an error happened. Note that this comes with a performance penalty, so it is disabled by default.
spring.cloud.dataflow.server.cloudfoundry.debugReactor == true-
Deployer and Cloud Foundry client library request and response logs. This feature allows seeing a detailed conversation between the Data Flow server and the Cloud Foundry Cloud Controller.
logging.level.cloudfoundry-client == DEBUG10.18. Spring Cloud Config Server
You can use Spring Cloud Config Server to centralize configuration properties for Spring Boot applications. Likewise, both Spring Cloud Data Flow and the applications orchestrated by Spring Cloud Data Flow can be integrated with a configuration server to use the same capabilities.
10.18.1. Stream, Task, and Spring Cloud Config Server
Similar to Spring Cloud Data Flow server, you can configure both the stream and task applications to resolve the centralized properties from the configuration server.
Setting the spring.cloud.config.uri property for the deployed applications is a common way to bind to the configuration server.
See the Spring Cloud Config Client reference guide for more information.
Since this property is likely to be used across all deployed applications, the Data Flow server’s spring.cloud.dataflow.applicationProperties.stream property for stream applications and spring.cloud.dataflow.applicationProperties.task property for task applications can be used to pass the uri of the Config Server to each deployed stream or task application. See the section on Common Application Properties for more information.
Note that, if you use the out-of-the-box Stream Applications, these applications already embed the spring-cloud-services-starter-config-client dependency.
If you build your application from scratch and want to add the client side support for config server, you can add a dependency reference to the config server client library. The following snippet shows a Maven example:
...
<dependency>
<groupId>io.pivotal.spring.cloud</groupId>
<artifactId>spring-cloud-services-starter-config-client</artifactId>
<version>CONFIG_CLIENT_VERSION</version>
</dependency>
...where CONFIG_CLIENT_VERSION can be the latest release of the Spring Cloud Config Server
client for Pivotal Cloud Foundry.
You may see a WARN logging message if the application that uses this library cannot connect to the configuration
server when the application starts and whenever the /health endpoint is accessed.
If you know that you are not using config server functionality, you can disable the client library by setting the
SPRING_CLOUD_CONFIG_ENABLED environment variable to false.
|
10.18.2. Sample Manifest Template
The following SCDF and Skipper manifest.yml templates includes the required environment variables for the Skipper and Spring Cloud Data Flow server and deployed applications and tasks to successfully run on Cloud Foundry and automatically resolve centralized properties from my-config-server at runtime:
applications:
- name: data-flow-server
host: data-flow-server
memory: 2G
disk_quota: 2G
instances: 1
path: {PATH TO SERVER UBER-JAR}
env:
SPRING_APPLICATION_NAME: data-flow-server
MAVEN_REMOTE_REPOSITORIES_CENTRAL_URL: https://repo.maven.apache.org/maven2
MAVEN_REMOTE_REPOSITORIES_REPO1_URL: https://repo.spring.io/snapshot
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_URL: https://api.sys.huron.cf-app.com
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_ORG: sabby20
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_SPACE: sabby20
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_DOMAIN: apps.huron.cf-app.com
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_USERNAME: admin
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_PASSWORD: ***
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_SKIP_SSL_VALIDATION: true
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_SERVICES: postgres
SPRING_CLOUD_SKIPPER_CLIENT_SERVER_URI: https://<skipper-host-name>/api
services:
- postgres
- my-config-serverapplications:
- name: skipper-server
host: skipper-server
memory: 1G
disk_quota: 1G
instances: 1
timeout: 180
buildpack: java_buildpack
path: <PATH TO THE DOWNLOADED SKIPPER SERVER UBER-JAR>
env:
SPRING_APPLICATION_NAME: skipper-server
SPRING_CLOUD_SKIPPER_SERVER_ENABLE_LOCAL_PLATFORM: false
SPRING_CLOUD_SKIPPER_SERVER_STRATEGIES_HEALTHCHECK_TIMEOUTINMILLIS: 300000
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_URL: https://api.local.pcfdev.io
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_ORG: pcfdev-org
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_SPACE: pcfdev-space
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_DOMAIN: cfapps.io
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_USERNAME: admin
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_PASSWORD: admin
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_SKIP_SSL_VALIDATION: false
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_DELETE_ROUTES: false
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_SERVICES: rabbit,my-config-server
services:
- postgres
my-config-serverwhere my-config-server is the name of the Spring Cloud Config Service instance running on Cloud Foundry.
By binding the service to Spring Cloud Data Flow server, Spring Cloud Task and via Skipper to all the Spring Cloud Stream applications respectively, we can now resolve centralized properties backed by this service.
10.18.3. Self-signed SSL Certificate and Spring Cloud Config Server
Often, in a development environment, we may not have a valid certificate to enable SSL communication between clients and the backend services. However, the configuration server for Pivotal Cloud Foundry uses HTTPS for all client-to-service communication, so we need to add a self-signed SSL certificate in environments with no valid certificates.
By using the same manifest.yml templates listed in the previous section for the server, we can provide the self-signed SSL certificate by setting TRUST_CERTS: <API_ENDPOINT>.
However, the deployed applications also require TRUST_CERTS as a flat environment variable (as opposed to being wrapped inside SPRING_APPLICATION_JSON), so we must instruct the server with yet another set of tokens (SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_USE_SPRING_APPLICATION_JSON: false) for tasks.
With this setup, the applications receive their application properties as regular environment variables.
The following listing shows the updated manifest.yml with the required changes. Both the Data Flow server and deployed applications
get their configuration from the my-config-server Cloud Config server (deployed as a Cloud Foundry service).
applications:
- name: test-server
host: test-server
memory: 1G
disk_quota: 1G
instances: 1
path: spring-cloud-dataflow-server-VERSION.jar
env:
SPRING_APPLICATION_NAME: test-server
MAVEN_REMOTE_REPOSITORIES_CENTRAL_URL: https://repo.maven.apache.org/maven2
MAVEN_REMOTE_REPOSITORIES_REPO1_URL: https://repo.spring.io/snapshot
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_URL: https://api.sys.huron.cf-app.com
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_ORG: sabby20
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_SPACE: sabby20
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_DOMAIN: apps.huron.cf-app.com
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_USERNAME: admin
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_PASSWORD: ***
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_SKIP_SSL_VALIDATION: true
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_SERVICES: postgres, config-server
SPRING_CLOUD_SKIPPER_CLIENT_SERVER_URI: https://<skipper-host-name>/api
TRUST_CERTS: <API_ENDPOINT> #this is for the server
SPRING_CLOUD_DATAFLOW_APPLICATION_PROPERTIES_TASK_TRUST_CERTS: <API_ENDPOINT> #this propagates to all tasks
services:
- postgres
- my-config-server #this is for the serverAlso add the my-config-server service to the Skipper’s manifest environment
applications:
- name: skipper-server
host: skipper-server
memory: 1G
disk_quota: 1G
instances: 1
timeout: 180
buildpack: java_buildpack
path: <PATH TO THE DOWNLOADED SKIPPER SERVER UBER-JAR>
env:
SPRING_APPLICATION_NAME: skipper-server
SPRING_CLOUD_SKIPPER_SERVER_ENABLE_LOCAL_PLATFORM: false
SPRING_CLOUD_SKIPPER_SERVER_STRATEGIES_HEALTHCHECK_TIMEOUTINMILLIS: 300000
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_URL: <URL>
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_ORG: <ORG>
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_SPACE: <SPACE>
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_DOMAIN: <DOMAIN>
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_USERNAME: <USER>
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_PASSWORD: <PASSWORD>
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_SERVICES: rabbit, my-config-server #this is so all stream applications bind to my-config-server
services:
- postgres
my-config-server10.19. Configure Scheduling
This section discusses how to configure Spring Cloud Data Flow to connect to the PCF-Scheduler as its agent to execute tasks.
|
Before following these instructions, be sure to have an instance of the PCF-Scheduler service running in your Cloud Foundry space.
To create a PCF-Scheduler in your space (assuming it is in your Market Place) execute the following from the CF CLI: |
For scheduling, you must add (or update) the following environment variables in your environment:
-
Enable scheduling for Spring Cloud Data Flow by setting
spring.cloud.dataflow.features.schedules-enabledtotrue. -
Bind the task deployer to your instance of PCF-Scheduler by adding the PCF-Scheduler service name to the
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_SERVICESenvironment variable. -
Establish the URL to the PCF-Scheduler by setting the
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_SCHEDULER_SCHEDULER_URLenvironment variable.
|
After creating the preceding configurations, you must create any task definitions that need to be scheduled. |
The following sample manifest shows both environment properties configured (assuming you have a PCF-Scheduler service available with the name myscheduler):
applications:
- name: data-flow-server
host: data-flow-server
memory: 2G
disk_quota: 2G
instances: 1
path: {PATH TO SERVER UBER-JAR}
env:
SPRING_APPLICATION_NAME: data-flow-server
SPRING_CLOUD_SKIPPER_SERVER_ENABLE_LOCAL_PLATFORM: false
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_URL: <URL>
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_ORG: <ORG>
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_SPACE: <SPACE>
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_DOMAIN: <DOMAIN>
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_USERNAME: <USER>
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_CONNECTION_PASSWORD: <PASSWORD>
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_DEPLOYMENT_SERVICES: rabbit, myscheduler
SPRING_CLOUD_DATAFLOW_FEATURES_SCHEDULES_ENABLED: true
SPRING_CLOUD_SKIPPER_CLIENT_SERVER_URI: https://<skipper-host-name>/api
SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]_SCHEDULER_SCHEDULER_URL: https://scheduler.local.pcfdev.io
services:
- postgresWhere the SPRING_CLOUD_DATAFLOW_TASK_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[default]SCHEDULER_SCHEDULER_URL has the following format: scheduler.<Domain-Name> (for
example, scheduler.local.pcfdev.io). Check the actual address from your _PCF environment.
| Detailed examples for launching and scheduling tasks across multiple platforms, are available in this section Multiple Platform Support for Tasks on dataflow.spring.io. |
11. Configuration - Kubernetes
This section describes how to configure Spring Cloud Data Flow features, such as deployer properties, tasks, and which relational database to use.
11.1. Feature Toggles
Data Flow server offers specific set of features that can be enabled or disabled when launching. These features include all the lifecycle operations, REST endpoints (server and client implementations including Shell and the UI) for:
-
Streams
-
Tasks
-
Schedules
You can enable or disable these features by setting the following boolean environment variables when launching the Data Flow server:
-
SPRING_CLOUD_DATAFLOW_FEATURES_STREAMS_ENABLED -
SPRING_CLOUD_DATAFLOW_FEATURES_TASKS_ENABLED -
SPRING_CLOUD_DATAFLOW_FEATURES_SCHEDULES_ENABLED
By default, all the features are enabled.
The /features REST endpoint provides information on the features that have been enabled and disabled.
11.2. Application and Server Properties
This section covers how you can customize the deployment of your applications. You can use a number of properties to influence settings for the applications that are deployed. Properties can be applied on a per-application basis or in the appropriate server configuration for all deployed applications.
| Properties set on a per-application basis always take precedence over properties set as the server configuration. This arrangement lets you override global server level properties on a per-application basis. |
Properties to be applied for all deployed Tasks are defined in the src/kubernetes/server/server-config-[binder].yaml file and for Streams in src/kubernetes/skipper/skipper-config-[binder].yaml. Replace [binder] with the messaging middleware you are using — for example, rabbit or kafka.
11.2.1. Memory and CPU Settings
Applications are deployed with default memory and CPU settings. If you need to, you can adjust these values. The following example shows how to set Limits to 1000m for CPU and 1024Mi for memory and Requests to 800m for CPU and 640Mi for memory:
deployer.<application>.kubernetes.limits.cpu=1000m
deployer.<application>.kubernetes.limits.memory=1024Mi
deployer.<application>.kubernetes.requests.cpu=800m
deployer.<application>.kubernetes.requests.memory=640MiThose values results in the following container settings being used:
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 800m
memory: 640MiYou can also control the default values to which to set the cpu and memory globally.
The following example shows how to set the CPU and memory for streams:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
limits:
memory: 640mi
cpu: 500mThe following example shows how to set the CPU and memory for tasks:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
limits:
memory: 640mi
cpu: 500mThe settings we have used so far affect only the settings for the container. They do not affect the memory setting for the JVM process in the container. If you would like to set JVM memory settings, you can set an environment variable to do so. See the next section for details.
11.2.2. Environment Variables
To influence the environment settings for a given application, you can use the spring.cloud.deployer.kubernetes.environmentVariables deployer property.
For example, a common requirement in production settings is to influence the JVM memory arguments.
You can do so by using the JAVA_TOOL_OPTIONS environment variable, as the following example shows:
deployer.<application>.kubernetes.environmentVariables=JAVA_TOOL_OPTIONS=-Xmx1024m
The environmentVariables property accepts a comma-delimited string. If an environment variable contains a value
that is also a comma-delimited string, it must be enclosed in single quotation marks — for example,
spring.cloud.deployer.kubernetes.environmentVariables=spring.cloud.stream.kafka.binder.brokers='somehost:9092,
anotherhost:9093'
|
This overrides the JVM memory setting for the desired <application> (replace <application> with the name of your application).
11.2.3. Liveness, Readiness and Startup Probes
The liveness and readiness probes use paths called /health and /info, respectively. They use a delay of 1 for both and a period of 60 and 10 respectively. You can change these defaults when you deploy the stream by using deployer properties. The liveness and readiness probes are applied only to streams.
The startup probe will use the /health path and a delay of 30 and period for 3 with a failure threshold of 20 times before the container restarts the application.
The following example changes the liveness and startup probes (replace <application> with the name of your application) by setting deployer properties:
deployer.<application>.kubernetes.livenessProbePath=/health
deployer.<application>.kubernetes.livenessProbeDelay=1
deployer.<application>.kubernetes.livenessProbePeriod=60
deployer.<application>.kubernetes.livenessProbeSuccess=1
deployer.<application>.kubernetes.livenessProbeFailure=3
deployer.<application>.kubernetes.startupHttpProbePath=/health
deployer.<application>.kubernetes.startupProbedelay=20
deployer.<application>.kubernetes.startupProbeSuccess=1
deployer.<application>.kubernetes.startupProbeFailure=30
deployer.<application>.kubernetes.startupProbePeriod=5
deployer.<application>.kubernetes.startupProbeTimeout=3You can declare the same as part of the server global configuration for streams, as the following example shows:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
livenessHttpProbePath: /health
livenessProbeDelay: 1
livenessProbePeriod: 60
livenessProbeSuccess: 1
livenessProbeFailure: 3
startupHttpProbePath: /health
startupProbeDelay: 20
startupProbeSuccess: 1
startupProbeFailure: 30
startupProbePeriod: 5
startupProbeTimeout: 3Similarly, you can swap liveness for readiness to override the default readiness settings.
By default, port 8080 is used as the probe port. You can change the defaults for both liveness and readiness probe ports by using deployer properties, as the following example shows:
deployer.<application>.kubernetes.readinessProbePort=7000
deployer.<application>.kubernetes.livenessProbePort=7000
deployer.<application>.kubernetes.startupProbePort=7000You can declare the same as part of the global configuration for streams, as the following example shows:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
readinessProbePort: 7000
livenessProbePort: 7000
startupProbePort: 7000|
By default, the The To automatically set both |
You can access secured probe endpoints by using credentials stored in a Kubernetes secret. You can use an existing secret, provided the credentials are contained under the credentials key name of the secret’s data block. You can configure probe authentication on a per-application basis. When enabled, it is applied to both the liveness and readiness probe endpoints by using the same credentials and authentication type. Currently, only Basic authentication is supported.
To create a new secret:
-
Generate the base64 string with the credentials used to access the secured probe endpoints.
Basic authentication encodes a username and a password as a base64 string in the format of
username:password.The following example (which includes output and in which you should replace
userandpasswith your values) shows how to generate a base64 string:$ echo -n "user:pass" | base64 dXNlcjpwYXNz -
With the encoded credentials, create a file (for example,
myprobesecret.yml) with the following contents:apiVersion: v1 kind: Secret metadata: name: myprobesecret type: Opaque data: credentials: GENERATED_BASE64_STRING -
Replace
GENERATED_BASE64_STRINGwith the base64-encoded value generated earlier. -
Create the secret by using
kubectl, as the following example shows:$ kubectl create -f ./myprobesecret.yml secret "myprobesecret" created -
Set the following deployer properties to use authentication when accessing probe endpoints, as the following example shows:
deployer.<application>.kubernetes.probeCredentialsSecret=myprobesecretReplace
<application>with the name of the application to which to apply authentication.
11.2.4. Using SPRING_APPLICATION_JSON
You can use a SPRING_APPLICATION_JSON environment variable to set Data Flow server properties (including the configuration of Maven repository settings) that are common across all of the Data Flow server implementations. These settings go at the server level in the container env section of a deployment YAML. The following example shows how to do so:
env:
- name: SPRING_APPLICATION_JSON
value: "{ \"maven\": { \"local-repository\": null, \"remote-repositories\": { \"central\": { \"url\": \"https://repo.maven.apache.org/maven2\" }, \"repo1\": { \"url\": \"https://repo.spring.io/snapshot\" } } } }"11.2.5. Private Docker Registry
You can pull Docker images from a private registry on a per-application basis. First, you must create a secret in the cluster. Follow the Pull an Image from a Private Registry guide to create the secret.
Once you have created the secret, you can use the imagePullSecret property to set the secret to use, as the following example shows:
deployer.<application>.kubernetes.imagePullSecret=mysecretReplace <application> with the name of your application and mysecret with the name of the secret you created earlier.
You can also configure the image pull secret at the global server level.
The following example shows how to do so for streams:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
imagePullSecret: mysecretThe following example shows how to do so for tasks:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
imagePullSecret: mysecretReplace mysecret with the name of the secret you created earlier.
11.2.6. Annotations
You can add annotations to Kubernetes objects on a per-application basis. The supported object types are pod Deployment, Service, and Job. Annotations are defined in a key:value format, allowing for multiple annotations separated by a comma. For more information and use cases on annotations, see Annotations.
The following example shows how you can configure applications to use annotations:
deployer.<application>.kubernetes.podAnnotations=annotationName:annotationValue
deployer.<application>.kubernetes.serviceAnnotations=annotationName:annotationValue,annotationName2:annotationValue2
deployer.<application>.kubernetes.jobAnnotations=annotationName:annotationValueReplace <application> with the name of your application and the value of your annotations.
11.2.7. Entry Point Style
An entry point style affects how application properties are passed to the container to be deployed. Currently, three styles are supported:
-
exec(default): Passes all application properties and command line arguments in the deployment request as container arguments. Application properties are transformed into the format of--key=value. -
shell: Passes all application properties and command line arguments as environment variables. Each of the applicationor command-line argument properties is transformed into an uppercase string and.characters are replaced with_. -
boot: Creates an environment variable calledSPRING_APPLICATION_JSONthat contains a JSON representation of all application properties. Command line arguments from the deployment request are set as container args.
| In all cases, environment variables defined at the server-level configuration and on a per-application basis are sent on to the container as is. |
You can configure an application as follows:
deployer.<application>.kubernetes.entryPointStyle=<Entry Point Style>Replace <application> with the name of your application and <Entry Point Style> with your desired entry point style.
You can also configure the entry point style at the global server level.
The following example shows how to do so for streams:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
entryPointStyle: entryPointStyleThe following example shows how to do so for tasks:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
entryPointStyle: entryPointStyleReplace entryPointStyle with the desired entry point style.
You should choose an Entry Point Style of either exec or shell, to correspond to how the ENTRYPOINT syntax is defined in the container’s Dockerfile. For more information and uses cases on exec versus shell, see the ENTRYPOINT section of the Docker documentation.
Using the boot entry point style corresponds to using the exec style ENTRYPOINT. Command line arguments from the deployment request are passed to the container, with the addition of application properties being mapped into the SPRING_APPLICATION_JSON environment variable rather than command line arguments.
When you use the boot Entry Point Style, the deployer.<application>.kubernetes.environmentVariables property must not contain SPRING_APPLICATION_JSON.
|
11.2.8. Deployment Service Account
You can configure a custom service account for application deployments through properties. You can use an existing service account or create a new one. One way to create a service account is by using kubectl, as the following example shows:
$ kubectl create serviceaccount myserviceaccountname
serviceaccount "myserviceaccountname" createdThen you can configure individual applications as follows:
deployer.<application>.kubernetes.deploymentServiceAccountName=myserviceaccountnameReplace <application> with the name of your application and myserviceaccountname with your service account name.
You can also configure the service account name at the global server level.
The following example shows how to do so for streams:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
deploymentServiceAccountName: myserviceaccountnameThe following example shows how to do so for tasks:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
deploymentServiceAccountName: myserviceaccountnameReplace myserviceaccountname with the service account name to be applied to all deployments.
11.2.9. Image Pull Policy
An image pull policy defines when a Docker image should be pulled to the local registry. Currently, three policies are supported:
-
IfNotPresent(default): Do not pull an image if it already exists. -
Always: Always pull the image regardless of whether it already exists. -
Never: Never pull an image. Use only an image that already exists.
The following example shows how you can individually configure applications:
deployer.<application>.kubernetes.imagePullPolicy=AlwaysReplace <application> with the name of your application and Always with your desired image pull policy.
You can configure an image pull policy at the global server level.
The following example shows how to do so for streams:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
imagePullPolicy: AlwaysThe following example shows how to do so for tasks:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
imagePullPolicy: AlwaysReplace Always with your desired image pull policy.
11.2.10. Deployment Labels
You can set custom labels on objects related to Deployment. See Labels for more information on labels. Labels are specified in key:value format.
The following example shows how you can individually configure applications:
deployer.<application>.kubernetes.deploymentLabels=myLabelName:myLabelValueReplace <application> with the name of your application, myLabelName with your label name, and myLabelValue with the value of your label.
Additionally, you can apply multiple labels, as the following example shows:
deployer.<application>.kubernetes.deploymentLabels=myLabelName:myLabelValue,myLabelName2:myLabelValue211.2.11. Tolerations
Tolerations work with taints to ensure pods are not scheduled onto particular nodes. Tolerations are set into the pod configuration while taints are set onto nodes. See the Taints and Tolerations section of the Kubernetes reference for more information.
The following example shows how you can individually configure applications:
deployer.<application>.kubernetes.tolerations=[{key: 'mykey', operator: 'Equal', value: 'myvalue', effect: 'NoSchedule'}]Replace <application> with the name of your application and the key-value pairs according to your desired toleration configuration.
You can configure tolerations at the global server level as well.
The following example shows how to do so for streams:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
tolerations:
- key: mykey
operator: Equal
value: myvalue
effect: NoScheduleThe following example shows how to do so for tasks:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
tolerations:
- key: mykey
operator: Equal
value: myvalue
effect: NoScheduleReplace the tolerations key-value pairs according to your desired toleration configuration.
11.2.12. Secret References
Secrets can be referenced and their entire data contents can be decoded and inserted into the pod environment as individual variables. See the Configure all key-value pairs in a Secret as container environment variables section of the Kubernetes reference for more information.
The following example shows how you can individually configure applications:
deployer.<application>.kubernetes.secretRefs=testsecretYou can also specify multiple secrets, as follows:
deployer.<application>.kubernetes.secretRefs=[testsecret,anothersecret]Replace <application> with the name of your application and the secretRefs attribute with the appropriate values for your application environment and secret.
You can configure secret references at the global server level as well.
The following example shows how to do so for streams:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
secretRefs:
- testsecret
- anothersecretThe following example shows how to do so for tasks:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
secretRefs:
- testsecret
- anothersecretReplace the items of secretRefs with one or more secret names.
11.2.13. Secret Key References
Secrets can be referenced and their decoded value can be inserted into the pod environment. See the Using Secrets as Environment Variables section of the Kubernetes reference for more information.
The following example shows how you can individually configure applications:
deployer.<application>.kubernetes.secretKeyRefs=[{envVarName: 'MY_SECRET', secretName: 'testsecret', dataKey: 'password'}]Replace <application> with the name of your application and the envVarName, secretName, and dataKey attributes with the appropriate values for your application environment and secret.
You can configure secret key references at the global server level as well.
The following example shows how to do so for streams:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
secretKeyRefs:
- envVarName: MY_SECRET
secretName: testsecret
dataKey: passwordThe following example shows how to do so for tasks:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
secretKeyRefs:
- envVarName: MY_SECRET
secretName: testsecret
dataKey: passwordReplace the envVarName, secretName, and dataKey attributes with the appropriate values for your secret.
11.2.14. ConfigMap References
A ConfigMap can be referenced and its entire data contents can be decoded and inserted into the pod environment as individual variables. See the Configure all key-value pairs in a ConfigMap as container environment variables section of the Kubernetes reference for more information.
The following example shows how you can individually configure applications:
deployer.<application>.kubernetes.configMapRefs=testcmYou can also specify multiple ConfigMap instances, as follows:
deployer.<application>.kubernetes.configMapRefs=[testcm,anothercm]Replace <application> with the name of your application and the configMapRefs attribute with the appropriate values for your application environment and ConfigMap.
You can configure ConfigMap references at the global server level as well.
The following example shows how to do so for streams. Edit the appropriate skipper-config-(binder).yaml, replacing (binder) with the corresponding binder in use:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
configMapRefs:
- testcm
- anothercmThe following example shows how to do so for tasks by editing the server-config.yaml file:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
configMapRefs:
- testcm
- anothercmReplace the items of configMapRefs with one or more secret names.
11.2.15. ConfigMap Key References
A ConfigMap can be referenced and its associated key value inserted into the pod environment. See the Define container environment variables using ConfigMap data section of the Kubernetes reference for more information.
The following example shows how you can individually configure applications:
deployer.<application>.kubernetes.configMapKeyRefs=[{envVarName: 'MY_CM', configMapName: 'testcm', dataKey: 'platform'}]Replace <application> with the name of your application and the envVarName, configMapName, and dataKey attributes with the appropriate values for your application environment and ConfigMap.
You can configure ConfigMap references at the global server level as well.
The following example shows how to do so for streams. Edit the appropriate skipper-config-(binder).yaml, replacing (binder) with the corresponding binder in use:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
configMapKeyRefs:
- envVarName: MY_CM
configMapName: testcm
dataKey: platformThe following example shows how to do so for tasks by editing the server-config.yaml file:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
configMapKeyRefs:
- envVarName: MY_CM
configMapName: testcm
dataKey: platformReplace the envVarName, configMapName, and dataKey attributes with the appropriate values for your ConfigMap.
11.2.16. Pod Security Context
The pod security context specifies security settings for a pod and its containers.
The configurable options are listed HERE (more details for each option can be found in the Pod Security Context section of the Kubernetes API reference).
The following example shows how you can configure the security context for an individual application pod:
deployer.<application>.kubernetes.podSecurityContext={runAsUser: 65534, fsGroup: 65534, supplementalGroups: [65534, 65535], seccompProfile: { type: 'RuntimeDefault' }}Replace <application> with the name of your application and any attributes with the appropriate values for your container environment.
You can configure the pod security context at the global server level as well.
The following example shows how to do so for streams. Edit the appropriate skipper-config-(binder).yaml, replacing (binder) with the corresponding binder in use:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
podSecurityContext:
runAsUser: 65534
fsGroup: 65534
supplementalGroups: [65534,65535]
seccompProfile:
type: Localhost
localhostProfile: my-profiles/profile-allow.jsonThe following example shows how to do so for tasks by editing the server-config.yaml file:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
podSecurityContext:
runAsUser: 65534
fsGroup: 65534
supplementalGroups: [65534,65535]
seccompProfile:
type: Localhost
localhostProfile: my-profiles/profile-allow.jsonAdjust the podSecurityContext attributes with the appropriate values for your container environment.
11.2.17. Container Security Context
The container security context specifies security settings for an individual container.
The configurable options are listed HERE (more details for each option can be found in the Container Security Context section of the Kubernetes API reference).
| The container security context is applied to all containers in your deployment unless they have their own security already explicitly defined, including regular init containers, stateful set init containers, and additional containers. |
The following example shows how you can configure the security context for containers in an individual application pod:
deployer.<application>.kubernetes.containerSecurityContext={allowPrivilegeEscalation: true, runAsUser: 65534}Replace <application> with the name of your application and any attributes with the appropriate values for your container environment.
You can configure the container security context at the global server level as well.
The following example shows how to do so for streams. Edit the appropriate skipper-config-(binder).yaml, replacing (binder) with the corresponding binder in use:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
containerSecurityContext:
allowPrivilegeEscalation: true
runAsUser: 65534The following example shows how to do so for tasks by editing the server-config.yaml file:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
containerSecurityContext:
allowPrivilegeEscalation: true
runAsUser: 65534Adjust the containerSecurityContext attributes with the appropriate values for your container environment.
11.2.18. Service Ports
When you deploy applications, a kubernetes Service object is created with a default port of 8080. If the server.port property is set, it overrides the default port value. You can add additional ports to the Service object on a per-application basis. You can add multiple ports with a comma delimiter.
The following example shows how you can configure additional ports on a Service object for an application:
deployer.<application>.kubernetes.servicePorts=5000
deployer.<application>.kubernetes.servicePorts=5000,9000Replace <application> with the name of your application and the value of your ports.
11.2.19. StatefulSet Init Container
When deploying an application by using a StatefulSet, an Init Container is used to set the instance index in the pod.
By default, the image used is busybox, which you can be customize.
The following example shows how you can individually configure application pods:
deployer.<application>.kubernetes.statefulSetInitContainerImageName=myimage:mylabelReplace <application> with the name of your application and the statefulSetInitContainerImageName attribute with the appropriate value for your environment.
You can configure the StatefulSet Init Container at the global server level as well.
The following example shows how to do so for streams. Edit the appropriate skipper-config-(binder).yaml, replacing (binder) with the corresponding binder in use:
data:
application.yaml: |-
spring:
cloud:
skipper:
server:
platform:
kubernetes:
accounts:
default:
statefulSetInitContainerImageName: myimage:mylabelThe following example shows how to do so for tasks by editing the server-config.yaml file:
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
statefulSetInitContainerImageName: myimage:mylabelReplace the statefulSetInitContainerImageName attribute with the appropriate value for your environment.
11.2.20. Init Containers
When you deploy applications, you can set a custom Init Container on a per-application basis. Refer to the Init Containers section of the Kubernetes reference for more information.
The following example shows how you can configure an Init Container for an application:
deployer.<application>.kubernetes.initContainer={containerName: 'test', imageName: 'busybox:latest', commands: ['sh', '-c', 'echo hello']}Replace <application> with the name of your application and set the values of the initContainer attributes appropriate for your Init Container.
11.2.21. Lifecycle Support
When you deploy applications, you may attach postStart and preStop Lifecycle handlers to execute commands.
The Kubernetes API supports other types of handlers besides exec. This feature may be extended to support additional actions in a future release.
To configure the Lifecycle handlers as shown in the linked page above,specify each command as a comma-delimited list, using the following property keys:
deployer.<application>.kubernetes.lifecycle.postStart.exec.command=/bin/sh,-c,'echo Hello from the postStart handler > /usr/share/message'
deployer.<application>.kubernetes.lifecycle.preStop.exec.command=/bin/sh,-c,'nginx -s quit; while killall -0 nginx; do sleep 1; done'11.2.22. Additional Containers
When you deploy applications, you may need one or more containers to be deployed along with the main container. This would allow you to adapt some deployment patterns such as sidecar, adapter in case of multi container pod setup.
The following example shows how you can configure additional containers for an application:
deployer.<application>.kubernetes.additionalContainers=[{name: 'c1', image: 'busybox:latest', command: ['sh', '-c', 'echo hello1'], volumeMounts: [{name: 'test-volume', mountPath: '/tmp', readOnly: true}]},{name: 'c2', image: 'busybox:1.26.1', command: ['sh', '-c', 'echo hello2']}]11.3. Deployer Properties
You can use the following configuration properties the Kubernetes deployer to customize how Streams and Tasks are deployed.
When deploying with the Data Flow shell, you can use the syntax deployer.<appName>.kubernetes.<deployerPropertyName>.
These properties are also used when configuring the Kubernetes task platforms in the Data Flow server and Kubernetes platforms in Skipper for deploying Streams.
| Deployer Property Name | Description | Default Value |
|---|---|---|
namespace |
Namespace to use |
environment variable |
deployment.nodeSelector |
The node selectors to apply to the deployment in |
<none> |
imagePullSecret |
Secrets for a access a private registry to pull images. |
<none> |
imagePullPolicy |
The Image Pull Policy to apply when pulling images. Valid options are |
IfNotPresent |
livenessProbeDelay |
Delay in seconds when the Kubernetes liveness check of the app container should start checking its health status. |
10 |
livenessProbePeriod |
Period in seconds for performing the Kubernetes liveness check of the app container. |
60 |
livenessProbeTimeout |
Timeout in seconds for the Kubernetes liveness check of the app container. If the health check takes longer than this value to return it is assumed as 'unavailable'. |
2 |
livenessProbePath |
Path that app container has to respond to for liveness check. |
<none> |
livenessProbePort |
Port that app container has to respond on for liveness check. |
<none> |
startupProbeDelay |
Delay in seconds when the Kubernetes startup check of the app container should start checking its health status. |
30 |
startupProbePeriod |
Period in seconds for performing the Kubernetes startup check of the app container. |
3 |
startupProbeFailure |
Number of probe failures allowed for the startup probe before the pod is restarted. |
20 |
startupHttpProbePath |
Path that app container has to respond to for startup check. |
<none> |
startupProbePort |
Port that app container has to respond on for startup check. |
<none> |
readinessProbeDelay |
Delay in seconds when the readiness check of the app container should start checking if the module is fully up and running. |
10 |
readinessProbePeriod |
Period in seconds to perform the readiness check of the app container. |
10 |
readinessProbeTimeout |
Timeout in seconds that the app container has to respond to its health status during the readiness check. |
2 |
readinessProbePath |
Path that app container has to respond to for readiness check. |
<none> |
readinessProbePort |
Port that app container has to respond on for readiness check. |
<none> |
probeCredentialsSecret |
The secret name containing the credentials to use when accessing secured probe endpoints. |
<none> |
limits.memory |
The memory limit, maximum needed value to allocate a pod, Default unit is mebibytes, 'M' and 'G" suffixes supported |
<none> |
limits.cpu |
The CPU limit, maximum needed value to allocate a pod |
<none> |
requests.memory |
The memory request, guaranteed needed value to allocate a pod. |
<none> |
requests.cpu |
The CPU request, guaranteed needed value to allocate a pod. |
<none> |
affinity.nodeAffinity |
The node affinity expressed in YAML format. e.g. |
<none> |
affinity.podAffinity |
The pod affinity expressed in YAML format. e.g. |
<none> |
affinity.podAntiAffinity |
The pod anti-affinity expressed in YAML format. e.g. |
<none> |
statefulSet.volumeClaimTemplate.storageClassName |
Name of the storage class for a stateful set |
<none> |
statefulSet.volumeClaimTemplate.storage |
The storage amount. Default unit is mebibytes, 'M' and 'G" suffixes supported |
<none> |
environmentVariables |
List of environment variables to set for any deployed app container |
<none> |
entryPointStyle |
Entry point style used for the Docker image. Used to determine how to pass in properties. Can be |
|
createLoadBalancer |
Create a "LoadBalancer" for the service created for each app. This facilitates assignment of external IP to app. |
false |
serviceAnnotations |
Service annotations to set for the service created for each application. String of the format |
<none> |
podAnnotations |
Pod annotations to set for the pod created for each deployment. String of the format |
<none> |
jobAnnotations |
Job annotations to set for the pod or job created for a job. String of the format |
<none> |
priorityClassName |
Pod Spec priorityClassName. Create a PriorityClass in Kubernetes before using this property. See Pod Priority and Preemption |
<none> |
shareProcessNamespace |
Will assign value to Pod.spec.shareProcessNamespace. See Share Process Namespace between Containers in a Pod |
<none> |
minutesToWaitForLoadBalancer |
Time to wait for load balancer to be available before attempting delete of service (in minutes). |
5 |
maxTerminatedErrorRestarts |
Maximum allowed restarts for app that fails due to an error or excessive resource use. |
2 |
maxCrashLoopBackOffRestarts |
Maximum allowed restarts for app that is in a CrashLoopBackOff. Values are |
|
volumeMounts |
volume mounts expressed in YAML format. e.g. |
<none> |
volumes |
The volumes that a Kubernetes instance supports specifed in YAML format. e.g. |
<none> |
hostNetwork |
The hostNetwork setting for the deployments, see kubernetes.io/docs/api-reference/v1/definitions/#_v1_podspec |
false |
createDeployment |
Create a "Deployment" with a "Replica Set" instead of a "Replication Controller". |
true |
createJob |
Create a "Job" instead of just a "Pod" when launching tasks. |
false |
containerCommand |
Overrides the default entry point command with the provided command and arguments. |
<none> |
containerPorts |
Adds additional ports to expose on the container. |
<none> |
createNodePort |
The explicit port to use when |
<none> |
deploymentServiceAccountName |
Service account name used in app deployments. Note: The service account name used for app deployments is derived from the Data Flow servers deployment. |
<none> |
deploymentLabels |
Additional labels to add to the deployment in |
<none> |
bootMajorVersion |
The Spring Boot major version to use. Currently only used to configure Spring Boot version specific probe paths automatically. Valid options are |
2 |
tolerations.key |
The key to use for the toleration. |
<none> |
tolerations.effect |
The toleration effect. See kubernetes.io/docs/concepts/configuration/taint-and-toleration for valid options. |
<none> |
tolerations.operator |
The toleration operator. See kubernetes.io/docs/concepts/configuration/taint-and-toleration/ for valid options. |
<none> |
tolerations.tolerationSeconds |
The number of seconds defining how long the pod will stay bound to the node after a taint is added. |
<none> |
tolerations.value |
The toleration value to apply, used in conjunction with |
<none> |
secretRefs |
The name of the secret(s) to load the entire data contents into individual environment variables. Multiple secrets may be comma separated. |
<none> |
secretKeyRefs.envVarName |
The environment variable name to hold the secret data |
<none> |
secretKeyRefs.secretName |
The secret name to access |
<none> |
secretKeyRefs.dataKey |
The key name to obtain secret data from |
<none> |
configMapRefs |
The name of the ConfigMap(s) to load the entire data contents into individual environment variables. Multiple ConfigMaps be comma separated. |
<none> |
configMapKeyRefs.envVarName |
The environment variable name to hold the ConfigMap data |
<none> |
configMapKeyRefs.configMapName |
The ConfigMap name to access |
<none> |
configMapKeyRefs.dataKey |
The key name to obtain ConfigMap data from |
<none> |
maximumConcurrentTasks |
The maximum concurrent tasks allowed for this platform instance |
20 |
podSecurityContext |
The security context applied to the pod expressed in YAML format. e.g. |
<none> |
podSecurityContext.runAsUser |
The numeric user ID to run pod container processes under |
<none> |
podSecurityContext.runAsGroup |
The numeric group id to run the entrypoint of the container process |
<none> |
podSecurityContext.runAsNonRoot |
Indicates that the container must run as a non-root user |
<none> |
podSecurityContext.fsGroup |
The numeric group ID for the volumes of the pod |
<none> |
podSecurityContext.fsGroupChangePolicy |
Defines behavior of changing ownership and permission of the volume before being exposed inside pod (only applies to volume types which support fsGroup based ownership and permissions) - possible values are "OnRootMismatch", "Always" |
<none> |
podSecurityContext.supplementalGroups |
The numeric group IDs applied to the pod container processes, in addition to the container’s primary group ID |
<none> |
podSecurityContext.seccompProfile |
The seccomp options to use for the pod containers expressed in YAML format. e.g. |
<none> |
podSecurityContext.seLinuxOptions |
The SELinux context to be applied to the pod containers expressed in YAML format. e.g. |
<none> |
podSecurityContext.sysctls |
List of namespaced sysctls used for the pod expressed in YAML format. e.g. |
<none> |
podSecurityContext.windowsOptions |
The Windows specific settings applied to all containers expressed in YAML format. e.g. |
<none> |
containerSecurityContext |
The security context applied to the containers expressed in YAML format. e.g. |
<none> |
containerSecurityContext.allowPrivilegeEscalation |
Whether a process can gain more privileges than its parent process |
<none> |
containerSecurityContext.capabilities |
The capabilities to add/drop when running the container expressed in YAML format. e.g. |
<none> |
containerSecurityContext.privileged |
Run container in privileged mode. |
<none> |
containerSecurityContext.procMount |
The type of proc mount to use for the container (only used when spec.os.name is not windows) |
<none> |
containerSecurityContext.readOnlyRootFilesystem |
Mounts the container’s root filesystem as read-only |
<none> |
containerSecurityContext.runAsUser |
The numeric user ID to run pod container processes under |
<none> |
containerSecurityContext.runAsGroup |
The numeric group id to run the entrypoint of the container process |
<none> |
containerSecurityContext.runAsNonRoot |
Indicates that the container must run as a non-root user |
<none> |
containerSecurityContext.seccompProfile |
The seccomp options to use for the pod containers expressed in YAML format. e.g. |
<none> |
containerSecurityContext.seLinuxOptions |
The SELinux context to be applied to the pod containers expressed in YAML format. e.g. |
<none> |
containerSecurityContext.sysctls |
List of namespaced sysctls used for the pod expressed in YAML format. e.g. |
<none> |
containerSecurityContext.windowsOptions |
The Windows specific settings applied to all containers expressed in YAML format. e.g. |
<none> |
statefulSetInitContainerImageName |
A custom image name to use for the StatefulSet Init Container |
<none> |
initContainer |
An Init Container expressed in YAML format to be applied to a pod. e.g. |
<none> |
additionalContainers |
Additional containers expressed in YAML format to be applied to a pod. e.g. |
<none> |
11.4. Tasks
The Data Flow server is responsible for deploying Tasks.
Tasks that are launched by Data Flow write their state to the same database that is used by the Data Flow server.
For Tasks which are Spring Batch Jobs, the job and step execution data is also stored in this database.
As with Skipper, Tasks can be launched to multiple platforms.
When Data Flow is running on Kubernetes, a Task platfom must be defined.
To configure new platform accounts that target Kubernetes, provide an entry under the spring.cloud.dataflow.task.platform.kubernetes section in your application.yaml file for via another Spring Boot supported mechanism.
In the following example, two Kubernetes platform accounts named dev and qa are created.
The keys such as memory and disk are Cloud Foundry Deployer Properties.
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
dev:
namespace: devNamespace
imagePullPolicy: Always
entryPointStyle: exec
limits:
cpu: 4
qa:
namespace: qaNamespace
imagePullPolicy: IfNotPresent
entryPointStyle: boot
limits:
memory: 2048m
By defining one platform as default allows you to skip using platformName where its use would otherwise be required.
|
When launching a task, pass the value of the platform account name using the task launch option --platformName If you do not pass a value for platformName, the value default will be used.
| When deploying a task to multiple platforms, the configuration of the task needs to connect to the same database as the Data Flow Server. |
You can configure the Data Flow server that is on Kubernetes to deploy tasks to Cloud Foundry and Kubernetes. See the section on Cloud Foundry Task Platform Configuration for more information.
Detailed examples for launching and scheduling tasks across multiple platforms, are available in this section Multiple Platform Support for Tasks on dataflow.spring.io.
11.5. General Configuration
The Spring Cloud Data Flow server for Kubernetes uses the spring-cloud-kubernetes module to process secrets that are mounted under /etc/secrets. ConfigMaps must be mounted as application.yaml in the /config directory that is processed by Spring Boot. To avoid access to the Kubernetes API server the SPRING_CLOUD_KUBERNETES_CONFIG_ENABLE_API and SPRING_CLOUD_KUBERNETES_SECRETS_ENABLE_API are set to false.
11.5.1. Using ConfigMap and Secrets
You can pass configuration properties to the Data Flow Server by using Kubernetes ConfigMap and secrets.
The following example shows one possible configuration, which enables MariaDB and sets a memory limit:
apiVersion: v1
kind: ConfigMap
metadata:
name: scdf-server
labels:
app: scdf-server
data:
application.yaml: |-
spring:
cloud:
dataflow:
task:
platform:
kubernetes:
accounts:
default:
limits:
memory: 1024Mi
datasource:
url: jdbc:mariadb://${MARIADB_SERVICE_HOST}:${MARIADB_SERVICE_PORT}/database
username: root
password: ${mariadb-root-password}
driverClassName: org.mariadb.jdbc.Driver
testOnBorrow: true
validationQuery: "SELECT 1"The preceding example assumes that MariaDB is deployed with mariadb as the service name. Kubernetes publishes the host and port values of these services as environment variables that we can use when configuring the apps we deploy.
We prefer to provide the MariaDB connection password in a Secrets file, as the following example shows:
apiVersion: v1
kind: Secret
metadata:
name: mariadb
labels:
app: mariadb
data:
mariadb-root-password: eW91cnBhc3N3b3JkThe password is a base64-encoded value.
11.6. Database
A relational database is used to store stream and task definitions as well as the state of executed tasks. Spring Cloud Data Flow provides schemas for MariaDB, MySQL, Oracle, PostgreSQL, Db2, SQL Server, and H2. The schema is automatically created when the server starts.
| The JDBC drivers for MariaDB, MySQL (via the MariaDB driver), PostgreSQL, SQL Server are available without additional configuration. To use any other database you need to put the corresponding JDBC driver jar on the classpath of the server as described here. |
To configure a database the following properties must be set:
-
spring.datasource.url -
spring.datasource.username -
spring.datasource.password -
spring.datasource.driver-class-name
The username and password are the same regardless of the database. However, the url and driver-class-name vary per database as follows.
| Database | spring.datasource.url | spring.datasource.driver-class-name | Driver included |
|---|---|---|---|
MariaDB 10.4+ |
jdbc:mariadb://${db-hostname}:${db-port}/${db-name} |
org.mariadb.jdbc.Driver |
Yes |
MySQL 5.7 |
jdbc:mysql://${db-hostname}:${db-port}/${db-name}?permitMysqlScheme |
org.mariadb.jdbc.Driver |
Yes |
MySQL 8.0+ |
jdbc:mysql://${db-hostname}:${db-port}/${db-name}?allowPublicKeyRetrieval=true&useSSL=false&autoReconnect=true&permitMysqlScheme[2] |
org.mariadb.jdbc.Driver |
Yes |
PostgresSQL |
jdbc:postgres://${db-hostname}:${db-port}/${db-name} |
org.postgresql.Driver |
Yes |
SQL Server |
jdbc:sqlserver://${db-hostname}:${db-port};databasename=${db-name}&encrypt=false |
com.microsoft.sqlserver.jdbc.SQLServerDriver |
Yes |
DB2 |
jdbc:db2://${db-hostname}:${db-port}/{db-name} |
com.ibm.db2.jcc.DB2Driver |
No |
Oracle |
jdbc:oracle:thin:@${db-hostname}:${db-port}/{db-name} |
oracle.jdbc.OracleDriver |
No |
11.6.1. H2
When no other database is configured then Spring Cloud Data Flow uses an embedded instance of the H2 database as the default.
| H2 is good for development purposes but is not recommended for production use nor is it supported as an external mode. |
11.6.2. Database configuration
When running in Kubernetes, the database properties are typically set in the ConfigMap. For instance, if you use MariaDB in addition to a password in the secrets file, you could provide the following properties in the ConfigMap:
data:
application.yaml: |-
spring:
datasource:
url: jdbc:mariadb://${MARIADB_SERVICE_HOST}:${MARIADB_SERVICE_PORT}/database
username: root
password: ${mariadb-root-password}
driverClassName: org.mariadb.jdbc.DriverSimilarly, for PostgreSQL you could use the following configuration:
data:
application.yaml: |-
spring:
datasource:
url: jdbc:postgresql://${PGSQL_SERVICE_HOST}:${PGSQL_SERVICE_PORT}/database
username: root
password: ${postgres-password}
driverClassName: org.postgresql.DriverThe following YAML snippet from a Deployment is an example of mounting a ConfigMap as application.yaml under /config where Spring Boot will process it plus a Secret mounted under /etc/secrets where it will get picked up by the spring-cloud-kubernetes library due to the environment variable SPRING_CLOUD_KUBERNETES_SECRETS_PATHS being set to /etc/secrets.
...
containers:
- name: scdf-server
image: springcloud/spring-cloud-dataflow-server:2.5.0.BUILD-SNAPSHOT
imagePullPolicy: Always
volumeMounts:
- name: config
mountPath: /config
readOnly: true
- name: database
mountPath: /etc/secrets/database
readOnly: true
ports:
...
volumes:
- name: config
configMap:
name: scdf-server
items:
- key: application.yaml
path: application.yaml
- name: database
secret:
secretName: mariadbYou can find migration scripts for specific database types in the spring-cloud-task repo.
11.7. Monitoring and Management
We recommend using the kubectl command for troubleshooting streams and tasks.
You can list all artifacts and resources used by using the following command:
kubectl get all,cm,secrets,pvcYou can list all resources used by a specific application or service by using a label to select resources. The following command lists all resources used by the mariadb service:
kubectl get all -l app=mariadbYou can get the logs for a specific pod by issuing the following command:
kubectl logs pod <pod-name>If the pod is continuously getting restarted, you can add -p as an option to see the previous log, as follows:
kubectl logs -p <pod-name>You can also tail or follow a log by adding an -f option, as follows:
kubectl logs -f <pod-name>A useful command to help in troubleshooting issues, such as a container that has a fatal error when starting up, is to use the describe command, as the following example shows:
kubectl describe pod ticktock-log-0-qnk7211.7.1. Inspecting Server Logs
You can access the server logs by using the following command:
kubectl get pod -l app=scdf=server
kubectl logs <scdf-server-pod-name>11.7.2. Streams
Stream applications are deployed with the stream name followed by the name of the application. For processors and sinks, an instance index is also appended.
To see all the pods that are deployed by the Spring Cloud Data Flow server, you can specify the role=spring-app label, as follows:
kubectl get pod -l role=spring-appTo see details for a specific application deployment you can use the following command:
kubectl describe pod <app-pod-name>To view the application logs, you can use the following command:
kubectl logs <app-pod-name>If you would like to tail a log you can use the following command:
kubectl logs -f <app-pod-name>11.7.3. Tasks
Tasks are launched as bare pods without a replication controller. The pods remain after the tasks complete, which gives you an opportunity to review the logs.
To see all pods for a specific task, use the following command:
kubectl get pod -l task-name=<task-name>To review the task logs, use the following command:
kubectl logs <task-pod-name>You have two options to delete completed pods. You can delete them manually once they are no longer needed or you can use the Data Flow shell task execution cleanup command to remove the completed pod for a task execution.
To delete the task pod manually, use the following command:
kubectl delete pod <task-pod-name>To use the task execution cleanup command, you must first determine the ID for the task execution. To do so, use the task execution list command, as the following example (with output) shows:
dataflow:>task execution list
╔═════════╤══╤════════════════════════════╤════════════════════════════╤═════════╗
║Task Name│ID│ Start Time │ End Time │Exit Code║
╠═════════╪══╪════════════════════════════╪════════════════════════════╪═════════╣
║task1 │1 │Fri May 05 18:12:05 EDT 2017│Fri May 05 18:12:05 EDT 2017│0 ║
╚═════════╧══╧════════════════════════════╧════════════════════════════╧═════════╝Once you have the ID, you can issue the command to cleanup the execution artifacts (the completed pod), as the following example shows:
dataflow:>task execution cleanup --id 1
Request to clean up resources for task execution 1 has been submittedDatabase Credentials for Tasks
By default Spring Cloud Data Flow passes database credentials as properties to the pod at task launch time.
If using the exec or shell entry point styles the DB credentials will be viewable if the user does a kubectl describe on the task’s pod.
To configure Spring Cloud Data Flow to use Kubernetes Secrets: Set spring.cloud.dataflow.task.use.kubernetes.secrets.for.db.credentials property to true. If using the yaml files provided by Spring Cloud Data Flow update the `src/kubernetes/server/server-deployment.yaml to add the following environment variable:
- name: SPRING_CLOUD_DATAFLOW_TASK_USE_KUBERNETES_SECRETS_FOR_DB_CREDENTIALS
value: 'true'If upgrading from a previous version of SCDF be sure to verify that spring.datasource.username and spring.datasource.password environment variables are present in the secretKeyRefs in the server-config.yaml. If not, add it as shown in the example below:
...
task:
platform:
kubernetes:
accounts:
default:
secretKeyRefs:
- envVarName: "spring.datasource.password"
secretName: mariadb
dataKey: mariadb-root-password
- envVarName: "spring.datasource.username"
secretName: mariadb
dataKey: mariadb-root-username
...Also verify that the associated secret(dataKey) is also available in secrets. SCDF provides an example of this for MariaDB here: src/kubernetes/mariadb/mariadb-svc.yaml.
| Passing of DB credentials via properties by default is to preserve to backwards compatibility. This will be feature will be removed in future release. |
11.8. Scheduling
This section covers customization of how scheduled tasks are configured. Scheduling of tasks is enabled by default in the Spring Cloud Data Flow Kubernetes Server. Properties are used to influence settings for scheduled tasks and can be configured on a global or per-schedule basis.
| Unless noted, properties set on a per-schedule basis always take precedence over properties set as the server configuration. This arrangement allows for the ability to override global server level properties for a specific schedule. |
11.8.1. Entry Point Style
An Entry Point Style affects how application properties are passed to the task container to be deployed. Currently, three styles are supported:
-
exec: (default) Passes all application properties as command line arguments. -
shell: Passes all application properties as environment variables. -
boot: Creates an environment variable calledSPRING_APPLICATION_JSONthat contains a JSON representation of all application properties.
You can configure the entry point style as follows:
deployer.kubernetes.entryPointStyle=<Entry Point Style>Replace <Entry Point Style> with your desired Entry Point Style.
You can also configure the Entry Point Style at the server level in the container env section of a deployment YAML, as the following example shows:
env:
- name: SPRING_CLOUD_DEPLOYER_KUBERNETES_ENTRY_POINT_STYLE
value: entryPointStyleReplace entryPointStyle with the desired Entry Point Style.
You should choose an Entry Point Style of either exec or shell, to correspond to how the ENTRYPOINT syntax is defined in the container’s Dockerfile. For more information and uses cases on exec vs shell, see the ENTRYPOINT section of the Docker documentation.
Using the boot Entry Point Style corresponds to using the exec style ENTRYPOINT. Command line arguments from the deployment request are passed to the container, with the addition of application properties mapped into the SPRING_APPLICATION_JSON environment variable rather than command line arguments.
ttlSecondsAfterFinished
When scheduling an application, You can clean up finished Jobs (either Complete or Failed) automatically by specifying ttlSecondsAfterFinished value.
The following example shows how you can individually configure application jobs:
deployer.<application>.kubernetes.cron.ttlSecondsAfterFinished=86400Replace <application> with the name of your application and the ttlSecondsAfterFinished attribute with the appropriate value for clean up finished Jobs.
You can configure the ttlSecondsAfterFinished at the global server level as well.
The following example shows how to do so for tasks:
You can configure an image pull policy at the server level in the container env section of a deployment YAML, as the following example shows:
env:
- name: SPRING_CLOUD_DEPLOYER_KUBERNETES_TTL_SECONDS_AFTER_FINISHED
value: 8640011.8.2. Environment Variables
To influence the environment settings for a given application, you can take advantage of the spring.cloud.deployer.kubernetes.environmentVariables property.
For example, a common requirement in production settings is to influence the JVM memory arguments.
You can achieve this by using the JAVA_TOOL_OPTIONS environment variable, as the following example shows:
deployer.kubernetes.environmentVariables=JAVA_TOOL_OPTIONS=-Xmx1024m
When deploying stream applications or launching task applications where some of the properties may contain sensitive information, use the shell or boot as the entryPointStyle. This is because the exec (default) converts all properties to command line arguments and thus may not be secure in some environments.
|
Additionally you can configure environment variables at the server level in the container env section of a deployment YAML, as the following example shows:
| When specifying environment variables in the server configuration and on a per-schedule basis, environment variables will be merged. This allows for the ability to set common environment variables in the server configuration and more specific at the specific schedule level. |
env:
- name: SPRING_CLOUD_DEPLOYER_KUBERNETES_ENVIRONMENT_VARIABLES
value: myVar=myValReplace myVar=myVal with your desired environment variables.
11.8.3. Image Pull Policy
An image pull policy defines when a Docker image should be pulled to the local registry. Currently, three policies are supported:
-
IfNotPresent: (default) Do not pull an image if it already exists. -
Always: Always pull the image regardless of whether it already exists. -
Never: Never pull an image. Use only an image that already exists.
The following example shows how you can individually configure containers:
deployer.kubernetes.imagePullPolicy=AlwaysReplace Always with your desired image pull policy.
You can configure an image pull policy at the server level in the container env section of a deployment YAML, as the following example shows:
env:
- name: SPRING_CLOUD_DEPLOYER_KUBERNETES_IMAGE_PULL_POLICY
value: AlwaysReplace Always with your desired image pull policy.
11.8.4. Private Docker Registry
Docker images that are private and require authentication can be pulled by configuring a Secret. First, you must create a Secret in the cluster. Follow the Pull an Image from a Private Registry guide to create the Secret.
Once you have created the secret, use the imagePullSecret property to set the secret to use, as the following example shows:
deployer.kubernetes.imagePullSecret=mysecretReplace mysecret with the name of the secret you created earlier.
You can also configure the image pull secret at the server level in the container env section of a deployment YAML, as the following example shows:
env:
- name: SPRING_CLOUD_DEPLOYER_KUBERNETES_IMAGE_PULL_SECRET
value: mysecretReplace mysecret with the name of the secret you created earlier.
11.8.5. Namespace
By default the namespace used for scheduled tasks is default. This value can be set at the server level configuration in the container env section of a deployment YAML, as the following example shows:
env:
- name: SPRING_CLOUD_DEPLOYER_KUBERNETES_NAMESPACE
value: mynamespace11.8.6. Service Account
You can configure a custom service account for scheduled tasks through properties. An existing service account can be used or a new one created. One way to create a service account is by using kubectl, as the following example shows:
$ kubectl create serviceaccount myserviceaccountname
serviceaccount "myserviceaccountname" createdThen you can configure the service account to use on a per-schedule basis as follows:
deployer.kubernetes.taskServiceAccountName=myserviceaccountnameReplace myserviceaccountname with your service account name.
You can also configure the service account name at the server level in the container env section of a deployment YAML, as the following example shows:
env:
- name: SPRING_CLOUD_DEPLOYER_KUBERNETES_TASK_SERVICE_ACCOUNT_NAME
value: myserviceaccountnameReplace myserviceaccountname with the service account name to be applied to all deployments.
For more information on scheduling tasks see Scheduling Tasks.
11.9. Debug Support
Debugging the Spring Cloud Data Flow Kubernetes Server and included components (such as the Spring Cloud Kubernetes Deployer) is supported through the Java Debug Wire Protocol (JDWP). This section outlines an approach to manually enable debugging and another approach that uses configuration files provided with Spring Cloud Data Flow Server Kubernetes to “patch” a running deployment.
| JDWP itself does not use any authentication. This section assumes debugging is being done on a local development environment (such as Minikube), so guidance on securing the debug port is not provided. |
11.9.1. Enabling Debugging Manually
To manually enable JDWP, first edit src/kubernetes/server/server-deployment.yaml and add an additional containerPort entry under spec.template.spec.containers.ports with a value of 5005. Additionally, add the JAVA_TOOL_OPTIONS environment variable under spec.template.spec.containers.env as the following example shows:
spec:
...
template:
...
spec:
containers:
- name: scdf-server
...
ports:
...
- containerPort: 5005
env:
- name: JAVA_TOOL_OPTIONS
value: '-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005'
The preceding example uses port 5005, but it can be any number that does not conflict with another port. The chosen port number must also be the same for the added containerPort value and the address parameter of the JAVA_TOOL_OPTIONS -agentlib flag, as shown in the preceding example.
|
You can now start the Spring Cloud Data Flow Kubernetes Server. Once the server is up, you can verify the configuration changes on the scdf-server deployment, as the following example (with output) shows:
kubectl describe deployment/scdf-server
...
...
Pod Template:
...
Containers:
scdf-server:
...
Ports: 80/TCP, 5005/TCP
...
Environment:
JAVA_TOOL_OPTIONS: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
...With the server started and JDWP enabled, you need to configure access to the port. In this example, we use the port-forward subcommand of kubectl. The following example (with output) shows how to expose a local port to your debug target by using port-forward:
$ kubectl get pod -l app=scdf-server
NAME READY STATUS RESTARTS AGE
scdf-server-5b7cfd86f7-d8mj4 1/1 Running 0 10m
$ kubectl port-forward scdf-server-5b7cfd86f7-d8mj4 5005:5005
Forwarding from 127.0.0.1:5005 -> 5005
Forwarding from [::1]:5005 -> 5005You can now attach a debugger by pointing it to 127.0.0.1 as the host and 5005 as the port. The port-forward subcommand runs until stopped (by pressing CTRL+c, for example).
You can remove debugging support by reverting the changes to src/kubernetes/server/server-deployment.yaml. The reverted changes are picked up on the next deployment of the Spring Cloud Data Flow Kubernetes Server. Manually adding debug support to the configuration is useful when debugging should be enabled by default each time the server is deployed.
11.9.2. Enabling Debugging with Patching
Rather than manually changing the server-deployment.yaml, Kubernetes objects can be “patched” in place. For convenience, patch files that provide the same configuration as the manual approach are included. To enable debugging by patching, use the following command:
kubectl patch deployment scdf-server -p "$(cat src/kubernetes/server/server-deployment-debug.yaml)"Running the preceding command automatically adds the containerPort attribute and the JAVA_TOOL_OPTIONS environment variable. The following example (with output) shows how toverify changes to the scdf-server deployment:
$ kubectl describe deployment/scdf-server
...
...
Pod Template:
...
Containers:
scdf-server:
...
Ports: 5005/TCP, 80/TCP
...
Environment:
JAVA_TOOL_OPTIONS: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
...To enable access to the debug port, rather than using the port-forward subcommand of kubectl, you can patch the scdf-server Kubernetes service object. You must first ensure that the scdf-server Kubernetes service object has the proper configuration. The following example (with output) shows how to do so:
kubectl describe service/scdf-server
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 30784/TCPIf the output contains <unset>, you must patch the service to add a name for this port. The following example shows how to do so:
$ kubectl patch service scdf-server -p "$(cat src/kubernetes/server/server-svc.yaml)"
A port name should only be missing if the target cluster had been created prior to debug functionality being added. Since multiple ports are being added to the scdf-server Kubernetes Service Object, each needs to have its own name.
|
Now you can add the debug port, as the following example shows:
kubectl patch service scdf-server -p "$(cat src/kubernetes/server/server-svc-debug.yaml)"The following example (with output) shows how to verify the mapping:
$ kubectl describe service scdf-server
Name: scdf-server
...
...
Port: scdf-server-jdwp 5005/TCP
TargetPort: 5005/TCP
NodePort: scdf-server-jdwp 31339/TCP
...
...
Port: scdf-server 80/TCP
TargetPort: 80/TCP
NodePort: scdf-server 30883/TCP
...
...The output shows that container port 5005 has been mapped to the NodePort of 31339. The following example (with output) shows how to get the IP address of the Minikube node:
$ minikube ip
192.168.99.100With this information, you can create a debug connection by using a host of 192.168.99.100 and a port of 31339.
The following example shows how to disable JDWP:
$ kubectl rollout undo deployment/scdf-server
$ kubectl patch service scdf-server --type json -p='[{"op": "remove", "path": "/spec/ports/0"}]'The Kubernetes deployment object is rolled back to its state before being patched. The Kubernetes service object is then patched with a remove operation to remove port 5005 from the containerPorts list.
kubectl rollout undo forces the pod to restart. Patching the Kubernetes Service Object does not re-create the service, and the port mapping to the scdf-server deployment remains the same.
|
See Rolling Back a Deployment for more information on deployment rollbacks, including managing history and Updating API Objects in Place Using kubectl Patch.