Shell

This section covers the options for starting the shell and more advanced functionality relating to how the shell handles whitespace, quotes, and interpretation of SpEL expressions. The introductory chapters to the Stream DSL and Composed Task DSL are good places to start for the most common usage of shell commands.spring-doc.cn

13. Shell Options

The shell is built upon the Spring Shell project. Some command-line options come from Spring Shell, and some are specific to Data Flow. The shell takes the following command line options:spring-doc.cn

unix:>java -jar spring-cloud-dataflow-shell-2.9.6.jar --help
Data Flow Options:
  --dataflow.uri=                              Address of the Data Flow Server [default: http://localhost:9393].
  --dataflow.username=                        Username of the Data Flow Server [no default].
  --dataflow.password=                    Password of the Data Flow Server [no default].
  --dataflow.credentials-provider-command= Executes an external command which must return an
                                                    OAuth Bearer Token (Access Token prefixed with 'Bearer '),
                                                    e.g. 'Bearer 12345'), [no default].
  --dataflow.skip-ssl-validation=       Accept any SSL certificate (even self-signed) [default: no].
  --dataflow.proxy.uri=                  Address of an optional proxy server to use [no default].
  --dataflow.proxy.username=        Username of the proxy server (if required by proxy server) [no default].
  --dataflow.proxy.password=        Password of the proxy server (if required by proxy server) [no default].
  --spring.shell.historySize=                 Default size of the shell log file [default: 3000].
  --spring.shell.commandFile=                 Data Flow Shell executes commands read from the file(s) and then exits.
  --help                                            This message.

You can use the spring.shell.commandFile option to point to an existing file that contains all the shell commands to deploy one or many related streams and tasks. Running multiple files is also supported. They should be passed as a comma-delimited string:spring-doc.cn

--spring.shell.commandFile=file1.txt,file2.txtspring-doc.cn

This option is useful when creating some scripts to help automate deployment.spring-doc.cn

Also, the following shell command helps to modularize a complex script into multiple independent files:spring-doc.cn

dataflow:>script --file <YOUR_AWESOME_SCRIPT>spring-doc.cn

14. Listing Available Commands

Typing help at the command prompt gives a listing of all available commands. Most of the commands are for Data Flow functionality, but a few are general purpose. The following listing shows the output of the help command:spring-doc.cn

! - Allows execution of operating system (OS) commands
clear - Clears the console
cls - Clears the console
date - Displays the local date and time
exit - Exits the shell
http get - Make GET request to http endpoint
http post - POST data to http endpoint
quit - Exits the shell
system properties - Shows the shells properties {JB - restore the apostrophe}
version - Displays shell version

Adding the name of the command to help shows additional information on how to invoke the command:spring-doc.cn

dataflow:>help stream create
Keyword:                   stream create
Description:               Create a new stream definition
 Keyword:                  ** default **
 Keyword:                  name
   Help:                   the name to give to the stream
   Mandatory:              true
   Default if specified:   '__NULL__'
   Default if unspecified: '__NULL__'

 Keyword:                  definition
   Help:                   a stream definition, using the DSL (e.g. "http --port=9000 | hdfs")
   Mandatory:              true
   Default if specified:   '__NULL__'
   Default if unspecified: '__NULL__'

 Keyword:                  deploy
   Help:                   whether to deploy the stream immediately
   Mandatory:              false
   Default if specified:   'true'
   Default if unspecified: 'false'

15. Tab Completion

You can complete the shell command options in the shell by pressing the TAB key after the leading --. For example, pressing TAB after stream create -- results in the following pair of suggestions:spring-doc.cn

dataflow:>stream create --
stream create --definition    stream create --name

If you type --de and then press tab, --definition expands.spring-doc.cn

Tab completion is also available inside the stream or composed task DSL expression for application or task properties. You can also use TAB to get hints in a stream DSL expression for the available sources, processors, or sinks that you can use.spring-doc.cn

16. Whitespace and Quoting Rules

You need to quote parameter values only if they contain spaces or the | character. The following example passes a SpEL expression (which is applied to any data it encounters) to a transform processor:spring-doc.cn

transform --expression='new StringBuilder(payload).reverse()'

If the parameter value needs to embed a single quote, use two single quotes, as follows:spring-doc.cn

// Query is: Select * from /Customers where name='Smith'
scan --query='Select * from /Customers where name=''Smith'''

16.1. Quotes and Escaping

There is a Spring Shell-based client that talks to the Data Flow Server and is responsible for parsing the DSL. In turn, applications may have application properties that rely on embedded languages, such as the Spring Expression Language.spring-doc.cn

The Shell, Data Flow DSL parser, and SpEL have rules about how they handle quotes and how syntax escaping works. When combined together, confusion may arise. This section explains the rules that apply and provides examples of the most complicated situations you may encounter when all three components are involved.spring-doc.cn

It is not always that complicated

If you do not use the Data Flow Shell (for example, if you use the REST API directly) or if application properties are not SpEL expressions, the escaping rules are simpler.spring-doc.cn

16.1.1. Shell Rules

Arguably, the most complex component when it comes to quotes is the Shell. The rules can be laid out quite simply, though:spring-doc.cn

  • A shell command is made of keys (--something) and corresponding values. There is a special, keyless mapping, though, which is described later.spring-doc.cn

  • A value cannot normally contain spaces, as space is the default delimiter for commands.spring-doc.cn

  • Spaces can be added though, by surrounding the value with quotes (either single (') or double (") quotes).spring-doc.cn

  • Values passed inside deployment properties (for example, deployment <stream-name> --properties " …​") should not be quoted again.spring-doc.cn

  • If surrounded with quotes, a value can embed a literal quote of the same kind by prefixing it with a backslash (\).spring-doc.cn

  • Other escapes are available, such as \t, \n, \r, \f and unicode escapes of the form \uxxxx.spring-doc.cn

  • The keyless mapping is handled in a special way such that it does not need quoting to contain spaces.spring-doc.cn

For example, the shell supports the ! command to execute native shell commands. The ! accepts a single keyless argument. This is why the following example works:spring-doc.cn

dataflow:>! rm something

The argument here is the whole rm something string, which is passed as is to the underlying shell.spring-doc.cn

As another example, the following commands are strictly equivalent, and the argument value is something (without the quotes):spring-doc.cn

dataflow:>stream destroy something
dataflow:>stream destroy --name something
dataflow:>stream destroy "something"
dataflow:>stream destroy --name "something"

16.1.2. Property Files Rules

The rules are relaxed when loading the properties from files.spring-doc.cn

  • The special characters used in property files (both Java and YAML) need to be escaped. For example \ should be replaced by \\, \t by \\t and so forth.spring-doc.cn

  • For Java property files (--propertiesFile <FILE_PATH>.properties), the property values should not be surrounded by quotes. It is not needed even if they contain spaces.spring-doc.cn

    filter.expression=payload > 5

  • For YAML property files (--propertiesFile <FILE_PATH>.yaml), though, the values need to be surrounded by double quotes.spring-doc.cn

    app:
    filter:
        filter:
            expression: "payload > 5"

16.1.3. DSL Parsing Rules

At the parser level (that is, inside the body of a stream or task definition), the rules are as follows:spring-doc.cn

  • Option values are normally parsed until the first space character.spring-doc.cn

  • They can be made of literal strings, though, surrounded by single or double quotes.spring-doc.cn

  • To embed such a quote, use two consecutive quotes of the desired kind.spring-doc.cn

As such, the values of the --expression option to the filter application are semantically equivalent in the following examples:spring-doc.cn

filter --expression=payload>5
filter --expression="payload>5"
filter --expression='payload>5'
filter --expression='payload > 5'

Arguably, the last one is more readable. It is made possible thanks to the surrounding quotes. The actual expression is payload > 5.spring-doc.cn

Now, imagine that we want to test against string messages. If we want to compare the payload to the SpEL literal string, "something", we could use the following:spring-doc.cn

filter --expression=payload=='something'           (1)
filter --expression='payload == ''something'''     (2)
filter --expression='payload == "something"'       (3)
1 This works because there are no spaces. It is not very legible, though.
2 This uses single quotes to protect the whole argument. Hence, the actual single quotes need to be doubled.
3 SpEL recognizes String literals with either single or double quotes, so this last method is arguably the most readable.

Note that the preceding examples are to be considered outside of the shell (for example, when calling the REST API directly). When entered inside the shell, chances are that the whole stream definition is itself inside double quotes, which would need to be escaped. The whole example then becomes the following:spring-doc.cn

dataflow:>stream create something --definition "http | filter --expression=payload='something' | log"

dataflow:>stream create something --definition "http | filter --expression='payload == ''something''' | log"

dataflow:>stream create something --definition "http | filter --expression='payload == \"something\"' | log"

16.1.4. SpEL Syntax and SpEL Literals

The last piece of the puzzle is about SpEL expressions. Many applications accept options that are to be interpreted as SpEL expressions, and, as seen earlier, String literals are handled in a special way there, too. The rules are as follows:spring-doc.cn

  • Literals can be enclosed in either single or double quotes.spring-doc.cn

  • Quotes need to be doubled to embed a literal quote. Single quotes inside double quotes need no special treatment, and the reverse is also true.spring-doc.cn

As a last example, assume you want to use the transform processor. This processor accepts an expression option which is a SpEL expression. It is to be evaluated against the incoming message, with a default of payload (which forwards the message payload untouched).spring-doc.cn

It is important to understand that the following statements are equivalent:spring-doc.cn

transform --expression=payload
transform --expression='payload'

However, they are different from the following (and variations upon them):spring-doc.cn

transform --expression="'payload'"
transform --expression='''payload'''

The first series evaluates to the message payload, while the latter examples evaluate to the literal string, payload.spring-doc.cn

16.1.5. Putting It All Together

As a last, complete example, consider how you could force the transformation of all messages to the string literal, hello world, by creating a stream in the context of the Data Flow shell:spring-doc.cn

dataflow:>stream create something --definition "http | transform --expression='''hello world''' | log" (1)

dataflow:>stream create something --definition "http | transform --expression='\"hello world\"' | log" (2)

dataflow:>stream create something --definition "http | transform --expression=\"'hello world'\" | log" (2)
1 In the first line, single quotes surround the string (at the Data Flow parser level), but they need to be doubled because they are inside a string literal (started by the first single quote after the equals sign).
2 The second and third lines use single and double quotes, respectively, to encompass the whole string at the Data Flow parser level. Consequently, the other kind of quote can be used inside the string. The whole thing is inside the --definition argument to the shell, though, which uses double quotes. Consequently, double quotes are escaped (at the shell level).