Plugin Configuration Basics

When you add or update storage plugin configurations on one Drill node in a cluster having multiple installations of Drill, Drill broadcasts the information to other Drill nodes to synchronize the storage plugin configurations. You do not need to restart any of the Drillbits when you add or update a storage plugin configuration.

Using the Drill Web Console

You can use the Drill Web Console to update or add a new storage plugin configuration. The Drill shell needs to be running to start the Web Console.

To create a name and new configuration:

1. Start the Drill shell.
2. Start the Web Console.
   

3. On the Storage tab, enter a name in New Storage Plugin. Each configuration registered with Drill must have a distinct name. Names are case-sensitive.
   

   Note

   The URL differs depending on your installation and configuration.

4. Click Create.

5. In Configuration, use JSON formatting to modify a copy of an existing configuration if possible.
   Using a copy of an existing configuration reduces the risk of JSON coding errors. Use the Storage Plugin Attributes table in the next section as a guide for making typical modifications.

6. Click Create.

Storage Plugin Attributes

The following graphic shows key attributes of a typical dfs-based storage plugin configuration:

List of Attributes and Definitions

The following table describes the attributes you configure for storage plugins installed with Drill.

Using the Formats Attributes

You set the formats attributes, such as skipFirstLine, in the formats area of the storage plugin configuration. When setting attributes for text files, such as CSV, you also need to set the sys.options property exec.storage.enable_new_text_reader to true (the default). For more information and examples of using formats for text files, see "Text Files: CSV, TSV, PSV".

Using the Formats Attributes as Table Function Parameters

In Drill version 1.4 and later, you can also set the formats attributes defined above on a per query basis. To pass parameters to the format plugin, use the table function syntax:

select a, b from table({table function name}(parameters))

The table function name is the table name, the type parameter is the format name, and the other parameters are the fields that the format plugin configuration accepts, as defined in the table above (except for extensions which do not apply in this context).

For example, to read a CSV file and parse the header:
select a, b from table(dfs.`path/to/data.csv`(type => 'text', fieldDelimiter => ',', extractHeader => true))

For more information about format plugin configuration see "Text Files: CSV, TSV, PSV".

Using Other Attributes

The configuration of other attributes, such as size.calculator.enabled in the hbase plugin and configProps in the hive plugin, are implementation-dependent and beyond the scope of this document.

Case-sensitive Names

As previously mentioned, workspace and storage plugin names are case-sensitive. For example, the following query uses a storage plugin name dfs and a workspace name clicks. When you refer to dfs.clicks in an SQL statement, use the defined case:

0: jdbc:drill:> USE dfs.clicks;

For example, using uppercase letters in the query after defining the storage plugin and workspace names using lowercase letters does not work.

Storage Plugin REST API

If you need to add a storage plugin configuration to Drill and do not want to use a web browser, you can use the Drill REST API to create a storage plugin configuration. Use a POST request and pass two properties:

• name
  The storage plugin configuration name.

• config
  The attribute settings as entered in the Web Console.

For example, this command creates a storage plugin named myplugin for reading files of an unknown type located on the root of the file system:

curl -X POST -H "Content-Type: application/json" -d '{"name":"myplugin", "config": {"type": "file", "enabled": false, "connection": "file:///", "workspaces": { "root": { "location": "/", "writable": false, "defaultInputFormat": null}}, "formats": null}}' http://localhost:8047/storage/myplugin.json

This example assumes HTTPS has not been enabled.

Bootstrapping a Storage Plugin

The REST API is recommended for programmatically adding a storage plugin configuration to Drill. An alternative for use in a distributed environment only is bootstrapping. You can create a bootstrap-storage-plugins.json file and include it on the classpath when starting Drill. The storage plugin configuration loads when Drill starts up.

Currently, bootstrapping a storage plugin configuration works only when the first Drillbit in the cluster first starts up. The configuration is stored in ZooKeeper, preventing Drill from picking up the bootstrap-storage-plugins.json again.

After cluster startup, you have to use the REST API or Drill Web Console to add a storage plugin configuration. Alternatively, you can modify the entry in ZooKeeper by uploading the json file for that plugin to the /drill directory of the zookeeper installation, or by just deleting the /drill directory if you do not have configuration properties to preserve.

If you load an HBase storage plugin configuration using bootstrap-storage-plugins.json file and HBase is not installed, you might experience a delay when executing the queries. Configure the HBase client timeout and retry settings in the config block of the HBase plugin configuration.