MS Azure Data Lake Storage (ADLS) Connector for Data Prep¶
User Persona: Data Prep User, Data Prep Admin, Data Source Admin, or IT/DevOps
This document covers all configuration fields available during connector setup. Some fields may have already been filled out by your Administrator at an earlier step of configuration and may not be visible to you. For more information on Data Prep's connector framework, see Data Prep Connector setup. Also, your Admin may have named this connector something else in the list of Data Sources.
Configure Data Prep¶
This connector allows you to connect to Azure Data Lake Storage (ADLS) for imports and exports. The following fields are used to define the connection parameters.
Name: Name of the data source as it will appear to users in the UI.
Description: Description of the data source as it will appear to users in the UI.
You can connect Data Prep to multiple Azure Data Lake Storage accounts. Using a descriptive name can be a big help to users in identifying the appropriate data source.
Azure Data Lake Storage Configuration¶
- ADL URI: The URI for the ADL site.
- Root Directory: Specifies the top-level of the directory structure from which import/export of data is enabled.
- Application ID: The application ID for the ADL site.
- OAUTH 2.0 Token Endpoint: The OAUTH 2.0 Token Endpoint for the ADL site.
- Application Access Key Value: The Application Access Key Value for the ADL site. See Issue 1 in the FAQ/Troubleshooting/Common Issues section for more information.
Can we have both ADLS Gen1 and ADLS Gen2 Connectors in the same Data Prep account?¶
Yes. The two Connectors can coexist and will not interfere with each other.
Issue: When you Test Connection, it fails and reports a "base64" issue.¶
How to fix it: In March 2020, Azure changed the format of the Application Access Key Value. The new format does not work for authentication, so you'll need to use the Azure command line to set the Base64 encoded version of the Application Access Key Value.
In the Azure Portal:
- Create a new Application Access Service Account.
- Copy the generated Access Key value.
- Base64 encode the password.
- Mac example: echo -n '
' | openssl base64
- Windows: Use a tool like Base64 Encoder.
- Create an Azure Command to reset the password to a Base64 encoded version of the password:
- az ad sp credential reset --name
--credential-description " " --append --years 2 -p " " -o=jsonc
- Open a Command prompt in the Azure Portal and paste the command from Step 4.
- Ensure that this service account has appropriate ACLs for the storage. You will receive ACL errors if permissions are not appropriate.
- Configure Application Access Key Value using the new Base64 encoded password.