Dataset Definition¶
The endpoints below outline how to create and manage data source definitions.
List all dataset definitions¶
Operation path: GET /api/v2/datasetDefinitions/
Authentication requirements: BearerAuth
List all dataset definitions.
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| offset | query | integer | false | This many results will be skipped. |
| limit | query | integer | false | At most this many results are returned. |
Example responses¶
200 Response
{
"properties": {
"count": {
"description": "Number of items returned on this page.",
"type": "integer"
},
"data": {
"description": "A list of user-defined dataset definitions.",
"items": {
"properties": {
"creatorUserId": {
"description": "The ID of the user who created the dataset definition.",
"type": "string"
},
"datasetInfo": {
"description": "Information about the dataset.",
"properties": {
"columns": {
"description": "List of the dataset column names, this field is auto-generated by the analysis job.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 1000,
"minItems": 2,
"type": "array"
},
"dataSourceId": {
"default": null,
"description": "The ID of the SQL table query and the database path, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dataStoreId": {
"default": null,
"description": "The ID of the SQL data store, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dialect": {
"description": "Source type data was retrieved from, this field is auto-generated by the analysis job.",
"enum": [
"snowflake",
"bigquery",
"databricks",
"spark",
"postgres"
],
"type": "string"
},
"estimatedSizePerRow": {
"description": "Estimated byte size per row of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"sourceSize": {
"description": "Total dataset byte size, this field is auto-generated by the analysis job.",
"type": "integer"
},
"totalRows": {
"description": "Total rows of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"version": {
"description": "The version of the dataset definition information.",
"type": "integer"
}
},
"required": [
"columns",
"dataSourceId",
"dataStoreId",
"dialect",
"estimatedSizePerRow",
"sourceSize",
"totalRows",
"version"
],
"type": "object",
"x-versionadded": "v2.37"
},
"datasetProps": {
"description": "Dataset properties.",
"properties": {
"datasetId": {
"description": "The ID of the AI Catalog dataset.",
"type": "string"
},
"datasetVersionId": {
"description": "The version ID of the AI Catalog dataset.",
"type": "string"
}
},
"required": [
"datasetId",
"datasetVersionId"
],
"type": "object",
"x-versionadded": "v2.37"
},
"dynamicDatasetProps": {
"description": "Dynamic dataset additional properties.",
"properties": {
"credentialsId": {
"default": null,
"description": "The ID of the credentials to access the data store.",
"type": [
"string",
"null"
]
}
},
"required": [
"credentialsId"
],
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the dataset definition.",
"type": "string"
},
"name": {
"description": "The name of the dataset definition.",
"type": "string"
}
},
"required": [
"creatorUserId",
"datasetInfo",
"datasetProps",
"dynamicDatasetProps",
"id",
"name"
],
"type": "object",
"x-versionadded": "v2.37"
},
"maxItems": 100,
"type": "array"
},
"next": {
"description": "URL pointing to the next page (if null, there is no next page).",
"format": "uri",
"type": [
"string",
"null"
]
},
"previous": {
"description": "URL pointing to the previous page (if null, there is no previous page).",
"format": "uri",
"type": [
"string",
"null"
]
},
"totalCount": {
"description": "The total number of items across all pages.",
"type": "integer"
}
},
"required": [
"data",
"next",
"previous",
"totalCount"
],
"type": "object",
"x-versionadded": "v2.37"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | none | DatasetDefPaginatedResponse |
Create a dataset definition¶
Operation path: POST /api/v2/datasetDefinitions/
Authentication requirements: BearerAuth
Create a dataset definition.
Body parameter¶
{
"properties": {
"credentialsId": {
"description": "The ID of the credentials to access the data store.",
"type": "string"
},
"datasetId": {
"description": "The ID of the AI Catalog dataset.",
"type": "string"
},
"datasetVersionId": {
"default": null,
"description": "The version ID of the AI Catalog dataset.",
"type": [
"string",
"null"
]
},
"name": {
"default": null,
"description": "The name of the dataset definition.",
"type": [
"string",
"null"
]
}
},
"required": [
"datasetId"
],
"type": "object",
"x-versionadded": "v2.37"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| body | body | DatasetDefCreate | false | none |
Example responses¶
201 Response
{
"properties": {
"creatorUserId": {
"description": "The ID of the user who created the dataset definition.",
"type": "string"
},
"datasetInfo": {
"description": "Information about the dataset.",
"properties": {
"columns": {
"description": "List of the dataset column names, this field is auto-generated by the analysis job.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 1000,
"minItems": 2,
"type": "array"
},
"dataSourceId": {
"default": null,
"description": "The ID of the SQL table query and the database path, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dataStoreId": {
"default": null,
"description": "The ID of the SQL data store, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dialect": {
"description": "Source type data was retrieved from, this field is auto-generated by the analysis job.",
"enum": [
"snowflake",
"bigquery",
"databricks",
"spark",
"postgres"
],
"type": "string"
},
"estimatedSizePerRow": {
"description": "Estimated byte size per row of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"sourceSize": {
"description": "Total dataset byte size, this field is auto-generated by the analysis job.",
"type": "integer"
},
"totalRows": {
"description": "Total rows of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"version": {
"description": "The version of the dataset definition information.",
"type": "integer"
}
},
"required": [
"columns",
"dataSourceId",
"dataStoreId",
"dialect",
"estimatedSizePerRow",
"sourceSize",
"totalRows",
"version"
],
"type": "object",
"x-versionadded": "v2.37"
},
"datasetProps": {
"description": "Dataset properties.",
"properties": {
"datasetId": {
"description": "The ID of the AI Catalog dataset.",
"type": "string"
},
"datasetVersionId": {
"description": "The version ID of the AI Catalog dataset.",
"type": "string"
}
},
"required": [
"datasetId",
"datasetVersionId"
],
"type": "object",
"x-versionadded": "v2.37"
},
"dynamicDatasetProps": {
"description": "Dynamic dataset additional properties.",
"properties": {
"credentialsId": {
"default": null,
"description": "The ID of the credentials to access the data store.",
"type": [
"string",
"null"
]
}
},
"required": [
"credentialsId"
],
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the dataset definition.",
"type": "string"
},
"name": {
"description": "The name of the dataset definition.",
"type": "string"
}
},
"required": [
"creatorUserId",
"datasetInfo",
"datasetProps",
"dynamicDatasetProps",
"id",
"name"
],
"type": "object",
"x-versionadded": "v2.37"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 201 | Created | Dataset definition created successfully. | DatasetDefResponse |
Soft delete a dataset definition based by dataset definition ID¶
Operation path: DELETE /api/v2/datasetDefinitions/{datasetDefinitionId}/
Authentication requirements: BearerAuth
Soft delete a dataset definition.
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| datasetDefinitionId | path | string | true | The ID of the dataset definition. |
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 204 | No Content | none | None |
Retrieve a dataset definition based by dataset definition ID¶
Operation path: GET /api/v2/datasetDefinitions/{datasetDefinitionId}/
Authentication requirements: BearerAuth
Retrieve a dataset definition.
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| version | query | integer | false | The version of the dataset definition information. |
| datasetDefinitionId | path | string | true | The ID of the dataset definition. |
Example responses¶
200 Response
{
"properties": {
"creatorUserId": {
"description": "The ID of the user who created the dataset definition.",
"type": "string"
},
"datasetInfo": {
"description": "Information about the dataset.",
"properties": {
"columns": {
"description": "List of the dataset column names, this field is auto-generated by the analysis job.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 1000,
"minItems": 2,
"type": "array"
},
"dataSourceId": {
"default": null,
"description": "The ID of the SQL table query and the database path, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dataStoreId": {
"default": null,
"description": "The ID of the SQL data store, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dialect": {
"description": "Source type data was retrieved from, this field is auto-generated by the analysis job.",
"enum": [
"snowflake",
"bigquery",
"databricks",
"spark",
"postgres"
],
"type": "string"
},
"estimatedSizePerRow": {
"description": "Estimated byte size per row of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"sourceSize": {
"description": "Total dataset byte size, this field is auto-generated by the analysis job.",
"type": "integer"
},
"totalRows": {
"description": "Total rows of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"version": {
"description": "The version of the dataset definition information.",
"type": "integer"
}
},
"required": [
"columns",
"dataSourceId",
"dataStoreId",
"dialect",
"estimatedSizePerRow",
"sourceSize",
"totalRows",
"version"
],
"type": "object",
"x-versionadded": "v2.37"
},
"datasetProps": {
"description": "Dataset properties.",
"properties": {
"datasetId": {
"description": "The ID of the AI Catalog dataset.",
"type": "string"
},
"datasetVersionId": {
"description": "The version ID of the AI Catalog dataset.",
"type": "string"
}
},
"required": [
"datasetId",
"datasetVersionId"
],
"type": "object",
"x-versionadded": "v2.37"
},
"dynamicDatasetProps": {
"description": "Dynamic dataset additional properties.",
"properties": {
"credentialsId": {
"default": null,
"description": "The ID of the credentials to access the data store.",
"type": [
"string",
"null"
]
}
},
"required": [
"credentialsId"
],
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the dataset definition.",
"type": "string"
},
"name": {
"description": "The name of the dataset definition.",
"type": "string"
}
},
"required": [
"creatorUserId",
"datasetInfo",
"datasetProps",
"dynamicDatasetProps",
"id",
"name"
],
"type": "object",
"x-versionadded": "v2.37"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | none | DatasetDefResponse |
Analyze a dataset definition by dataset definition ID¶
Operation path: POST /api/v2/datasetDefinitions/{datasetDefinitionId}/analyze/
Authentication requirements: BearerAuth
Analyze a dataset definition.
Body parameter¶
{
"type": "object"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| datasetDefinitionId | path | string | true | The ID of the dataset definition. |
| body | body | Empty | false | none |
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 204 | No Content | none | None |
Retrieve a list chunk definitions by dataset definition ID¶
Operation path: GET /api/v2/datasetDefinitions/{datasetDefinitionId}/chunkDefinitions/
Authentication requirements: BearerAuth
Retrieve a list chunk definitions.
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| offset | query | integer | false | This many results will be skipped. |
| limit | query | integer | false | At most this many results are returned. |
| datasetDefinitionId | path | string | true | The ID of the dataset definition. |
Example responses¶
200 Response
{
"properties": {
"count": {
"description": "Number of items returned on this page.",
"type": "integer"
},
"data": {
"description": "A list of chunk definitions.",
"items": {
"properties": {
"chunkDefinitionStats": {
"description": "Chunk definition stats. This field is auto-generated by the analysis job.",
"properties": {
"expectedChunkSize": {
"description": "Expected chunk size. this field is auto-generated by the analysis job.",
"type": "integer"
},
"numberOfRowsPerChunk": {
"description": "Number of rows per chunk. This field is auto-generated by the analysis job.",
"type": "integer"
},
"totalNumberOfChunks": {
"description": "Total rows of the chunks. This field is auto-generated by the analysis job.",
"type": "integer"
}
},
"required": [
"expectedChunkSize",
"numberOfRowsPerChunk",
"totalNumberOfChunks"
],
"type": "object",
"x-versionadded": "v2.37"
},
"chunkingStrategyType": {
"default": "rows",
"description": "The partition method.",
"enum": [
"features",
"rows"
],
"type": "string"
},
"datasetDefinitionId": {
"description": "The dataset definition ID the definition belongs.",
"type": "string"
},
"datasetDefinitionInfoVersion": {
"description": "The version of the dataset definition information.",
"type": "integer"
},
"featuresChunkDefinition": {
"description": "Feature chunk definition properties.",
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the chunk definition.",
"type": "string"
},
"isReadonly": {
"default": false,
"description": "Flag the allows or prevents updates.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": "string"
},
"partitionMethod": {
"default": "random",
"description": "The partition method.",
"enum": [
"random",
"stratified",
"date"
],
"type": "string"
},
"rowsChunkDefinition": {
"description": "Row chunk definition properties.",
"properties": {
"datetimePartitionColumn": {
"description": "Date partition column name.",
"type": [
"string",
"null"
]
},
"isDescendingOrder": {
"default": false,
"description": "The sorting order.",
"type": "boolean"
},
"orderByColumns": {
"default": [],
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"otvEarliestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvLatestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvTrainingEndDate": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationDownsamplingPct": {
"description": "Percent by which to downsample the validation data.",
"type": [
"number",
"null"
]
},
"otvValidationEndDate": {
"description": "The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationStartDate": {
"description": "The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationStartDate`, one must specify `ValidationEndDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"targetClass": {
"description": "Target Class.",
"type": [
"string",
"null"
]
},
"targetColumn": {
"description": "Target column name.",
"type": [
"string",
"null"
]
},
"userGroupColumn": {
"description": "User group column name.",
"type": [
"string",
"null"
]
}
},
"required": [
"datetimePartitionColumn",
"isDescendingOrder",
"orderByColumns",
"otvEarliestTimestamp",
"otvLatestTimestamp",
"otvTrainingEndDate",
"otvValidationDownsamplingPct",
"otvValidationEndDate",
"otvValidationStartDate",
"targetClass",
"targetColumn",
"userGroupColumn"
],
"type": "object",
"x-versionadded": "v2.37"
}
},
"required": [
"chunkDefinitionStats",
"chunkingStrategyType",
"datasetDefinitionId",
"datasetDefinitionInfoVersion",
"featuresChunkDefinition",
"id",
"isReadonly",
"name",
"partitionMethod",
"rowsChunkDefinition"
],
"type": "object",
"x-versionadded": "v2.37"
},
"maxItems": 100,
"type": "array"
},
"next": {
"description": "URL pointing to the next page (if null, there is no next page).",
"format": "uri",
"type": [
"string",
"null"
]
},
"previous": {
"description": "URL pointing to the previous page (if null, there is no previous page).",
"format": "uri",
"type": [
"string",
"null"
]
},
"totalCount": {
"description": "The total number of items across all pages.",
"type": "integer"
}
},
"required": [
"data",
"next",
"previous",
"totalCount"
],
"type": "object",
"x-versionadded": "v2.37"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | none | ChunkDefinitionPaginatedResponse |
Create a chunk definition based by dataset definition ID¶
Operation path: POST /api/v2/datasetDefinitions/{datasetDefinitionId}/chunkDefinitions/
Authentication requirements: BearerAuth
Create a chunk definition.
Body parameter¶
{
"discriminator": {
"propertyName": "partitionMethod"
},
"oneOf": [
{
"properties": {
"chunkingStrategyType": {
"default": "rows",
"description": "The partition method.",
"enum": [
"features",
"rows"
],
"type": "string"
},
"isDescendingOrder": {
"default": false,
"description": "The sorting order.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": [
"string",
"null"
]
},
"orderByColumns": {
"default": [],
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"partitionMethod": {
"description": "The partition method.",
"enum": [
"random"
],
"type": "string"
}
},
"required": [
"partitionMethod"
],
"type": "object"
},
{
"properties": {
"chunkingStrategyType": {
"default": "rows",
"description": "The partition method.",
"enum": [
"features",
"rows"
],
"type": "string"
},
"isDescendingOrder": {
"default": false,
"description": "The sorting order.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": [
"string",
"null"
]
},
"orderByColumns": {
"default": [],
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"partitionMethod": {
"description": "The partition method.",
"enum": [
"stratified"
],
"type": "string"
},
"targetClass": {
"description": "Target Class.",
"type": "string"
},
"targetColumn": {
"description": "Target column name.",
"type": "string"
}
},
"required": [
"partitionMethod",
"targetClass",
"targetColumn"
],
"type": "object"
},
{
"properties": {
"chunkingStrategyType": {
"default": "rows",
"description": "The partition method.",
"enum": [
"features",
"rows"
],
"type": "string"
},
"datetimePartitionColumn": {
"description": "Date partition column name.",
"type": "string"
},
"isDescendingOrder": {
"default": false,
"description": "The sorting order.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": [
"string",
"null"
]
},
"orderByColumns": {
"default": [],
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"otvTrainingEndDate": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationEndDate": {
"description": "The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationStartDate": {
"description": "The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationStartDate`, one must specify `ValidationEndDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"partitionMethod": {
"description": "The partition method.",
"enum": [
"date"
],
"type": "string"
}
},
"required": [
"datetimePartitionColumn",
"partitionMethod"
],
"type": "object"
}
],
"x-versionadded": "v2.37"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| datasetDefinitionId | path | string | true | The ID of the dataset definition. |
| body | body | ChunkDefinitionCreate | false | none |
Example responses¶
201 Response
{
"properties": {
"chunkDefinitionStats": {
"description": "Chunk definition stats. This field is auto-generated by the analysis job.",
"properties": {
"expectedChunkSize": {
"description": "Expected chunk size. this field is auto-generated by the analysis job.",
"type": "integer"
},
"numberOfRowsPerChunk": {
"description": "Number of rows per chunk. This field is auto-generated by the analysis job.",
"type": "integer"
},
"totalNumberOfChunks": {
"description": "Total rows of the chunks. This field is auto-generated by the analysis job.",
"type": "integer"
}
},
"required": [
"expectedChunkSize",
"numberOfRowsPerChunk",
"totalNumberOfChunks"
],
"type": "object",
"x-versionadded": "v2.37"
},
"chunkingStrategyType": {
"default": "rows",
"description": "The partition method.",
"enum": [
"features",
"rows"
],
"type": "string"
},
"datasetDefinitionId": {
"description": "The dataset definition ID the definition belongs.",
"type": "string"
},
"datasetDefinitionInfoVersion": {
"description": "The version of the dataset definition information.",
"type": "integer"
},
"featuresChunkDefinition": {
"description": "Feature chunk definition properties.",
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the chunk definition.",
"type": "string"
},
"isReadonly": {
"default": false,
"description": "Flag the allows or prevents updates.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": "string"
},
"partitionMethod": {
"default": "random",
"description": "The partition method.",
"enum": [
"random",
"stratified",
"date"
],
"type": "string"
},
"rowsChunkDefinition": {
"description": "Row chunk definition properties.",
"properties": {
"datetimePartitionColumn": {
"description": "Date partition column name.",
"type": [
"string",
"null"
]
},
"isDescendingOrder": {
"default": false,
"description": "The sorting order.",
"type": "boolean"
},
"orderByColumns": {
"default": [],
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"otvEarliestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvLatestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvTrainingEndDate": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationDownsamplingPct": {
"description": "Percent by which to downsample the validation data.",
"type": [
"number",
"null"
]
},
"otvValidationEndDate": {
"description": "The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationStartDate": {
"description": "The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationStartDate`, one must specify `ValidationEndDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"targetClass": {
"description": "Target Class.",
"type": [
"string",
"null"
]
},
"targetColumn": {
"description": "Target column name.",
"type": [
"string",
"null"
]
},
"userGroupColumn": {
"description": "User group column name.",
"type": [
"string",
"null"
]
}
},
"required": [
"datetimePartitionColumn",
"isDescendingOrder",
"orderByColumns",
"otvEarliestTimestamp",
"otvLatestTimestamp",
"otvTrainingEndDate",
"otvValidationDownsamplingPct",
"otvValidationEndDate",
"otvValidationStartDate",
"targetClass",
"targetColumn",
"userGroupColumn"
],
"type": "object",
"x-versionadded": "v2.37"
}
},
"required": [
"chunkDefinitionStats",
"chunkingStrategyType",
"datasetDefinitionId",
"datasetDefinitionInfoVersion",
"featuresChunkDefinition",
"id",
"isReadonly",
"name",
"partitionMethod",
"rowsChunkDefinition"
],
"type": "object",
"x-versionadded": "v2.37"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 201 | Created | Chunk definition created successfully. | ChunkDefinitionResponse |
Soft delete a chunk definition based by dataset definition ID¶
Operation path: DELETE /api/v2/datasetDefinitions/{datasetDefinitionId}/chunkDefinitions/{chunkDefinitionId}/
Authentication requirements: BearerAuth
Soft delete a chunk definition.
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| datasetDefinitionId | path | string | true | The ID of the dataset definition. |
| chunkDefinitionId | path | string | true | The ID of the chunk definition. |
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 204 | No Content | none | None |
Retrieve a chunk definition based by dataset definition ID¶
Operation path: GET /api/v2/datasetDefinitions/{datasetDefinitionId}/chunkDefinitions/{chunkDefinitionId}/
Authentication requirements: BearerAuth
Retrieve a chunk definition.
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| datasetDefinitionId | path | string | true | The ID of the dataset definition. |
| chunkDefinitionId | path | string | true | The ID of the chunk definition. |
Example responses¶
200 Response
{
"properties": {
"chunkDefinitionStats": {
"description": "Chunk definition stats. This field is auto-generated by the analysis job.",
"properties": {
"expectedChunkSize": {
"description": "Expected chunk size. this field is auto-generated by the analysis job.",
"type": "integer"
},
"numberOfRowsPerChunk": {
"description": "Number of rows per chunk. This field is auto-generated by the analysis job.",
"type": "integer"
},
"totalNumberOfChunks": {
"description": "Total rows of the chunks. This field is auto-generated by the analysis job.",
"type": "integer"
}
},
"required": [
"expectedChunkSize",
"numberOfRowsPerChunk",
"totalNumberOfChunks"
],
"type": "object",
"x-versionadded": "v2.37"
},
"chunkingStrategyType": {
"default": "rows",
"description": "The partition method.",
"enum": [
"features",
"rows"
],
"type": "string"
},
"datasetDefinitionId": {
"description": "The dataset definition ID the definition belongs.",
"type": "string"
},
"datasetDefinitionInfoVersion": {
"description": "The version of the dataset definition information.",
"type": "integer"
},
"featuresChunkDefinition": {
"description": "Feature chunk definition properties.",
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the chunk definition.",
"type": "string"
},
"isReadonly": {
"default": false,
"description": "Flag the allows or prevents updates.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": "string"
},
"partitionMethod": {
"default": "random",
"description": "The partition method.",
"enum": [
"random",
"stratified",
"date"
],
"type": "string"
},
"rowsChunkDefinition": {
"description": "Row chunk definition properties.",
"properties": {
"datetimePartitionColumn": {
"description": "Date partition column name.",
"type": [
"string",
"null"
]
},
"isDescendingOrder": {
"default": false,
"description": "The sorting order.",
"type": "boolean"
},
"orderByColumns": {
"default": [],
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"otvEarliestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvLatestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvTrainingEndDate": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationDownsamplingPct": {
"description": "Percent by which to downsample the validation data.",
"type": [
"number",
"null"
]
},
"otvValidationEndDate": {
"description": "The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationStartDate": {
"description": "The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationStartDate`, one must specify `ValidationEndDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"targetClass": {
"description": "Target Class.",
"type": [
"string",
"null"
]
},
"targetColumn": {
"description": "Target column name.",
"type": [
"string",
"null"
]
},
"userGroupColumn": {
"description": "User group column name.",
"type": [
"string",
"null"
]
}
},
"required": [
"datetimePartitionColumn",
"isDescendingOrder",
"orderByColumns",
"otvEarliestTimestamp",
"otvLatestTimestamp",
"otvTrainingEndDate",
"otvValidationDownsamplingPct",
"otvValidationEndDate",
"otvValidationStartDate",
"targetClass",
"targetColumn",
"userGroupColumn"
],
"type": "object",
"x-versionadded": "v2.37"
}
},
"required": [
"chunkDefinitionStats",
"chunkingStrategyType",
"datasetDefinitionId",
"datasetDefinitionInfoVersion",
"featuresChunkDefinition",
"id",
"isReadonly",
"name",
"partitionMethod",
"rowsChunkDefinition"
],
"type": "object",
"x-versionadded": "v2.37"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | none | ChunkDefinitionResponse |
Update a chunk definition based by dataset definition ID¶
Operation path: PATCH /api/v2/datasetDefinitions/{datasetDefinitionId}/chunkDefinitions/{chunkDefinitionId}/
Authentication requirements: BearerAuth
Update a chunk definition.
Body parameter¶
{
"properties": {
"operations": {
"description": "Operations to perform on the update chunk definition.",
"properties": {
"forceUpdate": {
"default": false,
"description": "Force update the chunk definition. If set to true, the analysis will be reset.",
"type": "boolean"
}
},
"type": "object",
"x-versionadded": "v2.37"
},
"updates": {
"description": "Fields to be updated in the chunk definition.",
"properties": {
"datetimePartitionColumn": {
"description": "Date partition column name.",
"type": "string"
},
"isDescendingOrder": {
"description": "The sorting order.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": "string"
},
"orderByColumns": {
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"otvTrainingEndDate": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": "string"
},
"otvValidationEndDate": {
"description": "The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": "string"
},
"otvValidationStartDate": {
"description": "The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationStartDate`, one must specify `ValidationEndDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": "string"
},
"targetClass": {
"description": "Target Class.",
"type": "string"
},
"targetColumn": {
"description": "Target column name.",
"type": "string"
},
"userGroupColumn": {
"description": "User group column name.",
"type": "string"
}
},
"type": "object",
"x-versionadded": "v2.37"
}
},
"required": [
"updates"
],
"type": "object",
"x-versionadded": "v2.37"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| datasetDefinitionId | path | string | true | The ID of the dataset definition. |
| chunkDefinitionId | path | string | true | The ID of the chunk definition. |
| body | body | ChunkDefinitionRowsUpdate | false | none |
Example responses¶
200 Response
{
"properties": {
"chunkDefinitionStats": {
"description": "Chunk definition stats. This field is auto-generated by the analysis job.",
"properties": {
"expectedChunkSize": {
"description": "Expected chunk size. this field is auto-generated by the analysis job.",
"type": "integer"
},
"numberOfRowsPerChunk": {
"description": "Number of rows per chunk. This field is auto-generated by the analysis job.",
"type": "integer"
},
"totalNumberOfChunks": {
"description": "Total rows of the chunks. This field is auto-generated by the analysis job.",
"type": "integer"
}
},
"required": [
"expectedChunkSize",
"numberOfRowsPerChunk",
"totalNumberOfChunks"
],
"type": "object",
"x-versionadded": "v2.37"
},
"chunkingStrategyType": {
"default": "rows",
"description": "The partition method.",
"enum": [
"features",
"rows"
],
"type": "string"
},
"datasetDefinitionId": {
"description": "The dataset definition ID the definition belongs.",
"type": "string"
},
"datasetDefinitionInfoVersion": {
"description": "The version of the dataset definition information.",
"type": "integer"
},
"featuresChunkDefinition": {
"description": "Feature chunk definition properties.",
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the chunk definition.",
"type": "string"
},
"isReadonly": {
"default": false,
"description": "Flag the allows or prevents updates.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": "string"
},
"partitionMethod": {
"default": "random",
"description": "The partition method.",
"enum": [
"random",
"stratified",
"date"
],
"type": "string"
},
"rowsChunkDefinition": {
"description": "Row chunk definition properties.",
"properties": {
"datetimePartitionColumn": {
"description": "Date partition column name.",
"type": [
"string",
"null"
]
},
"isDescendingOrder": {
"default": false,
"description": "The sorting order.",
"type": "boolean"
},
"orderByColumns": {
"default": [],
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"otvEarliestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvLatestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvTrainingEndDate": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationDownsamplingPct": {
"description": "Percent by which to downsample the validation data.",
"type": [
"number",
"null"
]
},
"otvValidationEndDate": {
"description": "The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationStartDate": {
"description": "The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationStartDate`, one must specify `ValidationEndDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"targetClass": {
"description": "Target Class.",
"type": [
"string",
"null"
]
},
"targetColumn": {
"description": "Target column name.",
"type": [
"string",
"null"
]
},
"userGroupColumn": {
"description": "User group column name.",
"type": [
"string",
"null"
]
}
},
"required": [
"datetimePartitionColumn",
"isDescendingOrder",
"orderByColumns",
"otvEarliestTimestamp",
"otvLatestTimestamp",
"otvTrainingEndDate",
"otvValidationDownsamplingPct",
"otvValidationEndDate",
"otvValidationStartDate",
"targetClass",
"targetColumn",
"userGroupColumn"
],
"type": "object",
"x-versionadded": "v2.37"
}
},
"required": [
"chunkDefinitionStats",
"chunkingStrategyType",
"datasetDefinitionId",
"datasetDefinitionInfoVersion",
"featuresChunkDefinition",
"id",
"isReadonly",
"name",
"partitionMethod",
"rowsChunkDefinition"
],
"type": "object",
"x-versionadded": "v2.37"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | Chunk definition updated successfully. | ChunkDefinitionResponse |
Analyze a chunk definition by dataset definition ID¶
Operation path: POST /api/v2/datasetDefinitions/{datasetDefinitionId}/chunkDefinitions/{chunkDefinitionId}/analyze/
Authentication requirements: BearerAuth
Analyze a chunk definition.
Body parameter¶
{
"type": "object"
}
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| datasetDefinitionId | path | string | true | The ID of the dataset definition. |
| chunkDefinitionId | path | string | true | The ID of the chunk definition. |
| body | body | Empty | false | none |
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 204 | No Content | none | None |
List all dataset definition versions by dataset definition ID¶
Operation path: GET /api/v2/datasetDefinitions/{datasetDefinitionId}/versions/
Authentication requirements: BearerAuth
List all dataset definition versions.
Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
| offset | query | integer | false | This many results will be skipped. |
| limit | query | integer | false | At most this many results are returned. |
| datasetDefinitionId | path | string | true | The ID of the dataset definition. |
Example responses¶
200 Response
{
"properties": {
"count": {
"description": "Number of items returned on this page.",
"type": "integer"
},
"data": {
"description": "A list of the dataset definition versions.",
"items": {
"properties": {
"datasetInfo": {
"description": "Information about the dataset.",
"properties": {
"columns": {
"description": "List of the dataset column names, this field is auto-generated by the analysis job.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 1000,
"minItems": 2,
"type": "array"
},
"dataSourceId": {
"default": null,
"description": "The ID of the SQL table query and the database path, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dataStoreId": {
"default": null,
"description": "The ID of the SQL data store, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dialect": {
"description": "Source type data was retrieved from, this field is auto-generated by the analysis job.",
"enum": [
"snowflake",
"bigquery",
"databricks",
"spark",
"postgres"
],
"type": "string"
},
"estimatedSizePerRow": {
"description": "Estimated byte size per row of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"sourceSize": {
"description": "Total dataset byte size, this field is auto-generated by the analysis job.",
"type": "integer"
},
"totalRows": {
"description": "Total rows of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"version": {
"description": "The version of the dataset definition information.",
"type": "integer"
}
},
"required": [
"columns",
"dataSourceId",
"dataStoreId",
"dialect",
"estimatedSizePerRow",
"sourceSize",
"totalRows",
"version"
],
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the dataset definition version.",
"type": "string"
}
},
"required": [
"datasetInfo",
"id"
],
"type": "object",
"x-versionadded": "v2.37"
},
"maxItems": 100,
"type": "array"
},
"next": {
"description": "URL pointing to the next page (if null, there is no next page).",
"format": "uri",
"type": [
"string",
"null"
]
},
"previous": {
"description": "URL pointing to the previous page (if null, there is no previous page).",
"format": "uri",
"type": [
"string",
"null"
]
},
"totalCount": {
"description": "The total number of items across all pages.",
"type": "integer"
}
},
"required": [
"data",
"next",
"previous",
"totalCount"
],
"type": "object",
"x-versionadded": "v2.37"
}
Responses¶
| Status | Meaning | Description | Schema |
|---|---|---|---|
| 200 | OK | none | DatasetDefVersionPaginatedResponse |
Schemas¶
ChunkDefinitionCreate
{
"discriminator": {
"propertyName": "partitionMethod"
},
"oneOf": [
{
"properties": {
"chunkingStrategyType": {
"default": "rows",
"description": "The partition method.",
"enum": [
"features",
"rows"
],
"type": "string"
},
"isDescendingOrder": {
"default": false,
"description": "The sorting order.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": [
"string",
"null"
]
},
"orderByColumns": {
"default": [],
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"partitionMethod": {
"description": "The partition method.",
"enum": [
"random"
],
"type": "string"
}
},
"required": [
"partitionMethod"
],
"type": "object"
},
{
"properties": {
"chunkingStrategyType": {
"default": "rows",
"description": "The partition method.",
"enum": [
"features",
"rows"
],
"type": "string"
},
"isDescendingOrder": {
"default": false,
"description": "The sorting order.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": [
"string",
"null"
]
},
"orderByColumns": {
"default": [],
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"partitionMethod": {
"description": "The partition method.",
"enum": [
"stratified"
],
"type": "string"
},
"targetClass": {
"description": "Target Class.",
"type": "string"
},
"targetColumn": {
"description": "Target column name.",
"type": "string"
}
},
"required": [
"partitionMethod",
"targetClass",
"targetColumn"
],
"type": "object"
},
{
"properties": {
"chunkingStrategyType": {
"default": "rows",
"description": "The partition method.",
"enum": [
"features",
"rows"
],
"type": "string"
},
"datetimePartitionColumn": {
"description": "Date partition column name.",
"type": "string"
},
"isDescendingOrder": {
"default": false,
"description": "The sorting order.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": [
"string",
"null"
]
},
"orderByColumns": {
"default": [],
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"otvTrainingEndDate": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationEndDate": {
"description": "The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationStartDate": {
"description": "The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationStartDate`, one must specify `ValidationEndDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"partitionMethod": {
"description": "The partition method.",
"enum": [
"date"
],
"type": "string"
}
},
"required": [
"datetimePartitionColumn",
"partitionMethod"
],
"type": "object"
}
],
"x-versionadded": "v2.37"
}
Properties¶
oneOf
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| anonymous | object | false | none | |
| » chunkingStrategyType | string | false | The partition method. | |
| » isDescendingOrder | boolean | false | The sorting order. | |
| » name | string,null | false | The name of the chunk definition. | |
| » orderByColumns | [string] | false | maxItems: 10 |
List of the sorting column names. |
| » partitionMethod | string | true | The partition method. |
xor
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| anonymous | object | false | none | |
| » chunkingStrategyType | string | false | The partition method. | |
| » isDescendingOrder | boolean | false | The sorting order. | |
| » name | string,null | false | The name of the chunk definition. | |
| » orderByColumns | [string] | false | maxItems: 10 |
List of the sorting column names. |
| » partitionMethod | string | true | The partition method. | |
| » targetClass | string | true | Target Class. | |
| » targetColumn | string | true | Target column name. |
xor
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| anonymous | object | false | none | |
| » chunkingStrategyType | string | false | The partition method. | |
| » datetimePartitionColumn | string | true | Date partition column name. | |
| » isDescendingOrder | boolean | false | The sorting order. | |
| » name | string,null | false | The name of the chunk definition. | |
| » orderByColumns | [string] | false | maxItems: 10 |
List of the sorting column names. |
| » otvTrainingEndDate | string,null(date-time) | false | The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying ValidationEndDate, one must specify ValidationStartDate. This attribute cannot be patched for non-OTV incremental learning projects. |
|
| » otvValidationEndDate | string,null(date-time) | false | The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying ValidationEndDate, one must specify ValidationStartDate. This attribute cannot be patched for non-OTV incremental learning projects. |
|
| » otvValidationStartDate | string,null(date-time) | false | The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying ValidationStartDate, one must specify ValidationEndDate. This attribute cannot be patched for non-OTV incremental learning projects. |
|
| » partitionMethod | string | true | The partition method. |
Enumerated Values¶
| Property | Value |
|---|---|
| chunkingStrategyType | [features, rows] |
| partitionMethod | random |
| chunkingStrategyType | [features, rows] |
| partitionMethod | stratified |
| chunkingStrategyType | [features, rows] |
| partitionMethod | date |
ChunkDefinitionPaginatedResponse
{
"properties": {
"count": {
"description": "Number of items returned on this page.",
"type": "integer"
},
"data": {
"description": "A list of chunk definitions.",
"items": {
"properties": {
"chunkDefinitionStats": {
"description": "Chunk definition stats. This field is auto-generated by the analysis job.",
"properties": {
"expectedChunkSize": {
"description": "Expected chunk size. this field is auto-generated by the analysis job.",
"type": "integer"
},
"numberOfRowsPerChunk": {
"description": "Number of rows per chunk. This field is auto-generated by the analysis job.",
"type": "integer"
},
"totalNumberOfChunks": {
"description": "Total rows of the chunks. This field is auto-generated by the analysis job.",
"type": "integer"
}
},
"required": [
"expectedChunkSize",
"numberOfRowsPerChunk",
"totalNumberOfChunks"
],
"type": "object",
"x-versionadded": "v2.37"
},
"chunkingStrategyType": {
"default": "rows",
"description": "The partition method.",
"enum": [
"features",
"rows"
],
"type": "string"
},
"datasetDefinitionId": {
"description": "The dataset definition ID the definition belongs.",
"type": "string"
},
"datasetDefinitionInfoVersion": {
"description": "The version of the dataset definition information.",
"type": "integer"
},
"featuresChunkDefinition": {
"description": "Feature chunk definition properties.",
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the chunk definition.",
"type": "string"
},
"isReadonly": {
"default": false,
"description": "Flag the allows or prevents updates.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": "string"
},
"partitionMethod": {
"default": "random",
"description": "The partition method.",
"enum": [
"random",
"stratified",
"date"
],
"type": "string"
},
"rowsChunkDefinition": {
"description": "Row chunk definition properties.",
"properties": {
"datetimePartitionColumn": {
"description": "Date partition column name.",
"type": [
"string",
"null"
]
},
"isDescendingOrder": {
"default": false,
"description": "The sorting order.",
"type": "boolean"
},
"orderByColumns": {
"default": [],
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"otvEarliestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvLatestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvTrainingEndDate": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationDownsamplingPct": {
"description": "Percent by which to downsample the validation data.",
"type": [
"number",
"null"
]
},
"otvValidationEndDate": {
"description": "The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationStartDate": {
"description": "The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationStartDate`, one must specify `ValidationEndDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"targetClass": {
"description": "Target Class.",
"type": [
"string",
"null"
]
},
"targetColumn": {
"description": "Target column name.",
"type": [
"string",
"null"
]
},
"userGroupColumn": {
"description": "User group column name.",
"type": [
"string",
"null"
]
}
},
"required": [
"datetimePartitionColumn",
"isDescendingOrder",
"orderByColumns",
"otvEarliestTimestamp",
"otvLatestTimestamp",
"otvTrainingEndDate",
"otvValidationDownsamplingPct",
"otvValidationEndDate",
"otvValidationStartDate",
"targetClass",
"targetColumn",
"userGroupColumn"
],
"type": "object",
"x-versionadded": "v2.37"
}
},
"required": [
"chunkDefinitionStats",
"chunkingStrategyType",
"datasetDefinitionId",
"datasetDefinitionInfoVersion",
"featuresChunkDefinition",
"id",
"isReadonly",
"name",
"partitionMethod",
"rowsChunkDefinition"
],
"type": "object",
"x-versionadded": "v2.37"
},
"maxItems": 100,
"type": "array"
},
"next": {
"description": "URL pointing to the next page (if null, there is no next page).",
"format": "uri",
"type": [
"string",
"null"
]
},
"previous": {
"description": "URL pointing to the previous page (if null, there is no previous page).",
"format": "uri",
"type": [
"string",
"null"
]
},
"totalCount": {
"description": "The total number of items across all pages.",
"type": "integer"
}
},
"required": [
"data",
"next",
"previous",
"totalCount"
],
"type": "object",
"x-versionadded": "v2.37"
}
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | false | Number of items returned on this page. | |
| data | [ChunkDefinitionResponse] | true | maxItems: 100 |
A list of chunk definitions. |
| next | string,null(uri) | true | URL pointing to the next page (if null, there is no next page). | |
| previous | string,null(uri) | true | URL pointing to the previous page (if null, there is no previous page). | |
| totalCount | integer | true | The total number of items across all pages. |
ChunkDefinitionResponse
{
"properties": {
"chunkDefinitionStats": {
"description": "Chunk definition stats. This field is auto-generated by the analysis job.",
"properties": {
"expectedChunkSize": {
"description": "Expected chunk size. this field is auto-generated by the analysis job.",
"type": "integer"
},
"numberOfRowsPerChunk": {
"description": "Number of rows per chunk. This field is auto-generated by the analysis job.",
"type": "integer"
},
"totalNumberOfChunks": {
"description": "Total rows of the chunks. This field is auto-generated by the analysis job.",
"type": "integer"
}
},
"required": [
"expectedChunkSize",
"numberOfRowsPerChunk",
"totalNumberOfChunks"
],
"type": "object",
"x-versionadded": "v2.37"
},
"chunkingStrategyType": {
"default": "rows",
"description": "The partition method.",
"enum": [
"features",
"rows"
],
"type": "string"
},
"datasetDefinitionId": {
"description": "The dataset definition ID the definition belongs.",
"type": "string"
},
"datasetDefinitionInfoVersion": {
"description": "The version of the dataset definition information.",
"type": "integer"
},
"featuresChunkDefinition": {
"description": "Feature chunk definition properties.",
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the chunk definition.",
"type": "string"
},
"isReadonly": {
"default": false,
"description": "Flag the allows or prevents updates.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": "string"
},
"partitionMethod": {
"default": "random",
"description": "The partition method.",
"enum": [
"random",
"stratified",
"date"
],
"type": "string"
},
"rowsChunkDefinition": {
"description": "Row chunk definition properties.",
"properties": {
"datetimePartitionColumn": {
"description": "Date partition column name.",
"type": [
"string",
"null"
]
},
"isDescendingOrder": {
"default": false,
"description": "The sorting order.",
"type": "boolean"
},
"orderByColumns": {
"default": [],
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"otvEarliestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvLatestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvTrainingEndDate": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationDownsamplingPct": {
"description": "Percent by which to downsample the validation data.",
"type": [
"number",
"null"
]
},
"otvValidationEndDate": {
"description": "The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationStartDate": {
"description": "The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationStartDate`, one must specify `ValidationEndDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"targetClass": {
"description": "Target Class.",
"type": [
"string",
"null"
]
},
"targetColumn": {
"description": "Target column name.",
"type": [
"string",
"null"
]
},
"userGroupColumn": {
"description": "User group column name.",
"type": [
"string",
"null"
]
}
},
"required": [
"datetimePartitionColumn",
"isDescendingOrder",
"orderByColumns",
"otvEarliestTimestamp",
"otvLatestTimestamp",
"otvTrainingEndDate",
"otvValidationDownsamplingPct",
"otvValidationEndDate",
"otvValidationStartDate",
"targetClass",
"targetColumn",
"userGroupColumn"
],
"type": "object",
"x-versionadded": "v2.37"
}
},
"required": [
"chunkDefinitionStats",
"chunkingStrategyType",
"datasetDefinitionId",
"datasetDefinitionInfoVersion",
"featuresChunkDefinition",
"id",
"isReadonly",
"name",
"partitionMethod",
"rowsChunkDefinition"
],
"type": "object",
"x-versionadded": "v2.37"
}
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| chunkDefinitionStats | ChunkDefinitionStatsResponse | true | Chunk definition stats. This field is auto-generated by the analysis job. | |
| chunkingStrategyType | string | true | The partition method. | |
| datasetDefinitionId | string | true | The dataset definition ID the definition belongs. | |
| datasetDefinitionInfoVersion | integer | true | The version of the dataset definition information. | |
| featuresChunkDefinition | FeaturesChunkDefinitionResponse | true | Feature chunk definition properties. | |
| id | string | true | The ID of the chunk definition. | |
| isReadonly | boolean | true | Flag the allows or prevents updates. | |
| name | string | true | The name of the chunk definition. | |
| partitionMethod | string | true | The partition method. | |
| rowsChunkDefinition | RowsChunkDefinitionResponse | true | Row chunk definition properties. |
Enumerated Values¶
| Property | Value |
|---|---|
| chunkingStrategyType | [features, rows] |
| partitionMethod | [random, stratified, date] |
ChunkDefinitionRowsUpdate
{
"properties": {
"operations": {
"description": "Operations to perform on the update chunk definition.",
"properties": {
"forceUpdate": {
"default": false,
"description": "Force update the chunk definition. If set to true, the analysis will be reset.",
"type": "boolean"
}
},
"type": "object",
"x-versionadded": "v2.37"
},
"updates": {
"description": "Fields to be updated in the chunk definition.",
"properties": {
"datetimePartitionColumn": {
"description": "Date partition column name.",
"type": "string"
},
"isDescendingOrder": {
"description": "The sorting order.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": "string"
},
"orderByColumns": {
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"otvTrainingEndDate": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": "string"
},
"otvValidationEndDate": {
"description": "The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": "string"
},
"otvValidationStartDate": {
"description": "The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationStartDate`, one must specify `ValidationEndDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": "string"
},
"targetClass": {
"description": "Target Class.",
"type": "string"
},
"targetColumn": {
"description": "Target column name.",
"type": "string"
},
"userGroupColumn": {
"description": "User group column name.",
"type": "string"
}
},
"type": "object",
"x-versionadded": "v2.37"
}
},
"required": [
"updates"
],
"type": "object",
"x-versionadded": "v2.37"
}
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| operations | ChunkDefinitionUpdateOperation | false | Operations to perform on the update chunk definition. | |
| updates | ChunkDefinitionRowsUpdateFields | true | Fields to be updated in the chunk definition. |
ChunkDefinitionRowsUpdateFields
{
"description": "Fields to be updated in the chunk definition.",
"properties": {
"datetimePartitionColumn": {
"description": "Date partition column name.",
"type": "string"
},
"isDescendingOrder": {
"description": "The sorting order.",
"type": "boolean"
},
"name": {
"description": "The name of the chunk definition.",
"type": "string"
},
"orderByColumns": {
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"otvTrainingEndDate": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": "string"
},
"otvValidationEndDate": {
"description": "The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": "string"
},
"otvValidationStartDate": {
"description": "The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationStartDate`, one must specify `ValidationEndDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": "string"
},
"targetClass": {
"description": "Target Class.",
"type": "string"
},
"targetColumn": {
"description": "Target column name.",
"type": "string"
},
"userGroupColumn": {
"description": "User group column name.",
"type": "string"
}
},
"type": "object",
"x-versionadded": "v2.37"
}
Fields to be updated in the chunk definition.
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datetimePartitionColumn | string | false | Date partition column name. | |
| isDescendingOrder | boolean | false | The sorting order. | |
| name | string | false | The name of the chunk definition. | |
| orderByColumns | [string] | false | maxItems: 10 |
List of the sorting column names. |
| otvTrainingEndDate | string(date-time) | false | The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying ValidationEndDate, one must specify ValidationStartDate. This attribute cannot be patched for non-OTV incremental learning projects. |
|
| otvValidationEndDate | string(date-time) | false | The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying ValidationEndDate, one must specify ValidationStartDate. This attribute cannot be patched for non-OTV incremental learning projects. |
|
| otvValidationStartDate | string(date-time) | false | The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying ValidationStartDate, one must specify ValidationEndDate. This attribute cannot be patched for non-OTV incremental learning projects. |
|
| targetClass | string | false | Target Class. | |
| targetColumn | string | false | Target column name. | |
| userGroupColumn | string | false | User group column name. |
ChunkDefinitionStatsResponse
{
"description": "Chunk definition stats. This field is auto-generated by the analysis job.",
"properties": {
"expectedChunkSize": {
"description": "Expected chunk size. this field is auto-generated by the analysis job.",
"type": "integer"
},
"numberOfRowsPerChunk": {
"description": "Number of rows per chunk. This field is auto-generated by the analysis job.",
"type": "integer"
},
"totalNumberOfChunks": {
"description": "Total rows of the chunks. This field is auto-generated by the analysis job.",
"type": "integer"
}
},
"required": [
"expectedChunkSize",
"numberOfRowsPerChunk",
"totalNumberOfChunks"
],
"type": "object",
"x-versionadded": "v2.37"
}
Chunk definition stats. This field is auto-generated by the analysis job.
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| expectedChunkSize | integer | true | Expected chunk size. this field is auto-generated by the analysis job. | |
| numberOfRowsPerChunk | integer | true | Number of rows per chunk. This field is auto-generated by the analysis job. | |
| totalNumberOfChunks | integer | true | Total rows of the chunks. This field is auto-generated by the analysis job. |
ChunkDefinitionUpdateOperation
{
"description": "Operations to perform on the update chunk definition.",
"properties": {
"forceUpdate": {
"default": false,
"description": "Force update the chunk definition. If set to true, the analysis will be reset.",
"type": "boolean"
}
},
"type": "object",
"x-versionadded": "v2.37"
}
Operations to perform on the update chunk definition.
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| forceUpdate | boolean | false | Force update the chunk definition. If set to true, the analysis will be reset. |
DatasetDefCreate
{
"properties": {
"credentialsId": {
"description": "The ID of the credentials to access the data store.",
"type": "string"
},
"datasetId": {
"description": "The ID of the AI Catalog dataset.",
"type": "string"
},
"datasetVersionId": {
"default": null,
"description": "The version ID of the AI Catalog dataset.",
"type": [
"string",
"null"
]
},
"name": {
"default": null,
"description": "The name of the dataset definition.",
"type": [
"string",
"null"
]
}
},
"required": [
"datasetId"
],
"type": "object",
"x-versionadded": "v2.37"
}
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| credentialsId | string | false | The ID of the credentials to access the data store. | |
| datasetId | string | true | The ID of the AI Catalog dataset. | |
| datasetVersionId | string,null | false | The version ID of the AI Catalog dataset. | |
| name | string,null | false | The name of the dataset definition. |
DatasetDefPaginatedResponse
{
"properties": {
"count": {
"description": "Number of items returned on this page.",
"type": "integer"
},
"data": {
"description": "A list of user-defined dataset definitions.",
"items": {
"properties": {
"creatorUserId": {
"description": "The ID of the user who created the dataset definition.",
"type": "string"
},
"datasetInfo": {
"description": "Information about the dataset.",
"properties": {
"columns": {
"description": "List of the dataset column names, this field is auto-generated by the analysis job.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 1000,
"minItems": 2,
"type": "array"
},
"dataSourceId": {
"default": null,
"description": "The ID of the SQL table query and the database path, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dataStoreId": {
"default": null,
"description": "The ID of the SQL data store, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dialect": {
"description": "Source type data was retrieved from, this field is auto-generated by the analysis job.",
"enum": [
"snowflake",
"bigquery",
"databricks",
"spark",
"postgres"
],
"type": "string"
},
"estimatedSizePerRow": {
"description": "Estimated byte size per row of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"sourceSize": {
"description": "Total dataset byte size, this field is auto-generated by the analysis job.",
"type": "integer"
},
"totalRows": {
"description": "Total rows of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"version": {
"description": "The version of the dataset definition information.",
"type": "integer"
}
},
"required": [
"columns",
"dataSourceId",
"dataStoreId",
"dialect",
"estimatedSizePerRow",
"sourceSize",
"totalRows",
"version"
],
"type": "object",
"x-versionadded": "v2.37"
},
"datasetProps": {
"description": "Dataset properties.",
"properties": {
"datasetId": {
"description": "The ID of the AI Catalog dataset.",
"type": "string"
},
"datasetVersionId": {
"description": "The version ID of the AI Catalog dataset.",
"type": "string"
}
},
"required": [
"datasetId",
"datasetVersionId"
],
"type": "object",
"x-versionadded": "v2.37"
},
"dynamicDatasetProps": {
"description": "Dynamic dataset additional properties.",
"properties": {
"credentialsId": {
"default": null,
"description": "The ID of the credentials to access the data store.",
"type": [
"string",
"null"
]
}
},
"required": [
"credentialsId"
],
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the dataset definition.",
"type": "string"
},
"name": {
"description": "The name of the dataset definition.",
"type": "string"
}
},
"required": [
"creatorUserId",
"datasetInfo",
"datasetProps",
"dynamicDatasetProps",
"id",
"name"
],
"type": "object",
"x-versionadded": "v2.37"
},
"maxItems": 100,
"type": "array"
},
"next": {
"description": "URL pointing to the next page (if null, there is no next page).",
"format": "uri",
"type": [
"string",
"null"
]
},
"previous": {
"description": "URL pointing to the previous page (if null, there is no previous page).",
"format": "uri",
"type": [
"string",
"null"
]
},
"totalCount": {
"description": "The total number of items across all pages.",
"type": "integer"
}
},
"required": [
"data",
"next",
"previous",
"totalCount"
],
"type": "object",
"x-versionadded": "v2.37"
}
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | false | Number of items returned on this page. | |
| data | [DatasetDefResponse] | true | maxItems: 100 |
A list of user-defined dataset definitions. |
| next | string,null(uri) | true | URL pointing to the next page (if null, there is no next page). | |
| previous | string,null(uri) | true | URL pointing to the previous page (if null, there is no previous page). | |
| totalCount | integer | true | The total number of items across all pages. |
DatasetDefResponse
{
"properties": {
"creatorUserId": {
"description": "The ID of the user who created the dataset definition.",
"type": "string"
},
"datasetInfo": {
"description": "Information about the dataset.",
"properties": {
"columns": {
"description": "List of the dataset column names, this field is auto-generated by the analysis job.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 1000,
"minItems": 2,
"type": "array"
},
"dataSourceId": {
"default": null,
"description": "The ID of the SQL table query and the database path, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dataStoreId": {
"default": null,
"description": "The ID of the SQL data store, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dialect": {
"description": "Source type data was retrieved from, this field is auto-generated by the analysis job.",
"enum": [
"snowflake",
"bigquery",
"databricks",
"spark",
"postgres"
],
"type": "string"
},
"estimatedSizePerRow": {
"description": "Estimated byte size per row of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"sourceSize": {
"description": "Total dataset byte size, this field is auto-generated by the analysis job.",
"type": "integer"
},
"totalRows": {
"description": "Total rows of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"version": {
"description": "The version of the dataset definition information.",
"type": "integer"
}
},
"required": [
"columns",
"dataSourceId",
"dataStoreId",
"dialect",
"estimatedSizePerRow",
"sourceSize",
"totalRows",
"version"
],
"type": "object",
"x-versionadded": "v2.37"
},
"datasetProps": {
"description": "Dataset properties.",
"properties": {
"datasetId": {
"description": "The ID of the AI Catalog dataset.",
"type": "string"
},
"datasetVersionId": {
"description": "The version ID of the AI Catalog dataset.",
"type": "string"
}
},
"required": [
"datasetId",
"datasetVersionId"
],
"type": "object",
"x-versionadded": "v2.37"
},
"dynamicDatasetProps": {
"description": "Dynamic dataset additional properties.",
"properties": {
"credentialsId": {
"default": null,
"description": "The ID of the credentials to access the data store.",
"type": [
"string",
"null"
]
}
},
"required": [
"credentialsId"
],
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the dataset definition.",
"type": "string"
},
"name": {
"description": "The name of the dataset definition.",
"type": "string"
}
},
"required": [
"creatorUserId",
"datasetInfo",
"datasetProps",
"dynamicDatasetProps",
"id",
"name"
],
"type": "object",
"x-versionadded": "v2.37"
}
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| creatorUserId | string | true | The ID of the user who created the dataset definition. | |
| datasetInfo | DatasetInformationResponse | true | Information about the dataset. | |
| datasetProps | DatasetPropsResponse | true | Dataset properties. | |
| dynamicDatasetProps | DynamicDatasetPropsResponse | true | Dynamic dataset additional properties. | |
| id | string | true | The ID of the dataset definition. | |
| name | string | true | The name of the dataset definition. |
DatasetDefVersionPaginatedResponse
{
"properties": {
"count": {
"description": "Number of items returned on this page.",
"type": "integer"
},
"data": {
"description": "A list of the dataset definition versions.",
"items": {
"properties": {
"datasetInfo": {
"description": "Information about the dataset.",
"properties": {
"columns": {
"description": "List of the dataset column names, this field is auto-generated by the analysis job.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 1000,
"minItems": 2,
"type": "array"
},
"dataSourceId": {
"default": null,
"description": "The ID of the SQL table query and the database path, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dataStoreId": {
"default": null,
"description": "The ID of the SQL data store, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dialect": {
"description": "Source type data was retrieved from, this field is auto-generated by the analysis job.",
"enum": [
"snowflake",
"bigquery",
"databricks",
"spark",
"postgres"
],
"type": "string"
},
"estimatedSizePerRow": {
"description": "Estimated byte size per row of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"sourceSize": {
"description": "Total dataset byte size, this field is auto-generated by the analysis job.",
"type": "integer"
},
"totalRows": {
"description": "Total rows of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"version": {
"description": "The version of the dataset definition information.",
"type": "integer"
}
},
"required": [
"columns",
"dataSourceId",
"dataStoreId",
"dialect",
"estimatedSizePerRow",
"sourceSize",
"totalRows",
"version"
],
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the dataset definition version.",
"type": "string"
}
},
"required": [
"datasetInfo",
"id"
],
"type": "object",
"x-versionadded": "v2.37"
},
"maxItems": 100,
"type": "array"
},
"next": {
"description": "URL pointing to the next page (if null, there is no next page).",
"format": "uri",
"type": [
"string",
"null"
]
},
"previous": {
"description": "URL pointing to the previous page (if null, there is no previous page).",
"format": "uri",
"type": [
"string",
"null"
]
},
"totalCount": {
"description": "The total number of items across all pages.",
"type": "integer"
}
},
"required": [
"data",
"next",
"previous",
"totalCount"
],
"type": "object",
"x-versionadded": "v2.37"
}
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| count | integer | false | Number of items returned on this page. | |
| data | [DatasetDefVersionResponse] | true | maxItems: 100 |
A list of the dataset definition versions. |
| next | string,null(uri) | true | URL pointing to the next page (if null, there is no next page). | |
| previous | string,null(uri) | true | URL pointing to the previous page (if null, there is no previous page). | |
| totalCount | integer | true | The total number of items across all pages. |
DatasetDefVersionResponse
{
"properties": {
"datasetInfo": {
"description": "Information about the dataset.",
"properties": {
"columns": {
"description": "List of the dataset column names, this field is auto-generated by the analysis job.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 1000,
"minItems": 2,
"type": "array"
},
"dataSourceId": {
"default": null,
"description": "The ID of the SQL table query and the database path, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dataStoreId": {
"default": null,
"description": "The ID of the SQL data store, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dialect": {
"description": "Source type data was retrieved from, this field is auto-generated by the analysis job.",
"enum": [
"snowflake",
"bigquery",
"databricks",
"spark",
"postgres"
],
"type": "string"
},
"estimatedSizePerRow": {
"description": "Estimated byte size per row of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"sourceSize": {
"description": "Total dataset byte size, this field is auto-generated by the analysis job.",
"type": "integer"
},
"totalRows": {
"description": "Total rows of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"version": {
"description": "The version of the dataset definition information.",
"type": "integer"
}
},
"required": [
"columns",
"dataSourceId",
"dataStoreId",
"dialect",
"estimatedSizePerRow",
"sourceSize",
"totalRows",
"version"
],
"type": "object",
"x-versionadded": "v2.37"
},
"id": {
"description": "The ID of the dataset definition version.",
"type": "string"
}
},
"required": [
"datasetInfo",
"id"
],
"type": "object",
"x-versionadded": "v2.37"
}
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetInfo | DatasetInformationResponse | true | Information about the dataset. | |
| id | string | true | The ID of the dataset definition version. |
DatasetInformationResponse
{
"description": "Information about the dataset.",
"properties": {
"columns": {
"description": "List of the dataset column names, this field is auto-generated by the analysis job.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 1000,
"minItems": 2,
"type": "array"
},
"dataSourceId": {
"default": null,
"description": "The ID of the SQL table query and the database path, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dataStoreId": {
"default": null,
"description": "The ID of the SQL data store, this field is auto-generated by the analysis job.",
"type": [
"string",
"null"
]
},
"dialect": {
"description": "Source type data was retrieved from, this field is auto-generated by the analysis job.",
"enum": [
"snowflake",
"bigquery",
"databricks",
"spark",
"postgres"
],
"type": "string"
},
"estimatedSizePerRow": {
"description": "Estimated byte size per row of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"sourceSize": {
"description": "Total dataset byte size, this field is auto-generated by the analysis job.",
"type": "integer"
},
"totalRows": {
"description": "Total rows of the dataset, this field is auto-generated by the analysis job.",
"type": "integer"
},
"version": {
"description": "The version of the dataset definition information.",
"type": "integer"
}
},
"required": [
"columns",
"dataSourceId",
"dataStoreId",
"dialect",
"estimatedSizePerRow",
"sourceSize",
"totalRows",
"version"
],
"type": "object",
"x-versionadded": "v2.37"
}
Information about the dataset.
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| columns | [string] | true | maxItems: 1000 minItems: 2 |
List of the dataset column names, this field is auto-generated by the analysis job. |
| dataSourceId | string,null | true | The ID of the SQL table query and the database path, this field is auto-generated by the analysis job. | |
| dataStoreId | string,null | true | The ID of the SQL data store, this field is auto-generated by the analysis job. | |
| dialect | string | true | Source type data was retrieved from, this field is auto-generated by the analysis job. | |
| estimatedSizePerRow | integer | true | Estimated byte size per row of the dataset, this field is auto-generated by the analysis job. | |
| sourceSize | integer | true | Total dataset byte size, this field is auto-generated by the analysis job. | |
| totalRows | integer | true | Total rows of the dataset, this field is auto-generated by the analysis job. | |
| version | integer | true | The version of the dataset definition information. |
Enumerated Values¶
| Property | Value |
|---|---|
| dialect | [snowflake, bigquery, databricks, spark, postgres] |
DatasetPropsResponse
{
"description": "Dataset properties.",
"properties": {
"datasetId": {
"description": "The ID of the AI Catalog dataset.",
"type": "string"
},
"datasetVersionId": {
"description": "The version ID of the AI Catalog dataset.",
"type": "string"
}
},
"required": [
"datasetId",
"datasetVersionId"
],
"type": "object",
"x-versionadded": "v2.37"
}
Dataset properties.
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datasetId | string | true | The ID of the AI Catalog dataset. | |
| datasetVersionId | string | true | The version ID of the AI Catalog dataset. |
DynamicDatasetPropsResponse
{
"description": "Dynamic dataset additional properties.",
"properties": {
"credentialsId": {
"default": null,
"description": "The ID of the credentials to access the data store.",
"type": [
"string",
"null"
]
}
},
"required": [
"credentialsId"
],
"type": "object",
"x-versionadded": "v2.37"
}
Dynamic dataset additional properties.
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| credentialsId | string,null | true | The ID of the credentials to access the data store. |
Empty
{
"type": "object"
}
Properties¶
None
FeaturesChunkDefinitionResponse
{
"description": "Feature chunk definition properties.",
"type": "object",
"x-versionadded": "v2.37"
}
Feature chunk definition properties.
Properties¶
None
RowsChunkDefinitionResponse
{
"description": "Row chunk definition properties.",
"properties": {
"datetimePartitionColumn": {
"description": "Date partition column name.",
"type": [
"string",
"null"
]
},
"isDescendingOrder": {
"default": false,
"description": "The sorting order.",
"type": "boolean"
},
"orderByColumns": {
"default": [],
"description": "List of the sorting column names.",
"items": {
"description": "Dataset column name.",
"type": "string"
},
"maxItems": 10,
"type": "array"
},
"otvEarliestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvLatestTimestamp": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvTrainingEndDate": {
"description": "The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationDownsamplingPct": {
"description": "Percent by which to downsample the validation data.",
"type": [
"number",
"null"
]
},
"otvValidationEndDate": {
"description": "The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationEndDate`, one must specify `ValidationStartDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"otvValidationStartDate": {
"description": "The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying `ValidationStartDate`, one must specify `ValidationEndDate`. This attribute cannot be patched for non-OTV incremental learning projects.",
"format": "date-time",
"type": [
"string",
"null"
]
},
"targetClass": {
"description": "Target Class.",
"type": [
"string",
"null"
]
},
"targetColumn": {
"description": "Target column name.",
"type": [
"string",
"null"
]
},
"userGroupColumn": {
"description": "User group column name.",
"type": [
"string",
"null"
]
}
},
"required": [
"datetimePartitionColumn",
"isDescendingOrder",
"orderByColumns",
"otvEarliestTimestamp",
"otvLatestTimestamp",
"otvTrainingEndDate",
"otvValidationDownsamplingPct",
"otvValidationEndDate",
"otvValidationStartDate",
"targetClass",
"targetColumn",
"userGroupColumn"
],
"type": "object",
"x-versionadded": "v2.37"
}
Row chunk definition properties.
Properties¶
| Name | Type | Required | Restrictions | Description |
|---|---|---|---|---|
| datetimePartitionColumn | string,null | true | Date partition column name. | |
| isDescendingOrder | boolean | true | The sorting order. | |
| orderByColumns | [string] | true | maxItems: 10 |
List of the sorting column names. |
| otvEarliestTimestamp | string,null(date-time) | true | The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying ValidationEndDate, one must specify ValidationStartDate. This attribute cannot be patched for non-OTV incremental learning projects. |
|
| otvLatestTimestamp | string,null(date-time) | true | The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying ValidationEndDate, one must specify ValidationStartDate. This attribute cannot be patched for non-OTV incremental learning projects. |
|
| otvTrainingEndDate | string,null(date-time) | true | The end date of training data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying ValidationEndDate, one must specify ValidationStartDate. This attribute cannot be patched for non-OTV incremental learning projects. |
|
| otvValidationDownsamplingPct | number,null | true | Percent by which to downsample the validation data. | |
| otvValidationEndDate | string,null(date-time) | true | The end date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying ValidationEndDate, one must specify ValidationStartDate. This attribute cannot be patched for non-OTV incremental learning projects. |
|
| otvValidationStartDate | string,null(date-time) | true | The start date of validation scoring data in string format. Format can be '%Y-%m-%d %H:%M%S' or '%Y-%m-%d', the timezone defaults to UTC.When specifying ValidationStartDate, one must specify ValidationEndDate. This attribute cannot be patched for non-OTV incremental learning projects. |
|
| targetClass | string,null | true | Target Class. | |
| targetColumn | string,null | true | Target column name. | |
| userGroupColumn | string,null | true | User group column name. |