Database inventory tools¶

DataRobot provides inventory tools for MongoDB and PostgreSQL to verify that no data is lost after a backup/restore process, upgrade, or migration. Run the applicable tool before and after the operation, then compare the two inventories for differences.

MongoDB inventory¶

The MongoDB inventory tool ensures that no data is lost after a backup/restore process or after upgrades and migrations. It was added in v9.0.1, and was backported to v8.0.16.

Usage¶

Script is located at DataRobot-RELEASE-10.X.X/installer_tools/scripts/ called mongodb_data_consistency.py on the k8s cluster

Copy this to the mmapp-app pod using the kubectl command below. Replace the namespace and pod name accordingly.

kubectl cp DataRobot-RELEASE-10.X.X/installer_tools/scripts/mongodb_data_consistency.py <your-kubernetes-namespace>/<mmapp-app-pod-name>:/mnt/local_file_storage/mongodb_data_consistency.py

Using the mmapp-app pod is strongly recommended that has DataRobot-runtime available or any other pod that has DataRobot-runtime and MongoDB secrets accessible to it

For default mongo-uri in dr-secrets: Run this command from source before backup restore process or before upgrade: python3 /mnt/local_file_storage/mongodb_data_consistency.py pre-upgrade Run this command from target after restore or upgrade: python3 /mnt/local_file_storage/mongodb_data_consistency.py post-upgrade --file <file/path/inventory_pre_upgrade.txt> For remote mongo-uri connections: Run this command from source before backup restore process or before upgrade: python3 /mnt/local_file_storage/mongodb_data_consistency.py pre-upgrade --mongo-uri 'mongodb://username:password@mongo_host:27017' Run this command from target after restore or upgrade: python3 /mnt/local_file_storage/mongodb_data_consistency.pypost-upgrade --file <file/path/inventory_pre_upgrade.txt> --post-mongo-uri 'mongodb://username:password@mongo_host:27017'

Parameters: * pre-upgrade option used to get MongoDB data inventory of the source before backup/restore, upgrade or migration process. By default inventory script is stored as inventory_pre_upgrade.txt in the current directory

• post-upgrade option is used to get MongoDB data inventory of target after restore, upgrade or migration completion. Requires --file parameter to be passed.

• --file option is mandatory for post-upgrade option, where pre inventory needs to be passed i.e inventory_pre_upgrade.txt

• --mongo-uri parameter can be passed if its remote database not available in dr-secrets

Inventory should be produced before and after running data migration. Once post-upgrade is run, it compares the two inventories and outputs the diff between them. Example output

['MMApp', 'admin', 'application_builder', 'audit', 'celery', 'common_infra', 'config', 'datasets', 'draudit', 'dss_profiling', 'identity', 'json_studio', 'loadtest', 'local', 'orm_next', 'prediction_optimization_ux', 'secure_configs', 'test', 'usersecrets', 'varietyResults']

Collection: reason_codes_job
Size Difference: 13230080 -> 13369344
Number of Documents Difference: 101872 -> 103024
Number of Indexes Difference: 5 -> 2

Collection: period_accuracy_period_metadata
Size Difference: 245760 -> 237568
Number of Documents Difference: 2886 -> 2921

Collection: project_clone
Size Difference: 3678208 -> 3764224

Number of Documents Difference: 101437 -> 119725
There are 3 collections that differ

The script also considers certain collections that keep varying which can be safely ignored during restores even if there are document/index mismatch. It prints the difference but ignores it to count towards an inconsistent state.

'job_process',
'qid_counter',
'compute_cluster_metrics',
'queue_monitor',
'queue',
'job_executions',
'execute_kubeworkers_health_checks'

At the end of the comparison, script prints whether the database is in a consistent state or not based on the differences.

PostgreSQL inventory¶

The PostgreSQL inventory tool ensures that no data is lost after a migration. It was added in v9.0, and was backported to v8.0.15.

Usage¶

Script can be run via: python3 ./support/postgres/model_monitoring_inventory.py inside DR environment

Optional parameters are: * -connection_string, that can be used to pass db connection parameters in the form of postgresql://scott:tiger@localhost/test (default value is pulled from the environment) * -output_file to change where inventory is written (default is inventory.json)

Output of the script is a json file in a form of

{
    "users": [
        "metering_api",
        "metering_worker",
        "modmon",
        "postgres"
    ],
    "tables": [
        {
            "schema": "public",
            "name": "modmon_batch",
            "row_count": 0,
            "indexes": [
                  {
                    "name": "modmon_batch_ts_max",
                    "unique": false,
                    "column_names": ["ts_max"]
                  }
            ]
        }
    ]
}

Inventory should be produced before and after running data migration. ./support/postgres/model_monitoring_compare_inventory.py <path_to_pre_migration_inventroy> <path_to_post_migration_inventory> can be used to compare two inventories and outputs the diff between them. Example output

DIFFERENT TABLES DETECTED
tables before not after:
tables after not before: mpcr_2
TABLE modmon_prediction_feature_stats is different before and after
TABLE modmon_training_classification_stats is different before and after

Scripts can also be used on prior 8.0 versions (8.0.x), for that contents of ./support/postgres/ directory need to be mounted in the DR environment.