Data Migration
This document details about migration of data from cloud absolute paths stored in the database with relative paths OR with new CSP absolute paths.
This document details about the migration of data with respect to
- Replace existing absolute paths in database with relative paths.
- Migration of data while changing the CSP providerExample: Moving from Azure to AWS service provider.
- neo4J fields based on objectType:
"asset": ["artifactUrl", "thumbnail", "downloadUrl"],
"content": ["appIcon", "artifactUrl", "posterImage", "previewUrl", "thumbnail", "assetsMap", "certTemplate", "itemSetPreviewUrl", "grayScaleAppIcon", "sourceURL", "variants", "downloadUrl","streamingUrl"],
"contentimage": ["appIcon", "artifactUrl", "posterImage", "previewUrl", "thumbnail", "assetsMap", "certTemplate", "itemSetPreviewUrl", "grayScaleAppIcon", "sourceURL", "variants", "downloadUrl","streamingUrl"],
"collection": ["appIcon", "artifactUrl", "posterImage", "previewUrl", "thumbnail", "toc_url", "grayScaleAppIcon", "variants", "downloadUrl"],
"collectionimage": ["appIcon", "artifactUrl", "posterImage", "previewUrl", "thumbnail", "toc_url", "grayScaleAppIcon", "variants", "downloadUrl"],
"plugins": ["artifactUrl"],
"itemset": ["previewUrl", "downloadUrl"],
"assessmentitem": ["data", "question", "solutions", "editorState", "media"],
"question": ["appIcon","artifactUrl", "posterImage", "previewUrl","downloadUrl", "variants","pdfUrl"],
"questionimage": ["appIcon","artifactUrl", "posterImage", "previewUrl","downloadUrl", "variants","pdfUrl"],
"questionset": ["appIcon","artifactUrl", "posterImage", "previewUrl","downloadUrl", "variants","pdfUrl"],
"questionsetimage": ["appIcon","artifactUrl", "posterImage", "previewUrl","downloadUrl", "variants","pdfUrl"]
Note: Above data is available as configuration(neo4j_fields_to_migrate) in 'csp-migrator' job.
- Cassandra data that will be migrated
Keyspace | Table | Column |
|---|---|---|
hierarchy_keystore | content_hierarchy | hierarchy |
content_keystore | content_data | body |
content_keystore | question_data | body, editorState, answer, solutions, instructions, media |
questionset_keystore | questionset_hierarchy | hierarchy |
dialcodes | dialcode_images | url |
dialcodes | dialcode_batch | url |
- ECAR (needs live nodes republishing)
- streamingUrl (needs regeneration based on new Media service provider)
Reference diagram to know how the migration of existing data with CNAME(storing relative path DB)
https://app.diagrams.net/#G1HzUob6a9_TrVWhcoo1nWoKPVN0PKzpmQ
app.diagrams.net
migration flow diagram (migration-job flow & migration-process workflow)
Note:
- Jenkins Job 'Neo4jElasticSearchSyncTool' is used to insert the events into 'csp-migrator' job input topic. 'csp-migrator' job will further insert topics into 'live-node-publisher' job and 'live-video-stream-generator' jobs based on conditions. Jenkins job command: migratecspdata
- 'cassandra-data-migration' job is to be used for migration of 'dialcode_images' and 'dialcode_batch' cassandra tables in 'dialcodes' keyspace.
- Run the migration flink jobs in a separate kafka setup with increased processing ability and storage for storing all kakfa events and logs.
- Increase the infra for neo4j. Also, increase the neo4j max heap size in neo4j conf file.
- Increase infra for logstash, search-indexer flink job and ElasticSearch to handle the neo4j transaction data sync.
The content migration should execute in the below order only. Otherwise there is a chances of migration failure because of dependent content is not yet migrated. more details
- Before running the migration steps, go here and run all the queries and keep the output to compare after migration.
- Go to Deploy/KnowledgePlatform/Neo4jElasticSearchSyncTool jenkins job.
- Select the command as migratecspdata
- And copy and paste the parameter one by one in parameter section in jenkins deployment job.
Sequence | Type | Sync Tool Jenkins Parameters |
|---|---|---|
1 | Video Asset | --graphId domain --objectType Asset --mimeType video/webm,video/mp4 --delay 2000 |
2 | Other Asset | --graphId domain --objectType Asset --delay 2000 |
3 | Video Content | --graphId domain --objectType Content,ContentImage --mimeType video/mp4,video/webm --delay 2000 |
4 | Plugin, Youtube Content, PDF Content,EPUB Content | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.plugin-archive,video/x-youtube,application/pdf,application/epub --delay 2000 |
5 | AssessmentItem | --graphId domain --objectType AssessmentItem --delay 2000 |
6 | ItemSet | --graphId domain --objectType ItemSet --delay 2000 |
7 | H5P Content | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.h5p-archive --delay 2000 |
8 | HTML | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.html-archive --delay 2000 |
9 | ECML | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.ecml-archive --delay 2000 |
10 | Remaining Contents | --graphId domain --objectType Content,ContentImage --delay 2000 |
11 | Collection | --graphId domain --objectType Content,ContentImage,Collection,CollectionImage --mimeType application/vnd.ekstep.content-collection --delay 2000 |
12 | dialcodes.dialcode_images | Push below event to "{{env}}.cassandra.data.migration.request" kafka topic
{"eid":"BE_JOB_REQUEST","ets":1619527882745,"mid":"LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36","actor":{"id":"cassandra-data-migration","type":"System"},"context":{"channel":"ORG_001","pdata":{"id":"org.sunbird.platform","ver":"1.0"},"env":"dev"},"edata":{"column":"url", "columnType":"String", "table": "dialcode_images", "keyspace": "dialcodes", "primaryKeyColumn": "filename", "primaryKeyColumnType": "String", "action":"migrate-cassandra","iteration":1}} |
13 | dialcodes.dialcode_batch | Push below event to "{{env}}.cassandra.data.migration.request" kafka topic
{"eid":"BE_JOB_REQUEST","ets":1619527882745,"mid":"LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36","actor":{"id":"cassandra-data-migration","type":"System"},"context":{"channel":"ORG_001","pdata":{"id":"org.sunbird.platform","ver":"1.0"},"env":"dev"},"edata":{"column":"url", "columnType":"String", "table": "dialcode_batch", "keyspace": "dialcodes", "primaryKeyColumn": "processid", "primaryKeyColumnType": "UUID", "action":"migrate-cassandra","iteration":1}} |
Note:
- If the 'migratecspdata' command stops before reaching 100%, please wait for 'csp-migrator' job lag to reach 0 before triggerring the same 'migratecspdata' command again.
- ECML migration can be triggered only after Asset, AssessmentItem and ItemSet migration is completed.
- For Collection migration to be triggered, pre-requisites are:
a. Question, QuestionSet migration should be completed (as part of Inquiry BB).
b. All assets and contents are to be migrated successfully.
c. All migrated contents data should be synced to ElasticSearch.
1.Neo4J & Cassandra data migration started
- no version => Data migration started for each Neo4j node. It will migrate the Neo4j data and Cassandra data migration failed for the object
- 1.0 => Success: Neo4j data and Cassandra data migration completed for the object(node)
- 0.1 => Fail: Data migration is failed for the Neo4J data or Cassandra data of the specific node(identifier is the key to know for which node it failed. We can check the logs of the service to know the reason of failure)
- 0.5 => Skipped: migration skipped for the object
2.ECAR Generation(after previous step of Neo4J & Cassandra data migration success)
- 1.1 => Success: Neo4j data and Cassandra data migration completed for the object and ECAR is republished.
- 0.2 => Fail: Neo4j data and Cassandra data migration completed for the object. But, ECAR republish has failed.
3.Video streaming generation (after previous step of ECAR generate is success)
- 1.2 => Success: Neo4j data and Cassandra data migration completed for video type of asset/content and streamingUrl is regenerated successfully.
More details of verification steps are added in the below confluence wiki
- 'cloudstorage_base_path' private/public repo variable will be updated with CNAME URL ONLY.
- 'valid_cloudstorage_base_urls' private/public repo update with CNAME URL along with Cloud storage BLOB URL
- Update of 'dial_service_schema_base_path' private repo variable with CNAME if it is having blob URL.
- 'sunbird_cloud_storage_urls' public/private devops repo variable update with CNAME URL addition.
- Restart all services and jobs (Services: Assessment service, Content Service, Taxonomy Service, Learning Service, DIAL Service. Jobs: asset-enrichment, content-auto-creator, content-publish, qrcode-image-generator, search-indexer)
- Run Sync tool using 'syncbyobjectType' command to sync all assets, contents and collections from neo4j to ElasticSearch with CNAME URLs.
- If CNAME is configured post migration, republishing of all Live Contents and Collections is to be triggered for ECARs to be updated with CNAME URLs.
Last modified 8mo ago