Configuration

This page lists the knowlg jobs' specific kafka topic, functionality configurations, sample event and dependencies (if any).

Configuration of all knowledge-platform-jobs during the time of deployment is referred from sunbird-learning-platform repository. However, configuration for local setup is referred from respective job folders in knowledge-platform-jobs repository.

🌠 dialcode-context-updater:

Kafka Topic:

kafka {
      input.topic = "{{ env_name }}.dialcode.context.job.request"
      groupId = "{{ env_name }}-dialcode-group"
    }

Job configuration variables:

Variable	Purpose
dialcode_context_updater.actions	Used to identify dial code context update action
dialcode_context_updater.search_mode	Used to set search mode for search API
dialcode_context_updater.context_map_path	Used to specify the path of context Mapping file. File used to specify the field mapping of context schema.jsonld to sunbird content/collection schema.json
dialcode_context_updater.identifier_search_fields	Used to specify the search fields when the content/collection details is fetched for primary category.
dialcode_context_updater.dial_code_context_read_api_path	Used to specify the api endpoint of the DIAL service read context API (/dialcode/v4/read)
dialcode_context_updater.dial_code_context_update_api_path	Used to specify the api endpoint of the DIAL service Update context API (/dialcode/v4/update)
service.search.basePath	Used to specify Search service base URL.
service.dial_service.basePath	Used to specify DIAL service base URL.
es_sync_wait_time	Used to specify wait time for collection nodes data to sync to ES after collection publish to reflect in search service results.

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1655804893687,
  "mid": "LP.1655804893687.e8e921df-f479-49a9-af5a-df8bbc4de70a",
  "actor": {
    "id": "DIAL code context update Job",
    "type": "System"
  },
  "context": {
    "pdata": {
      "ver": "1.0",
      "id": "org.ekstep.platform"
    },
    "channel": "01309282781705830427",
    "env": "dev"
  },
  "object": {
    "ver": "1.0",
    "id": "G3L9S2"
  },
  "edata": {
    "action": "dialcode-context-update",
    "iteration": 1,
    "dialcode": "G3L9S2",
    "identifier": "do_113556563202981888177"
  },
  "identifier": "do_113556563202981888177"
}

Dependency: Services: 1. Search Service - composite search API

2. DIAL service - DIAL context read and update API

🌠 cassandra-data-migration:

Kafka Topic:

kafka {
      input.topic = "{{ env_name }}.cassandra.data.migration.request"
      groupId = "{{ env_name }}-cassandra-data-migration-group"
    }

Job configuration variables:

Variable	Purpose
migrate.keyspace	Used to specify the keyspace of the cassandra database in which data is to be migrated.
migrate.table	Used to specify the table in the keyspace configured above; in which data is to be migrated.
migrate.primary_key_column	Used to specify the primay key column name of the table and the keyspace configured above; in which data is to be migrated.
migrate.primary_key_column_type	Used to specify the primay key column datatype in the table and the keyspace configured above; in which data is to be migrated.
migrate.column_to_migrate	Used to specify the name of the column to be migrated from the table and the keyspace configured above.
migrate.column_to_migrate_type	Used to specify the datatype of the column to be migrated from the table and the keyspace configured above.
migrate.key_value_strings_to_migrate	Used to specify the list of strings to be migrated in the column data.

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1619527882745,
  "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36",
  "actor": {
    "id": "cassandra-data-migration",
    "type": "System"
  },
  "context": {
    "channel": "ORG_001",
    "pdata": {
      "id": "org.sunbird.platform",
      "ver": "1.0"
    },
    "env": "dev"
  },
  "edata": {
    "column": "url",
    "table": "dialcode_images",
    "keyspace": "dialcodes",
    "action": "migrate-cassandra",
    "iteration": 1
  }
}

🌠 csp-migrator:

Kafka Topic:

kafka {
      input.topic = "{{ env_name }}.csp.migration.job.request"
      groupId = "{{ env_name }}-csp-migrator-group"
      failed.topic = "{{ csp_migrator_failed_topic_name }}"
      live_video_stream.topic = "{{ video_stream_topic_name }}"
      live_content_node_republish.topic = "{{ content_republish_topic_name }}"
      live_question_node_republish.topic = "{{ question_republish_topic_name }}"
    }

Job configuration variables:

Variable	Purpose
key_value_strings_to_migrate	Used to specify the list of strings to be migrated in the fields' data.
neo4j_fields_to_migrate	Used to specify the neo4j fields which are to be migrated based on the objectType
cassandra_fields_to_migrate	Used to specify the columns of assessmentItem which are to be migrated.
hierarchy.keyspace	Used to specify the keyspace containing collection hierarchy data table
hierarchy.table	Used to specify collection hierarchy data table
content.keyspace	Used to specify the keyspace containing content data table
content.content_table	Used to specify content data table
content.assessment_table	Used to specify assessment data table
questionset.hierarchy.keyspace	Used to specify the keyspace containing question set hierarchy table.
questionset.hierarchy.table	Used to specify the question set hierarchy table
migrationVersion	Used to specify the migration version

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1619527882745,
  "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36",
  "actor": {
    "id": "csp-migration",
    "type": "System"
  },
  "context": {
    "channel": "01309282781705830427",
    "pdata": {
      "id": "org.sunbird.platform",
      "ver": "1.0"
    },
    "env": "dev"
  },
  "object": {
    "id": "do_2136329211884830721183.img",
    "ver": "1619153418829"
  },
  "edata": {
    "metadata": {
      "mimeType": "application/vnd.ekstep.pdf",
      "objectType": "ContentImage",
      "status": "Live"
    },
    "action": "csp-migration",
    "iteration": 1
  }
}

Dependency: Sync tool (Jenkins Job - to trigger events for each content to be migrated).

🌠 live-node-publisher:

Kafka Topic:

kafka {
      input.topic = {{ env_name }}.republish.job.request
      groupId = {{ env_name }}-content-republish-group
      live_video_stream.topic = "{{ env_name }}.live.video.stream.request"
      error.topic = "{{ env_name }}.learning.events.failed"
      skipped.topic = "{{ env_name }}.learning.events.skipped"
    }

Job configuration variables:

Variable	Purpose
content.bundleLocation	Used to specify local/server folder location where artifacts are to be downloaded for ECAR bundling. Default value: "/data/contentBundle/"
content.isECARExtractionEnabled	Used to specify if the ECAR extraction is to be enabled to object 'version' and 'latest' cloud location using its 'snapshot' version. Default value: true
content.retry_asset_download_count	Used to specify number of times download attempt for assets part of content/collection object is to be done till it is successfully downloaded. Default value: 1
content.tmp_file_location	NOT USED
content.objectType	Used to specify list of valid objectTypes supported for publishing. Default value: ["Content", "ContentImage"]
content.mimeType	Used to specify list of valid mimeTypes supported for publishing. Default value: ["application/pdf"]
content.asset_download_duration	Used to specify time in seconds to wait for the asset download request to respond. Default value: "60 seconds"
content.stream.enabled	Used to check if streaming is enabled for published objects. If it is enabled, content rendenring is done using 'streamingUrl' attribute else via 'artifactUrl' Default value: false
content.stream.mimeType	Used to check if the mimeType of the object being published is of streamable type. If yes, event for video-stream-generator job is generated. Default value: ["video/mp4"]
content.artifact.size.for_online	Used to set the maximum size of the object (in bytes) that can be played by downloading beyond which "contentDisposition" is set to "online-only". Default value: 209715200
content.downloadFiles.spine	Used to specify list of attributes that store asset Urls which are to be downloaded from the mentioned Urls while packaging SPINE ECAR. Default value: ["appIcon"]
content.downloadFiles.full	Used to specify list of attributes that store asset Urls which are to be downloaded from the mentioned Urls while packaging FULL ECAR. Default value: ["appIcon", "grayScaleAppIcon", "artifactUrl", "itemSetPreviewUrl", "media"]
content.nested.fields	Used to specify the list of object properties that are of object types with nested attributes. Default value: ["badgeAssertions","targets","badgeAssociations"]
cloud_storage.folder.content	Used to specify the cloud store container folder name for content file storage/extraction etc. Default value: "container"
cloud_storage.folder.artifact	Used to specify the cloud store container folder name for artifact (media) files storage. Default value: "artifact"
contentTypeToPrimaryCategory	Used to specify the mapping between contentType and primaryCategory attributes using which object metadata is populated with the missing attribute among two. Default value: []
compositesearch.index.name	Used to specify the composite search index name where collection object nodes are synced with updated metadata. Default value: "compositesearch"
search.document.type	Used to specify the ElasticSearch document index type using which collection object nodes are synced with updated metadata. Default value: "cs"
master.category.validation.enabled	Used to specify whether object getting published is to be enriched with framework metadata. Default value: "Yes"
service.print.basePath	NOT USED
mimetype.allowed_extensions.word	Used to specify the list of file extensions allowed for uploaded content object. Default value: ["doc", "docx", "ppt", "pptx", "key", "odp", "pps", "odt", "wpd", "wps", "wks"]

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1619527882745,
  "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36",
  "actor": {
    "id": "content-publish",
    "type": "System"
  },
  "context": {
    "channel": "ORG_001",
    "pdata": {
      "id": "org.sunbird.platform",
      "ver": "1.0"
    },
    "env": "dev"
  },
  "object": {
    "id": "do_11363368845198131217",
    "ver": "1619153418829"
  },
  "edata": {
    "publish_type": "public",
    "metadata": {
      "identifier": "do_11363368845198131217",
      "mimeType": "application/vnd.ekstep.ecml-archive",
      "objectType": "Content",
      "lastPublishedBy": "",
      "pkgVersion": 4
    },
    "action": "republish",
    "iteration": 1
  }
}

Dependency:

Services: Search Service - composite search API

Jobs: 'csp-migrator'

🌠 live-video-stream-generator:

Kafka Topic:

kafka {
      input.topic = "{{ env_name }}.live.video.stream.request"
      groupId = "{{ env_name }}-live-video-stream-generator-group"
    }

Job configuration variables:

Variable	Purpose
service.content.basePath	Used to configure Content Service API base path for accessing System Update API (/content/v4/system/update/ - not exposed publicly).

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1649174914686,
  "mid": "LP.1649174914686.02c7ac3d-f7b2-46be-9771-e91645ecd632",
  "actor": {
    "id": "Post Publish Processor",
    "type": "System"
  },
  "context": {
    "pdata": {
      "ver": "1.0",
      "id": "org.sunbird.platform"
    },
    "channel": "01272777697873100812",
    "env": "sunbirdstaging"
  },
  "object": {
    "ver": "1649169311270",
    "id": "do_21350999440018636811684"
  },
  "edata": {
    "action": "post-publish-process",
    "iteration": 1,
    "identifier": "do_21350999440018636811684",
    "channel": "01272777697873100812",
    "mimeType": "application/vnd.ekstep.content-collection",
    "contentType": "Course",
    "pkgVersion": 1,
    "status": "Live",
    "name": "CourseMElenor",
    "trackable": {
      "enabled": "Yes",
      "autoBatch": "No"
    }
  }
}

Dependency:

Services: Content Service - system update API

Jobs: 'live-node-publisher'