Configuration

This page lists the knowlg jobs' specific kafka topic, functionality configurations, sample event and dependencies (if any).

Configuration of all knowledge-platform-jobs during the time of deployment is referred from sunbird-learning-platform repository. However, configuration for local setup is referred from respective job folders in knowledge-platform-jobs repository.

🌠 dialcode-context-updater:

Kafka Topic:

kafka {
      input.topic = "{{ env_name }}.dialcode.context.job.request"
      groupId = "{{ env_name }}-dialcode-group"
    }

Job configuration variables:

Variable	Purpose
dialcode_context_updater.actions	Used to identify dial code context update action
dialcode_context_updater.search_mode	Used to set search mode for search API
dialcode_context_updater.context_map_path	Used to specify the path of context Mapping file. File used to specify the field mapping of context schema.jsonld to sunbird content/collection schema.json
dialcode_context_updater.identifier_search_fields	Used to specify the search fields when the content/collection details is fetched for primary category.
dialcode_context_updater.dial_code_context_read_api_path	Used to specify the api endpoint of the DIAL service read context API (/dialcode/v4/read)
dialcode_context_updater.dial_code_context_update_api_path	Used to specify the api endpoint of the DIAL service Update context API (/dialcode/v4/update)
service.search.basePath	Used to specify Search service base URL.
service.dial_service.basePath	Used to specify DIAL service base URL.
es_sync_wait_time	Used to specify wait time for collection nodes data to sync to ES after collection publish to reflect in search service results.

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1655804893687,
  "mid": "LP.1655804893687.e8e921df-f479-49a9-af5a-df8bbc4de70a",
  "actor": {
    "id": "DIAL code context update Job",
    "type": "System"
  },
  "context": {
    "pdata": {
      "ver": "1.0",
      "id": "org.ekstep.platform"
    },
    "channel": "01309282781705830427",
    "env": "dev"
  },
  "object": {
    "ver": "1.0",
    "id": "G3L9S2"
  },
  "edata": {
    "action": "dialcode-context-update",
    "iteration": 1,
    "dialcode": "G3L9S2",
    "identifier": "do_113556563202981888177"
  },
  "identifier": "do_113556563202981888177"
}

Dependency: Services: 1. Search Service - composite search API

2. DIAL service - DIAL context read and update API

🌠 cassandra-data-migration:

Kafka Topic:

kafka {
      input.topic = "{{ env_name }}.cassandra.data.migration.request"
      groupId = "{{ env_name }}-cassandra-data-migration-group"
    }

Job configuration variables:

Variable	Purpose
migrate.keyspace	Used to specify the keyspace of the cassandra database in which data is to be migrated.
migrate.table	Used to specify the table in the keyspace configured above; in which data is to be migrated.
migrate.primary_key_column	Used to specify the primay key column name of the table and the keyspace configured above; in which data is to be migrated.
migrate.primary_key_column_type	Used to specify the primay key column datatype in the table and the keyspace configured above; in which data is to be migrated.
migrate.column_to_migrate	Used to specify the name of the column to be migrated from the table and the keyspace configured above.
migrate.column_to_migrate_type	Used to specify the datatype of the column to be migrated from the table and the keyspace configured above.
migrate.key_value_strings_to_migrate	Used to specify the list of strings to be migrated in the column data.

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1619527882745,
  "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36",
  "actor": {
    "id": "cassandra-data-migration",
    "type": "System"
  },
  "context": {
    "channel": "ORG_001",
    "pdata": {
      "id": "org.sunbird.platform",
      "ver": "1.0"
    },
    "env": "dev"
  },
  "edata": {
    "column": "url",
    "table": "dialcode_images",
    "keyspace": "dialcodes",
    "action": "migrate-cassandra",
    "iteration": 1
  }
}

🌠 csp-migrator:

Kafka Topic:

kafka {
      input.topic = "{{ env_name }}.csp.migration.job.request"
      groupId = "{{ env_name }}-csp-migrator-group"
      failed.topic = "{{ csp_migrator_failed_topic_name }}"
      live_video_stream.topic = "{{ video_stream_topic_name }}"
      live_content_node_republish.topic = "{{ content_republish_topic_name }}"
      live_question_node_republish.topic = "{{ question_republish_topic_name }}"
    }

Job configuration variables:

Variable	Purpose
key_value_strings_to_migrate	Used to specify the list of strings to be migrated in the fields' data.
neo4j_fields_to_migrate	Used to specify the neo4j fields which are to be migrated based on the objectType
cassandra_fields_to_migrate	Used to specify the columns of assessmentItem which are to be migrated.
hierarchy.keyspace	Used to specify the keyspace containing collection hierarchy data table
hierarchy.table	Used to specify collection hierarchy data table
content.keyspace	Used to specify the keyspace containing content data table
content.content_table	Used to specify content data table
content.assessment_table	Used to specify assessment data table
questionset.hierarchy.keyspace	Used to specify the keyspace containing question set hierarchy table.
questionset.hierarchy.table	Used to specify the question set hierarchy table
migrationVersion	Used to specify the migration version

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1619527882745,
  "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36",
  "actor": {
    "id": "csp-migration",
    "type": "System"
  },
  "context": {
    "channel": "01309282781705830427",
    "pdata": {
      "id": "org.sunbird.platform",
      "ver": "1.0"
    },
    "env": "dev"
  },
  "object": {
    "id": "do_2136329211884830721183.img",
    "ver": "1619153418829"
  },
  "edata": {
    "metadata": {
      "mimeType": "application/vnd.ekstep.pdf",
      "objectType": "ContentImage",
      "status": "Live"
    },
    "action": "csp-migration",
    "iteration": 1
  }
}

Dependency: Sync tool (Jenkins Job - to trigger events for each content to be migrated).

🌠 live-node-publisher:

Kafka Topic:

kafka {
      input.topic = {{ env_name }}.republish.job.request
      groupId = {{ env_name }}-content-republish-group
      live_video_stream.topic = "{{ env_name }}.live.video.stream.request"
      error.topic = "{{ env_name }}.learning.events.failed"
      skipped.topic = "{{ env_name }}.learning.events.skipped"
    }

Job configuration variables:

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1619527882745,
  "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36",
  "actor": {
    "id": "content-publish",
    "type": "System"
  },
  "context": {
    "channel": "ORG_001",
    "pdata": {
      "id": "org.sunbird.platform",
      "ver": "1.0"
    },
    "env": "dev"
  },
  "object": {
    "id": "do_11363368845198131217",
    "ver": "1619153418829"
  },
  "edata": {
    "publish_type": "public",
    "metadata": {
      "identifier": "do_11363368845198131217",
      "mimeType": "application/vnd.ekstep.ecml-archive",
      "objectType": "Content",
      "lastPublishedBy": "",
      "pkgVersion": 4
    },
    "action": "republish",
    "iteration": 1
  }
}

Dependency:

Services: Search Service - composite search API

Jobs: 'csp-migrator'

Kafka Topic:

kafka {
      input.topic = "{{ env_name }}.live.video.stream.request"
      groupId = "{{ env_name }}-live-video-stream-generator-group"
    }

Job configuration variables:

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1649174914686,
  "mid": "LP.1649174914686.02c7ac3d-f7b2-46be-9771-e91645ecd632",
  "actor": {
    "id": "Post Publish Processor",
    "type": "System"
  },
  "context": {
    "pdata": {
      "ver": "1.0",
      "id": "org.sunbird.platform"
    },
    "channel": "01272777697873100812",
    "env": "sunbirdstaging"
  },
  "object": {
    "ver": "1649169311270",
    "id": "do_21350999440018636811684"
  },
  "edata": {
    "action": "post-publish-process",
    "iteration": 1,
    "identifier": "do_21350999440018636811684",
    "channel": "01272777697873100812",
    "mimeType": "application/vnd.ekstep.content-collection",
    "contentType": "Course",
    "pkgVersion": 1,
    "status": "Live",
    "name": "CourseMElenor",
    "trackable": {
      "enabled": "Yes",
      "autoBatch": "No"
    }
  }
}

Dependency:

Services: Content Service - system update API

Jobs: 'live-node-publisher'