Configuration

This page lists the knowlg jobs' specific kafka topic, functionality configurations, sample event and dependencies (if any).

Configuration of all knowledge-platform-jobs during the time of deployment is referred from sunbird-learning-platform repository. However, configuration for local setup is referred from respective job folders in knowledge-platform-jobs repository.

🌠 dialcode-context-updater:

Kafka Topic:

kafka {
      input.topic = "{{ env_name }}.dialcode.context.job.request"
      groupId = "{{ env_name }}-dialcode-group"
    }

Job configuration variables:

VariablePurpose

dialcode_context_updater.actions

Used to identify dial code context update action

dialcode_context_updater.search_mode

Used to set search mode for search API

dialcode_context_updater.context_map_path

Used to specify the path of context Mapping file. File used to specify the field mapping of context schema.jsonld to sunbird content/collection schema.json

dialcode_context_updater.identifier_search_fields

Used to specify the search fields when the content/collection details is fetched for primary category.

dialcode_context_updater.dial_code_context_read_api_path

Used to specify the api endpoint of the DIAL service read context API (/dialcode/v4/read)

dialcode_context_updater.dial_code_context_update_api_path

Used to specify the api endpoint of the DIAL service Update context API (/dialcode/v4/update)

service.search.basePath

Used to specify Search service base URL.

service.dial_service.basePath

Used to specify DIAL service base URL.

es_sync_wait_time

Used to specify wait time for collection nodes data to sync to ES after collection publish to reflect in search service results.

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1655804893687,
  "mid": "LP.1655804893687.e8e921df-f479-49a9-af5a-df8bbc4de70a",
  "actor": {
    "id": "DIAL code context update Job",
    "type": "System"
  },
  "context": {
    "pdata": {
      "ver": "1.0",
      "id": "org.ekstep.platform"
    },
    "channel": "01309282781705830427",
    "env": "dev"
  },
  "object": {
    "ver": "1.0",
    "id": "G3L9S2"
  },
  "edata": {
    "action": "dialcode-context-update",
    "iteration": 1,
    "dialcode": "G3L9S2",
    "identifier": "do_113556563202981888177"
  },
  "identifier": "do_113556563202981888177"
}

Dependency: Services: 1. Search Service - composite search API

2. DIAL service - DIAL context read and update API

🌠 cassandra-data-migration:

Kafka Topic:

kafka {
      input.topic = "{{ env_name }}.cassandra.data.migration.request"
      groupId = "{{ env_name }}-cassandra-data-migration-group"
    }

Job configuration variables:

VariablePurpose

migrate.keyspace

Used to specify the keyspace of the cassandra database in which data is to be migrated.

migrate.table

Used to specify the table in the keyspace configured above; in which data is to be migrated.

migrate.primary_key_column

Used to specify the primay key column name of the table and the keyspace configured above; in which data is to be migrated.

migrate.primary_key_column_type

Used to specify the primay key column datatype in the table and the keyspace configured above; in which data is to be migrated.

migrate.column_to_migrate

Used to specify the name of the column to be migrated from the table and the keyspace configured above.

migrate.column_to_migrate_type

Used to specify the datatype of the column to be migrated from the table and the keyspace configured above.

migrate.key_value_strings_to_migrate

Used to specify the list of strings to be migrated in the column data.

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1619527882745,
  "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36",
  "actor": {
    "id": "cassandra-data-migration",
    "type": "System"
  },
  "context": {
    "channel": "ORG_001",
    "pdata": {
      "id": "org.sunbird.platform",
      "ver": "1.0"
    },
    "env": "dev"
  },
  "edata": {
    "column": "url",
    "table": "dialcode_images",
    "keyspace": "dialcodes",
    "action": "migrate-cassandra",
    "iteration": 1
  }
}

🌠 csp-migrator:

Kafka Topic:

kafka {
      input.topic = "{{ env_name }}.csp.migration.job.request"
      groupId = "{{ env_name }}-csp-migrator-group"
      failed.topic = "{{ csp_migrator_failed_topic_name }}"
      live_video_stream.topic = "{{ video_stream_topic_name }}"
      live_content_node_republish.topic = "{{ content_republish_topic_name }}"
      live_question_node_republish.topic = "{{ question_republish_topic_name }}"
    }

Job configuration variables:

VariablePurpose

key_value_strings_to_migrate

Used to specify the list of strings to be migrated in the fields' data.

neo4j_fields_to_migrate

Used to specify the neo4j fields which are to be migrated based on the objectType

cassandra_fields_to_migrate

Used to specify the columns of assessmentItem which are to be migrated.

hierarchy.keyspace

Used to specify the keyspace containing collection hierarchy data table

hierarchy.table

Used to specify collection hierarchy data table

content.keyspace

Used to specify the keyspace containing content data table

content.content_table

Used to specify content data table

content.assessment_table

Used to specify assessment data table

questionset.hierarchy.keyspace

Used to specify the keyspace containing question set hierarchy table.

questionset.hierarchy.table

Used to specify the question set hierarchy table

migrationVersion

Used to specify the migration version

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1619527882745,
  "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36",
  "actor": {
    "id": "csp-migration",
    "type": "System"
  },
  "context": {
    "channel": "01309282781705830427",
    "pdata": {
      "id": "org.sunbird.platform",
      "ver": "1.0"
    },
    "env": "dev"
  },
  "object": {
    "id": "do_2136329211884830721183.img",
    "ver": "1619153418829"
  },
  "edata": {
    "metadata": {
      "mimeType": "application/vnd.ekstep.pdf",
      "objectType": "ContentImage",
      "status": "Live"
    },
    "action": "csp-migration",
    "iteration": 1
  }
}

Dependency: Sync tool (Jenkins Job - to trigger events for each content to be migrated).

🌠 live-node-publisher:

Kafka Topic:

kafka {
      input.topic = {{ env_name }}.republish.job.request
      groupId = {{ env_name }}-content-republish-group
      live_video_stream.topic = "{{ env_name }}.live.video.stream.request"
      error.topic = "{{ env_name }}.learning.events.failed"
      skipped.topic = "{{ env_name }}.learning.events.skipped"
    }

Job configuration variables:

VariablePurpose

content.bundleLocation

Used to specify local/server folder location where artifacts are to be downloaded for ECAR bundling. Default value: "/data/contentBundle/"

content.isECARExtractionEnabled

Used to specify if the ECAR extraction is to be enabled to object 'version' and 'latest' cloud location using its 'snapshot' version. Default value: true

content.retry_asset_download_count

Used to specify number of times download attempt for assets part of content/collection object is to be done till it is successfully downloaded. Default value: 1

content.tmp_file_location

NOT USED

content.objectType

Used to specify list of valid objectTypes supported for publishing. Default value: ["Content", "ContentImage"]

content.mimeType

Used to specify list of valid mimeTypes supported for publishing. Default value: ["application/pdf"]

content.asset_download_duration

Used to specify time in seconds to wait for the asset download request to respond. Default value: "60 seconds"

content.stream.enabled

Used to check if streaming is enabled for published objects. If it is enabled, content rendenring is done using 'streamingUrl' attribute else via 'artifactUrl' Default value: false

content.stream.mimeType

Used to check if the mimeType of the object being published is of streamable type. If yes, event for video-stream-generator job is generated. Default value: ["video/mp4"]

content.artifact.size.for_online

Used to set the maximum size of the object (in bytes) that can be played by downloading beyond which "contentDisposition" is set to "online-only". Default value: 209715200

content.downloadFiles.spine

Used to specify list of attributes that store asset Urls which are to be downloaded from the mentioned Urls while packaging SPINE ECAR. Default value: ["appIcon"]

content.downloadFiles.full

Used to specify list of attributes that store asset Urls which are to be downloaded from the mentioned Urls while packaging FULL ECAR. Default value: ["appIcon", "grayScaleAppIcon", "artifactUrl", "itemSetPreviewUrl", "media"]

content.nested.fields

Used to specify the list of object properties that are of object types with nested attributes. Default value: ["badgeAssertions","targets","badgeAssociations"]

cloud_storage.folder.content

Used to specify the cloud store container folder name for content file storage/extraction etc. Default value: "container"

cloud_storage.folder.artifact

Used to specify the cloud store container folder name for artifact (media) files storage. Default value: "artifact"

contentTypeToPrimaryCategory

Used to specify the mapping between contentType and primaryCategory attributes using which object metadata is populated with the missing attribute among two. Default value: []

compositesearch.index.name

Used to specify the composite search index name where collection object nodes are synced with updated metadata. Default value: "compositesearch"

search.document.type

Used to specify the ElasticSearch document index type using which collection object nodes are synced with updated metadata. Default value: "cs"

master.category.validation.enabled

Used to specify whether object getting published is to be enriched with framework metadata. Default value: "Yes"

service.print.basePath

NOT USED

mimetype.allowed_extensions.word

Used to specify the list of file extensions allowed for uploaded content object. Default value: ["doc", "docx", "ppt", "pptx", "key", "odp", "pps", "odt", "wpd", "wps", "wks"]

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1619527882745,
  "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36",
  "actor": {
    "id": "content-publish",
    "type": "System"
  },
  "context": {
    "channel": "ORG_001",
    "pdata": {
      "id": "org.sunbird.platform",
      "ver": "1.0"
    },
    "env": "dev"
  },
  "object": {
    "id": "do_11363368845198131217",
    "ver": "1619153418829"
  },
  "edata": {
    "publish_type": "public",
    "metadata": {
      "identifier": "do_11363368845198131217",
      "mimeType": "application/vnd.ekstep.ecml-archive",
      "objectType": "Content",
      "lastPublishedBy": "",
      "pkgVersion": 4
    },
    "action": "republish",
    "iteration": 1
  }
}

Dependency:

Services: Search Service - composite search API

Jobs: 'csp-migrator'

🌠 live-video-stream-generator:

Kafka Topic:

kafka {
      input.topic = "{{ env_name }}.live.video.stream.request"
      groupId = "{{ env_name }}-live-video-stream-generator-group"
    }

Job configuration variables:

VariablePurpose

service.content.basePath

Used to configure Content Service API base path for accessing System Update API (/content/v4/system/update/ - not exposed publicly).

Sample kafka event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1649174914686,
  "mid": "LP.1649174914686.02c7ac3d-f7b2-46be-9771-e91645ecd632",
  "actor": {
    "id": "Post Publish Processor",
    "type": "System"
  },
  "context": {
    "pdata": {
      "ver": "1.0",
      "id": "org.sunbird.platform"
    },
    "channel": "01272777697873100812",
    "env": "sunbirdstaging"
  },
  "object": {
    "ver": "1649169311270",
    "id": "do_21350999440018636811684"
  },
  "edata": {
    "action": "post-publish-process",
    "iteration": 1,
    "identifier": "do_21350999440018636811684",
    "channel": "01272777697873100812",
    "mimeType": "application/vnd.ekstep.content-collection",
    "contentType": "Course",
    "pkgVersion": 1,
    "status": "Live",
    "name": "CourseMElenor",
    "trackable": {
      "enabled": "Yes",
      "autoBatch": "No"
    }
  }
}

Dependency:

Services: Content Service - system update API

Jobs: 'live-node-publisher'

Last updated