# Configuration Configuration of all knowledge-platform-jobs during the time of deployment is referred from [sunbird-learning-platform](https://github.com/project-sunbird/sunbird-learning-platform/blob/release-4.8.0/kubernetes/helm_charts/datapipeline_jobs/values.j2) repository. However, configuration for local setup is referred from respective job folders in [knowledge-platform-jobs](https://github.com/project-sunbird/knowledge-platform-jobs) repository. ### :stars: dialcode-context-updater: **Kafka Topic:** ``` kafka { input.topic = "{{ env_name }}.dialcode.context.job.request" groupId = "{{ env_name }}-dialcode-group" } ``` **Job configuration variables:**
VariablePurpose
dialcode_context_updater.actionsUsed to identify dial code context update action
dialcode_context_updater.search_modeUsed to set search mode for search API
dialcode_context_updater.context_map_pathUsed to specify the path of context Mapping file. File used to specify the field mapping of context schema.jsonld to sunbird content/collection schema.json
dialcode_context_updater.identifier_search_fieldsUsed to specify the search fields when the content/collection details is fetched for primary category.
dialcode_context_updater.dial_code_context_read_api_pathUsed to specify the api endpoint of the DIAL service read context API (/dialcode/v4/read)
dialcode_context_updater.dial_code_context_update_api_pathUsed to specify the api endpoint of the DIAL service Update context API (/dialcode/v4/update)
service.search.basePathUsed to specify Search service base URL.
service.dial_service.basePathUsed to specify DIAL service base URL.
es_sync_wait_timeUsed to specify wait time for collection nodes data to sync to ES after collection publish to reflect in search service results.
**Sample kafka event:** ``` { "eid": "BE_JOB_REQUEST", "ets": 1655804893687, "mid": "LP.1655804893687.e8e921df-f479-49a9-af5a-df8bbc4de70a", "actor": { "id": "DIAL code context update Job", "type": "System" }, "context": { "pdata": { "ver": "1.0", "id": "org.ekstep.platform" }, "channel": "01309282781705830427", "env": "dev" }, "object": { "ver": "1.0", "id": "G3L9S2" }, "edata": { "action": "dialcode-context-update", "iteration": 1, "dialcode": "G3L9S2", "identifier": "do_113556563202981888177" }, "identifier": "do_113556563202981888177" } ``` {% hint style="info" %} ***Dependency:***\ \&#xNAN;** Services:** \ 1\. Search Service - composite search API 2\. DIAL service - DIAL context read and update API {% endhint %} ### :stars: cassandra-data-migration: **Kafka Topic:** ``` kafka { input.topic = "{{ env_name }}.cassandra.data.migration.request" groupId = "{{ env_name }}-cassandra-data-migration-group" } ``` **Job configuration variables:** | Variable | Purpose | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | migrate.keyspace | Used to specify the keyspace of the cassandra database in which data is to be migrated. | | migrate.table | Used to specify the table in the keyspace configured above; in which data is to be migrated. | | migrate.primary\_key\_column | Used to specify the primay key column name of the table and the keyspace configured above; in which data is to be migrated. | | migrate.primary\_key\_column\_type | Used to specify the primay key column datatype in the table and the keyspace configured above; in which data is to be migrated. | | migrate.column\_to\_migrate | Used to specify the name of the column to be migrated from the table and the keyspace configured above. | | migrate.column\_to\_migrate\_type | Used to specify the datatype of the column to be migrated from the table and the keyspace configured above. | | migrate.key\_value\_strings\_to\_migrate | Used to specify the list of strings to be migrated in the column data. | **Sample kafka event:** ``` { "eid": "BE_JOB_REQUEST", "ets": 1619527882745, "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36", "actor": { "id": "cassandra-data-migration", "type": "System" }, "context": { "channel": "ORG_001", "pdata": { "id": "org.sunbird.platform", "ver": "1.0" }, "env": "dev" }, "edata": { "column": "url", "table": "dialcode_images", "keyspace": "dialcodes", "action": "migrate-cassandra", "iteration": 1 } } ``` ### :stars: csp-migrator: **Kafka Topic:** ``` kafka { input.topic = "{{ env_name }}.csp.migration.job.request" groupId = "{{ env_name }}-csp-migrator-group" failed.topic = "{{ csp_migrator_failed_topic_name }}" live_video_stream.topic = "{{ video_stream_topic_name }}" live_content_node_republish.topic = "{{ content_republish_topic_name }}" live_question_node_republish.topic = "{{ question_republish_topic_name }}" } ``` **Job configuration variables:** | Variable | Purpose | | -------------------------------- | --------------------------------------------------------------------------------- | | key\_value\_strings\_to\_migrate | Used to specify the list of strings to be migrated in the fields' data. | | neo4j\_fields\_to\_migrate | Used to specify the neo4j fields which are to be migrated based on the objectType | | cassandra\_fields\_to\_migrate | Used to specify the columns of assessmentItem which are to be migrated. | | hierarchy.keyspace | Used to specify the keyspace containing collection hierarchy data table | | hierarchy.table | Used to specify collection hierarchy data table | | content.keyspace | Used to specify the keyspace containing content data table | | content.content\_table | Used to specify content data table | | content.assessment\_table | Used to specify assessment data table | | questionset.hierarchy.keyspace | Used to specify the keyspace containing question set hierarchy table. | | questionset.hierarchy.table | Used to specify the question set hierarchy table | | migrationVersion | Used to specify the migration version | | | | | | | | | | **Sample kafka event:** ``` { "eid": "BE_JOB_REQUEST", "ets": 1619527882745, "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36", "actor": { "id": "csp-migration", "type": "System" }, "context": { "channel": "01309282781705830427", "pdata": { "id": "org.sunbird.platform", "ver": "1.0" }, "env": "dev" }, "object": { "id": "do_2136329211884830721183.img", "ver": "1619153418829" }, "edata": { "metadata": { "mimeType": "application/vnd.ekstep.pdf", "objectType": "ContentImage", "status": "Live" }, "action": "csp-migration", "iteration": 1 } } ``` {% hint style="info" %} ***Dependency:***** Sync tool (Jenkins Job - to trigger events for each content to be migrated).** {% endhint %} ### :stars: live-node-publisher: **Kafka Topic:** ``` kafka { input.topic = {{ env_name }}.republish.job.request groupId = {{ env_name }}-content-republish-group live_video_stream.topic = "{{ env_name }}.live.video.stream.request" error.topic = "{{ env_name }}.learning.events.failed" skipped.topic = "{{ env_name }}.learning.events.skipped" } ``` **Job configuration variables:** | Variable | Purpose | | ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | content.bundleLocation |

Used to specify local/server folder location where artifacts are to be downloaded for ECAR bundling.
Default value: "/data/contentBundle/"

| | content.isECARExtractionEnabled |

Used to specify if the ECAR extraction is to be enabled to object 'version' and 'latest' cloud location using its 'snapshot' version.
Default value: true

| | content.retry\_asset\_download\_count |

Used to specify number of times download attempt for assets part of content/collection object is to be done till it is successfully downloaded.
Default value: 1

| | content.tmp\_file\_location | NOT USED | | content.objectType |

Used to specify list of valid objectTypes supported for publishing.
Default value: \["Content", "ContentImage"]

| | content.mimeType |

Used to specify list of valid mimeTypes supported for publishing.
Default value: \["application/pdf"]

| | content.asset\_download\_duration |

Used to specify time in seconds to wait for the asset download request to respond.
Default value: "60 seconds"

| | content.stream.enabled |

Used to check if streaming is enabled for published objects. If it is enabled, content rendenring is done using 'streamingUrl' attribute else via 'artifactUrl'
Default value: false

| | content.stream.mimeType |

Used to check if the mimeType of the object being published is of streamable type. If yes, event for video-stream-generator job is generated.
Default value: \["video/mp4"]

| | content.artifact.size.for\_online |

Used to set the maximum size of the object (in bytes) that can be played by downloading beyond which "contentDisposition" is set to "online-only".
Default value: 209715200

| | content.downloadFiles.spine |

Used to specify list of attributes that store asset Urls which are to be downloaded from the mentioned Urls while packaging SPINE ECAR.
Default value: \["appIcon"]

| | content.downloadFiles.full |

Used to specify list of attributes that store asset Urls which are to be downloaded from the mentioned Urls while packaging FULL ECAR.
Default value: \["appIcon", "grayScaleAppIcon", "artifactUrl", "itemSetPreviewUrl", "media"]

| | content.nested.fields |

Used to specify the list of object properties that are of object types with nested attributes.
Default value: \["badgeAssertions","targets","badgeAssociations"]

| | cloud\_storage.folder.content |

Used to specify the cloud store container folder name for content file storage/extraction etc.
Default value: "container"

| | cloud\_storage.folder.artifact |

Used to specify the cloud store container folder name for artifact (media) files storage.
Default value: "artifact"

| | contentTypeToPrimaryCategory |

Used to specify the mapping between contentType and primaryCategory attributes using which object metadata is populated with the missing attribute among two.
Default value: \[]

| | compositesearch.index.name |

Used to specify the composite search index name where collection object nodes are synced with updated metadata.
Default value: "compositesearch"

| | search.document.type |

Used to specify the ElasticSearch document index type using which collection object nodes are synced with updated metadata.
Default value: "cs"

| | master.category.validation.enabled |

Used to specify whether object getting published is to be enriched with framework metadata.
Default value: "Yes"

| | service.print.basePath | NOT USED | | mimetype.allowed\_extensions.word |

Used to specify the list of file extensions allowed for uploaded content object.
Default value: \["doc", "docx", "ppt", "pptx", "key", "odp", "pps", "odt", "wpd", "wps", "wks"]

| **Sample kafka event:** ``` { "eid": "BE_JOB_REQUEST", "ets": 1619527882745, "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36", "actor": { "id": "content-publish", "type": "System" }, "context": { "channel": "ORG_001", "pdata": { "id": "org.sunbird.platform", "ver": "1.0" }, "env": "dev" }, "object": { "id": "do_11363368845198131217", "ver": "1619153418829" }, "edata": { "publish_type": "public", "metadata": { "identifier": "do_11363368845198131217", "mimeType": "application/vnd.ekstep.ecml-archive", "objectType": "Content", "lastPublishedBy": "", "pkgVersion": 4 }, "action": "republish", "iteration": 1 } } ``` {% hint style="info" %} ***Dependency:*** **Services:** Search Service - composite search API **Jobs:** 'csp-migrator' {% endhint %} ### :stars: live-video-stream-generator: **Kafka Topic:** ``` kafka { input.topic = "{{ env_name }}.live.video.stream.request" groupId = "{{ env_name }}-live-video-stream-generator-group" } ``` **Job configuration variables:** | Variable | Purpose | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | service.content.basePath | Used to configure Content Service API base path for accessing System Update API (/content/v4/system/update/ - not exposed publicly). | | | | **Sample kafka event:** ``` { "eid": "BE_JOB_REQUEST", "ets": 1649174914686, "mid": "LP.1649174914686.02c7ac3d-f7b2-46be-9771-e91645ecd632", "actor": { "id": "Post Publish Processor", "type": "System" }, "context": { "pdata": { "ver": "1.0", "id": "org.sunbird.platform" }, "channel": "01272777697873100812", "env": "sunbirdstaging" }, "object": { "ver": "1649169311270", "id": "do_21350999440018636811684" }, "edata": { "action": "post-publish-process", "iteration": 1, "identifier": "do_21350999440018636811684", "channel": "01272777697873100812", "mimeType": "application/vnd.ekstep.content-collection", "contentType": "Course", "pkgVersion": 1, "status": "Live", "name": "CourseMElenor", "trackable": { "enabled": "Yes", "autoBatch": "No" } } } ``` {% hint style="info" %} ***Dependency:*** **Services:** Content Service - system update API **Jobs:** 'live-node-publisher' {% endhint %} ###