Configuration
This page lists the knowlg jobs' specific kafka topic, functionality configurations, sample event and dependencies (if any).
Configuration of all knowledge-platform-jobs during the time of deployment is referred from sunbird-learning-platform repository. However, configuration for local setup is referred from respective job folders in knowledge-platform-jobs repository.
🌠 dialcode-context-updater:
Kafka Topic:
Job configuration variables:
Variable | Purpose |
---|---|
dialcode_context_updater.actions | Used to identify dial code context update action |
dialcode_context_updater.search_mode | Used to set search mode for search API |
dialcode_context_updater.context_map_path | Used to specify the path of context Mapping file. File used to specify the field mapping of context schema.jsonld to sunbird content/collection schema.json |
dialcode_context_updater.identifier_search_fields | Used to specify the search fields when the content/collection details is fetched for primary category. |
dialcode_context_updater.dial_code_context_read_api_path | Used to specify the api endpoint of the DIAL service read context API (/dialcode/v4/read) |
dialcode_context_updater.dial_code_context_update_api_path | Used to specify the api endpoint of the DIAL service Update context API (/dialcode/v4/update) |
service.search.basePath | Used to specify Search service base URL. |
service.dial_service.basePath | Used to specify DIAL service base URL. |
es_sync_wait_time | Used to specify wait time for collection nodes data to sync to ES after collection publish to reflect in search service results. |
Sample kafka event:
Dependency: Services: 1. Search Service - composite search API
2. DIAL service - DIAL context read and update API
🌠 cassandra-data-migration:
Kafka Topic:
Job configuration variables:
Variable | Purpose |
---|---|
migrate.keyspace | Used to specify the keyspace of the cassandra database in which data is to be migrated. |
migrate.table | Used to specify the table in the keyspace configured above; in which data is to be migrated. |
migrate.primary_key_column | Used to specify the primay key column name of the table and the keyspace configured above; in which data is to be migrated. |
migrate.primary_key_column_type | Used to specify the primay key column datatype in the table and the keyspace configured above; in which data is to be migrated. |
migrate.column_to_migrate | Used to specify the name of the column to be migrated from the table and the keyspace configured above. |
migrate.column_to_migrate_type | Used to specify the datatype of the column to be migrated from the table and the keyspace configured above. |
migrate.key_value_strings_to_migrate | Used to specify the list of strings to be migrated in the column data. |
Sample kafka event:
🌠 csp-migrator:
Kafka Topic:
Job configuration variables:
Variable | Purpose |
---|---|
key_value_strings_to_migrate | Used to specify the list of strings to be migrated in the fields' data. |
neo4j_fields_to_migrate | Used to specify the neo4j fields which are to be migrated based on the objectType |
cassandra_fields_to_migrate | Used to specify the columns of assessmentItem which are to be migrated. |
hierarchy.keyspace | Used to specify the keyspace containing collection hierarchy data table |
hierarchy.table | Used to specify collection hierarchy data table |
content.keyspace | Used to specify the keyspace containing content data table |
content.content_table | Used to specify content data table |
content.assessment_table | Used to specify assessment data table |
questionset.hierarchy.keyspace | Used to specify the keyspace containing question set hierarchy table. |
questionset.hierarchy.table | Used to specify the question set hierarchy table |
migrationVersion | Used to specify the migration version |
Sample kafka event:
Dependency: Sync tool (Jenkins Job - to trigger events for each content to be migrated).
🌠 live-node-publisher:
Kafka Topic:
Job configuration variables:
Variable | Purpose |
---|---|
content.bundleLocation | Used to specify local/server folder location where artifacts are to be downloaded for ECAR bundling. Default value: "/data/contentBundle/" |
content.isECARExtractionEnabled | Used to specify if the ECAR extraction is to be enabled to object 'version' and 'latest' cloud location using its 'snapshot' version. Default value: true |
content.retry_asset_download_count | Used to specify number of times download attempt for assets part of content/collection object is to be done till it is successfully downloaded. Default value: 1 |
content.tmp_file_location | NOT USED |
content.objectType | Used to specify list of valid objectTypes supported for publishing. Default value: ["Content", "ContentImage"] |
content.mimeType | Used to specify list of valid mimeTypes supported for publishing. Default value: ["application/pdf"] |
content.asset_download_duration | Used to specify time in seconds to wait for the asset download request to respond. Default value: "60 seconds" |
content.stream.enabled | Used to check if streaming is enabled for published objects. If it is enabled, content rendenring is done using 'streamingUrl' attribute else via 'artifactUrl' Default value: false |
content.stream.mimeType | Used to check if the mimeType of the object being published is of streamable type. If yes, event for video-stream-generator job is generated. Default value: ["video/mp4"] |
content.artifact.size.for_online | Used to set the maximum size of the object (in bytes) that can be played by downloading beyond which "contentDisposition" is set to "online-only". Default value: 209715200 |
content.downloadFiles.spine | Used to specify list of attributes that store asset Urls which are to be downloaded from the mentioned Urls while packaging SPINE ECAR. Default value: ["appIcon"] |
content.downloadFiles.full | Used to specify list of attributes that store asset Urls which are to be downloaded from the mentioned Urls while packaging FULL ECAR. Default value: ["appIcon", "grayScaleAppIcon", "artifactUrl", "itemSetPreviewUrl", "media"] |
content.nested.fields | Used to specify the list of object properties that are of object types with nested attributes. Default value: ["badgeAssertions","targets","badgeAssociations"] |
cloud_storage.folder.content | Used to specify the cloud store container folder name for content file storage/extraction etc. Default value: "container" |
cloud_storage.folder.artifact | Used to specify the cloud store container folder name for artifact (media) files storage. Default value: "artifact" |
contentTypeToPrimaryCategory | Used to specify the mapping between contentType and primaryCategory attributes using which object metadata is populated with the missing attribute among two. Default value: [] |
compositesearch.index.name | Used to specify the composite search index name where collection object nodes are synced with updated metadata. Default value: "compositesearch" |
search.document.type | Used to specify the ElasticSearch document index type using which collection object nodes are synced with updated metadata. Default value: "cs" |
master.category.validation.enabled | Used to specify whether object getting published is to be enriched with framework metadata. Default value: "Yes" |
service.print.basePath | NOT USED |
mimetype.allowed_extensions.word | Used to specify the list of file extensions allowed for uploaded content object. Default value: ["doc", "docx", "ppt", "pptx", "key", "odp", "pps", "odt", "wpd", "wps", "wks"] |
Sample kafka event:
Dependency:
Services: Search Service - composite search API
Jobs: 'csp-migrator'
🌠 live-video-stream-generator:
Kafka Topic:
Job configuration variables:
Variable | Purpose |
---|---|
service.content.basePath | Used to configure Content Service API base path for accessing System Update API (/content/v4/system/update/ - not exposed publicly). |
Sample kafka event:
Dependency:
Services: Content Service - system update API
Jobs: 'live-node-publisher'
Last updated