Skip to main content
Version: 16.1 🔚

Configuration

Utilities​

Utilities settings are available for all modules.

KeyTypeDefault ValuePossible ValuesFor
LOGGER_LEVELstrERRORDEBUG, INFO, WARN, ERROR, FATAL
LOGGER_TO_FILEboolFalseTrue/False
LOGGER_FILEstrtemp/rotating.log
LOGGER_FILE_SIZEint10242880
LOGGER_FILE_BACKUP_COUNTint5
LOGGER_FILE_ENCODINGstrutf-8
LOGGER_JSON_FORMATboolFalseTrue/False
LOGGER_FORMATstr%(asctime)s - %(process)d - %(threadName)s - %(name)s - %(levelname)s - %(message)s

Storage​

Storage settings are available for all modules.

KeyTypeDefault ValuePossible ValuesFor
DECIMAL_INTEGRAL_DIGITSint24
DECIMAL_FRACTION_DIGITSint8
tip

Settings of decimal digits are used in MySQL only for cast string value to decimal value.

Meta​

KeyTypeDefault ValuePossible ValuesFor
SUPER_ADMIN_TENANT_IDstr1
SUPER_ADMIN_USER_IDstr1
SUPER_ADMIN_USER_NAMEstrimma-super
SUPER_ADMIN_USER_NICKNAMEstrIMMA Super
META_STORAGE_TYPEstrmysqlmysql, oracle, mongodb, mssql
META_STORAGE_USER_NAMEstrwatchmen
META_STORAGE_PASSWORDstrwatchmen
META_STORAGE_HOSTstrlocalhost
META_STORAGE_PORTint3306
META_STORAGE_NAMEstrwatchmen
META_STORAGE_ECHOboolFalseTrue/FalseEnable meta storage sql logging
SNOWFLAKE_DATA_CENTER_IDint00 ~ 3Data center id
SNOWFLAKE_WORKER_IDint00 ~ 1023Worker id
SNOWFLAKE_COMPETITIVE_WORKERSboolTrueTrue/FalseEnable competitive snowflake worker
SNOWFLAKE_COMPETITIVE_WORKER_HEART_BEAT_INTERVALint60Competitive worker heart beat interval, in seconds
SNOWFLAKE_COMPETITIVE_WORKER_CREATION_RETRY_TIMESint3Competitive worker creation max retry times
SNOWFLAKE_COMPETITIVE_WORKER_RESTART_ON_SHOWDOWNboolFalseTrue/FalseCompetitive worker restart automatically on shutdown
ENGINE_INDEXboolTrueTrue/FalseEnable meta data auto index

Data Kernel​

KeyTypeDefault ValuePossible ValuesFor
STORAGE_ECHOboolFalseTrue/FalseEnable data storage sql logging
FULL_DATETIME_FORMATSSet[str]%Y%m%d%H%M%S%f,
%d%m%Y%H%M%S%f,
%m%d%Y%H%M%S%f
Follow python date time formats
DATETIME_FORMATSSet[str]%Y%m%d%H%M%S,
%d%m%Y%H%M%S,
%m%d%Y%H%M%S,
%Y%m%d%H%M,
%d%m%Y%H%M,
%m%d%Y%H%M
Follow python date time formats
DATE_FORMATSSet[str]%Y%m%d,
%d%m%Y,
%m%d%Y
Follow python date time formats
TIME_FORMATSSet[str]%H%M%S,
%H%M
Follow python date time formats
ABANDON_DATE_TIME_ON_PARSE_FAILboolFalseTrue/FalseAbandon value or not when it cannot be parsed to date or time by designated factor type,
works only on topic data.
ENCRYPT_AES_KEYstr32 chars strDefault AES key,
MUST BE REPLACED
ENCRYPT_AES_IVstr16 chars strDefault AES iv,
MUST BE REPLACED
IGNORE_DEFAULT_ON_RAWboolTrueTrue/FalseDefault value of factor will be ignored on the raw topic
KERNEL_CACHEboolTrueTrue/FalseEnable kernel cache, keep it enabled in production
KERNEL_CACHE_HEART_BEATboolTrueTrue/FalseEnable kernel cache heart beat, keep it disabled in production
KERNEL_CACHE_HEART_BEAT_INTERVALint60Cache heart beat interval, in seconds
SYNC_TOPIC_TO_STORAGEboolFalseTrue/FalseSynchronize topic structure change to storage entity
REPLACE_TOPIC_TO_STORAGEboolFalseTrue/FalseForce replace existing topic entity (drop and recreate)

Data Surface​

KeyTypeDefault ValuePossible ValuesFor
TRUNCATE_TOPIC_DATAboolTrueTrue/FalseEnable topic data truncate rest api

Inquiry Trino​

KeyTypeDefault ValuePossible ValuesFor
TRINO_HOSTstr127.0.0.1
TRINO_PORTint5678
TRINO_USERstradmin
TRINO_PASSWORDOptional[str]None

Inquiry Kernel​

KeyTypeDefault ValuePossible ValuesFor
USE_STORAGE_DIRECTLYboolTrueTrue/FalseEnable use storage directly for inquiry, save Trino cost when all topics are from same storage.
TRINOboolTrueTrue/FalseEnable Trino

Inquiry Surface​

KeyTypeDefault ValuePossible ValuesFor
DATASET_PAGE_MAX_ROWSint10000Default max rows for dataset page

Pipeline Kernel​

KeyTypeDefault ValuePossible ValuesFor
DECRYPT_FACTOR_VALUEboolFalseTrue/FalseDecrypt encrypted factor value on read
PIPELINE_PARALLEL_ACTIONS_IN_LOOP_UNITboolFalseTrue/False
PIPELINE_PARALLEL_ACTIONS_COUNTint8
PIPELINE_PARALLEL_ACTIONS_DASK_TEMP_DIROptional[str]None
PIPELINE_PARALLEL_ACTIONS_DASK_USE_PROCESSboolTrueTrue/False
PIPELINE_STANDARD_EXTERNAL_WRITERboolTrueTrue/False
PIPELINE_ELASTIC_SEARCH_EXTERNAL_WRITERboolFalseTrue/False
PIPELINE_UPDATE_RETRYboolTrueTrue/FalseEnable pipeline update retry if it is failed on optimistic lock
PIPELINE_UPDATE_RETRY_TIMESint3Optimistic lock retry times
PIPELINE_UPDATE_RETRY_INTERVALint10Retry interval in milliseconds
PIPELINE_UPDATE_RETRY_FORCEboolTrueTrue/FalseEnable force retry after all retries failed
PIPELINE_ASYNC_HANDLE_MONITOR_LOGboolTrueTrue/FalseHandle monitor log (might with pipelines) asynchronized

Pipeline Surface​

KeyTypeDefault ValuePossible ValuesFor
RABBITMQ_CONNECTORboolFalseTrue/FalseEnable trigger pipeline from RabbitMQ
RABBITMQ_HOSTstrEmpty string
RABBITMQ_PORTstr5672
RABBITMQ_USERNAMEstrEmpty string
RABBITMQ_PASSWORDstrEmpty string
RABBITMQ_VIRTUALHOSTstrEmpty string
RABBITMQ_QUEUEstrEmpty string
RABBITMQ_DURABLEboolTrueTrue/False
RABBITMQ_AUTO_DELETEboolFalseTrue/False
KAFKA_CONNECTORboolFalseTrue/FalseEnable trigger pipeline from Kafka
KAFKA_BOOTSTRAP_SERVERstrlocalhost:9092
KAFKA_TOPICSstrEmpty string

Rest​

KeyTypeDefault ValuePossible ValuesFor
APP_NAMEstrWatchmen REST AppApplication name
VERSIONstr16.0.0Application version
DESCRIPTIONstrA lighter platform for data analyticsA short description of application
API_VERSION_STRstrEmpty strFor FastAPI docs
JWT_SECRET_KEYstr43 chars stringDefault JWT key,
MUST BE REPLACED
JWT_ALGORITHMstrHS256
ACCESS_TOKEN_EXPIRE_MINUTESint11520
CORSboolTrueTrue/False
CORS_ALLOWED_ORIGINSSet[str]*
CORS_ALLOW_CREDENTIALSboolTrue
CORS_ALLOWED_METHODSSet[str]*
CORS_ALLOWED_HEADERSSet[str]*
PROMETHEUSboolFalseTrue/False
PROMETHEUS_CONTEXTstr/metrics

Rest Doll​

KeyTypeDefault ValuePossible ValuesFor
APP_NAMEstrWatchmen DollOverwrite Rest's
TUPLE_DELETABLEboolFalseTrue/FalseEnable tuple deletable rest apis
CREATE_PIPELINE_MONITOR_TOPICS_ON_TENANT_CREATEboolTrueTrue/FalseEnable create pipeline monitor log topics on create tenant
CREATE_DQC_TOPICS_ON_TENANT_CREATEboolTrueTrue/FalseEnable create DQC topics on create tenant
SSO_ONboolFalseTrue/FalseEnable SSO integration
SSO_PROVIDERstrsaml2saml2single sign-on (SSO) login protocol
SAML_STRICTboolTrueTrue/FalseIf strict is True, then the Python Toolkit will reject unsigned
SAML_DEBUGboolTrueTrue/FalseEnable debug mode (outputs errors).
SAML_IDP_ENTITY_IDstrIdentifier of the IdP entity
SAML_IDP_SSO_URLstrURLURL Target of the IdP where the Authentication Request Message
SAML_IDP_SSO_BINDINGstrHTTP-RedirectSAML binding standardSAML protocol binding to be used when returning
SAML_IDP_X509CERTstrPublic X.509 certificate of the IdP
SAML_SP_ENTITY_IDstrIdentifier of the SP entity
SAML_SP_ASSERT_URLstrURLURL Location where the <Response> from the IdP will be returned
SAML_SP_ASSERT_BINDINGstrHTTP-RedirectSAML binding standardSAML protocol binding to be used when returning
SAML_SP_X509CERTstrX.509 certificate of the SP

DQC​

KeyTypeDefault ValuePossible ValuesFor
MONITOR_JOBSboolFalseTrue/FalseEnable DQC jobs
MONITOR_JOB_TRIGGERstrcronJob trigger scheduler
MONITOR_JOB_DAILY_DAY_OF_WEEKstrmon-sunCron daily
MONITOR_JOB_DAILY_HOURSint0Trigger hour of daily job
MONITOR_JOB_DAILY_MINUTESint1Trigger minute of daily job
MONITOR_JOB_WEEKLY_DAY_OF_WEEKstrsunCron weekly, assume week starts from sunday
MONITOR_JOB_WEEKLY_HOURSint0Trigger hour of daily job
MONITOR_JOB_WEEKLY_MINUTESint1Trigger hour of daily job
MONITOR_JOB_MONTHLY_DAYint1Cron mongly
MONITOR_JOB_MONTHLY_HOURSint0Trigger hour of daily job
MONITOR_JOB_MONTHLY_MINUTESint1Trigger hour of daily job
MONITOR_RESULT_PIPELINE_ASYNCboolFalseTrue/FalseEnable asynchronized pipeline trigger for data which produced by DQC rules

Rest DQC​

KeyTypeDefault ValuePossible ValuesFor
APP_NAMEstrWatchmen DQCOverwrite Rest's