close
Jump to content

Wikidata Platform

From Wikitech

Future home for technical documentation for the Wikidata Platform team.

Components

Production

  • WDQS streaming producer
  • WDQS streaming consumer
  • WDQS streaming updater/reconciler
  • Blazegraph
  • Blazegraph extensions
  • Query UI
  • Airflow DAGs

Experiments, testing, analytics

Query data

See Wikidata_Platform/Query_Data for tables you can use to analyze WDQS requests.

Monitoring, SLO

Runbooks

These runbooks are intended for WDP engineers:

SREs also maintain various runbooks for WDQS alerts that might be of interest for WDP engineers as well.

Alerts w/ config links and their corresponding runbooks
Alert Runbook
BlazegraphFailedClientRatioIncrease Wikidata_Query_Service/Runbook#Increased_ratio_of_failed_requests_(4xx)
BlazegraphFailedServerRatioIncrease Wikidata_Query_Service/Runbook/High_replication_lag_and_query_timeout
BlazegraphFreeAllocatorsDecreasingRapidly, BlazegraphFreeAllocatorsRunningOut Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly
BlazegraphMainTriplesDeltaIncreasing, BlazegraphCategoryTriplesDeltaIncreasing Wikidata_Query_Service/Runbook#Incomplete/problematic_data_reload
CategoriesQueryServiceUpdateLagTooHigh Wikidata_Query_Service/Runbook#Categories_update_lag
ElevatedMaxLagWDQS Wikidata_Query_Service/Runbook#Timeouts and Wikidata_Query_Service/Runbook/High_replication_lag_and_query_timeout
RdfStreamingUpdaterFlinkJobUnstable Wikidata_Query_Service/Streaming_Updater#The_job_is_not_starting
RdfStreamingUpdaterFlinkProcessingLatencyIsHigh Wikidata_Query_Service/Streaming_Updater#The_job_processing_latency_is_high
RdfStreamingUpdaterHighConsumerUpdateLag Wikidata_Query_Service/Streaming_Updater#The_consumers_are_backlogged
WdqsStreamingUpdaterFlinkJobNotRunning Wikidata_Query_Service/Streaming_Updater#The_job_is_not_running