GitLab v17.7.0, introduced JWT for CI jobs to improve the security of CI jobs. This subsequently broke job execution in some cases, causing Jobs to get stuck in the Running state, eventually timing out.
On affected Runners, the effect of this issue can be seen by debugging the runner.
gitlab-runner stop && \
gitlab-runner --debug runAfter starting a Job on the runner, several WARNING: Appending trace to coordinator... errors will appear indicating a 500 error code from GitLab.
Watch GitLab's logs while the Runner is attempting to run a Job
tail -f /var/log/gitlab/gitlab-rails/api_json.logThe following log line indicates the real issue
...
"exception.class":"RuntimeError","exception.message":"CI job token signing key is not set"
...Login to the Rails console
gitlab-rails consoleConfirm the issue - if the following command returns nil, the key will need to be generated manually.
Gitlab::CurrentSettings.ci_job_token_signing_keyGenerate the key
settings = Gitlab::CurrentSettings.current_application_settings
settings.ci_job_token_signing_key = OpenSSL::PKey::RSA.new(2048).to_pem
settings.save