====== Gitlab issues ======

===== One project failed its last repository check.  =====

<code bash>cat /var/log/gitlab/gitlab-rails/repocheck.log</code>

''error in blob 31ae455954e2ab24f917f0cad3c9dd9b0e029be4: gitmodulesMissing: unable to read .gitmodules blob''

Solution:
<code bash>
su - git
cd /home/git-data/repositories/\@hashed\..\..\...git
git fsck
git gc
git fsck
</code>


===== Failed to add control inotify watch descriptor for control group =====
<code bash>
gru 28 18:36:15 gitlab systemd[1]: user@997.service: Failed to add control inotify watch descriptor for control group /user.slice/user-997.slice/user@997.service: No space left on device
gru 28 18:36:15 gitlab systemd[1]: user@997.service: Failed to add memory inotify watch descriptor for control group /user.slice/user-997.slice/user@997.service: No space left on device
</code>

===== Permission denied, please try again. =====

rate_limit_gitlab_shell:

Gitlab runner cannot clone repository:

<code>
Permission denied, please try again.
Permission denied, please try again.
git@gitlab.example.com: Permission denied (publickey,password).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
</code>

**Tracing**
Upon SSH connection ''gitlab-shell'' is executed. Shell makes internal API connection to ''http://unix/api/v4/internal/allowed'' endpoint. 
''Puma'' web server is responsible for handling request.

[[https://docs.gitlab.com/ee/architecture/blueprints/pods/pods-feature-git-access.html#22-git-clone-over-ssh|2.2. Git clone over SSH]]

**HINT:** [[https://docs.gitlab.com/ee/administration/operations/puma.html#gitlab-api-is-not-accessible|GitLab: API is not accessible]]

**Reason:**
Rate limit introduced with v14.8 with (28 Jan, 2022) [[https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79419|Enable Gitlab Shell rate limit by default]].

More info:
  * [[https://docs.gitlab.com/ee/security/rate_limits.html#git-operations-using-ssh|Git operations using SSH]]
  * [[https://docs.gitlab.com/ee/administration/feature_flags.html|Enable and disable GitLab features deployed behind feature flags]]

How to disable rate limits - how to disable ''rate_limit_gitlab_shell'':
<code bash>gitlab-rails console</code>
and then
<code ruby>


Feature.enabled?(:rate_limit_gitlab_shell)

Feature.disable(:rate_limit_gitlab_shell)

Feature.enabled?(:rate_limit_gitlab_shell)
</code>

Still problems, try to figure out:
  * Gitlab-shell rate limiter: [[https://docs.gitlab.com/ee/development/gitlab_shell/#rate-limiting|Rate limiting]]
  * There is another rate limiter in Gitaly: ''Gitaly also has a rate-limiter in place, but calls are never made to Gitaly if the rate limit is exceeded in GitLab Shell (Rails).''
    * It is disabled by default. Need to specify RPC endpoint and params to create Gitaly limiter: [[https://gitlab.com/gitlab-org/gitaly/-/blob/master/internal/middleware/limithandler/concurrency_limiter.go#L144|L144 of code]]
    * [[https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/backpressure.md|Request limiting in Gitaly]]
    * [[https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#limit-rpc-concurrency|Limit RPC concurrency]]
      * ''When these limits are reached, users are disconnected.''
    * [[https://docs.gitlab.com/ee/administration/gitaly/monitoring.html#monitor-gitaly-rate-limiting|Monitor Gitaly rate limiting]]
    * [[https://docs.gitlab.com/ee/administration/gitaly/monitoring.html#monitor-gitaly-concurrency-limiting|Monitor Gitaly concurrency limiting]]
    * [[https://docs.gitlab.com/ee/administration/gitaly/troubleshooting.html#500-and-fetching-folder-content-errors-on-repository-pages|Troubleshooting Gitaly and Gitaly Cluster]]
    * [[https://docs.gitlab.com/ee/administration/gitaly/reference.html|Gitaly reference]]
    * [[https://gitlab.com/gitlab-org/gitaly/-/tree/master|Gitaly source]]
  * [[https://docs.gitlab.com/ee/user/admin_area/settings/user_and_ip_rate_limits.html#allow-specific-users-to-bypass-authenticated-request-rate-limiting|Allow specific users to bypass authenticated request rate limiting]]