meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Next revisionBoth sides next revision
git:gitlab:issues [2022/12/28 14:38] – niziakgit:gitlab:issues [2022/12/28 22:39] – niziak
Line 1: Line 1:
 ====== Gitlab issues ====== ====== Gitlab issues ======
  
 +rate_limit_gitlab_shell:
 +
 +===== Failed to add control inotify watch descriptor for control group =====
 +<code bash>
 +gru 28 18:36:15 gitlab systemd[1]: user@997.service: Failed to add control inotify watch descriptor for control group /user.slice/user-997.slice/user@997.service: No space left on device
 +gru 28 18:36:15 gitlab systemd[1]: user@997.service: Failed to add memory inotify watch descriptor for control group /user.slice/user-997.slice/user@997.service: No space left on device
 +</code>
  
 ===== Permission denied, please try again. ===== ===== Permission denied, please try again. =====
Line 14: Line 21:
 and the repository exists. and the repository exists.
 </code> </code>
 +
 +**Tracing**
 +Upon SSH connection ''gitlab-shell'' is executed. Shell makes internal API connection to ''http://unix/api/v4/internal/allowed'' endpoint. 
 +''Puma'' web server is responsible for handling request.
 +
 +[[https://docs.gitlab.com/ee/architecture/blueprints/pods/pods-feature-git-access.html#22-git-clone-over-ssh|2.2. Git clone over SSH]]
 +
 +**HINT:** [[https://docs.gitlab.com/ee/administration/operations/puma.html#gitlab-api-is-not-accessible|GitLab: API is not accessible]]
  
 **Reason:** **Reason:**
Line 35: Line 50:
 </code> </code>
  
 +Still problems, try to figure out:
 +  * Gitlab-shell rate limiter: [[https://docs.gitlab.com/ee/development/gitlab_shell/#rate-limiting|Rate limiting]]
 +  * There is another rate limiter in Gitaly: ''Gitaly also has a rate-limiter in place, but calls are never made to Gitaly if the rate limit is exceeded in GitLab Shell (Rails).''
 +    * It is disabled by default. Need to specify RPC endpoint and params to create Gitaly limiter: [[https://gitlab.com/gitlab-org/gitaly/-/blob/master/internal/middleware/limithandler/concurrency_limiter.go#L144|L144 of code]]
 +    * [[https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/backpressure.md|Request limiting in Gitaly]]
 +    * [[https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#limit-rpc-concurrency|Limit RPC concurrency]]
 +      * ''When these limits are reached, users are disconnected.''
 +    * [[https://docs.gitlab.com/ee/administration/gitaly/monitoring.html#monitor-gitaly-rate-limiting|Monitor Gitaly rate limiting]]
 +    * [[https://docs.gitlab.com/ee/administration/gitaly/monitoring.html#monitor-gitaly-concurrency-limiting|Monitor Gitaly concurrency limiting]]
 +    * [[https://docs.gitlab.com/ee/administration/gitaly/troubleshooting.html#500-and-fetching-folder-content-errors-on-repository-pages|Troubleshooting Gitaly and Gitaly Cluster]]
 +    * [[https://docs.gitlab.com/ee/administration/gitaly/reference.html|Gitaly reference]]
 +    * [[https://gitlab.com/gitlab-org/gitaly/-/tree/master|Gitaly source]]
 +  * [[https://docs.gitlab.com/ee/user/admin_area/settings/user_and_ip_rate_limits.html#allow-specific-users-to-bypass-authenticated-request-rate-limiting|Allow specific users to bypass authenticated request rate limiting]]