Read the permissions.
We evaluated seven robots profiles for each of three root URLs. No matching prohibition applied to those paths.
INDEPENDENT CASE STUDY · 06 SEP 2026
GitLab: AI crawler access.
We checked GitLab’s commercial homepage, documentation and handbook. Their robots rules allowed the sampled root paths. Our crawler user-agent requests received different responses across the three sites.
3root URLs examined
18initial crawler-string requests
5HTTP 403 responses, all on the commercial root
Ordinary-IP user-agent simulations. They do not establish authenticated crawler access or GitLab’s presence in AI answers.
Explore the findings ↓THE QUESTION
Do published permissions match our requests?
A public rule and a server response describe different parts of access. We compared both, keeping search crawlers separate from training controls.
Compare the responses.
We sent six crawler user-agent strings to each root. Five requests received HTTP 403, all on the commercial homepage.
Keep the boundary clear.
These requests came from an ordinary IP. The real providers’ crawlers may be treated differently.
THE RECORD
One company. Three different origins.
Initial run: 6 September 2026, approximately 17:41 UTC. HTTP 200 means the request received a success status; HTTP 403 means it was refused. Only headers were inspected, so 200 does not verify the page content.
Swipe the table to compare all three sites →
| User-agent profile | Commercial site about.gitlab.com | Documentation docs.gitlab.com | Handbook handbook.gitlab.com |
|---|---|---|---|
| SEARCH CRAWLER STRINGS | |||
| OAI-SearchBot | 403 | 200 | 200 |
| Googlebot | 200 | 200 | 200 |
| PerplexityBot | 403 | 200 | 200 |
| Claude-SearchBot | 403 | 200 | 200 |
| TRAINING CRAWLER STRINGS | |||
| GPTBot | 403 | 200 | 200 |
| ClaudeBot | 403 | 200 | 200 |
Search profiles: three of four received 403 on the commercial root; all four received 200 on documentation and handbook. The other two refusals were training profiles and are reported separately.
Robots rules: seven profiles per root, all allowed for the sampled path. Google-Extended was evaluated only as a policy token; it has no separate HTTP crawler request.
Rechecks: the commercial-root statuses repeated at 17:42 UTC. Before publication, all 18 crawler-string requests were repeated at 17:47 UTC with the same results. Three browser-user-agent controls returned 200, one per root.
WHAT THE FINDING MEANS
Read each layer in context.
Permission for the tested path.
The fetched files contained no matching prohibition for the seven profiles at /. This is a path-level reading, not permission for every page. The marketing and documentation files restrict other paths; a wildcard group does not explicitly endorse individual bots.
A difference worth checking in logs.
The same crawler strings received different responses across the three origins. Correct rejection of non-provider IPs, bot-management rules, CDN routing or request-specific treatment could explain the refusals. We did not inspect GitLab’s configuration. An authorized owner could compare real crawler visits in server or CDN logs with provider verification guidance.
AI-answer presence was not measured.
These checks do not tell us whether GitLab is mentioned, recommended or cited by an AI product. No AI-answer sessions, traffic or commercial outcomes were measured. Nothing in this record demonstrates a visibility loss, an improvement or a need to rewrite a page.
METHOD & RECEIPTS
Inspect the dated evidence.
We selected GitLab for its public commercial, documentation and handbook sites. The marketing homepage and its robots file were inspected during selection; the other origins and probe profiles were fixed before their results were examined. Selection did not depend on a failed request or missing AI mention.
The initial denominator is 18 requests: three roots × six crawler strings. Its 13 HTTP 200 and five HTTP 403 responses are separate from the six-request commercial-root repeat, the 18-request publication check and browser controls. Seven robots evaluations per root use three files, not 21 independent policies.
Requests used the existing Deep Ocean AI Access Scan engine from a local runtime with bounded HTTPS fetching. Egress IP and location were not independently recorded. The first attempt failed inside a network-restricted local environment; those failures are excluded from website findings. Initial policy bodies were archived immediately afterward; the publication check captured them during the same scan. Two or three nearby observations do not establish long-term or geographic consistency.
CSV includes rounds, URLs, profiles, rules, statuses and UTC timestamps. JSON includes exact user-agent strings and the policy bodies captured during publication verification.
- Public rules: commercial site · documentation · handbook.
- Provider documentation: OpenAI · Google · Perplexity · Anthropic.
Evidence file checksums · This is a technical research record. No intervention was performed.