# Titleman — collaborative AI for US commercial real estate professionals. # AI assistants and their crawlers are welcome on the marketing pages: # accurate answers about Titleman are in everyone's interest. # App, checkout and utility routes are excluded for every crawler below. User-agent: * Disallow: /chat Disallow: /deal-document Disallow: /for-friends Disallow: /first Disallow: /adv Disallow: /_nuxt Disallow: /api Disallow: /_graphql Disallow: /.well-known Disallow: /services Disallow: /checkout/ Disallow: /signup Disallow: /friends Disallow: /drive # ⛔ THESE TWO LINES LOOK LIKE A CONTRADICTION. THEY ARE NOT. READ THIS FIRST. # # Disallow: /pages/ — below, and it is TEMPORARILY LIFTED (16.09.2026) # because every customer page on it is now 404 and # the block is what kept the stale ones in search. # Restore it once they have dropped. See the note # at that line. # Allow: /answers/ — below, and it is the point of this block # # /pages/ and /answers/ are served by the SAME table and the SAME code. What # separates them is WHO the page belongs to, and that is the only thing that # has ever mattered here. # # MEASURED ON PRODUCTION 15.09.2026, conversations_room_state_pages: # # publish = true : 70 rows — 14 ours (user_id IS NULL), 56 CUSTOMERS' # publish = false: 3 598 rows — 403 ours, 3 195 customers' # # Four fifths of the published set is somebody else's paid work: a report a # named client commissioned, about a building they are buying. Owner directive # 11.09.2026 #7 — «клиентские и кейсовые страницы не должны индексироваться» — # covers every one of them, and `Disallow: /pages/` is what holds that line. # # The other 14 are ours: answer pages we wrote to be found. They are the # surface AI engines actually cite — over nine months of ahoy_visits /pages/* # carried 147 visits referred by chatgpt, perplexity and claude against 137 for # THE WHOLE REST OF THE SITE — and this same line was hiding them. # # ⛔ WHY WE DID NOT SIMPLY LIFT IT. Frontend !1691 proposed commenting this line # out. Measured against the table above, that submits 56 customers' reports to # search and breaks directive #7 the day it deploys. There is no version of # lifting this line that does not do so while both kinds share the prefix. # # ⛔ AND WHY WE DID NOT LIST OUR 14 SLUGS AS `Allow:` LINES EITHER. It works # once. The generator writes new pages continuously (403 of ours are waiting # unpublished), and the first page nobody remembers to add here is invisible — # a failure that is silent, permanent and invisible to CI. # # SO THE PATHS WERE SPLIT INSTEAD. A page with no user_id is served from # /answers/, a page with one from /pages/; the prefix is computed from user_id # and from nothing else (Conversations::RoomStatePage#public_path, backend # !1903). There is no flag to set and no list to maintain, so a customer's page # cannot arrive on the allowed path by anybody's oversight. The two lines below # then say exactly one thing: index what we wrote, never what a client paid for. # # ONE THING THIS BLOCK DOES NOT FIX, AND SOMEBODY SHOULD DECIDE IT SEPARATELY. # A Disallow stops a crawler RE-FETCHING a URL, so anything already indexed # under /pages/ stays in the index with its old title until the rule comes off # — the same trap recorded for /case/ twenty lines below. /pages/ now also # answers with X-Robots-Tag: noindex (nuxt.config.ts routeRules), which is the # instruction that actually removes a URL, but no crawler may read it while # this line stands. Taking the already-indexed customer URLs out of search is # therefore still an owner action in Google Search Console, or a deliberate, # time-boxed lift of this line exactly as /case/ got one. # ⛔ `Disallow: /pages/` IS TEMPORARILY LIFTED (16.09.2026, operator shift), for # exactly the reason /case/ got the same treatment on 11.09, and only AFTER the # thing it was protecting stopped existing. # # WHAT CHANGED FIRST. Every customer page under this path is now unpublished: # 56 rows of conversations_room_state_pages that carried publish = true and a # named user_id were set to false on 16.09.2026. They were never published by # their owners — the generator used to read # `page.publish = user.subscriptions_helper.free?` # so every run by every free-plan user became a public, crawlable page, and a # published page also made the whole room anonymously readable. Owner's ruling # of 15 Aug ("мы не публикуем чужие исследования") removed that rule for NEW # pages and deliberately left existing flags alone; this finishes it. # Verified live after the change: every one of the customer URLs that Google # still lists answers 404. The full list is kept in ops_facts under # privacy.client_pages_published_snapshot_before_unpublish_16_09, which is also # the recipe for putting any of them back. # # WHY THE LINE HAS TO COME OFF. A Disallow stops a crawler FETCHING a URL, so # the six customer pages measured in public search on 16.09 — among them an # office valuation naming a London address and square footage — keep their old # title and snippet in the index for as long as this line stands. The 404 and # the X-Robots-Tag: noindex that /pages/** now answers with are the instructions # that actually remove them, and no crawler may read either one while the path # is blocked. Blocking is what KEPT them in search. # # WHAT A CRAWLER FINDS HERE NOW: 404 for every customer page, and a 301 to # /answers/ for the fourteen that are ours. Nothing to index either way. # # ⏳ RESTORE THIS LINE once the customer URLs have dropped out of search — # check with `site:titleman.ai/pages` — so the path goes back to being closed by # default. Until then it is open on purpose. # Disallow: /pages/ # OURS. The 14 published answer pages and everything published after them. # Submitted in /answers-sitemap.xml, which is built from an anonymous GraphQL # field scoped to user_id IS NULL, so it cannot contain a customer's report. Allow: /answers/ Disallow: /slides/ Disallow: /pm/ Disallow: /preview/ Disallow: /hero-lab Disallow: /runs/ # `Disallow: /case/` is TEMPORARILY LIFTED (owner, 11.09.2026: cases must not # be indexed, and the 11-12 already-indexed /case/ URLs taken out of # search). /case/ now answers with X-Robots-Tag: noindex, nofollow, noarchive # on both the ?locale 302 and the page, and the page repeats it in a robots # meta. A crawler blocked here would see NEITHER and would keep the URLs it # already has. RESTORE THIS LINE once they have dropped out. # Disallow: /case/ # feat/report-share-link — a shared report is private to the client it was # sent to. Belt and braces: the page also carries and an X-Robots-Tag header, and /shared/ is absent from # server/routes/site-sitemap.xml.js (a hardcoded list). Disallow: /shared/ # P8-4 (Wave 5) — a shared deal status is private to whoever the link was # sent to. Same three belt-and-braces layers as /shared/ above. Disallow: /shared-deal/ # deal-package D2 (28.09.2026) — «Your deal package» is private to the person the # letter went to. Same three layers as /shared/ above. Disallow: /deal-package/ # scout package (06.10.2026) — /scout-package/ is one Scout search shared by its author. Same layers: this line, the page's # robots meta, an X-Robots-Tag header, no sitemap entry. Disallow: /scout-package/ # upload link (05.10.2026) — /upload/ is one client's drop box for one deal. Same layers: this line, the page's robots meta, # an X-Robots-Tag header, no sitemap entry. Disallow: /upload/ # web deck link (growth1010 A6, 10.10.2026): /deck/ is one tracked link to one recipient. Same layers: this line, # the page's robots meta, an X-Robots-Tag header, no sitemap entry. Disallow: /deck/ # «25 per cent» (01.10.2026) — an analysis holds the address an investor is # looking at and the price they would pay; /25/me is their own list behind the # emailed key. Same three layers as /shared/ above. /25 and /25/pricing stay open. Disallow: /25/deal/ Disallow: /25/me Disallow: /product_hunt Disallow: /product-hunt # 24 random six-character paths used to be listed here: aliases of the old # private /for-friends landing (2025), since removed. They answer 404 and no # route defines them (checked 17.09.2026), and naming a would-be secret path in # a public file announces it rather than hiding it. Removed, not replaced. Sitemap: https://titleman.ai/site-sitemap.xml Sitemap: https://titleman.ai/resources-sitemap.xml Sitemap: https://titleman.ai/answers-sitemap.xml # The index names the lists a crawler cannot find from the three lines above — # the «25 per cent» door (/25/sitemap.xml) among them (product 04.10 13:03Z: /25 # was reachable only by back-links) — plus the tools and maps lists. Sitemap: https://titleman.ai/sitemap.xml