Understand crawling restrictions and indexing instructions, and check the right settings when moving a temporary site to production.
Temporary websites may need different search settings from production. Two commonly confused controls are robots.txt crawling rules and a page's noindex directive. They are not interchangeable switches.
Understand the distinction
Robots.txt guides crawler access; it does not reliably prevent a URL appearing in search results by itself. Noindex instructs a search engine not to index a page. Google must be able to crawl the page to read that directive. Blocking access can prevent a new noindex instruction being seen.
Do not use them as access control
Protect administration areas, customer documents and test data with authentication where needed. Being absent from search results does not prevent someone with the address from opening a page.
Prepare the production checklist
Check HTTPS, working pages, canonical addresses and language links. Then review crawling restrictions and noindex on pages intended for publication. Treat the temporary domain separately.
Verify the actual response
A configuration change does not instantly change search results. Inspect live response headers and HTML, then monitor reprocessing. Diagnose from current output rather than an old screenshot.