id
type
5 (blog/news article)
status
21 (imported old-v2, waiting for another import)
review version
0
cleanup version
0
pending deletion
0 (-)
created at
2025-05-07 03:32:36
updated at
2025-11-07 14:06:39
url
https://blog.mestwin.net/robots-txt-spring-boot/
url length
48
url crc
11432
url crc32
142290088
location type
1 (url matches target location, page_location is empty)
canonical status
10 (verified canonical url)
canonical page id
domain id
domain tld
0
domain parts
0
originating warc id
-
originating url
https://blog.mestwin.net/post-sitemap.xml
source type
1 (sitemap)
server ip
Publication date
2025-11-07 14:06:39
Fetch attempts
1
Original html size
40643
Normalized and saved size
14337
title
Serving robots.txt in Spring Boot application - Mestwin Blog
excerpt
content

 
 
 
 
 What is robots.txt file? The robots.txt file is a text file standard widely used in websites. Basically it tells search engine robots and other crawlers which pages and areas on your websites are allowed/disallowed for them to follow. Serving robots.txt in Spring Boot Depending on your project configuration there are couple of ways to serve robots.txt file using Spring Boot. In my case I was using Thymeleaf as default template engine but for robots.txt serving you’ll need to create simple method in your controller which serves plain text response. Let’s say we don’t want robots to visit our /admin path. So we need to specify it in our robots.txt file. You can do it like this: User-agent: * Disallow: /admin One of the quickest way to serve robots.txt file is to create a method in your controller class which returns a String object with proper content. Please note that your robots.txt file should be placed in root path of your website. @Reques...
author
updated
1768404013
block type
0
extracted fields
233
extracted bits
featured image
title
full content
content was extracted heuristically
OpenGraph suggests this is an article
detected location
0
detected language
1 (English)
category id
-
index version
1
paywall score
0
spam phrases
0
text nonlatin
0
text cyrillic
0
text characters
1150
text words
242
text unique words
128
text lines
1
text sentences
11
text paragraphs
1
text words per sentence
22
text matched phrases
0
text matched dictionaries
0
image author