简介
今天去定位一个nodeport的问题,发现curl 127.0.0.1:32000 访问nodeport的时候会规律的hang住,本来以为是后端服务的问题,但是curl管理ip:nodeport 是正常的。这个就奇怪了,深入研究了下发现 ipvs模式下是不支持这样访问的,如果想使用 localhost: 得使用iptables模式。
下面是一个歪果仁的解释
After a little dig into the kernel, failing to connect localhost: can be explained.
Assume we are visiting http://127.0.0.1:, every packet will first pass through ip_vs_nat_xmit. Then after some check, it runs to https://github.com/torvalds/linux/blob/v4.18/net/netfilter/ipvs/ip_vs_xmit.c#L756. __ip_vs_get_out_rt is used to search for route to remote server, k8s pods in our case. Take a deeper look at __ip_vs_get_out_rt, after validating route cache or finding route via do_output_route4, it comes to crosses_local_route_boundary to judge whether the searched route can pass cross-local-route-boundary check.
Copy the code of crosses_local_route_boundary here and go deeper.
static inline bool crosses_local_route_boundar服务器托管网y(int skb_af, struct sk_buff *skb,
int rt_mode,
bool new_rt_is_local)
{
bool rt_mode_allow_local = !!(rt_mode & IP_VS_RT_MODE_LOCAL);
bool rt_mode_allow_non_local = !!(rt_mode & IP_VS_RT_MODE_NON_LOCAL);
bool rt_mode_allow_redirect = !!(rt_mode & IP_VS_RT_MODE_RDR);
bool source_is_loopback;
bool old_rt_is_local;
#ifdef CONFIG_IP_VS_IPV6
/* omit ipv6 */
#endif
{
source_is_loopback = ipv4_is_loopback(ip_hdr(skb)->saddr);
old_rt_is_local = skb_rtable(skb)->rt_flags & RTCF_LOCAL;
}
if (unlikely(new_rt_is_local)) {
if (!rt_mode_allow_local)
return true;
if (!rt_mode_allow_redirect && !old_rt_is_local)
return true;
} else {
if (!rt_mode_allow_non_local)
return true;
if (source_is_loopback)
return true;
}
return false;
}
For nat mode of IPVS, rt_mode is assigned as IP_VS_RT_MODE_LOCAL | IP_VS_RT_MODE_NON_LOCAL | IP_VS_RT_MODE_RDR as https://github.com/torvalds/linux/blob/v4.18/net/netfilter/ipvs/ip_vs_xmit.c#L757-L759 indicates and new_rt_is_local is 0 due to https://github.com/torvalds/linux/blob/v4.18/net/netfilter/ipvs/ip_vs_xmit.c#L363.
source_is_loopback will be true because source address ip_hdr(skb)->saddr is 127.0.0.1. The five booleans defined in crosses_local_route_boundary will all be true in this case. So we finally fall into here
if (source_is_loopback)
return true;
and never pass the cross-local-route-boundary check.
解决方案
目前看如果想这样使用1是修改内核,这个可能比较难实现,另一个方案是修改路由表,修改源ip的地址,也是一种曲线救国的方式
sudo ip route change 127.0.0.1 dev lo proto kern服务器托管网el scope host src node ip> table local
引用
https://github.com/kubernetes/kubernetes/issues/96879
https://github.com/kubernetes/kubernetes/issues/67730
https://serverfault.com/questions/851221/what-is-the-local-routing-table-used-for
服务器托管,北京服务器托管,服务器租用 http://www.fwqtg.net