LVS-Tun offers the opportunity for your realservers to be anywhere (including on different continents).
Even better would be the ability to determine the closest server (by some internet metric) to any particular client. This has been done at least for http by Horms (of Ultra Monkey fame) with the Super Sparrow Project. Super Sparrow works differently than and is incompatible with LVS, and because of it's success, it's unlikely that anyone will go to the pain of developing a similar capability for LVS.
Super Sparrow uses zebra to determine the number of AS hops between client and server using BGP4 routing information. Documentation on BGP is hard to find (the early zebra docs had none). Horms suggests a BGP Routing Part 1 by Avi Freeman of Akami. It's somewhat Cisco centric and there is no part 2 yet but is applicable to zebra.
However many people thought about geographically distributed LVSs. For historical completeness here are some of their musings.
Michael Sparkszathras@epsilon3.mcc.ac.uk
I'm curious about the physical architecture of a cluster of servers where "the realservers have their own route to the client." (Like in LVS-DR and LVS-Tun) How have people achived this in real life? Does each real server actually have it's own dedicated router and Internet connection? Do you set up groups of real servers where each group shares one line?
It could do or it can share things. We've got 3 LVS based clusters, based around LVS-Tun. The reason for this is because one of the clusters is at a different location (about 200 miles from where I'm sitting) , and this allows us to configure all the realservers in the same way thus:
tunl0:1 - IP of LVS balanced cluster1 tunl0:2 - IP of LVS balanced cluster2 tunl0:3 - IP of LVS balanced cluster3 (remote)
The only machines that ends up getting configured differently then are just the directors.
So whilst machines are nominally in one of the three clusters, if (say) the remote cluster is overloaded, it can take advantage of the extra machines in the other two clusters, which then reply directly back to the client - and vice versa.
In that situation a client in (say) Edinburgh, could request an object via the director at Manchester, and if the machines are overloaded there, have the request forwarded to London, which then requests the object via a network completely separate from the director's and returns the object to the client.
That UK Nat cache likely to be introducing another node at another location in the country at some point in the near future which will be very useful. (The key advantage is that at each location we gain X more Mbit/s of bandwidth to utilise making service better for users.)
Joe
how do I get BGP info from a BGP router to the director?
Lars Marowsky-Breelmb@teuto.net
23 Jul 1999If you telnet to the BGP4 port of the the router running BGP4, and do a "sh ip route www.yahoo.com" for example, you will get something like this
Routing entry for 204.71.192.0/20, supernet Known via "bgp 8925", distance 20, metric 0 Tag 1270, type external Last update from 139.4.18.2 1w5d ago Routing Descriptor Blocks: * 139.4.18.2, from 139.4.18.2, 1w5d ago Route metric is 0, traffic share count is 1 AS Hops 4This address is 4 AS hops away from me. You can also find out this information using SNMP if I recall correctly.The most cool idea would be to actually run a routing daemon on the cluster manager (like gated or Zebra (see www.zebra.org)), then we wouldn't even need to telnet to the router but could run fully self contained using an IBGP feed. Zebra is quite modular even and could possibly be made to integrated more tightly with the dispatcher...
It must have been your other mail where you said that this was simple but not everyone knew about it. I just found out why. My cisco tcipip routing book has 2 pages on BGP. They told me to find a cisco engineer to "discuss my needs" with if I wanted to know more about BGP
There is actually some sort of nice introduction hidden on www.cisco.com, search for BGP4. "Internet Routing Architecture" from Cisco Press covers everything you might want to know about BGP4._All_ routers, participating in global Internet routing, hold a full view of the routing table, reflecting their view of the network. They know how many AS (autonomous systems) are in between them and any reachable destination on the network. If they have multiple choices (ie multile connections, so called multi homed providers), they select the one with the shortest AS path and install it into their routing table.
Now, one sets up a dispatcher which has BGP4 peerings with all participatin g clusters. Since the dispatcher only installs the best routes to all targets in it's routing table, it is a simple lookup to see which cluster is closest to the client.
If a cluster fails, the peering with the dispatcher is shutdown and the backup routes (the views learned from other clusters) take over.
This is actually very nice and well tested technology, since it is what makes the internet work.
It requires cooperation on part of the ISP hosting the cluster, that he provides a read-only "IBGP" feed to the cluster manager inside his AS.
BGP4 AS hops may not be directly related to latency. However, it tells you how many backbones are in between you and how many are not, which does have a tight relationship to the latency. And you can use the BGP4 route-maps etc to influnce the load balancing on an easy way - if you got one cluster from which a certain part of the Internet is reached via a slow satellite link, you can automatically lower the preference for all routes comeing in via that satellite link and not use that.
Ted Pavlic tpavlic@netwalk.com
9 Sep 1999
For now AS hops probably are useful - we have two mirrors on different continents.
You do NOT and cannot run OSPF here. OSPF is an "IGP" (interior routing protocol) which can't be usefully applied here.
I suppose I figured large networks might all share OSPF information, but I guess that they wouldn't share too much OSPF information between different geographical locations. (And I'm guessing that the latency between the load balancer and the user will PROABABLY almost exactly the same as the latency between the end-servers and the user...so...) I never claimed that I knew much of anything about BGP or OSPF, but thought that if BGP wasn't very helpful... OSPF might be. :) (It was a shot in the dark - a request for comments, if anything)
Of course, you not only want to figure BGP4, but also load and availability. We should investigate what other geogrpahical load balancers do. A lot of them setup large proxy'ing networks.
AFAICT, a lot of geographical load balancing systems seem to use their own means of finding which server is best per end-user. I think, for decent load balancing on that sort of scale, most balancers have to invent their own wheel.
This by no means is an easy task -- doing decent geographical load balancing. Companies put in a good deal of R&D to do just this and customers pay a good deal of money for the results.
it wasn't a serious suggestion for production. Just simply a way to divy out which mirror got which request.Worse comes to worst, have a perl script look-up the name of the machine that made the request... grab the first-level domain... figure out which continent it should go on.DNS has no guaranteed reply times at all.