When did we switch storage systems.... I'm not sure I can tell.....

So as part of the Storage evolution within the UK, the Birmingham site is switching SEs from DPM to EOS for the ATLAS experiment. However this blog isn't about that, this blog is about what is happening in the meantime... Similar to the way in which we pointed UCL WNs to use QMUL storage in London, ATLAS have switched the 'Brum' WNs to use storage at Manchester. This "greatly" increases the rtt between WN and SE from ~0.1ms to ~4ms. So I got to wandering, is this increase in latency noticeable in Job efficiency for the VOs. Here are the graphs from the VO for Birmingham and Manchester for the last month..
For Manchester a a control set for SE performance we have:

And for Birmingham we have:

I leave it to the reader to come to conclusions about wether Birmingham efficiencies have dropped since changing SEs. (Even if that conclusion is "you need more data to make a proper comparison",) but at least its something to get things going...

What can a lossy fibre in the middle of you r network infrastructure do to your network rate.......

Just a quick post regarding how we noticed packet errors on one of a balanced pair of links within our network. By taking the link down (at  ~1700 Wednesday) and  re-seating the end connections lead to a mark improvement in rate and reduction in packet loss.

Here is a plot of the number of packets on the link:

Packets on the link

This clearly shows the period with no packets on the link corresponding to when the link was down.

Now look at the packet errors and discards:

Error/Discards on the link

 This clearly shows packet errors/discards before the change; and none after the change. The packet "loss" rate was ~ 6k/240k or ~14k/560k which equates to ~2.5 %!!The really interest comes when looking at the data rate through the combined link during this period:

Rate

The data rate through the link pair increases by ~10x when the link is removed (form ~0.6Gbps to 6Gbps.) Also note that there is no dramatic loss when the secondary link is re-nabled  (at ~1130 on Thursday.)

Lambda station

So what did CMS say at GridPP33?  Having looked ahead to the future, they came up with more speculative suggestions. Like FNAL's Lambda Station in the past, one suggestion was to look again at scheduling network for transfers, what we might nowadays call network-as-a-service (well, near enough): since we schedule transfers, it would indeed make sense to integrate networks more closely with the pre-allocation at the endpoints (where you'd bringOnline() at the source and schedule the transfer to avoid saturating the channel.) Phoebus is a related approach from Internet2. 

ss a tool to debug sockets

In the effort to explain why a 1M default buffer size works better than the more canonic 87k set also by the system and suggested in every network optimization site, as I wrote in this post about sonar tests to BNL, I tried the ss command suggested by John Green. Below are two results with 87k and 1M default buffer size respectively there are two things that jump to the eye for me

1) cwnd doesn't go beyond 30 in the first test, while in the second test cwnd is at least 556 and up to 1783. In my wikipedia knowledge of how this all really works it means the TCP window is not scaling with the first settings and it is with the second.

2) the memory field is completely different in the first case we have bytes in memory in some state in one (more often two streams, sometimes three) and nothing else. In the second case we have memory fields equally filled. This mirrors what observed with netstat and the Send-Q field.

Different state of the connection might report additional fields. For example when the connection is in CLOSE_WAIT state an ato (ack timeout) parameter appears. Also in CLOSE_WAIT the memory is fields are different r and f are uniformly filled instead of w and f.


Which confirms that with the fasterdata settings the TCP window doesn't scale (at least beyond a very small value such as 30).

Some interesting links I found to explain the fields

mem r,w,f,t values
cwnd or slow start and congestion control
wscale value
rto retransmission timeout

87k settings

ss -timeo|grep -A1 192.12.15.235|grep -v 192.12
mem:(r0,w0,f0,t0) ts sack htcp wscale:7,9 rto:292 rtt:92/0.75 cwnd:31 send 3.9Mbps rcv_space:14600
mem:(r0,w0,f0,t0) ts sack htcp wscale:7,9 rto:292 rtt:92/0.75 cwnd:30 send 3.8Mbps rcv_space:14600
mem:(r0,w0,f0,t0) ts sack htcp wscale:7,9 rto:307 rtt:104.125/0.75 cwnd:30 send 3.3Mbps rcv_space:14600
mem:(r0,w106648,f73576,t0) ts sack htcp wscale:7,9 rto:307 rtt:104/0.75 cwnd:21 send 2.3Mbps rcv_space:14600
mem:(r0,w0,f0,t0) ts sack htcp wscale:7,9 rto:307 rtt:104.125/0.75 cwnd:30 send 3.3Mbps rcv_space:14600
mem:(r0,w0,f0,t0) ts sack htcp wscale:7,9 rto:307 rtt:104/0.75 cwnd:30 send 3.3Mbps rcv_space:14600
mem:(r0,w0,f0,t0) ts sack htcp wscale:7,9 rto:292 rtt:92.125/0.75 cwnd:29 send 3.6Mbps rcv_space:14600
mem:(r0,w0,f0,t0) ts sack htcp wscale:7,9 rto:291 rtt:91.875/0.75 cwnd:30 send 3.8Mbps rcv_space:14600
mem:(r0,w0,f0,t0) ts sack htcp wscale:7,9 rto:292 rtt:92/0.75 cwnd:28 send 3.5Mbps rcv_space:14600

1MB settings

ss -timeo|grep -A1 192.12.15.226|grep -v 192.12
     mem:(r0,w1620945,f267311,t0) ts sack htcp wscale:9,8 rto:312 rtt:112.5/26 cwnd:621 send 63.9Mbps rcv_space:14600
     mem:(r0,w5074901,f213035,t0) ts sack htcp wscale:9,8 rto:320 rtt:120.5/34.5 cwnd:638 send 61.3Mbps rcv_space:14600
     mem:(r0,w269841,f495,t0) ts sack htcp wscale:9,8 rto:333 rtt:130.625/18 cwnd:356 send 31.6Mbps rcv_space:14600
     mem:(r0,w269841,f495,t0) ts sack htcp wscale:9,8 rto:319 rtt:119.5/18.75 cwnd:345 send 33.4Mbps rcv_space:14600
     mem:(r0,w2944464,f266800,t0) ts sack htcp wscale:9,8 rto:317 rtt:117.125/34.5 cwnd:1236 send 122.2Mbps rcv_space:14600
     mem:(r0,w269841,f495,t0) ts sack htcp wscale:9,8 rto:319 rtt:119.75/18.75 cwnd:320 send 31.0Mbps rcv_space:14600
     mem:(r0,w5621967,f239409,t0) ts sack htcp wscale:9,8 rto:313 rtt:113.125/24.25 cwnd:624 send 63.9Mbps rcv_space:14600
     mem:(r0,w269841,f495,t0) ts sack htcp wscale:9,8 rto:322 rtt:122.25/17.25 cwnd:318 send 30.1Mbps rcv_space:14600
     mem:(r0,w2943432,f800312,t0) ts sack htcp wscale:9,8 rto:314 rtt:114.375/21.75 cwnd:655 send 66.3Mbps rcv_space:14600

Connections in different states (additional fields)


ss -timeo|grep -A1 192.12.15
ESTAB      0      1587665     195.194.108.50:43684        192.12.15.233:41628    timer:(on,299ms,0) uid:19536 ino:20107501 sk:ffff88010b09cb00
     mem:(r0,w1620945,f267311,t0) ts sack htcp wscale:9,8 rto:312 rtt:112.5/26 cwnd:621 send 63.9Mbps rcv_space:14600
[...]

ss -timeo|grep -A1 192.12.15
CLOSE-WAIT 1      0          195.194.108.50:43684        192.12.15.233:41628    uid:19536 ino:20107501 sk:ffff88010b09cb00
     mem:(r4352,w0,f3840,t0) ts sack htcp wscale:9,8 rto:300 rtt:100.5/9 ato:40 cwnd:1924 send 221.8Mbps rcv_space:14600
[....]
 

Busy afternoon for ATLAS Storage at the RAL T1

Lunchtime on the 8th March 2012 was a busy time for the disk servers in the ATLAS instance of CASTOR at the Tier1 at RAL. Servers were delivering approximately 45Gbps as a source:



FTS controlled transfers at this as controlled by the RAL FTS WAN transfers were negligible; since the majority of "WAN" transfer via FTS (RAL) system were actually an intra-CASTOR transfer.

"WAN Rate"


"Intra-CASTOR" rate shown below shows that of the 4-5GB/s only 2.5Gbps is internal FTS traffic. ( ~5% of traffic.)

However this is only ~1/3 of the outbound traffic over the WAN. The other traffic is controlled by other FTS servers.
ATLAS aggregate this into the following ATLAS specific plot:

This shows the outbound traffic was actually 625MB/s (~5Gbps)
 ATLAS have a graph showing amount of data processed . This peaked at over 8TB in one hour.


However 8TB/hour is approximately 25Gbps
So what was using the remaining bandwidth...

Well we also need to include traffic from disk servers to tape servers (which are LAN transfers that neither ATLAS nor FTS know about. This rate can be seen here:


This shows there is an extra 600MB/s (~5Gbps) of rate unaccounted for. With FTS+WN+Tape network loads; the known bandwidth usage is 35Gbps. As a rough estimation, this explains how the majority of the bandwidth was used.