14 December 2010

My Birthday Calendar...

You know that feeling of trying to get a birthday card for a relative, can you imagine the trouble I have when I have so many children! Here is my current calendar with all the days I have a relatives' birthday marked (and the number of children on each day!)

What this calendar does not take into account are early datasets and defunct reprocessing datasets which have been deleted. I would love know what makes October 3rd and June 9th so popular??? Of course some of my children are alot bigger in terms of size and files ( and importance) which is lost in this plot.

Almost Nine months old.. But still going strong.

Its been a busy year for me,
I was born nearly nine months ago now and seem to have been kept spreading.
My avatar has been remiss in reporting my exploits so I am forcing him to give you an update.
I am now only in 19 countries. These are
Australia, Austria, Canada, China, Czech republic, Denmark, France, Germany, India, Israel, Italy, Japan, Netherlands, Portugal, Spain, Russia, Taiwan, UK and the USA.
I am at 68 physical sites (134 ATLAS endpoints) .
I am in a total of 2555 datasets ( but only 857 are unique).
The top 10 popular datasets have on averaged 20.7 copies on the grid. ( Ignoring these datasets leads to an average number of copies for all off Dave's datasets is 2.7.)
In total these unique datasets now are 33.4TB ( slightly more than the original 3TB that I was to start with!!!) So an increase in ~ a factor of 11. However my modest 1779 number of files has now increased to 698749 files; (an increase factor of 392).
My next post will show my birthday calendar. ( hoping to see if there is clustering around the time before conferences...)

08 December 2010

So who's winning the transfer rate race for the Tier2 sites in the UK

Well its not really a race but I thought it would be interested to see how the rates to and from the Tier2 with RAL.
So the clear winner is ( and probably always going to be) RALPPD. This should seem obvious since they are co-located so various factors (small rtt, large bandwidth link for examples;) which would lead to a high rates. This lead to the rate plot as shown below:

Here you can see the rate has got to over 250MB/s
A close second I thought was going to be Glasgow, (they have a longer rtt and lower bandwidth pipe so you would expect them to fair worse) :

This shows that they have got to 140MB/s
Sometimes it is unfair to compare these numbers since the number of concurrent transfers on the FTS channels varies,. Glasgow actually have more concurrent transfers ( ~140MB/s with 42 concurrent transfers compare to ~250MB/s with fewer concurrent transfers.) It is because Glasgow had the most concurrent transfers that I thought it would have the highest rates:
But the peak rates I see for the sites appears to be the following (in the last month at least). If a site thinks they have seen better rates then comment on this Post!!!
( rtt time between SEs is shown in brackets after each record. Well its actually the rtt to the closest router that traceroute can resolve.)

RALPP 250 MB/s ( .2ms)
Imperial 250 MB/s (5.0ms)
Manchester 200MB/s (8.0ms)
Glasgow 140Mb/s (10.9ms)
Lancaster 100MB/s (6.4ms)
QMUL 100MB/s (5.7ms)
Birmingham 90MB/s (8.8ms)
Brunel 90MB/s (6.4ms)
Oxford 90MB/s (8.9ms)
Sheffiled 80Mb/s (9.1ms)
Liverpool 75MB/s (9.3ms)
RHUL 40MB/s (8.2ms)
Cambridge 30MB/s (10.3ms)
ECDF 20MB/s (12.9ms)
Bristol 17MB/s (4.0ms)
UCL 15MB/s (7.3ms)
Durham 12 MB/s (14.8ms)

For some sites the limiting factor seems to be the link (ie transfers are running at line speed.) For other sites the limiting factor is the number of concurrent transfers currently set in FTS. Some thing to tweak further....( Something really interesting is that the top two sites are the only dCache we have, but this could just be coincidence since they are also the shortest and close to second shortest rtt times of any site.

ATLAS have also started their sonar T2-T2 mesh of testing inter cloud T2 transfers. this made me think of the work I had done (but not reported) about work I had done in looking in splitting STAR-T2 channels at RAL into a slow medium and fast channels. the rough split is western European sites in fast channel, north America in the medium channel and south America/Asia-pacific in slow channel ( going from rtt time) . This would be an initial split and then some tweaking if some sites were slower than their rtt would suggest.
Interesting to see if ATLAS list of slow transfers for UK sites match mine.
my list of slow sites would be:


Medium sites ( North America)would be:


Plus I am working with Glasgow to see how much of their 6Gbps can be used; but more on that in my next post.