Pages: [1] 2 3 4
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-25 08:47:34

I'm trying to get BOINCstats working on Amazons' cloud platform. It's pretty expensive so I'm starting all tests with the cheapest virtual machines and/or database instances.

The XML import is working with pretty decent performance. Single project user update also works but only tested a small project since the data isn't transferred yet.

I'm now working on the host update. The database is importing right now and I'm running a query to get the BOINC Combined totals. It's really slow. As this is done purely in SQL I could easily double the database performance, however this also doubles the cost to $431 per month. And that's only the database. The VM to run the stats on also needs to be upgraded, the default only has 1GB of RAM and the dataset for hosts is 4GB.

More later.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-25 09:02:48
last modified: 2015-12-25 11:39:03

Combining host totals. This is without the ranking and history calculations. Just getting the total and average credit for hosts from all projects combined:

Current metal server:
Seconds needed to fetch data: 45
Seconds needed to order data: 396
Seconds needed update table: 756
Total: 1197 second.

On AWS I couldn't run the script because of memory constraints so I reprogrammed it to run completely in SQL:
Seconds needed to update: 2835

Without upgrading AWS this needs ~27 minutes more time to complete.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-25 11:47:08
last modified: 2015-12-25 11:49:37

Everything is ready to do a full host update. However, it's now time to be with my family. And having a little thing to eat. Something like:


Tomorrow morning I'll run the update.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
Dr Who Fan
BAM!ID: 1075
Joined: 2006-05-31
Posts: 599
Credits: 64,294,124
World-rank: 10,008

2015-12-25 19:00:12

Merry Christmas to Willy & his Family!!! Thank you for all your hard work over the years to keep BoincStats running!
Tuna Ertemalp
    Donator
Tester
BAM!ID: 37744
Joined: 2007-10-31
Posts: 531
Credits: 9,639,484,557
World-rank: 276

2015-12-25 19:07:39

[BOINCstats Willy wrote:
However, it's now time to be with my family. And having a little thing to eat.


Merry XMas to you & yours!


[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-26 08:11:39

Tested a host project update on LHC. The update needed a little bit longer than on the metal server but not too much so that's OK.

However, the AWS instance doesn't have enough memory to run the update. I added a 3GB SWAP file and then it ran correctly. Upgrading the AWS instance to have 4GB quadruples the cost for the instance.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-26 15:38:38

SETI host update takes like forever, I think I need to upgrade the instance, will test that tomorrow.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
TeeVeeEss
  Donator
Tester - Translator - BOINCstats SOFA member
BAM!ID: 2648
Joined: 2006-06-27
Posts: 755
Credits: 12,062,140,682
World-rank: 246

2015-12-26 15:43:05

Enjoy your 2nd xmas meal ;-)
Tuna Ertemalp
    Donator
Tester
BAM!ID: 37744
Joined: 2007-10-31
Posts: 531
Credits: 9,639,484,557
World-rank: 276

2015-12-26 21:52:34

[BOINCstats Willy wrote:

SETI host update takes like forever, I think I need to upgrade the instance, will test that tomorrow.

Wouldn't that eventually raise the cost from 300 to 1200 per month, if I understood your previous message correctly?
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-26 22:05:53

Not really. Different types of servers have different pricing. Databases are the most expensive. The server that needs upgrading means an increase to 40 euros for this instance. I'll probably need four like these plus the database engine.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
Tuna Ertemalp
    Donator
Tester
BAM!ID: 37744
Joined: 2007-10-31
Posts: 531
Credits: 9,639,484,557
World-rank: 276

2015-12-26 22:07:04

[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-27 08:39:25

The quadrupled instance (2 cores, 4GB RAM) is running quite a lot better. The host update is chugging along without the server becoming unresponsive.

I've been testing now for three days, these are the cost so far:
RDS: $29.36
DataTransfer: $1.31
EC2: $0.05
Tax: $6.45
Total: $37.17

RDS = database, EC2 is the server instance (5ct because the initial instance was free for one year).
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-27 17:55:27

Not good: combining the hosts for boinc combined took five hours today
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
Tuna Ertemalp
    Donator
Tester
BAM!ID: 37744
Joined: 2007-10-31
Posts: 531
Credits: 9,639,484,557
World-rank: 276

2015-12-27 18:13:04

What does it take on the metal? Would increasing the cores and/or memory help? Do you know here the bottleneck is from any perf counters they might provide?
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-27 18:28:24

About half an hour on metal. In this case it was the database instance which was too slow.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
Tuna Ertemalp
    Donator
Tester
BAM!ID: 37744
Joined: 2007-10-31
Posts: 531
Credits: 9,639,484,557
World-rank: 276

2015-12-27 18:37:24

[BOINCstats Willy wrote:
About half an hour on metal.
Zzzzooommmmgggg....


[BOINCstats Willy wrote:
In this case it was the database instance which was too slow.

Don't know much about (modern) database optimization. What are the options short of data partitioning, incremental caching, more SQL optimizations, more use of precompiled Stored Procedures?

[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-27 19:16:09

That brought a smile to my face. I've been optimizing this sucker for ten years now.

I already partitioned the queries but no improvement.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
Tuna Ertemalp
    Donator
Tester
BAM!ID: 37744
Joined: 2007-10-31
Posts: 531
Credits: 9,639,484,557
World-rank: 276

2015-12-27 19:23:51

Bummer. More cores & memory on the database server? Yeah, would increase the price, but to get an idea...
TeeVeeEss
  Donator
Tester - Translator - BOINCstats SOFA member
BAM!ID: 2648
Joined: 2006-06-27
Posts: 755
Credits: 12,062,140,682
World-rank: 246

2015-12-27 23:05:32

How is the metal-config (cpu, ram, ssd) compared to the aws-config? Should be more or less similar to get same kind of expected values on the different parts of the update?
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-28 07:45:26

The metal server has 8 cores, 32GB RAM and uses MariaDB. The AWS database has 2 cores, 15GB and uses Aurora. According to Amazon, Aurora is five times faster than MySQL/MariaDB. Let's say they exaggerate and that it is three times faster. It could/should be on par with the old server.

Now I expect that it will be somewhat slower. Doesn't bother me that much for the host update. But this is way slower.

The first upgrade to the database server gives it 4 cores, 30.5GB and improved network speed. This doubles the price (pricing).
On demand it goes from $215 to $431 per month. If I could pay three years in advance it goes from $75 to $150 (that's affordable but requires $5314 to be paid at once). Or, yet another option, paying $2500 in advance and $87.60 per month for a total of €160 per month.

Of course, before I pay so much money I want to know if it works but Amazon doesn't see this as testing. If testing lasts a month the full $431 is due.

So far I've spend $55 for testing.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-28 09:30:50
last modified: 2015-12-28 10:11:59

Upgraded the database instance, the host combine update now took 51 minutes.

Now doing the same update for non-asic -> that took 39 minutes.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
TeeVeeEss
  Donator
Tester - Translator - BOINCstats SOFA member
BAM!ID: 2648
Joined: 2006-06-27
Posts: 755
Credits: 12,062,140,682
World-rank: 246

2015-12-28 16:32:20

ok, the aws-test looks like a reasonable performance at a considerable cost: $431 a month is roughly €395, that's more than the current monthly cost for metal?
Tuna Ertemalp
    Donator
Tester
BAM!ID: 37744
Joined: 2007-10-31
Posts: 531
Credits: 9,639,484,557
World-rank: 276

2015-12-28 17:08:10

Well, Willy asks for €300 monthly donation for the maintenance/datacenter costs of the metal, but he said somewhere that they are more like €320-€350 in reality. Plus the Bitcoin Utopia goal of €2100 (excluding the €420-ish campaign late last year), which equals to €175/month. The more realistic total per month for metal would then be ~€500. Plus all the times Willy needs to drive back and forth to replace stuff, buy stuff, hit buttons, etc.

Plus, paying three years up front would seem to bring the AWS cost down to $5314/3/12*0.91=€134/month.

If all these above numbers are correct, then it would totally be worth it. Amazon would take care of all the hardware issues, most times without even Willy hearing about it, he would never have to replace another part himself, and never need to drive to the datacenter.

In any case, it is too early to calculate and compare costs without testing AWS with full data, full functionality and some sort of full user traffic.

Tuna
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-29 07:36:22

Tuna Ertemalp wrote:

Plus, paying three years up front would seem to bring the AWS cost down to $5314/3/12*0.91=€134/month.

That is only the database instance. I'm going to need at least two EC2 't2.medium' instances (updaters), two EC2 't2.micro' instances (website, maybe they need to be upgraded) and a load balancer. EC2 pricing, load balancer pricing (BOINCstats transfers about 400GB/month via LB ).
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9137
Credits: 348,241,820
World-rank: 2,997

2015-12-29 14:25:37
last modified: 2015-12-29 15:53:49

Just ran an incremental update without issues. Speed was fine too. Now trying a daily update.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
Pages: [1] 2 3 4

Index :: BOINCstats general :: BOINCstats in the cloud test results
Reason: