Project News » DistributedDataMining:

DistributedDataMining:

Skip to page: 1 2 3

Aborted WUs and granted credit

Posted: August 30, 2010, 10:27pm CEST

Today, I've noticed that about 30 WUs got cancelled recently due to an unknown server problem. Because of the long runtime of these WUs, I added the complete credit to the related user accounts anyway. Now, I am going to have a look into the problem in order to fix it. Sorry for the inconvenience.
More RAM for the dDM servers

Posted: August 30, 2010, 10:53am CEST

Recently, I've noticed some performance problems of the dDM database. In some cases these problems might lead to slow or delayed connections of the boinc clients. Worst case scenario was as follows: Clients couldn't connect to the server and wait for one hour (because of load reducing) before they try again. In order to speed up the data base, I've spent more RAM to the servers. This includes the frontend server that is responsible for the websites and the backend server that carries the dDM data base. Let's see if it helps to normalize situation.
New images for the Boinc Manager

Posted: August 27, 2010, 7:00pm CEST

Today, I've added two new images that are shown in the simple view of the Boinc Manager. I've chosen these images in order to symbolize the Stock Price Prediction Application. The first one is a coloured version of the SPP logo that is also used on the website. The second one is less abstract and shows a Stock Price Diagnosis. I hope the new images pleases you even if the most dDM members prefer the advanced view in the BM.
New AppVersion 4.30: 32bit for Win and Linux + 64bit for linux

Posted: August 1, 2010, 6:01pm CEST

Today, I've released AppVersion 4.30 for Linux and Windows systems. It's the first time, dDM releases a 64bit version for Linux. Besides that, there are no big changes - just small bug fixes. The most mentionable new feature addresses the problem of multiple java processes that remain after a crash of the java wrapper: Every time a java process is create the java wrapper stores the java process ID in the checkpoint file. After a restart of the wrapper (for whatever reason) it is checking if the old java process (based on the stored PID) is still running. Doing so we can avoid that two java processes doing the same work and consume the double cpu time.
Suggestions for website improvements

Posted: July 24, 2010, 9:02am CEST

Our new website is on-line for several days and dDM members could participate in a poll about the new layout. Most of the participants like the new website but 20% don't like the new layout. Therefore, I'd like to invite you to post your comments, feedback and suggestions how we could improve our new website furthermore. Please, use the Forum Website Issues to share your ideas.
New operation system for the dDM backend server

Posted: July 18, 2010, 7:26pm CEST

Today, I've updated the operation system of the dDM backend server. From now on, a 64 bit debian (lenny) system is running in a XEN virtual machine. Doing so, the dDM software fully supports the 64 bit architecture of the XEN host system. That should end in a better overall performance. In addition, the switch to a 64bit OS makes the development of 64 bit applications for dDM much easier. Because of unforeseen problems regarding file and database permissions the dDM backend server was off-line for about 1 hour. During this time, also the boinc functionality that is included in the new dDM website was not available. Today's architecture switch was the last step in a batch of changes. During the last days the dDM project got a new website, a new project URL and a better server infrastructure. I hope any experienced inconvenience is compensated by the new dDM features.
New dDM website

Posted: July 16, 2010, 9:22pm CEST

Recently, I was working on a new dDM website. After finishing the beta test phase our new site is now on-line. The objective of the new site is to separate it from my personal website in order to focus on the project part. Therefore, we have a new domain name: [www.distributeddatamining.org] In addition, I tried to realize a better integration of the boinc functionality. Therefore, I've implemented a module for the popular content management system drupal. From now on, any boinc content is shown inside the dDM website instead of opening a new tab or window. To reach the non-public part of the website (your boinc settings, private messages ...) you have to login by using your usual boinc account data (email + password). Any comments, hints or feedback is welcome in the Number crunching forum.
New AppVersion 429

Posted: June 22, 2010, 5:00pm CEST
Today, I've released AppVersion 4.29. The new version fixes the following bugs:
- The java process did not finish after finishing the boinc manager (reported by bloodrain).
- The java process did allocate to much memory (reported by Augustine).
- The trickle message frequency was too high (reported by ritterm). Now, it's set to once per hour for each WU.
- The update frequency of the status messages sent to the boinc manager was to low. Now it's dynamically adapted based on the WU running time.
- In case of errors the WU was restarted even if the error was unrecoverable. From now on, an unrecoverable error will quit the WU immediately.
- There is still the problem of the 0xc0000005 errors. It seems to be caused by memory problems. Quite often they are recoverable. From now on, the WU will wait 120 seconds before restarting the WU. This should help to normalize the memory situation on the affected systems.
- In rare cases the checkpoint file can't be read (reported by magyarficko) As consequence the cputime isn't counted correctly. I added some debug code to find out what causes this problem.
- There was a bug in the on-demand trickle message mechanism that sends the error log file back to the server. Now, its working for linux as well as windows machines.
New AppVersion 429

Posted: June 22, 2010, 5:00pm CEST
Today, I've released AppVersion 4.29. The new version fixes the following bugs:
- The java process did not finish after finishing the boinc manager (reported by bloodrain).
- The java process did allocate to much memory (reported by Augustine).
- The trickle message frequency was too high (reported by ritterm). Now, it's set to once per hour for each WU.
- The update frequency of the status messages sent to the boinc manager was to low. Now it's dynamically adapted based on the WU running time.
- In case of errors the WU was restarted even if the error was unrecoverable. From now on, an unrecoverable error will quit the WU immediately.
- There is still the problem of the 0xc0000005 errors. It seems to be caused by memory problems. Quite often they are recoverable. From now on, the WU will wait 120 seconds before restarting the WU. This should help to normalize the memory situation on the affected systems.
- In rare cases the checkpoint file can't be read (reported by magyarficko) As consequence the cputime isn't counted correctly. I added some debug code to find out what causes this problem.
- There was a bug in the on-demand trickle message mechanism that sends the error log file back to the server. Now, its working for linux as well as windows machines.
Bug fixing efforts and WU error rate

Posted: May 24, 2010, 2:31pm CEST

During the last weeks, I was wondering if our bug fixing efforts were successful and if we were able to improve the dDM application. Therefore, I've implemented a script that calculates the fraction of cpu time that got lost because of WU errors. This calculation is done - once per day - for each single AppVersion that has processed a significant number of WUs. Thereby, beta-WUs and application independent errors are not taken into account. The results are daily updated and shown at [www.nicoschlitter.de] . I am glad of reaching a state where less than 0.5 percent of the total cpu time gets lost because of WU aborts. Of course, we will continue our efforts of decreasing the number of errors. I'd like to thank you for your patience, feedback and bug reports. Without your help dDM couldn't be successful.
Bug fixing efforts and WU error rate

Posted: May 24, 2010, 2:31pm CEST

During the last weeks, I was wondering if our bug fixing efforts were successful and if we were able to improve the dDM application. Therefore, I've implemented a script that calculates the fraction of cpu time that got lost because of WU errors. This calculation is done - once per day - for each single AppVersion that has processed a significant number of WUs. Thereby, beta-WUs and application independent errors are not taken into account. The results are daily updated and shown at [www.nicoschlitter.de] . I am glad of reaching a state where less than 0.5 percent of the total cpu time gets lost because of WU aborts. Of course, we will continue our efforts of decreasing the number of errors. I'd like to thank you for your patience, feedback and bug reports. Without your help dDM couldn't be successful.
WUs with beta-flag for a better quality control

Posted: May 21, 2010, 6:22pm CEST

Recently, we had to face some WUs that didn't work correctly because of a bug in the underlying data mining framework (RapidMiner). Regarding WUs, dDM is a special project, because the dDM-WUs differ not just in data - they also differ in the applied data processing algorithm. Therefore, distributing WUs with new algorithms brings always the risk of WU errors. To minimize these risks, I test a small batch of the new WUs before I distribute them. As usual it runs without having any errors. Later on some dDM members report problems that are documented here and here. The problem occurs in WUs that uses Support Vector Machines with Polynomial and Epnechnikov Kernels for time series analysis. I thought about how to avoid such problems in the future and came up with the following idea. From now on, each WU that uses new algorithms is tagged with a flag. The new dDM scheduler distributes these beta-WUs only to users that have agreed to receive beta applications. Doing so, we can test more of the new WUs on different platforms while non-beta tester are not involved at all. After a certain time without problems regarding the beta WUs they will loose its beta state and will be distributed to all dDM members. If you are willing to act as a beta tester please agree to Run test applications? in your account settings. Consequently, being a beta tester results in a higher probability to get WUs that finish with an error. Therefore, beta testers get credits for beta-WUs - even if they error out. If you don't want to be a beta tester for WUs please reject running test applications.
WUs with beta-flag for a better quality control

Posted: May 21, 2010, 6:22pm CEST

Recently, we had to face some WUs that didn't work correctly because of a bug in the underlying data mining framework (RapidMiner). Regarding WUs, dDM is a special project, because the dDM-WUs differ not just in data - they also differ in the applied data processing algorithm. Therefore, distributing WUs with new algorithms brings always the risk of WU errors. To minimize these risks, I test a small batch of the new WUs before I distribute them. As usual it runs without having any errors. Later on some dDM members report problems that are documented here and here. The problem occurs in WUs that uses Support Vector Machines with Polynomial and Epnechnikov Kernels for time series analysis. I thought about how to avoid such problems in the future and came up with the following idea. From now on, each WU that uses new algorithms is tagged with a flag. The new dDM scheduler distributes these beta-WUs only to users that have agreed to receive beta applications. Doing so, we can test more of the new WUs on different platforms while non-beta tester are not involved at all. After a certain time without problems regarding the beta WUs they will loose its beta state and will be distributed to all dDM members. If you are willing to act as a beta tester please agree to Run test applications? in your account settings. Consequently, being a beta tester results in a higher probability to get WUs that finish with an error. Therefore, beta testers get credits for beta-WUs - even if they error out. If you don't want to be a beta tester for WUs please reject running test applications.
New AppVersion 421

Posted: May 20, 2010, 5:56pm CEST
I've released AppVersion 4.21. I did some bug fixes and extended the trickle message features:
- Now, the App is able to react to a specific server command and sends the current error logfile (stderr.txt) back to the server. Doing so, I will be able to check the state of WUs in case there are noticeable problems.
- The problems regarding error code -1073741819 seems to be solved. It was caused by the decompressing (unzip) of the input files.
- There was a problem that WUs end in an infinite loop. Now it should be solved.
- I've changed the mechanism that detects WUs with no more progress during a long time.
- In addition, I did some performance improvements of the java wrapper: Uncompressing of input files is done just once and IO-load is reduced by better data caching.
New AppVersion 421

Posted: May 20, 2010, 5:56pm CEST
I've released AppVersion 4.21. I did some bug fixes and extended the trickle message features:
- Now, the App is able to react to a specific server command and sends the current error logfile (stderr.txt) back to the server. Doing so, I will be able to check the state of WUs in case there are noticeable problems.
- The problems regarding error code -1073741819 seems to be solved. It was caused by the decompressing (unzip) of the input files.
- There was a problem that WUs end in an infinite loop. Now it should be solved.
- I've changed the mechanism that detects WUs with no more progress during a long time.
- In addition, I did some performance improvements of the java wrapper: Uncompressing of input files is done just once and IO-load is reduced by better data caching.
New AppVersion 419

Posted: May 13, 2010, 3:16pm CEST
I've released AppVersion 4.19. The new version deals with trickle messages. It is able to send and receive messages to/from the server. The messages will be exchanged when the boinc client sends an update request to server. I've increased the update frequency to ensure a steady communication. Because of this mechanism the following features are provided:
- In order to monitor the behaviour of single WUs the application reports the cputime and the progress of a WU in answer to a specific server request. This feature helps to detect failing WUs.
- The application reports the start of a new WU to the dDM server. Therefore, it's now possible to abort downloaded but unstarted WUs by sending a dedicated message from the server to the application. This will be used in case of failing WUs. This feature is a consequence of a this situation.
In addition, the handling of the memory problem (described here and here) was improved. Doing so, the progress of an WU shouldn't slow down after a long runtime. Unfortunately, the new version brings some negative side effects. In some rare cases there is a problem with errors (0xc0000005 and -1073741819). So far, I don't know the cause but I am going to look into it. The error occurs at the start of a WU and doesn't cost much cputime. Credit for this failing WUs will be granted. In spite of this known problems, I've released this version anyway because the advantages will outstrip the drawbacks.
New AppVersion 419

Posted: May 13, 2010, 3:16pm CEST
I've released AppVersion 4.19. The new version deals with trickle messages. It is able to send and receive messages to/from the server. The messages will be exchanged when the boinc client sends an update request to server. I've increased the update frequency to ensure a steady communication. Because of this mechanism the following features are provided:
- In order to monitor the behaviour of single WUs the application reports the cputime and the progress of a WU in answer to a specific server request. This feature helps to detect failing WUs.
- The application reports the start of a new WU to the dDM server. Therefore, it's now possible to abort downloaded but unstarted WUs by sending a dedicated message from the server to the application. This will be used in case of failing WUs. This feature is a consequence of a this situation.
In addition, the handling of the memory problem (described here and here) was improved. Doing so, the progress of an WU shouldn't slow down after a long runtime. Unfortunately, the new version brings some negative side effects. In some rare cases there is a problem with errors (0xc0000005 and -1073741819). So far, I don't know the cause but I am going to look into it. The error occurs at the start of a WU and doesn't cost much cputime. Credit for this failing WUs will be granted. In spite of this known problems, I've released this version anyway because the advantages will outstrip the drawbacks.
New AppVersion 412

Posted: May 8, 2010, 12:24am CEST

Today, I've released AppVersion 4.12. Besides some bugfixes, version 4.12 takes care of a problem that is related to the java garbage collector. Long running WUs consume a lot of memory and memory management becomes more complex. At some point the java memory manager (garbage collector) takes the most CPU time and the experiment doesn't go further. This isn't a general java issue - there might be an error in the RapidMiner framework. To overcome this problem the new java wrapper divides the WU in several parts and each part is iteratively processed by RapidMiner. The number of splits is handled dynamical - depending on the WU runtime and the current progress. The speed-up factor is quite good. The runtime of the current WUs (SVM-Learner with parameter optimization) are reduced down to 25% of the original duration! Consequently, we save 75% of the computational power. All users are encouraged to abort waiting tasks that are assigned to earlier application versions.
New AppVersion 412

Posted: May 8, 2010, 12:24am CEST

Today, I've released AppVersion 4.12. Besides some bugfixes, version 4.12 takes care of a problem that is related to the java garbage collector. Long running WUs consume a lot of memory and memory management becomes more complex. At some point the java memory manager (garbage collector) takes the most CPU time and the experiment doesn't go further. This isn't a general java issue - there might be an error in the RapidMiner framework. To overcome this problem the new java wrapper divides the WU in several parts and each part is iteratively processed by RapidMiner. The number of splits is handled dynamical - depending on the WU runtime and the current progress. The speed-up factor is quite good. The runtime of the current WUs (SVM-Learner with parameter optimization) are reduced down to 25% of the original duration! Consequently, we save 75% of the computational power. All users are encouraged to abort waiting tasks that are assigned to earlier application versions.
New AppVersion 409

Posted: May 4, 2010, 11:35am CEST
Yesterday, I've released AppVersion 409 for Windows and Linux. It brings the following new features:
- In order to avoid short-running WUs (less then a minute), the new version supports a parameter optimization of the underlying learning algorithms. Doing so, it will be possible to perform different experiments on the same data. Thereby, the experiment setup will be the same but the used parameters will differ. As a consequence the runtime of a WU is increased by a factor that depends on the number of different parameter settings. Currently the parameter optimization is done by a simple grid search but it's easily extendible to other approaches e.g. genetic algorithms.
- In addition, the new version allows to specify the amount of RAM that is usable by the java processes. Hopefully, the better memory management will reduce the number of -226 errors.
- The new version sends more information back to the server. Besides the experiment results (output) and the optimization related files (opt_final,opt_grid) also logfiles (stderr.txt, stdout) will be transmitted to the server in order to find the cause of the -226 errors.
- In former versions, the consumed CPU time was counted incorrectly if the java process was killed while the java wrapper was suspended. The reported cpu time and the resulting credit was much too high and I had to correct them manually. The new version solves this problem.

Skip to page: 1 2 3

← CzechNationalTeam project DNETC4GPU: →

Project News » DistributedDataMining:

Feeds

DistributedDataMining:

Posted: August 30, 2010, 10:27pm CEST

Posted: August 30, 2010, 10:53am CEST

Posted: August 27, 2010, 7:00pm CEST

Posted: August 1, 2010, 6:01pm CEST

Posted: July 24, 2010, 9:02am CEST

Posted: July 18, 2010, 7:26pm CEST

Posted: July 16, 2010, 9:22pm CEST

Posted: June 22, 2010, 5:00pm CEST

Posted: June 22, 2010, 5:00pm CEST

Posted: May 24, 2010, 2:31pm CEST

Posted: May 24, 2010, 2:31pm CEST

Posted: May 21, 2010, 6:22pm CEST

Posted: May 21, 2010, 6:22pm CEST

Posted: May 20, 2010, 5:56pm CEST

Posted: May 20, 2010, 5:56pm CEST

Posted: May 13, 2010, 3:16pm CEST

Posted: May 13, 2010, 3:16pm CEST

Posted: May 8, 2010, 12:24am CEST

Posted: May 8, 2010, 12:24am CEST

Posted: May 4, 2010, 11:35am CEST