artima - Article Discussion - Flat View

Article Discussion

Linux Clustering with Ruby Queue: Small is Beautiful

View Threaded

Summary: Ruby Queue software package lowers the barriers scientists need to overcome in order to realize the power of Linux clusters. It provides an extremely simple, economic, and easy-to-understand tool that harnesses the power of many CPUs while simultaneously allowing researchers to shift their focus away from the mundane details of complicated distributed computing systems and back to the task of actually doing science. The tool set is designed with a K.I.S.S, research-focused, philosophy that enables any ordinary (non-root) user to set up a zero-admin Linux cluster in 10 minutes or less. It is currently being used successfully in such diverse fields as bio-chemical research at the University of Toronto, geo-mechanical modeling at IGEOSS, and studying the nighttime lights of the world at the National Geophysical Data Center.

7 posts on 1 page.

« Previous 1 Next »

The ability to add new comments in this discussion is temporarily disabled.

Most recent reply: November 6, 2005 3:46 PM by Zach

James

Posts: 9 / Nickname: jbritt / Registered: March 24, 2003 4:10 PM

Linux Clustering with Ruby Queue: Small is Beautiful

October 9, 2005 8:00 PM

Ruby Queue software package lowers the barriers scientists need to overcome in order to realize the power of Linux clusters. The toolset is designed with a K.I.S.S,
research focused, philosophy that enables any ordinary (non-root) user to set
up a zero-admin Linux cluster in 10 minutes or less.

Read this Artima article by Ara T. Howard

http://www.artima.com/rubycs/articles/rubyqueue.html

What do you think about Linux clustering with Ruby?

Jamie

Posts: 2 / Nickname: metalii / Registered: October 12, 2005 4:35 PM

Re: Linux Clustering with Ruby Queue: Small is Beautiful

October 12, 2005 8:58 PM

Great Article. I am currently working on some clustering on my home network. I have a mixed platform environment though. I am considering adapting Ruby Queue to Work on a mixed platform environment. I let you know how my progress goes.

Jamie

Posts: 2 / Nickname: metalii / Registered: October 12, 2005 4:35 PM

Re: Linux Clustering with Ruby Queue: Small is Beautiful

October 12, 2005 9:00 PM

Though it will probably be a few months before I begin. I have some other projects on the que with higher priority. Though I did love your article and the thought of what could be implimented with Ruby Queues.

ara

Posts: 2 / Nickname: ahoward / Registered: October 13, 2005 4:29 AM

Re: Linux Clustering with Ruby Queue: Small is Beautiful

October 13, 2005 8:39 AM

sounds great. my environment is extremely limited in that i can't get in/out on any ports. making rq be cross platform would be straightforward. essentially you'd want a drb listener that took requests to insert/delete/update jobs and jobrunners would simply connect to it. it would actually be much easier than the nfs approach i had to take due to limited networking. if any other are interested in such a project let me know - other ideas i've talked with people about include a totally decentralized approach where each node was both a job runner and job submitter: each node would manage it's own work queue and return the status of jobs to whomever sent that particular job. it could be quite powerful.

Mark

Posts: 1 / Nickname: hubrix / Registered: October 19, 2005 6:45 AM

Re: Linux Clustering with Ruby Queue: Small is Beautiful

October 19, 2005 10:47 AM

We have an approach for running large batch job chunks that only involves Drb and a database, no complex NFS setup, we're considering open sourcing it or perhaps selling licenses. Contact me if interested.

ara

Posts: 2 / Nickname: ahoward / Registered: October 13, 2005 4:29 AM

Re: Linux Clustering with Ruby Queue: Small is Beautiful

October 27, 2005 6:17 AM

hi-

sounds very interesting. does your system work with ssl authentication or through ssh tunnels? if you aren't using nfs, where does the code that is run live - must it be installed locally on all compute nodes? is the database a single point of failure in the setup?

it sounds interesting - i've got a prototype of something similar myself but haven't gotten it running with ssl and resovled the issue of job collection. in any case i'd love to check out the code and/or talk with you about it. feel free to contact me on or offline - ara.t.howard@gmail.com.

regards.

-a

Zach

Posts: 2 / Nickname: zdennis / Registered: September 22, 2004 8:40 AM

Re: Linux Clustering with Ruby Queue: Small is Beautiful

November 6, 2005 3:46 PM

> hi-
>
> sounds very interesting. does your system work with ssl
> authentication or through ssh tunnels? if you aren't
> using nfs, where does the code that is run live - must it
> be installed locally on all compute nodes? is the
> database a single point of failure in the setup?

With urirequire recently released, it doesn't seem long off that a 'dbrequire' would be in the works. All code could live in the database. I don't know what Mark's set up is...but your article has got me thinking... Thanks Ara!

7 posts on 1 page.

« Previous 1 Next »