ARCHITECTURAL DESIGN

Architectural Design



Abstract

HTC (High Throughput Computing) is an environment that can provide large amounts of processing capacity over long periods of time. To HTC, users are more concerned about how many jobs can be completed in a long period, but not how fast can a single job be finished. Condor, an implementation of HTC, is constructed by commodity CPUs and memory. As long as the Condor nodes are controlled by the Central Management Node, its reliability and scalability had been restricted. Based on the concept of DHT (Distributed Hash Table), this paper presents a new distributed HTC architecture, named MiPad, which has no central parts, and its metadata is distributed across all nodes in the system. Theoretical analysis and the simulation results proved MiPad to be an efficient architecture with high scalability and reliability.

Table of Content

Introduction4

Related work4

Design Analysis5

Route table6

New node Joining7

Normal node leaving9

System Structure Maintenance9

Node failure10

Job matching and dispatching11

Optimization to MiPad12

Cache Mechanism13

Theoretical analysis in MiPad14

Route Cost14

Maintenance Cost15

Enterprise to system transition15

Functional entities17

Functional behaviour17

Interaction specialisation18

Interaction point19

Interaction patterns20

Decomposed required service perspective21

Reliability Analysis22

Simulations and results24

Simulation for node joining cost24

Simulation for route cost24

High Performance SOA25

Description of CLI applications26

Resources allocation27

Conclusions29

References30

Architectural Design

Introduction

In many applications, most researchers are concerned with how many jobs per month or per year they can finish rather than FLOPS. We refer to such environments as High Throughput Computing (HTC) environments. A common solution of HTC is using the spare time of many PCs. As a result, the more computers the system contains the more computing resources it can provide. However, if nodes in the system are organized in traditional centralized mode, its reliability and scalability will be restricted. DHT (Distributed Hash Table) is a resources organization and location method in structured overlay network. It breaks the hash table held by central node into many pieces, and scatters them to all the nodes in the system. In such scheme, each node contains information about a certain part of whole system. When a resource location request arrived, it will be routed to its target through a series of nodes. DHT has good scalability and reliability. Based on the idea of DHT, a new HTC architecture with no central parts, named MiPad, is presented in this paper. In MiPad, the metadata is distributed across nodes in the system. MiPad is an efficient architecture with high scalability and reliability.

Related work

SETI@home is a distributed computing project using Internet-connected computers, hosted by the Space Sciences Laboratory in Berkeley. Users who want to participate can download and install SETI@home client software, which repeatedly downloads jobs from a central server and reports any signals found in radio telescope data. SETI@home has over 500,000 active users and provides computing power over 20 TeraFlop/sec cumulatively. The cost of setting up and maintaining SETI@home is only £500 thousand. The overwhelming success of SETI@home project has demonstrated the tremendous capabilities of Internet-connected commodity resources. Condor is a typical HTC. It organizes many PCs together to form a Condor Pool. Nodes in Condor Pool can run tasks from other users or submit its tasks to ...