Skip to content
This repository has been archived by the owner on Jan 2, 2020. It is now read-only.
/ go-web-mapreduce Public archive

A MapReduce server using web browsers as workers, written in Go.

Notifications You must be signed in to change notification settings

oryband/go-web-mapreduce

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Web MapReduce GoDoc Build Status Coverage Status

A MapReduce server using web browsers as workers, written in Go.

This is a proof of concept.

gopher worker gopher worker gopher worker gopher master

Abstract

This is an implementation of a MapReduce master server, able to utilize the free computing capabilities of web browsers as MapReduce workers.

Communication is done via WebSockets, which allow consistent full-duplex comminucation between master and workers.

Worker computation is done via WebWorkers, which allow background processing to take place, while the main UI thread is left uninterrupted.

The server is written in Go and is highly concurrent, able to make full use of all available CPUs and memory. Every worker connection is managed by a cheap goroutine. This makes the master able to handle hundreds of thousands of connections at any given moment on a modern machine.

Resources

Resources used in the implementation:

To-Do List

The things left "unticked" are potential improvements, as a future project.

  • Master-Worker protocol
    • Protocol unit tests
  • Master algorithm and job management
    • Algorithm unit tests
    • Algorithm benchmark tests
  • Websocket worker management using SockJS
    • Unit tests
    • Websocket integration tests
  • Master
    • Master unit tests
    • Master benchmark tests
    • WebRTC - Necessary in order to fully implement MapReduce
  • HTTP Server
    • Server unit tests
    • Website pages (HTML, CSS).
  • Client (Javascript)
    • Client unit tests
    • Master-Worker integration tests
  • Extras
    • Travis
    • Coveralls

Develop

Prerequisites

  1. Install Go: You should probably either use [gvm][gvm] or install version >= 1.4 manually.
  2. Clone this repo into: $GOPATH/src/github.com/oryband/go-web-mapreduce.

Build

# download dependencies
$ go get -t -u -v ./...

# execute
$ go run server/server.go server/api.go server/views.go

Test

# run all tests including sub-packages, and output coverage.
$ go test -v -cover ./...

About

A MapReduce server using web browsers as workers, written in Go.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published