Engines¶

An engine is used to execute the calculation on specialized infrastructure: multi-core, HT condor or in a Cloud environment

Multi-Core Engine¶

The Multi-Core engine uses a process pool to manage the calculation tasks locally:

engine = femagtools.multiproc.Engine()

Docker Engine¶

The Docker engine uses a load balancer (dispatcher):

engine = femagtools.docker.Engine(dispatcher='localhost', port=5000, num_threads=5)

Condor Engine¶

The Condor engine uses a HT Condor pool to manage the calculation tasks in a corporate network:

engine = femagtools.condor.Engine()

Amazon Engine¶

The Amazon engine executes the FEMAG tasks in the Amazon EC2 cloud. The files for the calculation are uploaded to an unique S3 Bucket.

Prerequisite¶

To use Amazon as a calculation engine you have to:

create an account and setup a user with the correct authority.
configured your AWS credentials under ~/.aws/credentialsarv

Fast start¶

To use Amazon as engine setup a configuration and a cloud init file and use Amazon as engine:

import femagtools
engine = femagtools.amazon.Engine(buckets=None, configfile='config.ini')

If you already uploaded some FEMAG files you can add the bucket as list:

buckets = [{'id': '926c39a7-db69-42f8-bf86-2fd1f8559188-0', 'folder': '~/parvar/0'},
           {'id': '99ec9af5-ecb5-4029-88d6-70da9841ef91-1', 'folder': '~/parvar/1'},
           {'id': '4c61ebb3-89df-4e3a-ba46-288f8e0672a6-2', 'folder': '~/parvar/2'}]

Configuration file¶

You have to setup a configuration file with an amazon Section, where you define some Amazon EC2 Instance attribute.

Name	Example value	Description
iam_instance_profile	ecsInstanceRole	An Amazon Role name. The Role should at least have the AmazonEC2ContainerServiceforEC2Role and AmazonS3FullAccess to create EC2 Instances and S3 Buckets
key_name	test-key-pair-eucentral-1	SSH Key name for login over SSH. It is not neccessary but useful if something does not work as expected on the instance.
security_group_ids	sg-8475b5dd,	A list of security group ids. If you only have one security group it is important to add a , at the end.
server_location	eu-central-1	Server location of your instances
instance_type	t2.small	Instance type. Femag does not run if you use an instance type under the t2.small
image_id*	ami-6d4ceb03	The image of the instance. It is very advisable to use a custom image where femag and his dependencies are installed.
cloud_init	./cloud_init.txt	A path to a cloud_init file which will be run when the instance is started. Usually here you define your calculation for femag
delete_buckets	1	Delete the buckets after calculation is finished. 1 Delete 0 or no entry means no delete

All configuration options are optional.

Example of a configuration file:

[amazon]
iam_instance_profile = ecsInstanceRole
key_name = test-key-pair-eucentral-1
security_group_ids = sg-8475b5dd,
server_location = eu-central-1
instance_type = t2.small
image_id =  ami-6d4ceb03
cloud_init = ./cloud_init.txt

Note

The security_group_ids is a list. If you only have one security id you have to put a ‘,’ at the end

Cloud Init¶

With the cloud init file you can define the command to be excuted after the instance has started up. This is a good place to start the calculation.

The special entry {{ENV}} indicates the femagtools module to put all the configuration options as environment variables. Additionally it adds the bucket name as BUCKET_NAME environment to download the files from the correct Bucket.

Note

The files for each calculation are transferred in a tar.gz file

Example:

#!/bin/bash

{{ENV}}

mkdir ~/.aws
echo -e '[default]\nregion = '$SERVER_LOCATION'\noutput = json' > ~/.aws/config

yum install -y aws-cli libquadmath

mkdir ~/data
aws s3 sync s3://$BUCKET_NAME/ ~/data
cd ~/data
tar -xzf *.tar.gz
/usr/local/bin/xfemag -b femag.fsl </dev/null
echo $? > ~/data/exit_code
aws s3 sync ~/data s3://$BUCKET_NAME

Google Engine¶

The Google engine calculates the FEMAG tasks in the Google Cloud. The files for calculation are uploaded to an unique Google bucket.

Prerequisite¶

To use Google as a calculation engine you have to:

create an account and setup a user with the correct authority.
configured your google credentials with the google command tool

Fast start¶

To use Google Cloud as engine, setup a configuartion and a startup bash file and use google as engine:

import femagtools
engine = femagtools.google.Engine()
# Load config file
engine.config.from_ini_file('config.ini')

If you already uploaded some FEMAG files you can add the buckets as list:

buckets = [{'id': '926c39a7-db69-42f8-bf86-2fd1f8559188-0', 'folder': '~/parvar/0'},
           {'id': '99ec9af5-ecb5-4029-88d6-70da9841ef91-1', 'folder': '~/parvar/1'},
           {'id': '4c61ebb3-89df-4e3a-ba46-288f8e0672a6-2', 'folder': '~/parvar/2'}]

Configuration file¶

You have to setup a configuration file with an google Section, where you define some Options for the Google instance.

Name	Example value	Description
server_location	us-east1-b	Server location of your instances
instance_type	n1-standard-1	Google instance type
image_id*	femag	The image of the instance. It is very advisable to use a custom image where femag and his dependencies are installed.
startup	./startup.sh	A path to a startup.sh file which will be run when the instance is started. Usually here you define your calculation for femag
delete_buckets	1	Delete the buckets after calculation is finished. 1 Delete 0 or no entry means no delete
server_config	./gcloud.json	A json file where you can define some attributes for the instances

All configuration options are optional.

Example of a configuration file:

[google]
SERVER_LOCATION = us-east1-b
instance_type = n1-standard-1
image_id =  femag-v1
startup = ./cloud_init.txt
company_name = Semafor
delete_buckets = 1
server_config = ./gcloud.json
startup_script = ./startup.sh

startup.sh¶

With the startup.sh file you can define the processing sequence after the instance has started up. This is a good place to start the calculation.

Note

The files for one calculation are transferred in a tar.gz file

Example:

#!/bin/bash

# [START startup_script]
CS_BUCKET=$(curl http://metadata/computeMetadata/v1/instance/attributes/bucket -H "Metadata-Flavor: Google")

mkdir data
gsutil rsync gs://$CS_BUCKET data
cd data
tar -xzf *.tar.gz
xfemag -b femag.fsl </dev/null
echo $? > exit_code
gsutil rsync /data gs://$CS_BUCKET

# [END startup_script]

Server configuration¶

The Google Cloud allows you to set the configuartion over a JSON file. It is recommendable to set only attributes which are valid for all instances.

These attributes are automaticly set during runtime if they are missing in the json configuration:

Value	Descritpion
sourceImage	Your project id with your image name
machineType	Your server location with your instance type
name	Same name as the buckets with your company name first
bucket_id	The bucket id to download the correct data bucket

If you want more data in your instance, please set them as metadata items, and load them in the instance.

Example:

{
  "disks": [
      {
          "boot": true,
          "autoDelete": true
      }
  ],
  "networkInterfaces": [{
      "network": "global/networks/default",
      "accessConfigs": [
          {"type": "ONE_TO_ONE_NAT", "name": "External NAT"}
      ]
  }],
  "serviceAccounts": [{
      "email": "default",
      "scopes": [
          "https://www.googleapis.com/auth/devstorage.read_write",
          "https://www.googleapis.com/auth/logging.write"
      ]
  }],


  "metadata": {
      "items": [{
          "key": "text",
          "value": "Run femag"
      }]
  }
}

Engines¶

Multi-Core Engine¶

Docker Engine¶

Condor Engine¶

Amazon Engine¶

Prerequisite¶

Fast start¶

Configuration file¶

Cloud Init¶

Google Engine¶

Prerequisite¶

Fast start¶

Configuration file¶

startup.sh¶

Server configuration¶

Table of Contents

Previous topic

Next topic

This Page