max/poet
1
0
Fork 0
mirror of https://git.gfz-potsdam.de/naaice/poet.git synced 2025-12-15 12:28:22 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'add-phreeqcrm' into 'main'

Browse Source 
Refactor ChemistryModule and implement PhreeqcRM parallelization

See merge request sec34/port!22
This commit is contained in:
Max Lübke 2023-03-07 13:58:50 +01:00
commit 881ed1fea9
45 changed files with 6290 additions and 7851 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
CMakeLists.txt
View File

@ -14,6 +14,8 @@ set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
include("CMake/POET_Scripts.cmake")
list(APPEND CMAKE_MODULE_PATH "${POET_SOURCE_DIR}/CMake")
get_poet_version()
# set(GCC_CXX_FLAGS "-D STRICT_R_HEADERS") add_definitions(${GCC_CXX_FLAGS})
find_package(MPI REQUIRED)

146
README.md
View File

@ -1,10 +1,6 @@
<!--
Time-stamp: "Last modified 2023-01-19 12:06:10 delucia"
-->
**Po**tsdamer **R**eactive **T**ransport
# Forked Project
*PORT* is a fork of [POET](https://doi.org/10.5281/zenodo.4757913)
@ -19,11 +15,20 @@ also applicable for this project.
POET is a coupled reactive transport simulator implementing a parallel
architecture and a fast, original MPI-based Distributed Hash Table.
## Parsed code documentiation
A parsed version of POET's documentiation can be found at [Gitlab
pages](https://sec34.git-pages.gfz-potsdam.de/port).
## External Libraries
The following external header library is shipped with POET:
- **argh** - https://github.com/adishavit/argh (BSD license)
- **PhreeqcRM** with patches from GFZ -
https://www.usgs.gov/software/phreeqc-version-3 -
https://git.gfz-potsdam.de/mluebke/phreeqcrm-gfz
- **tug** - https://git.gfz-potsdam.de/sec34/tug
## Installation
@ -35,23 +40,18 @@ To compile POET you need several software to be installed:
- MPI-Implementation (tested with OpenMPI and MVAPICH)
- R language and environment
- CMake 3.9+
If you want to build documentation during compilation, `doxygen`and
`graphviz` must be provided too.
- *optional*: `doxygen` with `dot` bindings for documentiation
The following R libraries must then be installed, which will get the
needed dependencies automatically:
- [devtools](https://www.r-project.org/nosvn/pandoc/devtools.html)
- [Rcpp](https://cran.r-project.org/web/packages/Rcpp/index.html)
- [RInside](https://cran.r-project.org/web/packages/RInside/index.html)
- [RedModRphree](https://git.gfz-potsdam.de/delucia/RedModRphree)
- [Rmufits](https://git.gfz-potsdam.de/delucia/Rmufits)
### Compiling source code
The generation of makefiles is done with CMake. If you obtained POET
from git, you should be able to generate Makefiles by running
The generation of makefiles is done with CMake. You should be able to generate
Makefiles by running:
```sh
mkdir build && cd build
@ -62,34 +62,21 @@ This will create the directory `build` and processes the CMake files
and generate Makefiles from it. You're now able to run `make` to start
build process.
If POET was obtained from the official SVN repository or the redmine
at <https://redmine.cs.uni-potsdam.de/projects/poet> the branch or tag
to be used have to be set via
```sh
mkdir build && cd build
cmake -D POET_SET_BRANCH="<BRANCH>" ..
```
where currently available branches/tags are:
- dev
If everything went well you'll find the executable at
`build/src/poet`, but it is recommended to install the POET project
`build/app/poet`, but it is recommended to install the POET project
structure to a desired `CMAKE_INSTALL_PREFIX` with `make install`.
During the generation of Makefiles, various options can be specified
via `cmake -D <option>=<value> [...]`. Currently there are the
via `cmake -D <option>=<value> [...]`. Currently, there are the
following available options:
- **DHT_Debug**=_boolean_ - toggles the output of detailed statistics
about DHT usage (`cmake -D DHT_Debug=ON`). Defaults to _OFF_.
- **BUILD_DOC**=_boolean_ - toggles the generation of documantiation
during compilation process. Defaults to _ON_.
- _only from svn version:_ **POET_SET_BRANCH**=_string_ - set branch
or tag whose code is used
- **POET_DHT_Debug**=_boolean_ - toggles the output of detailed statistics about
DHT usage. Defaults to _OFF_.
- **POET_ENABLE_TESTING**=_boolean_ - enables small set of unit tests (more to
come). Defaults to _OFF_.
- **POET_USE_PRM_BACKEND**=_bollean_ - use the PhreeqcRM parallelization instead
of POET's one. Intended for debugging purposes for modellers.
### Example: Build from scratch
Assuming that only the C/C++ compiler, MPI libraries, R runtime
@ -101,9 +88,7 @@ follows:
$ R
# install R dependencies
> install.packages(c("devtools", "Rcpp", "RInside"))
> devtools::install_gitlab("delucia/RedModRphree", host="https://git.gfz-potsdam.de")
> devtools::install_gitlab("delucia/Rmufits", host="https://git.gfz-potsdam.de")
> install.packages(c("Rcpp", "RInside"))
> q(save="no")
# cd into POET project root
@ -124,60 +109,53 @@ POET we **do not recommend** to install to hierarchies like
The correspondending directory tree would look like this:
```sh
.
└── poet/
├── bin/
│ └── poet
├── data/
│ └── SimDol2D.R
├── docs/
│ └── html/
│ ├── index.html
│ └── ...
└── R_lib/
├── kin_r_library.R
└── parallel_r_library.R
poet
├── bin
│ └── poet
├── R_lib
│ └── kin_r_library.R
└── share
└── poet
├── bench
│ ├── dolo_diffu_inner_large.R
│ ├── dolo_diffu_inner.R
│ └── dolo_inner.pqi
└── examples
├── dol.pqi
├── phreeqc_kin.dat
├── SimDol1D_diffu.R
└── SimDol2D_diffu.R
```
The R libraries will be loaded at runtime and the paths are hardcoded
absolute paths inside `poet.cpp`. So, if you consider to move
`bin/poet` either change paths of the R source files and recompile
POET or also move `R_lib/*` according to the binary.
To display the generated html documentation just open
`docs/html/index.html` with the browser of your choice.
POET or also move `R_lib/*` relative to the binary.
## Running
Before POET is ready to run, a working directory must be created. In
this directory you should find the executable file, the R scripts
`<POET_ROOT>/R_lib/kin_r_library.R` and
`<POET_ROOT>/R_lib/parallel_r_library.R` and the simulation
description e.g. `<POET_ROOT>/data/chem_problems/SimDol2D.R`.
Run POET by `mpirun ./poet <OPTIONS> <SIMFILE> <OUTPUT_DIRECTORY>`
where:
- **OPTIONS** - runtime parameters (explained below)
- **SIMFILE** - simulation described as R script (currently supported:
`<POET_INSTALL_DIR>/data/SimDol2D.R`)
- **SIMFILE** - simulation described as R script (e.g.
`<POET_INSTALL_DIR>/share/examples/SimDol2D_diffu.R`)
- **OUTPUT_DIRECTORY** - path, where all output of POET should be stored
### Runtime options
The following parameters can be set:
| Option | Value | Description |
| ------------------------ | ------------ | -------------------------------------------------------------- |
| **--work-package-size=** | _1..n_ | size of work packages (defaults to _5_) |
| **--ignore-result** | | disables store of simulation resuls |
| **--dht** | | enabling DHT usage (defaults to _OFF_) |
| **--dht-nolog** | | disabling applying of logarithm before rounding |
| **--dht-signif=** | _1..n_ | set rounding to number of significant digits (defaults to _5_) |
| **--dht-strategy=** | _0-1_ | change DHT strategy. **NOT IMPLEMENTED YET** (Defaults to _0_) |
| **--dht-size=** | _1-n_ | size of DHT per process involved in byte (defaults to _1 GiB_) |
| **--dht-snaps=** | _0-2_ | disable or enable storage of DHT snapshots |
| **--dht-file=** | `<SNAPSHOT>` | initializes DHT with the given snapshot file |
| Option | Value | Description |
|--------------------------|--------------|--------------------------------------------------------------------------------------------------------------------------|
| **--work-package-size=** | _1..n_ | size of work packages (defaults to _5_) |
| **--ignore-result** | | disables store of simulation resuls |
| **--dht** | | enabling DHT usage (defaults to _OFF_) |
| **--dht-signif=** | _1..n_ | set rounding to number of significant digits (defaults to _5_) (it is recommended to use `signif_vec` in R input script) |
| **--dht-strategy=** | _0-1_ | change DHT strategy. **NOT IMPLEMENTED YET** (Defaults to _0_) |
| **--dht-size=** | _1-n_ | size of DHT per process involved in megabyte (defaults to _1000 MByte_) |
| **--dht-snaps=** | _0-2_ | disable or enable storage of DHT snapshots |
| **--dht-file=** | `<SNAPSHOT>` | initializes DHT with the given snapshot file |
#### Additions to `dht-signif`
@ -197,26 +175,26 @@ Following values can be set:
### Example: Running from scratch
We will continue the above example and start a simulation with
`SimDol2D.R`, which is the only simulation supported at this moment.
The required flow velocities snapshots are included in the R package
Rmufits. It's a 2D, 50x50 grid, with 20 time steps. To start the
simulation with 4 processes `cd` into your previously installed
POET-dir `<POET_INSTALL_DIR>/bin` and run:
`SimDol2D_diffu.R`. As transport a simple fixed-coefficient diffusion is used.
It's a 2D, 100x100 grid, simulating 10 time steps. To start the simulation with
4 processes `cd` into your previously installed POET-dir
`<POET_INSTALL_DIR>/bin` and run:
```sh
mpirun -n 4 ./poet ../data/SimDol2D.R output
mpirun -n 4 ./poet ../share/poet/examples/SimDol2D_diffu.R output
```
After a finished simulation all data generated by POET will be found
in the directory `output`.
You might want to use the DHT to cache previously simulated data and
reuse them in further time-steps. Just append `--dht` to the options
of POET to activate the usage of the DHT. The resulting call would
look like this:
You might want to use the DHT to cache previously simulated data and reuse them
in further time-steps. Just append `--dht` to the options of POET to activate
the usage of the DHT. Also, after each iteration a DHT snapshot shall be
produced. This is done by appending the `--dht-snaps=<value>` option. The
resulting call would look like this:
```sh
mpirun -n 4 ./poet --dht SimDol2D.R output
mpirun -n 4 ./poet --dht --dht-snaps=2 ../share/poet/examples/SimDol2D_diffu.R output
```
## About the usage of MPI_Wtime()

12
app/CMakeLists.txt
View File

@ -1,10 +1,14 @@
get_poet_version()
configure_file(poet.h.in poet.h)
add_executable(poet poet.cpp)
if(POET_USE_PRM_BACKEND)
set(poet_SRC poet_prm.cpp)
else()
set(poet_SRC poet.cpp)
endif()
add_executable(poet ${poet_SRC})
target_include_directories(poet PUBLIC "${CMAKE_CURRENT_BINARY_DIR}")
target_link_libraries(poet PUBLIC poet_lib MPI::MPI_C)
target_link_libraries(poet PUBLIC poet_lib MPI::MPI_CXX)
#target_compile_definitions(poet PRIVATE OMPI_SKIP_MPICXX)
install(TARGETS poet DESTINATION bin)

262
app/poet.cpp
View File

@ -18,11 +18,12 @@
** Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include "poet/ChemistryModule.hpp"
#include <RInside.h>
#include <Rcpp.h>
#include <Rcpp/internal/wrap.h>
#include <cstdint>
#include <poet/ChemSimPar.hpp>
#include <poet/ChemSimSeq.hpp>
#include <cstdlib>
#include <poet/DiffusionModule.hpp>
#include <poet/Grid.hpp>
#include <poet/SimParams.hpp>
@ -39,9 +40,65 @@ using namespace std;
using namespace poet;
using namespace Rcpp;
template <class ChemistryInstance>
poet::ChemistryModule::SingleCMap DFToHashMap(const Rcpp::DataFrame &df) {
std::unordered_map<std::string, double> out_map;
vector<string> col_names = Rcpp::as<vector<string>>(df.names());
for (const auto &name : col_names) {
double val = df[name.c_str()];
out_map.insert({name, val});
}
return out_map;
}
void set_chem_parameters(poet::ChemistryModule &chem, uint32_t wp_size,
const std::string &database_path) {
chem.SetErrorHandlerMode(1);
chem.SetComponentH2O(false);
chem.SetRebalanceFraction(0.5);
chem.SetRebalanceByCell(true);
chem.UseSolutionDensityVolume(false);
chem.SetPartitionUZSolids(false);
// Set concentration units
// 1, mg/L; 2, mol/L; 3, kg/kgs
chem.SetUnitsSolution(2);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
chem.SetUnitsPPassemblage(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
chem.SetUnitsExchange(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
chem.SetUnitsSurface(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
chem.SetUnitsGasPhase(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
chem.SetUnitsSSassemblage(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
chem.SetUnitsKinetics(1);
// Set representative volume
std::vector<double> rv;
rv.resize(wp_size, 1.0);
chem.SetRepresentativeVolume(rv);
// Set initial porosity
std::vector<double> por;
por.resize(wp_size, 1);
chem.SetPorosity(por);
// Set initial saturation
std::vector<double> sat;
sat.resize(wp_size, 1.0);
chem.SetSaturation(sat);
// Load database
chem.LoadDatabase(database_path);
}
inline double RunMasterLoop(SimParams &params, RInside &R, Grid &grid,
ChemistryParams &chem_params) {
ChemistryParams &chem_params,
const GridParams &g_params, uint32_t nxyz_master) {
DiffusionModule diffusion(poet::DiffusionParams(R), grid);
/* Iteration Count is dynamic, retrieving value from R (is only needed by
@ -50,8 +107,31 @@ inline double RunMasterLoop(SimParams &params, RInside &R, Grid &grid,
double sim_time = .0;
ChemistryInstance C(params, R, grid);
C.InitModule(chem_params);
ChemistryModule chem(nxyz_master, params.getNumParams().wp_size,
MPI_COMM_WORLD);
set_chem_parameters(chem, nxyz_master, chem_params.database_path);
chem.RunInitFile(chem_params.input_script);
chem.InitializeField(grid.GetTotalCellCount(), DFToHashMap(g_params.init_df));
if (params.getNumParams().dht_enabled) {
chem.SetDHTEnabled(true, params.getNumParams().dht_size_per_process);
if (!params.getDHTSignifVector().empty()) {
chem.SetDHTSignifVector(params.getDHTSignifVector());
}
if (!params.getDHTPropTypeVector().empty()) {
chem.SetDHTPropTypeVector(params.getDHTPropTypeVector());
}
if (!params.getDHTFile().empty()) {
chem.ReadDHTFile(params.getDHTFile());
}
if (params.getNumParams().dht_snaps > 0) {
chem.SetDHTSnaps(params.getNumParams().dht_snaps, params.getOutDir());
}
}
StateMemory *chem_state = grid.RegisterState("state_C", chem.GetPropNames());
chem_state->mem = chem.GetField();
/* SIMULATION LOOP */
double dStartTime = MPI_Wtime();
@ -77,7 +157,14 @@ inline double RunMasterLoop(SimParams &params, RInside &R, Grid &grid,
cout << "CPP: Chemistry" << endl;
C.Simulate(dt);
chem.SetTimeStep(dt);
chem.SetField(chem_state->mem);
chem.SetTimeStep(dt);
chem.RunCells();
chem_state->mem = chem.GetField();
grid.WriteFieldsToR(R);
grid.PreModuleFieldCopy(tick++);
R["req_dt"] = dt;
@ -97,12 +184,50 @@ inline double RunMasterLoop(SimParams &params, RInside &R, Grid &grid,
<< endl
<< endl;
MPI_Barrier(MPI_COMM_WORLD);
// MPI_Barrier(MPI_COMM_WORLD);
} // END SIMULATION LOOP
R.parseEvalQ("profiling <- list()");
C.End();
R["simtime_chemistry"] = chem.GetChemistryTime();
R.parseEvalQ("profiling$simtime_chemistry <- simtime_chemistry");
R["chemistry_loop"] = chem.GetMasterLoopTime();
R.parseEvalQ("profiling$chemistry_loop <- chemistry_loop");
R["chemistry_sequential"] = chem.GetMasterSequentialTime();
R.parseEvalQ("profiling$simtime_sequential <- chemistry_sequential");
R["idle_master"] = chem.GetMasterIdleTime();
R.parseEvalQ("profiling$idle_master <- idle_master");
R["idle_worker"] = Rcpp::wrap(chem.GetWorkerIdleTimings());
R.parseEvalQ("profiling$idle_worker <- idle_worker");
R["phreeqc_time"] = Rcpp::wrap(chem.GetWorkerPhreeqcTimings());
R.parseEvalQ("profiling$phreeqc <- phreeqc_time");
R["simtime_transport"] = diffusion.getTransportTime();
R.parseEvalQ("profiling$simtime_transport <- simtime_transport");
// R["phreeqc_count"] = phreeqc_counts;
// R.parseEvalQ("profiling$phreeqc_count <- phreeqc_count");
if (params.getNumParams().dht_enabled) {
R["dht_hits"] = Rcpp::wrap(chem.GetWorkerDHTHits());
R.parseEvalQ("profiling$dht_hits <- dht_hits");
R["dht_miss"] = Rcpp::wrap(chem.GetWorkerDHTMiss());
R.parseEvalQ("profiling$dht_miss <- dht_miss");
R["dht_evictions"] = Rcpp::wrap(chem.GetWorkerDHTEvictions());
R.parseEvalQ("profiling$dht_evictions <- dht_evictions");
R["dht_get_time"] = Rcpp::wrap(chem.GetWorkerDHTGetTimings());
R.parseEvalQ("profiling$dht_get_time <- dht_get_time");
R["dht_fill_time"] = Rcpp::wrap(chem.GetWorkerDHTFillTimings());
R.parseEvalQ("profiling$dht_fill_time <- dht_fill_time");
}
chem.MasterLoopBreak();
diffusion.end();
return MPI_Wtime() - dStartTime;
@ -119,20 +244,36 @@ int main(int argc, char *argv[]) {
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
/*Create custom Communicator with all processes except 0 (the master) for DHT
* storage */
MPI_Comm dht_comm;
if (world_rank == 0) {
MPI_Comm_split(MPI_COMM_WORLD, MPI_UNDEFINED, world_rank, &dht_comm);
} else {
MPI_Comm_split(MPI_COMM_WORLD, 1, world_rank, &dht_comm);
}
if (world_rank == 0) {
cout << "Running POET in version " << poet_version << endl << endl;
}
if (world_rank > 0) {
{
uint32_t c_size;
MPI_Bcast(&c_size, 1, MPI_UINT32_T, 0, MPI_COMM_WORLD);
char *buffer = new char[c_size + 1];
MPI_Bcast(buffer, c_size, MPI_CHAR, 0, MPI_COMM_WORLD);
buffer[c_size] = '\0';
// ChemistryModule worker(nxyz, nxyz, MPI_COMM_WORLD);
ChemistryModule worker =
poet::ChemistryModule::createWorker(MPI_COMM_WORLD);
set_chem_parameters(worker, worker.GetWPSize(), std::string(buffer));
delete[] buffer;
worker.WorkerLoop();
}
MPI_Barrier(MPI_COMM_WORLD);
cout << "CPP: finished, cleanup of process " << world_rank << endl;
MPI_Finalize();
return EXIT_SUCCESS;
}
/* initialize R runtime */
RInside R(argc, argv);
@ -154,42 +295,18 @@ int main(int argc, char *argv[]) {
cout << "CPP: R Init (RInside) on process " << world_rank << endl;
// HACK: we disable master_init and dt_differ propagation here for testing
// purposes
//
bool dt_differ = false;
R.parseEvalQ("mysetup <- setup");
if (world_rank == 0) { // get timestep vector from
// grid_init function ... //
std::string master_init_code = "mysetup <- master_init(setup=setup)";
R.parseEval(master_init_code);
// dt_differ = R.parseEval("mysetup$dt_differ");
// // ... and broadcast it to every other rank unequal to 0
MPI_Bcast(&dt_differ, 1, MPI_C_BOOL, 0, MPI_COMM_WORLD);
}
/* workers will only read the setup DataFrame defined by input file */
else {
// R.parseEval("mysetup <- setup");
MPI_Bcast(&dt_differ, 1, MPI_C_BOOL, 0, MPI_COMM_WORLD);
}
params.setDtDiffer(dt_differ);
// initialize chemistry on all processes
// TODO: einlesen einer initial matrix (DataFrame)
// std::string init_chemistry_code = "mysetup <-
// init_chemistry(setup=mysetup)"; R.parseEval(init_chemistry_code);
// TODO: Grid anpassen
// if (world_rank == 0) { // get timestep vector from
// grid_init function ... //
std::string master_init_code = "mysetup <- master_init(setup=setup)";
R.parseEval(master_init_code);
Grid grid;
GridParams g_params(R);
grid.InitModuleFromParams(GridParams(R));
grid.PushbackModuleFlow(poet::DIFFUSION_MODULE_NAME,
poet::BaseChemModule::CHEMISTRY_MODULE_NAME);
grid.PushbackModuleFlow(poet::BaseChemModule::CHEMISTRY_MODULE_NAME,
poet::DIFFUSION_MODULE_NAME);
grid.InitModuleFromParams(g_params);
grid.PushbackModuleFlow(poet::DIFFUSION_MODULE_NAME, CHEMISTRY_MODULE_NAME);
grid.PushbackModuleFlow(CHEMISTRY_MODULE_NAME, poet::DIFFUSION_MODULE_NAME);
params.initVectorParams(R, grid.GetSpeciesCount());
@ -203,38 +320,35 @@ int main(int argc, char *argv[]) {
cout << "CPP: Init done on process with rank " << world_rank << endl;
}
MPI_Barrier(MPI_COMM_WORLD);
// MPI_Barrier(MPI_COMM_WORLD);
poet::ChemistryParams chem_params(R);
/* THIS IS EXECUTED BY THE MASTER */
if (world_rank == 0) {
if (world_size == 1) {
dSimTime = RunMasterLoop<ChemSeq>(params, R, grid, chem_params);
} else {
dSimTime = RunMasterLoop<ChemMaster>(params, R, grid, chem_params);
}
std::string db_path = chem_params.database_path;
uint32_t c_size = db_path.size();
MPI_Bcast(&c_size, 1, MPI_UINT32_T, 0, MPI_COMM_WORLD);
cout << "CPP: finished simulation loop" << endl;
MPI_Bcast(db_path.data(), c_size, MPI_CHAR, 0, MPI_COMM_WORLD);
uint32_t nxyz_master = (world_size == 1 ? grid.GetTotalCellCount() : 1);
cout << "CPP: start timing profiling" << endl;
dSimTime = RunMasterLoop(params, R, grid, chem_params, g_params, nxyz_master);
R["simtime"] = dSimTime;
R.parseEvalQ("profiling$simtime <- simtime");
cout << "CPP: finished simulation loop" << endl;
string r_vis_code;
r_vis_code = "saveRDS(profiling, file=paste0(fileout,'/timings.rds'));";
R.parseEval(r_vis_code);
cout << "CPP: start timing profiling" << endl;
cout << "CPP: Done! Results are stored as R objects into <"
<< params.getOutDir() << "/timings.rds>" << endl;
}
/* THIS IS EXECUTED BY THE WORKERS */
else {
ChemWorker worker(params, R, grid, dht_comm);
worker.InitModule(chem_params);
worker.loop();
}
R["simtime"] = dSimTime;
R.parseEvalQ("profiling$simtime <- simtime");
string r_vis_code;
r_vis_code = "saveRDS(profiling, file=paste0(fileout,'/timings.rds'));";
R.parseEval(r_vis_code);
cout << "CPP: Done! Results are stored as R objects into <"
<< params.getOutDir() << "/timings.rds>" << endl;
MPI_Barrier(MPI_COMM_WORLD);
cout << "CPP: finished, cleanup of process " << world_rank << endl;
MPI_Finalize();

4
app/poet.h.in
View File

@ -1,6 +1,10 @@
#ifndef POET_H
#define POET_H
#include "poet/ChemistryModule.hpp"
#include <Rcpp.h>
const char *poet_version = "@POET_VERSION@";
const char *CHEMISTRY_MODULE_NAME = "state_C";
#endif // POET_H

287
app/poet_prm.cpp Normal file
View File

@ -0,0 +1,287 @@
/*
** Copyright (C) 2018-2021 Alexander Lindemann, Max Luebke (University of
** Potsdam)
**
** Copyright (C) 2018-2022 Marco De Lucia, Max Luebke (GFZ Potsdam)
**
** POET is free software; you can redistribute it and/or modify it under the
** terms of the GNU General Public License as published by the Free Software
** Foundation; either version 2 of the License, or (at your option) any later
** version.
**
** POET is distributed in the hope that it will be useful, but WITHOUT ANY
** WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
** A PARTICULAR PURPOSE. See the GNU General Public License for more details.
**
** You should have received a copy of the GNU General Public License along with
** this program; if not, write to the Free Software Foundation, Inc., 51
** Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include "poet/ChemistryModule.hpp"
#include <RInside.h>
#include <Rcpp.h>
#include <Rcpp/internal/wrap.h>
#include <cstdint>
#include <cstdlib>
#include <poet/DiffusionModule.hpp>
#include <poet/Grid.hpp>
#include <poet/SimParams.hpp>
#include <cstring>
#include <iostream>
#include <string>
#include <vector>
#include <mpi.h>
#include <poet.h>
using namespace std;
using namespace poet;
using namespace Rcpp;
void set_chem_parameters(poet::ChemistryModule &chem, uint32_t wp_size,
const std::string &database_path) {
chem.SetErrorHandlerMode(1);
chem.SetComponentH2O(false);
chem.SetRebalanceFraction(0.5);
chem.SetRebalanceByCell(true);
chem.UseSolutionDensityVolume(false);
chem.SetPartitionUZSolids(false);
// Set concentration units
// 1, mg/L; 2, mol/L; 3, kg/kgs
chem.SetUnitsSolution(2);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
chem.SetUnitsPPassemblage(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
chem.SetUnitsExchange(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
chem.SetUnitsSurface(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
chem.SetUnitsGasPhase(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
chem.SetUnitsSSassemblage(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
chem.SetUnitsKinetics(1);
// Set representative volume
std::vector<double> rv;
rv.resize(wp_size, 1.0);
chem.SetRepresentativeVolume(rv);
// Set initial porosity
std::vector<double> por;
por.resize(wp_size, 1);
chem.SetPorosity(por);
// Set initial saturation
std::vector<double> sat;
sat.resize(wp_size, 1.0);
chem.SetSaturation(sat);
// Load database
chem.LoadDatabase(database_path);
}
inline double RunMasterLoop(SimParams &params, RInside &R, Grid &grid,
ChemistryParams &chem_params,
const GridParams &g_params, uint32_t nxyz_master) {
DiffusionModule diffusion(poet::DiffusionParams(R), grid);
/* Iteration Count is dynamic, retrieving value from R (is only needed by
* master for the following loop) */
uint32_t maxiter = R.parseEval("mysetup$iterations");
double sim_time = .0;
ChemistryModule chem(grid.GetTotalCellCount(), MPI_COMM_WORLD);
set_chem_parameters(chem, nxyz_master, chem_params.database_path);
chem.RunInitFile(chem_params.input_script);
chem.SetSelectedOutputOn(true);
chem.SetTimeStep(0);
chem.RunCells();
StateMemory *chem_state = grid.RegisterState("state_C", chem.GetPropNames());
auto &chem_field = chem_state->mem;
chem_field = chem.GetField();
/* SIMULATION LOOP */
double dStartTime = MPI_Wtime();
for (uint32_t iter = 1; iter < maxiter + 1; iter++) {
uint32_t tick = 0;
// cout << "CPP: Evaluating next time step" << endl;
// R.parseEvalQ("mysetup <- master_iteration_setup(mysetup)");
double dt = Rcpp::as<double>(
R.parseEval("mysetup$timesteps[" + std::to_string(iter) + "]"));
cout << "CPP: Next time step is " << dt << "[s]" << endl;
/* displaying iteration number, with C++ and R iterator */
cout << "CPP: Going through iteration " << iter << endl;
cout << "CPP: R's $iter: " << ((uint32_t)(R.parseEval("mysetup$iter")))
<< ". Iteration" << endl;
/* run transport */
// TODO: transport to diffusion
diffusion.simulate(dt);
grid.PreModuleFieldCopy(tick++);
cout << "CPP: Chemistry" << endl;
chem.SetTimeStep(dt);
chem.SetConcentrations(chem_field);
chem.SetTimeStep(dt);
chem.RunCells();
chem_field = chem.GetField();
grid.WriteFieldsToR(R);
grid.PreModuleFieldCopy(tick++);
R["req_dt"] = dt;
R["simtime"] = (sim_time += dt);
R.parseEval("mysetup$req_dt <- req_dt");
R.parseEval("mysetup$simtime <- simtime");
// MDL master_iteration_end just writes on disk state_T and
// state_C after every iteration if the cmdline option
// --ignore-results is not given (and thus the R variable
// store_result is TRUE)
R.parseEvalQ("mysetup <- master_iteration_end(setup=mysetup)");
cout << endl
<< "CPP: End of *coupling* iteration " << iter << "/" << maxiter
<< endl
<< endl;
// MPI_Barrier(MPI_COMM_WORLD);
} // END SIMULATION LOOP
R.parseEvalQ("profiling <- list()");
R["simtime_chemistry"] = chem.GetChemistryTime();
R.parseEvalQ("profiling$simtime_chemistry <- simtime_chemistry");
chem.MpiWorkerBreak();
diffusion.end();
return MPI_Wtime() - dStartTime;
}
int main(int argc, char *argv[]) {
double dSimTime, sim_end;
int world_size, world_rank;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
if (world_rank == 0) {
cout << "Running POET in version " << poet_version << endl << endl;
}
if (world_rank > 0) {
{
uint32_t nxyz;
MPI_Bcast(&nxyz, 1, MPI_UINT32_T, 0, MPI_COMM_WORLD);
ChemistryModule worker(nxyz, MPI_COMM_WORLD);
worker.MpiWorker();
}
MPI_Barrier(MPI_COMM_WORLD);
cout << "CPP: finished, cleanup of process " << world_rank << endl;
MPI_Finalize();
return EXIT_SUCCESS;
}
/* initialize R runtime */
RInside R(argc, argv);
/*Loading Dependencies*/
// TODO: kann raus
std::string r_load_dependencies = "source('../R_lib/kin_r_library.R');";
R.parseEvalQ(r_load_dependencies);
SimParams params(world_rank, world_size);
int pret = params.parseFromCmdl(argv, R);
if (pret == poet::PARSER_ERROR) {
MPI_Finalize();
return EXIT_FAILURE;
} else if (pret == poet::PARSER_HELP) {
MPI_Finalize();
return EXIT_SUCCESS;
}
cout << "CPP: R Init (RInside) on process " << world_rank << endl;
R.parseEvalQ("mysetup <- setup");
// if (world_rank == 0) { // get timestep vector from
// grid_init function ... //
std::string master_init_code = "mysetup <- master_init(setup=setup)";
R.parseEval(master_init_code);
Grid grid;
GridParams g_params(R);
grid.InitModuleFromParams(g_params);
grid.PushbackModuleFlow(poet::DIFFUSION_MODULE_NAME, CHEMISTRY_MODULE_NAME);
grid.PushbackModuleFlow(CHEMISTRY_MODULE_NAME, poet::DIFFUSION_MODULE_NAME);
params.initVectorParams(R, grid.GetSpeciesCount());
// MDL: store all parameters
if (world_rank == 0) {
cout << "CPP: Calling R Function to store calling parameters" << endl;
R.parseEvalQ("StoreSetup(setup=mysetup)");
}
if (world_rank == 0) {
cout << "CPP: Init done on process with rank " << world_rank << endl;
}
// MPI_Barrier(MPI_COMM_WORLD);
poet::ChemistryParams chem_params(R);
/* THIS IS EXECUTED BY THE MASTER */
uint32_t nxyz = grid.GetTotalCellCount();
MPI_Bcast(&nxyz, 1, MPI_UINT32_T, 0, MPI_COMM_WORLD);
uint32_t nxyz_master = grid.GetTotalCellCount();
dSimTime = RunMasterLoop(params, R, grid, chem_params, g_params, nxyz_master);
cout << "CPP: finished simulation loop" << endl;
cout << "CPP: start timing profiling" << endl;
R["simtime"] = dSimTime;
R.parseEvalQ("profiling$simtime <- simtime");
string r_vis_code;
r_vis_code = "saveRDS(profiling, file=paste0(fileout,'/timings.rds'));";
R.parseEval(r_vis_code);
cout << "CPP: Done! Results are stored as R objects into <"
<< params.getOutDir() << "/timings.rds>" << endl;
MPI_Barrier(MPI_COMM_WORLD);
cout << "CPP: finished, cleanup of process " << world_rank << endl;
MPI_Finalize();
if (world_rank == 0) {
cout << "CPP: done, bye!" << endl;
}
exit(0);
}

7
bench/dolo_diffu_inner/dolo_inner.pqi
View File

@ -1,13 +1,6 @@
SELECTED_OUTPUT
-high_precision true
-reset false
-time
-soln
-temperature true
-water true
-pH
-pe
-totals C Ca Cl Mg
-kinetic_reactants Calcite Dolomite
-equilibrium O2g

3718
bench/surfex/SMILE_2021_11_01_TH.dat Normal file
View File

File diff suppressed because it is too large Load Diff

56
bench/surfex/SurfExBase.pqi Normal file
View File

@ -0,0 +1,56 @@
## Time-stamp: "Last modified 2023-02-27 14:31:11 delucia"
KNOBS
-logfile false
-iterations 10000
-convergence_tolerance 1E-12
-step_size 2
-pe_step_size 2
SELECTED_OUTPUT
-reset false
-high_precision true
-solution true
-state true
-step true
-pH true
-pe true
-ionic_strength true
-water true
USER_PUNCH
-head total_o total_h cb C(-4) C(4) Ca Cl Fe(2) Fe(3) H(0) K Mg Na S(-2) S(2) S(4) S(6) Sr U(4) U(5) U(6) UO2(am,hyd) KdU
-start
5 w=TOT("water")
10 PUNCH TOTMOLE("O"), TOTMOLE("H"), CHARGE_BALANCE, w*TOT("C(-4)"), w*TOT("C(4)"), w*TOT("Ca"), w*TOT("Cl"), w*TOT("Fe(2)"), w*TOT("Fe(3)"), w*TOT("H(0)"), w*TOT("K"), w*TOT("Mg"), w*TOT("Na"), w*TOT("S(-2)"), w*TOT("S(2)"), w*TOT("S(4)"), w*TOT("S(6)"), w*TOT("Sr"), w*TOT("U(4)"), w*TOT("U(5)"), w*TOT("U(6)"), EQUI("UO2(am,hyd)")
20 PUNCH ((SURF("U, Ill")+SURF("U, Mll")+SURF("U, Kln")+EDL("U, Ill")+EDL("U, Mll")+EDL("U, Kln"))/((TOT("U")*1.01583)))/(0.002251896406*1000)
-end
SOLUTION 1
temp 13
units mol/kgw
pH 7.06355
pe -2.626517
C(4) 0.001990694
Ca 0.02172649
Cl 0.3227673 charge
Fe 0.0001434717
K 0.001902357
Mg 0.01739704
Na 0.2762882
S(6) 0.01652701
Sr 0.0004520361
U(4) 8.147792e-12
U(6) 2.237946e-09
-water 0.00133
SURFACE 1
-equil 1
-sites_units density
-donnan 4.9e-10
Kln_aOH 1.155 11. 5.0518
Kln_siOH 1.155
Ill_sOH 0.05 100. 5.5931
Ill_wOH 2.26
Mll_sOH 0.05 100. 1.0825
Mll_wOH 2.26
EXCHANGE 1
-equil 1
Z 0.0012585
Y 0.0009418
END

142
bench/surfex/surfex.R Normal file
View File

@ -0,0 +1,142 @@
## Time-stamp: "Last modified 2023-02-27 18:33:30 delucia"
database <- normalizePath("./SMILE_2021_11_01_TH.dat")
input_script <- normalizePath("./SurfExBase.pqi")
cat(paste(":: R This is a test 1\n"))
#################################################################
## Section 1 ##
## Grid initialization ##
#################################################################
n <- 10
m <- 10
types <- c("scratch", "phreeqc", "rds")
init_cell <- list(H = 1.476571028625e-01,
O = 7.392297218936e-02,
Charge = -1.765225732724e-18,
`C(-4)` = 2.477908970828e-21,
`C(4)` = 2.647623016916e-06,
Ca = 2.889623169138e-05,
Cl = 4.292806181039e-04,
`Fe(2)` =1.908142472666e-07,
`Fe(3)` =3.173306589931e-12,
`H(0)` =2.675642675119e-15,
K = 2.530134809667e-06,
Mg =2.313806319294e-05,
Na =3.674633059628e-04,
`S(-2)` = 8.589766637180e-15,
`S(2)` = 1.205284362720e-19,
`S(4)` = 9.108958772790e-18,
`S(6)` = 2.198092329098e-05,
Sr = 6.012080128154e-07,
`U(4)` = 1.039668623852e-14,
`U(5)` = 1.208394829796e-15,
`U(6)` = 2.976409147150e-12)
grid <- list(
n_cells = c(n, m),
s_cells = c(1, 1),
type = "scratch",
init_cell = as.data.frame(init_cell, check.names = FALSE),
props = names(init_cell),
database = database,
input_script = input_script
)
##################################################################
## Section 2 ##
## Diffusion parameters and boundary conditions ##
##################################################################
vecinj_diffu <- list(
list(H = 0.147659686316291,
O = 0.0739242798146046,
Charge = 7.46361643222701e-20,
`C(-4)` = 2.92438561098248e-21,
`C(4)` = 2.65160558871092e-06,
Ca = 2.89001071336443e-05,
Cl = 0.000429291158114428,
`Fe(2)` = 1.90823391198114e-07,
`Fe(3)` = 3.10832423034763e-12,
`H(0)` = 2.7888235127385e-15,
K = 2.5301787e-06,
Mg = 2.31391999937907e-05,
Na = 0.00036746969,
`S(-2)` = 1.01376078438546e-14,
`S(2)` = 1.42247026981542e-19,
`S(4)` = 9.49422092568557e-18,
`S(6)` = 2.19812504654191e-05,
Sr = 6.01218519999999e-07,
`U(4)` = 4.82255946569383e-12,
`U(5)` = 5.49050615347901e-13,
`U(6)` = 1.32462838991902e-09)
)
vecinj <- do.call(rbind.data.frame, vecinj_diffu)
names(vecinj) <- grid$props
## diffusion coefficients
alpha_diffu <- c(H = 1E-6, O = 1E-6, Charge = 1E-6, `C(-4)` = 1E-6,
`C(4)` = 1E-6, Ca = 1E-6, Cl = 1E-6, `Fe(2)` = 1E-6,
`Fe(3)` = 1E-6, `H(0)` = 1E-6, K = 1E-6, Mg = 1E-6,
Na = 1E-6, `S(-2)` = 1E-6, `S(2)` = 1E-6,
`S(4)` = 1E-6, `S(6)` = 1E-6, Sr = 1E-6,
`U(4)` = 1E-6, `U(5)` = 1E-6, `U(6)` = 1E-6)
## list of boundary conditions/inner nodes
## vecinj_inner <- list(
## list(1,1,1)
## )
boundary <- list(
"N" = rep(1, n),
"E" = rep(0, n),
"S" = rep(0, n),
"W" = rep(0, n)
)
diffu_list <- names(alpha_diffu)
diffusion <- list(
init = init_cell,
vecinj = vecinj,
# vecinj_inner = vecinj_inner,
vecinj_index = boundary,
alpha = alpha_diffu
)
#################################################################
## Section 3 ##
## Chemistry module (Phreeqc) ##
#################################################################
chemistry <- list(
database = database,
input_script = input_script
)
#################################################################
## Section 4 ##
## Putting all those things together ##
#################################################################
iterations <- 10
dt <- 200
setup <- list(
grid = grid,
diffusion = diffusion,
chemistry = chemistry,
iterations = iterations,
timesteps = rep(dt, iterations),
store_result = TRUE,
out_save = c(5, iterations)
)

128
data/SimComp2D.R
View File

@ -1,128 +0,0 @@
## chemical database
db <- RPhreeFile("mdl_quint_kin.dat", is.db=TRUE)
phreeqc::phrLoadDatabaseString(db)
## only the directory
demodir <- system.file("extdata", "demo_rtwithmufits", package="Rmufits")
prop <- c("Al", "C","Ca","Cl","Fe", "K", "Mg","Na", "Si", "pH", ## "pe",
"Albite", "Calcite", "Chlorite", "Illite", "Kaolinite")
signif_vector <- c(7,7,7,7,7,7,7,7,7,6, 5,5,5,5,5)
prop_type <- rep("normal", length(signif_vector))
base <- c("SOLUTION 1",
"units mol/kgw",
"pH 6.77",
"temp 35",
"-water 1",
"Al 8.06386e-09",
"C 0.0006108294",
"Ca 0.09709463",
"Cl 4.340042",
"Fe 1.234357e-05",
"K 0.01117434",
"Mg 0.0406959",
"Na 4.189209",
"Si 0.0001935754",
"INCREMENTAL_REACTIONS true",
"KINETICS 1 ",
"-steps 86400",
"-bad_step_max 10000",
"-cvode true",
"Albite",
"-m 8.432165", ## 1540.0",
"-parms 01.54 100",
"Calcite",
"-m 0.0",
"-parms 10 100",
"Chlorite",
"-m 1.106585", ## 202.100",
"-parms 64.84 100",
"Illite",
"-m 0.9549153", ## 174.400",
"-parms 43.38 100",
"Kaolinite",
"-m 0.0",
"-parms 29.17 100",
"END")
selout <- c("KNOBS",
"-convergence_tolerance 1E-6",
"SELECTED_OUTPUT",
"-reset false",
"USER_PUNCH",
"-head Al C Ca Cl Fe K Mg Na Si pH Albite Calcite Chlorite Illite Kaolinite", ## pe
"10 PUNCH TOT(\"Al\"), TOT(\"C\"), TOT(\"Ca\"), TOT(\"Cl\"), TOT(\"Fe\"), TOT(\"K\"), TOT(\"Mg\"), TOT(\"Na\"), TOT(\"Si\"), -LA(\"H+\"), KIN(\"Albite\"), KIN(\"Calcite\"), KIN(\"Chlorite\"), KIN(\"Illite\"), KIN(\"Kaolinite\")" )
## Define initial conditions as equilibrium with primary minerals
ipr <- c(Al = 8.689e-10,
C = 0.0006108,
Ca = 0.09709,
Cl = 4.34,
Fe = 1.802e-06,
K = 0.01131,
Mg = 0.04074,
Na = 4.189,
Si = 7.653e-05,
pH = 6.889,
Albite = 5.0,
Calcite = 0.0,
Chlorite = 10.0,
Illite = 2.0,
Kaolinite = 0.0
)
initstate <- matrix(rep(ipr, 2500), byrow=TRUE, ncol=length(ipr))
colnames(initstate) <- names(ipr)
vecinj <- c(Al= 8.694e-10,
C = 8.182e-01,
Ca= 9.710e-02,
Cl= 4.340e+00,
Fe= 1.778e-06,
K = 1.131e-02,
Mg= 4.074e-02,
Na= 4.189e+00,
Si= 7.652e-05,
pH= 2.556228)
## setup boundary conditions for transport - we have already read the
## GRID with the following code:
## grid <- Rmufits::ReadGrid(paste0(demodir,"/d2ascii.run.GRID.SUM"))
## cbound <- which(grid$cell$ACTNUM == 2)
## dput(cbound)
cbound <- c(1L, 50L, 100L, 150L, 200L, 250L, 300L, 350L, 400L, 450L, 500L,
550L, 600L, 650L, 700L, 750L, 800L, 850L, 900L, 950L, 1000L,
1050L, 1100L, 1150L, 1200L, 1250L, 1300L, 1350L, 1400L, 1450L,
1500L, 1550L, 1600L, 1650L, 1700L, 1750L, 1800L, 1850L, 1900L,
1950L, 2000L, 2050L, 2100L, 2150L, 2200L, 2250L, 2300L, 2350L,
2400L, 2450L, 2451L, 2452L, 2453L, 2454L, 2455L, 2456L, 2457L,
2458L, 2459L, 2460L, 2461L, 2462L, 2463L, 2464L, 2465L, 2466L,
2467L, 2468L, 2469L, 2470L, 2471L, 2472L, 2473L, 2474L, 2475L,
2476L, 2477L, 2478L, 2479L, 2480L, 2481L, 2482L, 2483L, 2484L,
2485L, 2486L, 2487L, 2488L, 2489L, 2490L, 2491L, 2492L, 2493L,
2494L, 2495L, 2496L, 2497L, 2498L, 2499L, 2500L)
boundary_matrix <- matrix(rep(ipr[1:10], length(cbound)), byrow=TRUE, nrow=length(cbound))
colnames(boundary_matrix) <- names(ipr[1:10])
boundary_matrix[1, ] <- vecinj
boundary_matrix <- cbind(cbound,boundary_matrix)
setup <- list(n = 2500,
bound = boundary_matrix,
base = base,
first = selout,
initsim = initstate,
Cf = 1,
prop = prop,
immobile = seq(11,15),
kin = seq(11,15),
phase = "FLUX1",
density = "DEN1",
reduce = FALSE,
snapshots = demodir, ## directory where we will read MUFITS SUM files
gridfile = paste0(demodir,"/d2ascii.run.GRID.SUM")
)

131
data/SimCompKtz.R
View File

@ -1,131 +0,0 @@
db <- RPhreeFile("mdl_quint_kin.dat", is.db=TRUE)
phreeqc::phrLoadDatabaseString(db)
## only the directory
demodir <- "./snaps/"
base <- c("SOLUTION 1",
"units mol/kgw",
"pH 6.77",
"temp 35",
"-water 1",
"Al 8.06386e-09",
"C 0.0006108294",
"Ca 0.09709463",
"Cl 4.340042",
"Fe 1.234357e-05",
"K 0.01117434",
"Mg 0.0406959",
"Na 4.189209",
"Si 0.0001935754",
"INCREMENTAL_REACTIONS true",
"KINETICS 1 ",
"-steps 86400",
"-bad_step_max 10000",
"-cvode true",
"Albite",
"-m 8.432165", ## 1540.0",
"-parms 01.54 100",
"Calcite",
"-m 0.0",
"-parms 10 100",
"Chlorite",
"-m 1.106585", ## 202.100",
"-parms 64.84 100",
"Illite",
"-m 0.9549153", ## 174.400",
"-parms 43.38 100",
"Kaolinite",
"-m 0.0",
"-parms 29.17 100",
"END")
selout <- c("KNOBS",
"-convergence_tolerance 1E-6",
"SELECTED_OUTPUT",
"-reset false",
"USER_PUNCH",
"-head Al C Ca Cl Fe K Mg Na Si pH Albite Calcite Chlorite Illite Kaolinite", ## pe
"10 PUNCH TOT(\"Al\"), TOT(\"C\"), TOT(\"Ca\"), TOT(\"Cl\"), TOT(\"Fe\"), TOT(\"K\"), TOT(\"Mg\"), TOT(\"Na\"), TOT(\"Si\"), -LA(\"H+\"), KIN(\"Albite\"), KIN(\"Calcite\"), KIN(\"Chlorite\"), KIN(\"Illite\"), KIN(\"Kaolinite\")" )
## Define initial conditions as equilibrium with primary minerals
ipr <- c(Al = 8.689e-10,
C = 0.0006108,
Ca = 0.09709,
Cl = 4.34,
Fe = 1.802e-06,
K = 0.01131,
Mg = 0.04074,
Na = 4.189,
Si = 7.653e-05,
pH = 6.889,
Albite = 5.0,
Calcite = 0.0,
Chlorite = 10.0,
Illite = 2.0,
Kaolinite = 0.0
)
initstate <- matrix(rep(ipr, 648420), byrow=TRUE, ncol=length(ipr))
colnames(initstate) <- names(ipr)
vecinj <- c(Al= 8.694e-10,
C = 8.182e-01,
Ca= 9.710e-02,
Cl= 4.340e+00,
Fe= 1.778e-06,
K = 1.131e-02,
Mg= 4.074e-02,
Na= 4.189e+00,
Si= 7.652e-05,
pH= 2.556228)
prop <- c("Al", "C","Ca","Cl","Fe", "K", "Mg","Na", "Si", "pH", ## "pe",
"Albite", "Calcite", "Chlorite", "Illite", "Kaolinite")
bound_elm <- c(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L, 11L, 12L, 13L, 14L,
15L, 16L, 17L, 18L, 19L, 20L, 21L, 22L, 23L, 24L, 25L, 26L, 27L,
28L, 29L, 30L, 31L, 32L, 33L, 34L, 35L, 36L, 37L, 38L, 39L, 40L,
41L, 42L, 43L, 44L, 45L, 46L, 47L, 48L, 49L, 50L, 51L, 52L, 53L,
54L, 55L, 56L, 57L, 58L, 59L, 60L, 61L, 62L, 63L, 64L, 65L, 66L,
67L, 68L, 69L, 70L, 71L, 72L, 73L, 74L, 75L, 76L, 77L, 78L, 79L,
80L, 81L, 82L, 83L, 84L, 85L, 86L, 87L, 88L, 89L, 90L, 91L, 92L,
93L, 94L, 95L, 96L, 97L, 98L, 99L, 100L, 101L, 102L, 103L, 104L,
105L, 106L, 107L, 108L, 214L, 215L)
inj_elm <- c(7426L, 18233L, 29040L, 39847L,
50654L, 61461L, 72268L, 83075L, 93882L, 104689L, 115496L, 126303L,
137110L, 147917L, 158724L, 169531L, 180338L, 191145L, 201952L,
212759L, 223566L, 234373L, 245180L, 255987L, 266794L, 277601L,
288408L, 299215L, 310022L, 320829L, 331636L, 342443L, 353250L,
364057L, 374864L, 385671L, 396478L, 407285L, 418092L, 428899L,
439706L, 450513L, 461320L, 472127L, 482934L, 493741L, 504548L,
515355L)
cbound <- inj_elm
boundinit <- matrix(rep(vecinj, length(cbound)), ncol=length(vecinj), byrow=TRUE)
myboundmat <- cbind(cbound,boundinit)
## distinguish between injection and real boundaries
colnames(myboundmat) <- c("cbound", names(vecinj))
setup <- list(n=648420,
base=base,
bound=myboundmat,
first=selout,
initsim=initstate,
Cf=1,
prop=prop,
immobile=seq(11,15),
kin= seq(11,15),
phase="FLUX1",
density="DENS",
reduce=TRUE,
snapshots="snaps/AllSnaps_cmp_v3.rds",
gridfile ="snaps/GridKtz_cmp_v3.rds")

120
data/SimDol2D.R
View File

@ -1,120 +0,0 @@
## chemical database
db <- RPhreeFile(system.file("extdata", "phreeqc_kin.dat",
package="RedModRphree"), is.db=TRUE)
phreeqc::phrLoadDatabaseString(db)
## only the directory
demodir <- system.file("extdata", "demo_rtwithmufits", package="Rmufits")
prop <- c("C","Ca","Cl","Mg","pH","pe","O2g", "Calcite","Dolomite")
signif_vector <- c(7,7,7,7,7,7,7,5,5)
prop_type <- c("act","act","act","act","logact","logact","ignore","act","act")
base <- c("SOLUTION 1",
"units mol/kgw",
"temp 25.0",
"water 1",
"pH 9.91 charge",
"pe 4.0",
"C 1.2279E-04",
"Ca 1.2279E-04",
"Mg 0.001",
"Cl 0.002",
"PURE 1",
"O2g -0.1675 10",
"KINETICS 1",
"-steps 100",
"-step_divide 100",
"-bad_step_max 2000",
"Calcite", "-m 0.000207",
"-parms 0.0032",
"Dolomite",
"-m 0.0",
"-parms 0.00032",
"END")
selout <- c("SELECTED_OUTPUT", "-high_precision true", "-reset false",
"-time", "-soln", "-temperature true", "-water true",
"-pH", "-pe", "-totals C Ca Cl Mg",
"-kinetic_reactants Calcite Dolomite", "-equilibrium O2g")
initsim <- c("SELECTED_OUTPUT", "-high_precision true",
"-reset false",
"USER_PUNCH",
"-head C Ca Cl Mg pH pe O2g Calcite Dolomite",
"10 PUNCH TOT(\"C\"), TOT(\"Ca\"), TOT(\"Cl\"), TOT(\"Mg\"), -LA(\"H+\"), -LA(\"e-\"), EQUI(\"O2g\"), EQUI(\"Calcite\"), EQUI(\"Dolomite\")",
"SOLUTION 1",
"units mol/kgw",
"temp 25.0", "water 1",
"pH 9.91 charge",
"pe 4.0",
"C 1.2279E-04",
"Ca 1.2279E-04",
"Cl 1E-12",
"Mg 1E-12",
"PURE 1",
"O2g -0.6788 10.0",
"Calcite 0.0 2.07E-3",
"Dolomite 0.0 0.0",
"END")
vecinj <- c("C"= 0,
"Ca" = 0,
"Cl" = 0.002,
"Mg" = 0.001,
"pe" = 4,
"pH" = 7)
init <- c("C(4)"= 1.2279E-4,
"Ca" =1.2279E-4,
"Cl" =0,
"Mg" =0,
"pe" =4,
"pH" =7,
"Calcite"= 2.07e-4,
"Dolomite"= 0)
## setup boundary conditions for transport - we have already read the
## GRID with the following code:
## grid <- Rmufits::ReadGrid(paste0(demodir,"/d2ascii.run.GRID.SUM"))
## cbound <- which(grid$cell$ACTNUM == 2)
## dput(cbound)
cbound <- c(1L, 50L, 100L, 150L, 200L, 250L, 300L, 350L, 400L, 450L, 500L,
550L, 600L, 650L, 700L, 750L, 800L, 850L, 900L, 950L, 1000L,
1050L, 1100L, 1150L, 1200L, 1250L, 1300L, 1350L, 1400L, 1450L,
1500L, 1550L, 1600L, 1650L, 1700L, 1750L, 1800L, 1850L, 1900L,
1950L, 2000L, 2050L, 2100L, 2150L, 2200L, 2250L, 2300L, 2350L,
2400L, 2450L, 2451L, 2452L, 2453L, 2454L, 2455L, 2456L, 2457L,
2458L, 2459L, 2460L, 2461L, 2462L, 2463L, 2464L, 2465L, 2466L,
2467L, 2468L, 2469L, 2470L, 2471L, 2472L, 2473L, 2474L, 2475L,
2476L, 2477L, 2478L, 2479L, 2480L, 2481L, 2482L, 2483L, 2484L,
2485L, 2486L, 2487L, 2488L, 2489L, 2490L, 2491L, 2492L, 2493L,
2494L, 2495L, 2496L, 2497L, 2498L, 2499L, 2500L)
boundinit <- matrix(rep(init[-c(7,8)], length(cbound)), byrow=TRUE, nrow=length(cbound))
myboundmat <- cbind(cbound,boundinit)
myboundmat[cbound==1, c(2:7)] <- vecinj
colnames(myboundmat) <- c("cbound", names(vecinj))
# TODO: dt and iterations
setup <- list(n=2500,
bound=myboundmat,
base=base,
first=selout,
initsim=initsim,
Cf=1,
prop=prop,
immobile=c(7,8,9),
kin= c(8,9),
ann=list(O2g=-0.1675),
phase="FLUX1",
density="DEN1",
reduce=FALSE,
snapshots=demodir, ## directory where we will read MUFITS SUM files
gridfile=paste0(demodir,"/d2ascii.run.GRID.SUM")
)

130
data/SimDolKtz.R
View File

@ -1,130 +0,0 @@
# library(RedModRphree)
# library(Rmufits)
# library(RcppVTK)
db <- RPhreeFile(system.file("extdata", "phreeqc_kin.dat",
package="RedModRphree"), is.db=TRUE)
phreeqc::phrLoadDatabaseString(db)
prop <- c("C","Ca","Cl","Mg","pH","pe","O2g", "Calcite","Dolomite")
signif_vector <- c(7,7,7,7,7,7,7,5,5)
prop_type <- c("act","act","act","act","logact","logact","ignore","act","act")
base <- c("SOLUTION 1",
"units mol/kgw",
"temp 25.0",
"water 1",
"pH 9.91 charge",
"pe 4.0",
"C 1.2279E-04",
"Ca 1.2279E-04",
"Mg 0.001",
"Cl 0.002",
"PURE 1",
"O2g -0.1675 10",
"KINETICS 1",
"-steps 100",
"-step_divide 100",
"-bad_step_max 2000",
"Calcite", "-m 0.000207",
"-parms 0.0032",
"Dolomite",
"-m 0.0",
"-parms 0.00032",
"END")
selout <- c("SELECTED_OUTPUT",
"-high_precision true",
"-reset false",
"-time",
"-soln",
"-temperature true",
"-water true",
"-pH",
"-pe",
"-totals C Ca Cl Mg",
"-kinetic_reactants Calcite Dolomite",
"-equilibrium O2g")
initsim <- c("SELECTED_OUTPUT",
"-high_precision true",
"-reset false",
"USER_PUNCH",
"-head C Ca Cl Mg pH pe O2g Calcite Dolomite",
"10 PUNCH TOT(\"C\"), TOT(\"Ca\"), TOT(\"Cl\"), TOT(\"Mg\"), -LA(\"H+\"), -LA(\"e-\"), EQUI(\"O2g\"), EQUI(\"Calcite\"), EQUI(\"Dolomite\")",
"SOLUTION 1",
"units mol/kgw",
"temp 25.0", "water 1",
"pH 9.91 charge",
"pe 4.0",
"C 1.2279E-04",
"Ca 1.2279E-04",
"Cl 1E-12",
"Mg 1E-12",
"PURE 1",
"O2g -0.6788 10.0",
"Calcite 0.0 2.07E-3",
"Dolomite 0.0 0.0",
"END")
vecinj <- c("C"= 0,
"Ca" = 0,
"Cl" = 0.002,
"Mg" = 0.001,
"pe" = 4,
"pH" = 7)
init <- c("C(4)"= 1.2279E-4,
"Ca" =1.2279E-4,
"Cl" =0,
"Mg" =0,
"pe" =4,
"pH" =7,
"Calcite"= 2.07e-4,
"Dolomite"= 0)
bound_elm <- c(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L, 11L, 12L, 13L, 14L,
15L, 16L, 17L, 18L, 19L, 20L, 21L, 22L, 23L, 24L, 25L, 26L, 27L,
28L, 29L, 30L, 31L, 32L, 33L, 34L, 35L, 36L, 37L, 38L, 39L, 40L,
41L, 42L, 43L, 44L, 45L, 46L, 47L, 48L, 49L, 50L, 51L, 52L, 53L,
54L, 55L, 56L, 57L, 58L, 59L, 60L, 61L, 62L, 63L, 64L, 65L, 66L,
67L, 68L, 69L, 70L, 71L, 72L, 73L, 74L, 75L, 76L, 77L, 78L, 79L,
80L, 81L, 82L, 83L, 84L, 85L, 86L, 87L, 88L, 89L, 90L, 91L, 92L,
93L, 94L, 95L, 96L, 97L, 98L, 99L, 100L, 101L, 102L, 103L, 104L,
105L, 106L, 107L, 108L, 214L, 215L)
inj_elm <- c(7426L, 18233L, 29040L, 39847L,
50654L, 61461L, 72268L, 83075L, 93882L, 104689L, 115496L, 126303L,
137110L, 147917L, 158724L, 169531L, 180338L, 191145L, 201952L,
212759L, 223566L, 234373L, 245180L, 255987L, 266794L, 277601L,
288408L, 299215L, 310022L, 320829L, 331636L, 342443L, 353250L,
364057L, 374864L, 385671L, 396478L, 407285L, 418092L, 428899L,
439706L, 450513L, 461320L, 472127L, 482934L, 493741L, 504548L,
515355L)
cbound <- inj_elm
boundinit <- matrix(rep(vecinj, length(cbound)), ncol=length(vecinj), byrow=TRUE)
myboundmat <- cbind(cbound,boundinit)
## distinguish between injection and real boundaries
colnames(myboundmat) <- c("cbound", names(vecinj))
setup <- list(n=648420,
bound=myboundmat,
base=base,
first=selout,
initsim=initsim,
Cf=1,
prop=prop,
immobile=c(7,8,9),
kin= c(8,9),
ann=list(O2g=-0.1675),
phase="FLUX1",
density="DENS",
reduce=FALSE,
snapshots="snaps/AllSnaps_cmp_v3.rds",
gridfile ="snaps/GridKtz_cmp_v3.rds"
)

2625
data/mdl_quint_kin.dat
View File

File diff suppressed because it is too large Load Diff

2
docs/CMakeLists.txt
View File

@ -9,10 +9,12 @@ if(DOXYGEN_FOUND)
set(DOXYGEN_DISABLE_INDEX NO)
set(DOXYGEN_GENERATE_TREEVIEW YES)
set(DOXYGEN_FULL_SIDEBAR YES)
set(DOXYGEN_PROJECT_NUMBER ${POET_VERSION})
doxygen_add_docs(doxygen
${PROJECT_SOURCE_DIR}/include
${PROJECT_SOURCE_DIR}/README.md
${PROJECT_SOURCE_DIR}/docs/Input_Scripts.md
${PROJECT_SOURCE_DIR}/docs/Output.md
COMMENT "Generate html pages")
endif()

2633
docs/Doxyfile.in
View File

File diff suppressed because it is too large Load Diff

95
docs/Input_Scripts.md Normal file
View File

@ -0,0 +1,95 @@
# Input Scripts
In the following the expected schemes of the input scripts is described.
Therefore, each section of the input script gets its own chapter. All sections
should return a `list` as results, which are concatenated to one setup list at
the end of the file. All values must have the same name in order to get parsed
by POET.
## Grid initialization
| name | type | description |
|----------------|----------------|-----------------------------------------------------------------------|
| `n_cells` | Numeric Vector | Number of cells in each direction |
| `s_cells` | Numeric Vector | Spatial resolution of grid in each direction |
| `type` | String | Type of initialization, can be set to *scratch*, *phreeqc* or *rds* |
| `init_cell` | Data Frame | Containing all exactly one value per species to initialize the field. |
| `props` | String Vector | Names of all species |
| `database` | String | Path to Phreeqc database |
| `input_script` | String | Path to Phreeqc initial script |
## Diffusion parameters
| name | type | description |
|----------------|----------------------|-------------------------------------------|
| `init` | Named Numeric Vector | Initial state for each diffused species |
| `vecinj` | Data Frame | Defining all boundary conditions row wise |
| `vecinj_inner` | List of Triples | Inner boundaries |
| `vecinj_index` | List of 4 elements | Ghost nodes boundary conditions |
| `alpha` | Named Numeric Vector | Constant alpha for each species |
### Remark on boundary conditions
Each boundary condition should be defined in `vecinj` as a data frame, where one
row holds one boundary condition.
To define inner (constant) boundary conditions, use a list of triples in
`vecinj_inner`, where each triples is defined by $(i,x,y)$. $i$ is defining the
boundary condition, referencing to the row in `vecinj`. $x$ and $y$ coordinates
then defining the position inside the grid.
Ghost nodes are set by `vecinj_index` which is a list containing boundaries for
each celestial direction (**important**: named by `N, E, S, W`). Each direction
is a numeric vector, also representing a row index of the `vecinj` data frame
for each ghost node, starting at the left-most and upper cell respectively. By
setting the boundary condition to $0$, the ghost node is set as closed boundary.
#### Example
Suppose you have a `vecinj` data frame defining 2 boundary conditions and a grid
consisting of $10 \times 10$ grid cells. Grid cell $(1,1)$ should be set to the
first boundary condition and $(5,6)$ to the second. Also, all boundary
conditions for the ghost nodes should be closed. Except the southern boundary,
which should be set to the first boundary condition injection. The following
setup describes how to setup your initial script, where `n` and `m` are the
grids cell count for each direction ($n = m = 10$):
```R
vecinj_inner <- list (
l1 = c(1, 1, 1),
l2 = c(2, 5, 6)
)
vecinj_index <- list(
"N" = rep(0, n),
"E" = rep(0, m),
"S" = rep(1, n),
"W" = rep(0, m)
)
```
## Chemistry parameters
| name | type | description |
|----------------|--------|-----------------------------------|
| `database` | String | Path to the Phreeqc database |
| `input_script` | String | Path the the Phreeqc input script |
## Final setup
| name | type | description |
|----------------|----------------|------------------------------------------------------------|
| `grid` | List | Grid parameter list |
| `diffusion` | List | Diffusion parameter list |
| `chemistry` | List | Chemistry parameter list |
| `iterations` | Numeric Value | Count of iterations |
| `timesteps` | Numeric Vector | $\Delta t$ to use for specific iteration |
| `store_result` | Boolean | Indicates if results should be stored |
| `out_save` | Numeric Vector | *optional:* At which iteration the states should be stored |
### DHT setup
| name | type | description |
|-----------------|----------------|---------------------------------------------------------------------------------|
| `signif_vector` | Numeric Vector | Indicates significant digits to use for DHT rounding. Order of `props` vector. |
| `prop_type` | String Vector | Set type of species for rounding, can be left blank or set to *act* or *ignore* |

12
docs/Output.md
View File

@ -46,11 +46,13 @@ and possible to read out within a R runtime with
If running parallel there are also measured timings which are subsets of
*simtime\_​chemistry*.
| Value | Description |
|----------------------------|----------------------------------------------------|
| simtime\_workers | time spent in send/​recv loop of master |
| simtime\_chemistry\_master | sequential part of master chemistry |
| phreeqc | measured time of each worker in PHREEQC subroutine |
| Value | Description |
|-----------------------|-----------------------------------------------------------|
| chemistry\_loop | time spent in send/​recv loop of master |
| chemistry\_sequential | sequential part of master chemistry |
| idle\_master | idling time (waiting for any free worker) of the master |
| idle\_worker | idling time (waiting for work from master) of the workers |
| phreeqc\_time | accumulated times for Phreeqc calls of every worker |
### DHT usage {#DHT-usage}

2
ext/phreeqcrm

@ -1 +1 @@
Subproject commit cc5ef0ff37233d24108a793a74abe649dcae54b7
Subproject commit f862889b693507458d450c6319abe5e1dce030f7

273
include/poet/ChemSimPar.hpp
View File

@ -1,273 +0,0 @@
/*
** Copyright (C) 2018-2021 Alexander Lindemann, Max Luebke (University of
** Potsdam)
**
** Copyright (C) 2018-2022 Marco De Lucia, Max Luebke (GFZ Potsdam)
**
** POET is free software; you can redistribute it and/or modify it under the
** terms of the GNU General Public License as published by the Free Software
** Foundation; either version 2 of the License, or (at your option) any later
** version.
**
** POET is distributed in the hope that it will be useful, but WITHOUT ANY
** WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
** A PARTICULAR PURPOSE. See the GNU General Public License for more details.
**
** You should have received a copy of the GNU General Public License along with
** this program; if not, write to the Free Software Foundation, Inc., 51
** Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#ifndef CHEMSIMPAR_H
#define CHEMSIMPAR_H
#include "ChemSimSeq.hpp"
#include "DHT_Wrapper.hpp"
#include "Grid.hpp"
#include "RInside.h"
#include "SimParams.hpp"
#include "poet/PhreeqcWrapper.hpp"
#include <array>
#include <cstdint>
#include <mpi.h>
#include <string>
#include <vector>
/** Number of data elements that are kept free at each work package */
#define BUFFER_OFFSET 5
/** Message tag indicating work */
#define TAG_WORK 42
/** Message tag indicating to finish loop */
#define TAG_FINISH 43
/** Message tag indicating timing profiling */
#define TAG_TIMING 44
/** Message tag indicating collecting DHT performance */
#define TAG_DHT_PERF 45
/** Message tag indicating simulation reached the end of an itertation */
#define TAG_DHT_ITER 47
namespace poet {
/**
* @brief Class providing execution of master chemistry
*
* Providing member functions to run an iteration and to end a simulation. Also
* a loop to send and recv pkgs from workers is implemented.
*
*/
class ChemMaster : public BaseChemModule {
public:
/**
* @brief Construct a new ChemMaster object
*
* The following steps are executed to create a new object of ChemMaster:
* -# all needed simulation parameters are extracted
* -# memory is allocated
* -# distribution of work packages is calculated
*
* @param params Simulation parameters as SimParams object
* @param R_ R runtime
* @param grid_ Grid object
*/
ChemMaster(SimParams &params, RInside &R_, Grid &grid_);
/**
* @brief Destroy the ChemMaster object
*
* By freeing ChemMaster all buffers allocated in the Constructor are freed.
*
*/
~ChemMaster();
/**
* @brief Run iteration of simulation in parallel mode
*
* To run the chemistry simulation parallel following steps are done:
*
* -# 'Shuffle' the grid by previously calculated distribution of work
* packages. Convert R grid to C memory area.
* -# Start the send/recv loop.
* Detailed description in sendPkgs respectively in recvPkgs.
* -# 'Unshuffle'
* the grid and convert C memory area to R grid.
* -# Run 'master_chemistry'
*
* The main tasks are instrumented with time measurements.
*
*/
void Simulate(double dt);
/**
* @brief End chemistry simulation.
*
* Notify the worker to finish their 'work'-loop. This is done by sending
* every worker an empty message with the tag TAG_FINISH. Now the master will
* receive measured times and DHT metrics from all worker one after another.
* Finally he will write all data to the R runtime and return this function.
*
*/
void End();
/**
* @brief Get the send time
*
* Time spent in send loop.
*
* @return double sent time in seconds
*/
double getSendTime();
/**
* @brief Get the recv time
*
* Time spent in recv loop.
*
* @return double recv time in seconds
*/
double getRecvTime();
/**
* @brief Get the idle time
*
* Time master was idling in MPI_Probe of recv loop.
*
* @return double idle time in seconds
*/
double getIdleTime();
/**
* @brief Get the Worker time
*
* Time spent in whole send/recv loop.
*
* @return double worker time in seconds
*/
double getWorkerTime();
/**
* @brief Get the ChemMaster time
*
* Time spent in 'master_chemistry' R function.
*
* @return double ChemMaster time in seconds
*/
double getChemMasterTime();
/**
* @brief Get the sequential time
*
* Time master executed code which must be run sequential.
*
* @return double seqntial time in seconds.
*/
double getSeqTime();
private:
void printProgressbar(int count_pkgs, int n_wp, int barWidth = 70);
inline void sendPkgs(int &pkg_to_send, int &count_pkgs, int &free_workers,
double *&work_pointer, const double &dt,
const uint32_t iteration);
inline void recvPkgs(int &pkg_to_recv, bool to_send, int &free_workers);
void shuffleField(const std::vector<double> &in_field, uint32_t size_per_prop,
uint32_t prop_count, double *out_buffer);
void unshuffleField(const double *in_buffer, uint32_t size_per_prop,
uint32_t prop_count, std::vector<double> &out_field);
uint32_t wp_size;
bool dht_enabled;
double send_t = 0.f;
double recv_t = 0.f;
double master_idle = 0.f;
double worker_t = 0.f;
double chem_master = 0.f;
double seq_t = 0.f;
typedef struct {
char has_work;
double *send_addr;
} worker_struct;
worker_struct *workerlist;
double *send_buffer;
double *mpi_buffer;
std::vector<uint32_t> wp_sizes_vector;
poet::StateMemory *state;
};
/**
* @brief Class providing execution of worker chemistry
*
* Providing mainly a function to loop and wait for messages from the master.
*
*/
class ChemWorker : public BaseChemModule {
public:
/**
* @brief Construct a new ChemWorker object
*
* The following steps are executed to create a new object of ChemWorker:
* -# all needed simulation parameters are extracted
* -# memory is allocated
* -# Preparetion to create a DHT
* -# and finally create a new DHT_Wrapper
*
* @param params Simulation parameters as SimParams object
* @param R_ R runtime
* @param grid_ Grid object
* @param dht_comm Communicator addressing all processes marked as worker
*/
ChemWorker(SimParams &params, RInside &R_, Grid &grid_, MPI_Comm dht_comm);
void InitModule(poet::ChemistryParams &chem_params);
/**
* @brief Destroy the ChemWorker object
*
* Therefore all buffers are freed and the DHT_Wrapper object is destroyed.
*
*/
~ChemWorker();
/**
* @brief Start the 'work' loop
*
* Loop in an endless loop. At the beginning probe for a message from the
* master process. If there is a receivable message evaluate the message tag.
*
*/
void loop();
private:
void doWork(MPI_Status &probe_status);
void postIter();
void finishWork();
void writeFile();
void readFile();
uint32_t wp_size;
uint32_t dht_size_per_process;
uint32_t iteration = 0;
bool dht_enabled;
bool dt_differ;
int dht_snaps;
std::string dht_file;
std::vector<bool> dht_flags;
double *mpi_buffer_results;
DHT_Wrapper *dht;
std::array<double, 3> timing;
double idle_t = 0.f;
uint32_t phreeqc_count = 0;
double *mpi_buffer;
uint32_t ncomps;
std::string out_dir;
PhreeqcWrapper *phreeqc_rm = std::nullptr_t();
};
} // namespace poet
#endif // CHEMSIMPAR_H

131
include/poet/ChemSimSeq.hpp
View File

@ -1,131 +0,0 @@
/*
** Copyright (C) 2018-2021 Alexander Lindemann, Max Luebke (University of
** Potsdam)
**
** Copyright (C) 2018-2022 Marco De Lucia, Max Luebke (GFZ Potsdam)
**
** POET is free software; you can redistribute it and/or modify it under the
** terms of the GNU General Public License as published by the Free Software
** Foundation; either version 2 of the License, or (at your option) any later
** version.
**
** POET is distributed in the hope that it will be useful, but WITHOUT ANY
** WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
** A PARTICULAR PURPOSE. See the GNU General Public License for more details.
**
** You should have received a copy of the GNU General Public License along with
** this program; if not, write to the Free Software Foundation, Inc., 51
** Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#ifndef CHEMSIMSEQ_H
#define CHEMSIMSEQ_H
#include "DHT_Wrapper.hpp"
#include "Grid.hpp"
#include "PhreeqcWrapper.hpp"
#include "RInside.h"
#include "SimParams.hpp"
#include <bits/stdint-uintn.h>
#include <cstddef>
#include <cstdint>
#include <mpi.h>
#include <string>
#include <vector>
/** Number of data elements that are kept free at each work package */
#define BUFFER_OFFSET 5
/** Message tag indicating work */
#define TAG_WORK 42
/** Message tag indicating to finish loop */
#define TAG_FINISH 43
/** Message tag indicating timing profiling */
#define TAG_TIMING 44
/** Message tag indicating collecting DHT performance */
#define TAG_DHT_PERF 45
/** Message tag indicating simulation reached the end of an itertation */
#define TAG_DHT_ITER 47
namespace poet {
/**
* @brief Base class of the chemical simulation
*
* Providing member functions to run an iteration and to end a simulation. Also
* containing basic parameters for simulation.
*
*/
class BaseChemModule {
public:
BaseChemModule(SimParams &params, RInside &R_, Grid &grid_);
virtual void InitModule(poet::ChemistryParams &chem_params){};
static constexpr const char *CHEMISTRY_MODULE_NAME = "state_c";
protected:
int world_rank;
int world_size;
RInside &R;
Grid &grid;
double chem_t = 0.f;
double current_sim_time = 0;
uint32_t n_cells_per_prop;
std::vector<std::string> prop_names;
};
class ChemSeq : public BaseChemModule {
public:
/**
* @brief Construct a new ChemSim object
*
* Creating a new instance of class ChemSim will just extract simulation
* parameters from SimParams object.
*
* @param params SimParams object
* @param R_ R runtime
* @param grid_ Initialized grid
*/
ChemSeq(SimParams &params, RInside &R_, Grid &grid_);
~ChemSeq();
void InitModule(poet::ChemistryParams &chem_params);
/**
* @brief Run iteration of simulation in sequential mode
*
* This will call the correspondending R function slave_chemistry, followed by
* the execution of master_chemistry.
*
* @todo change function name. Maybe 'slave' to 'seq'.
*
*/
void Simulate(double dt);
/**
* @brief End simulation
*
* End the simulation by distribute the measured runtime of simulation to the
* R runtime.
*
*/
void End();
/**
* @brief Get the Chemistry Time
*
* @return double Runtime of sequential chemistry simulation in seconds
*/
double getChemistryTime();
private:
poet::StateMemory *state;
PhreeqcWrapper *phreeqc_rm = std::nullptr_t();
};
} // namespace poet
#endif // CHEMSIMSEQ_H

413
include/poet/ChemistryModule.hpp Normal file
View File

@ -0,0 +1,413 @@
#ifndef CHEMISTRYMODULE_H_
#define CHEMISTRYMODULE_H_
#include "IrmResult.h"
#include "PhreeqcRM.h"
#include "poet/DHT_Wrapper.hpp"
#include <cstddef>
#include <cstdint>
#include <mpi.h>
#include <string>
#include <unordered_map>
#include <vector>
namespace poet {
/**
* \brief Wrapper around PhreeqcRM to provide POET specific parallelization with
* easy access.
*/
class ChemistryModule : public PhreeqcRM {
public:
#ifdef POET_USE_PRM
/**
* Creates a new instance of Chemistry module with given grid cell count and
* using MPI communicator.
*
* Constructor is just a wrapper around PhreeqcRM's constructor, so just calls
* the base class constructor.
*
* \param nxyz Count of grid cells.
* \param communicator MPI communicator where work will be distributed.
*/
ChemistryModule(uint32_t nxyz, MPI_Comm communicator)
: PhreeqcRM(nxyz, communicator), n_cells(nxyz){};
#else
/**
* Creates a new instance of Chemistry module with given grid cell count, work
* package size and communicator.
*
* This constructor shall only be called by the master. To create workers, see
* ChemistryModule::createWorker .
*
* When the use of parallelization is intended, the nxyz value shall be set to
* 1 to save memory and only one node is needed for initialization.
*
* \param nxyz Count of grid cells to allocate and initialize for each
* process. For parellel use set to 1 at the master.
* \param wp_size Count of grid cells to fill each work package at maximum.
* \param communicator MPI communicator to distribute work in.
*/
ChemistryModule(uint32_t nxyz, uint32_t wp_size, MPI_Comm communicator);
/**
* Deconstructor, which frees DHT data structure if used.
*/
~ChemistryModule();
#endif
/**
* Parses the input script and extract information needed during runtime.
*
* **Only run by master**.
*
* Database must be loaded beforehand.
*
* \param input_script_path Path to input script to parse.
*/
void RunInitFile(const std::string &input_script_path);
/**
* Run the chemical simulation with parameters set.
*/
void RunCells();
/**
* Returns the chemical field.
*/
auto GetField() const { return this->field; }
/**
* Returns all known species names, including not only aqueous species, but
* also equilibrium, exchange, surface and kinetic reactants.
*/
auto GetPropNames() const { return this->prop_names; }
/**
* Return the accumulated runtime in seconds for chemical simulation.
*/
auto GetChemistryTime() const { return this->chem_t; }
#ifndef POET_USE_PRM
/**
* Create a new worker instance inside given MPI communicator.
*
* Wraps communication needed before instanciation can take place.
*
* \param communicator MPI communicator to distribute work in.
*
* \returns A worker instance with fixed work package size.
*/
static ChemistryModule createWorker(MPI_Comm communicator);
/**
* Default work package size.
*/
static constexpr uint32_t CHEM_DEFAULT_WP_SIZE = 5;
/**
* Intended to alias input parameters for grid initialization with a single
* value per species.
*/
using SingleCMap = std::unordered_map<std::string, double>;
/**
* Intended to alias input parameters for grid initialization with mutlitple
* values per species.
*/
using VectorCMap = std::unordered_map<std::string, std::vector<double>>;
/**
* Enumerating DHT file options
*/
enum {
DHT_FILES_DISABLED = 0, //!< disabled file output
DHT_FILES_SIMEND, //!< only output of snapshot after simulation
DHT_FILES_ITEREND //!< output snapshots after each iteration
};
/**
* Initializes field with a "DataFrame" containing a single value for each
* species.
*
* Each species must have been previously found by
* ChemistryModule::RunInitFile.
*
* \exception std::domain_error Species name of map is not defined.
*
* \param n_cells Count of cells for each species.
* \param mapped_values Unordered map containing one double value for each
* specified species.
*/
void InitializeField(uint32_t n_cells, const SingleCMap &mapped_values);
/**
* Initializes field with a "DataFrame" containing a vector of values for each
* species.
*
* Each species must have been previously found by
* ChemistryModule::RunInitFile.
*
* There is no check if vector length matches count of grid cells defined.
*
* \exception std::domain_error Species name of map is not defined.
*
* \param mapped_values Unordered map containing a vector of multiple values.
* Size of the vectors shall be the count of grid cells defined previously.
*/
void InitializeField(const VectorCMap &mapped_values);
/**
* **Only called by workers!** Start the worker listening loop.
*/
void WorkerLoop();
/**
* **Called by master** Advise the workers to break the loop.
*/
void MasterLoopBreak();
/**
* **Master only** Enables DHT usage for the workers.
*
* If called multiple times enabling the DHT, the current state of DHT will be
* overwritten with a new instance of the DHT.
*
* \param enable Enables or disables the usage of the DHT.
* \param size_mb Size in megabyte to allocate for the DHT if enabled.
*/
void SetDHTEnabled(bool enable, uint32_t size_mb);
/**
* **Master only** Set DHT snapshots to specific mode.
*
* \param type DHT snapshot mode.
* \param out_dir Path to output DHT snapshots.
*/
void SetDHTSnaps(int type, const std::string &out_dir);
/**
* **Master only** Set the vector with significant digits to round before
* inserting into DHT.
*
* \param signif_vec Vector defining significant digit for each species. Order
* is defined by prop_type vector (ChemistryModule::GetPropNames).
*/
void SetDHTSignifVector(std::vector<uint32_t> signif_vec);
/**
* **Master only** Set the DHT rounding type of each species. See
* DHT_PROP_TYPES enumemartion for explanation.
*
* \param proptype_vec Vector defining DHT prop type for each species.
*/
void SetDHTPropTypeVector(std::vector<uint32_t> proptype_vec);
/**
* **Master only** Load the state of the DHT from given file.
*
* \param input_file File to load data from.
*/
void ReadDHTFile(const std::string &input_file);
/**
* Overwrite the current field state by another field.
*
* There are no checks if new vector dimensions matches expected sizes etc.
*
* \param field New input field. Current field state of the instance will be
* overwritten.
*/
void SetField(const std::vector<double> &field) { this->field = field; }
/**
* **Master only** Return count of grid cells.
*/
auto GetNCells() const { return this->n_cells; }
/**
* **Master only** Return work package size.
*/
auto GetWPSize() const { return this->wp_size; }
/**
* **Master only** Return the time in seconds the master spent waiting for any
* free worker.
*/
auto GetMasterIdleTime() const { return this->idle_t; }
/**
* **Master only** Return the time in seconds the master spent in sequential
* part of the simulation, including times for shuffling/unshuffling field
* etc.
*/
auto GetMasterSequentialTime() const { return this->seq_t; }
/**
* **Master only** Return the time in seconds the master spent in the
* send/receive loop.
*/
auto GetMasterLoopTime() const { return this->send_recv_t; }
/**
* **Master only** Collect and return all accumulated timings recorded by
* workers to run Phreeqc simulation.
*
* \return Vector of all accumulated Phreeqc timings.
*/
std::vector<double> GetWorkerPhreeqcTimings() const;
/**
* **Master only** Collect and return all accumulated timings recorded by
* workers to get values from the DHT.
*
* \return Vector of all accumulated DHT get times.
*/
std::vector<double> GetWorkerDHTGetTimings() const;
/**
* **Master only** Collect and return all accumulated timings recorded by
* workers to write values to the DHT.
*
* \return Vector of all accumulated DHT fill times.
*/
std::vector<double> GetWorkerDHTFillTimings() const;
/**
* **Master only** Collect and return all accumulated timings recorded by
* workers waiting for work packages from the master.
*
* \return Vector of all accumulated waiting times.
*/
std::vector<double> GetWorkerIdleTimings() const;
/**
* **Master only** Collect and return DHT hits of all workers.
*
* \return Vector of all count of DHT hits.
*/
std::vector<uint32_t> GetWorkerDHTHits() const;
/**
* **Master only** Collect and return DHT misses of all workers.
*
* \return Vector of all count of DHT misses.
*/
std::vector<uint32_t> GetWorkerDHTMiss() const;
/**
* **Master only** Collect and return DHT evictions of all workers.
*
* \return Vector of all count of DHT evictions.
*/
std::vector<uint32_t> GetWorkerDHTEvictions() const;
#endif
protected:
#ifdef POET_USE_PRM
void PrmToPoetField(std::vector<double> &field);
#else
enum {
CHEM_INIT,
CHEM_WP_SIZE,
CHEM_DHT_ENABLE,
CHEM_DHT_SIGNIF_VEC,
CHEM_DHT_PROP_TYPE_VEC,
CHEM_DHT_SNAPS,
CHEM_DHT_READ_FILE,
CHEM_WORK_LOOP,
CHEM_PERF,
CHEM_BREAK_MAIN_LOOP
};
enum { LOOP_WORK, LOOP_END };
enum {
WORKER_PHREEQC,
WORKER_DHT_GET,
WORKER_DHT_FILL,
WORKER_IDLE,
WORKER_DHT_HITS,
WORKER_DHT_MISS,
WORKER_DHT_EVICTIONS
};
struct worker_s {
double phreeqc_t = 0.;
double dht_get = 0.;
double dht_fill = 0.;
double idle_t = 0.;
};
struct worker_info_s {
char has_work = 0;
double *send_addr;
};
using worker_list_t = std::vector<struct worker_info_s>;
using workpointer_t = std::vector<double>::iterator;
void MasterRunParallel();
void MasterRunSequential();
void MasterSendPkgs(worker_list_t &w_list, workpointer_t &work_pointer,
int &pkg_to_send, int &count_pkgs, int &free_workers,
double dt, uint32_t iteration,
const std::vector<uint32_t> &wp_sizes_vector);
void MasterRecvPkgs(worker_list_t &w_list, int &pkg_to_recv, bool to_send,
int &free_workers);
std::vector<double> MasterGatherWorkerTimings(int type) const;
std::vector<uint32_t> MasterGatherWorkerMetrics(int type) const;
void WorkerProcessPkgs(struct worker_s &timings, uint32_t &iteration);
void WorkerDoWork(MPI_Status &probe_status, int double_count,
struct worker_s &timings);
void WorkerPostIter(MPI_Status &prope_status, uint32_t iteration);
void WorkerPostSim(uint32_t iteration);
void WorkerWriteDHTDump(uint32_t iteration);
void WorkerReadDHTDump(const std::string &dht_input_file);
void WorkerPerfToMaster(int type, const struct worker_s &timings);
void WorkerMetricsToMaster(int type);
IRM_RESULT WorkerRunWorkPackage(std::vector<double> &vecWP,
std::vector<int32_t> &vecMapping,
double dSimTime, double dTimestep);
void GetWPFromInternals(std::vector<double> &vecWP, uint32_t wp_size);
void SetInternalsFromWP(const std::vector<double> &vecWP, uint32_t wp_size);
std::vector<uint32_t> CalculateWPSizesVector(uint32_t n_cells,
uint32_t wp_size) const;
int comm_size, comm_rank;
MPI_Comm group_comm;
bool is_sequential;
bool is_master;
uint32_t wp_size;
bool dht_enabled = false;
int dht_snaps_type = DHT_FILES_DISABLED;
std::string dht_file_out_dir;
poet::DHT_Wrapper *dht = std::nullptr_t();
static constexpr uint32_t BUFFER_OFFSET = 5;
inline void ChemBCast(void *buf, int count, MPI_Datatype datatype) const {
MPI_Bcast(buf, count, datatype, 0, this->group_comm);
}
inline void PropagateFunctionType(int &type) const {
ChemBCast(&type, 1, MPI_INT);
}
double simtime = 0.;
double idle_t = 0.;
double seq_t = 0.;
double send_recv_t = 0.;
#endif
double chem_t = 0.;
uint32_t n_cells = 0;
uint32_t prop_count = 0;
std::vector<std::string> prop_names;
std::vector<double> field;
std::vector<uint32_t> speciesPerModule;
static constexpr uint32_t MODULE_COUNT = 5;
};
} // namespace poet
#endif // CHEMISTRYMODULE_H_

8
include/poet/DHT_Types.hpp Normal file
View File

@ -0,0 +1,8 @@
#ifndef DHT_TYPES_H_
#define DHT_TYPES_H_
namespace poet {
enum DHT_PROP_TYPES { DHT_TYPE_DEFAULT, DHT_TYPE_ACT, DHT_TYPE_IGNORE };
}
#endif // DHT_TYPES_H_

134
include/poet/DHT_Wrapper.hpp
View File

@ -21,8 +21,6 @@
#ifndef DHT_WRAPPER_H
#define DHT_WRAPPER_H
#include "SimParams.hpp"
#include <cstdint>
#include <string>
#include <vector>
@ -46,7 +44,7 @@ using DHT_ResultObject = struct DHTResobj {
std::vector<std::vector<double>> results;
std::vector<bool> needPhreeqc;
void ResultsToMapping(std::vector<int32_t> &curr_mapping);
void ResultsToMapping(std::vector<int32_t> &curr_mapping);
void ResultsToWP(std::vector<double> &curr_wp);
};
@ -79,15 +77,14 @@ public:
* calling DHT_create with all given parameters. Also the fuzzing buffer will
* be allocated and all needed parameters extracted from simparams struct.
*
* @param params Simulation parameter object
* @param dht_comm Communicator which addresses all participating DHT
* processes
* @param buckets_per_process Count of buckets to allocate for each process
* @param key_count Count of key entries
* @param data_count Count of data entries
*/
DHT_Wrapper(const poet::SimParams &params, MPI_Comm dht_comm,
uint32_t dht_size, uint32_t key_count, uint32_t data_count);
DHT_Wrapper(MPI_Comm dht_comm, uint32_t dht_size, uint32_t key_count,
uint32_t data_count);
/**
* @brief Destroy the dht wrapper object
*
@ -173,138 +170,43 @@ public:
*
* @return uint64_t Count of hits
*/
uint64_t getHits();
auto getHits() { return this->dht_hits; };
/**
* @brief Get the Misses object
*
* @return uint64_t Count of read misses
*/
uint64_t getMisses();
auto getMisses() { return this->dht_miss; };
/**
* @brief Get the Evictions object
*
* @return uint64_t Count of evictions
*/
uint64_t getEvictions();
auto getEvictions() { return this->dht_evictions; };
void SetSignifVector(std::vector<uint32_t> signif_vec);
void SetPropTypeVector(std::vector<uint32_t> prop_type_vec);
private:
uint32_t key_count;
uint32_t data_count;
/**
* @brief Transform given workpackage into DHT key
*
* A given workpackage will be transformed into a DHT key by rounding each
* value of a workpackage to a given significant digit. Three different types
* of variables 'act', 'logact' and 'ignore' are used. Those types are given
* via the dht_signif_vector.
*
* If a variable is defined as 'act', dht_log is true and non-negative, the
* logarithm with base 10 will be applied. After that the value is negated. In
* case the value is 0 the fuzzing_buffer is also set to 0 at this position.
* If the value is negative a correspondending warning will be printed to
* stderr and the fuzzing buffer will be set to 0 at this index.
*
* If a variable is defined as 'logact' the value will be cut after the
* significant digit.
*
* If a variable ist defined as 'ignore' the fuzzing_buffer will be set to 0
* at the index of the variable.
*
* If dt_differ is true the current time step of the simulation will be set at
* the end of the fuzzing_buffer.
*
* @param var_count Count of variables for the current work package
* @param key Pointer to work package handled as the key
* @param dt Current time step of the simulation
*/
std::vector<DHT_Keyelement> fuzzForDHT(int var_count, void *key, double dt);
/**
* @brief DHT handle
*
* Stores information about the DHT. Will be used as a handle for each DHT
* library call.
*
*/
DHT *dht_object;
/**
* @brief Count of hits
*
* The counter will be incremented if a previously simulated workpackage can
* be retrieved with a given key.
*
*/
uint64_t dht_hits = 0;
std::vector<DHT_Keyelement> fuzzForDHT(int var_count, void *key, double dt);
/**
* @brief Count of read misses
*
* The counter will be incremented if a given key doesn't retrieve a value
* from the DHT.
*
*/
uint64_t dht_miss = 0;
uint32_t dht_hits = 0;
uint32_t dht_miss = 0;
uint32_t dht_evictions = 0;
/**
* @brief Count of evictions
*
* If a value in the DHT must be evicted because of lack of space/reaching the
* last index etc., this counter will be incremented.
*
*/
uint64_t dht_evictions = 0;
std::vector<uint32_t> dht_signif_vector;
std::vector<uint32_t> dht_prop_type_vector;
/**
* @brief Rounded work package values
*
* Stores rounded work package values and serves as the DHT key pointer.
*
*/
double *fuzzing_buffer;
/**
* @brief Indicates change in time step during simulation
*
* If set to true, the time step of simulation will differ between iterations,
* so the current time step must be stored inside the DHT key. Otherwise wrong
* values would be obtained.
*
* If set to false the time step doesn't need to be stored in the DHT key.
*
*/
bool dt_differ;
/**
* @brief Logarithm before rounding
*
* Indicates if the logarithm with base 10 will be applied to a variable
* before rounding.
*
* Defaults to true.
*
*/
bool dht_log;
/**
* @brief Significant digits for each variable
*
* Stores the rounding/significant digits for each variable of the work
* package.
*
*/
std::vector<int> dht_signif_vector;
/**
* @brief Type of each variable
*
* Defines the type of each variable of the work package.
*
*/
std::vector<std::string> dht_prop_type_vector;
static constexpr int DHT_KEY_SIGNIF_DEFAULT = 5;
static constexpr int DHT_KEY_SIGNIF_TOTALS = 12;
static constexpr int DHT_KEY_SIGNIF_CHARGE = 3;
};
} // namespace poet

2
include/poet/DiffusionModule.hpp
View File

@ -39,7 +39,7 @@ namespace poet {
*
*/
constexpr const char *DIFFUSION_MODULE_NAME = "state_t";
constexpr const char *DIFFUSION_MODULE_NAME = "state_T";
class DiffusionModule {
public:

3
include/poet/Grid.hpp
View File

@ -22,6 +22,7 @@
#define GRID_H
#include "poet/SimParams.hpp"
#include <RInside.h>
#include <Rcpp.h>
#include <array>
#include <cstddef>
@ -97,6 +98,8 @@ public:
std::string module_name = poet::GRID_MODULE_NAME) const
-> std::vector<double>;
void WriteFieldsToR(RInside &R);
private:
std::vector<FlowInputOutputInfo> flow_vec;

50
include/poet/PhreeqcWrapper.hpp
View File

@ -1,50 +0,0 @@
#ifndef PHREEQCWRAPPER_H_
#define PHREEQCWRAPPER_H_
#include "IrmResult.h"
#include "poet/SimParams.hpp"
#include <PhreeqcRM.h>
#include <cstdint>
#include <string>
#include <vector>
namespace poet {
class PhreeqcWrapper : public PhreeqcRM {
public:
PhreeqcWrapper(uint32_t inxyz);
void SetupAndLoadDB(const poet::ChemistryParams &chemPar);
void InitFromFile(const std::string &strInputFile);
auto GetSpeciesCount() -> uint32_t;
auto GetSpeciesCountByModule() -> const std::vector<uint32_t> &;
auto GetSpeciesNamesFull() -> const std::vector<std::string> &;
void SetInternalsFromWP(const std::vector<double> &vecWP,
uint32_t iCurrWPSize);
void GetWPFromInternals(std::vector<double> &vecWP, uint32_t iCurrWPSize);
auto ReplaceTotalsByPotentials(const std::vector<double> &vecWP,
uint32_t iCurrWPSize) -> std::vector<double>;
IRM_RESULT RunWorkPackage(std::vector<double> &vecWP,
std::vector<int32_t> &vecMapping, double dSimTime,
double dTimestep);
private:
void InitInternals();
inline void SetMappingForWP(uint32_t iCurrWPSize);
static constexpr int MODULE_COUNT = 5;
static constexpr uint32_t DHT_SELOUT = 100;
uint32_t iWPSize;
bool bInactiveCells = false;
uint32_t iSpeciesCount;
std::vector<uint32_t> vecSpeciesPerModule;
std::vector<std::string> vecSpeciesNames;
std::vector<int> vecDefMapping;
};
} // namespace poet
#endif // PHREEQCWRAPPER_H_

86
include/poet/SimParams.hpp
View File

@ -31,8 +31,8 @@
#include <Rcpp.h>
// BSD-licenced
/** Standard DHT Size (Defaults to 1 GiB) */
#define DHT_SIZE_PER_PROCESS 1073741824
/** Standard DHT Size. Defaults to 1 GB (1000 MB) */
constexpr uint32_t DHT_SIZE_PER_PROCESS_MB = 1E3;
/** Standard work package size */
#define WORK_PACKAGE_SIZE_DEFAULT 5
@ -167,16 +167,6 @@ public:
*/
void initVectorParams(RInside &R, int col_count);
/**
* @brief Set if dt differs
*
* Set a boolean variable if the timestep differs between iterations of
* simulation.
*
* @param dt_differ Boolean value, if dt differs
*/
void setDtDiffer(bool dt_differ);
/**
* @brief Get the numerical params struct
*
@ -185,7 +175,7 @@ public:
*
* @return t_simparams Parameter struct
*/
t_simparams getNumParams() const;
auto getNumParams() const { return this->simparams; };
/**
* @brief Get the DHT_Signif_Vector
@ -196,7 +186,7 @@ public:
* @return std::vector<int> Vector of integers containing information about
* significant digits
*/
std::vector<int> getDHTSignifVector() const;
auto getDHTSignifVector() const { return this->dht_signif_vector; };
/**
* @brief Get the DHT_Prop_Type_Vector
@ -206,7 +196,7 @@ public:
* @return std::vector<std::string> Vector if strings defining a type of a
* variable
*/
std::vector<std::string> getDHTPropTypeVector() const;
auto getDHTPropTypeVector() const { return this->dht_prop_type_vector; };
/**
* @brief Return name of DHT snapshot.
@ -216,7 +206,7 @@ public:
*
* @return std::string Absolute paht to the DHT snapshot
*/
std::string_view getDHTFile() const;
auto getDHTFile() const { return this->dht_file; };
/**
* @brief Get the filesim name
@ -226,7 +216,7 @@ public:
*
* @return std::string Absolute path to R file
*/
std::string_view getFilesim() const;
auto getFilesim() const { return this->filesim; };
/**
* @brief Get the output directory
@ -236,71 +226,23 @@ public:
*
* @return std::string Absolute path to output path
*/
std::string_view getOutDir() const;
auto getOutDir() const { return this->out_dir; };
private:
/**
* @brief Validate program parameters and flags
*
* Therefore this function iterates over the list of flags and parameters and
* compare them to the class member flagList and paramList. If a program
* argument is not included it is put to a list. This list will be returned.
*
* @return std::list<std::string> List with all unknown parameters. Might be
* empty.
*/
std::list<std::string> validateOptions(argh::parser cmdl);
/**
* @brief Contains all valid program flags.
*
*/
std::set<std::string> flaglist{"ignore-result", "dht", "dht-nolog"};
const std::set<std::string> flaglist{"ignore-result", "dht", "dht-nolog"};
const std::set<std::string> paramlist{"work-package-size", "dht-signif",
"dht-strategy", "dht-size",
"dht-snaps", "dht-file"};
/**
* @brief Contains all valid program parameters.
*
*/
std::set<std::string> paramlist{"work-package-size", "dht-signif",
"dht-strategy", "dht-size",
"dht-snaps", "dht-file"};
/**
* @brief Struct containing all simulation parameters
*
* Contains only those values which are standard arithmetic C types.
*
*/
t_simparams simparams;
/**
* @brief Defines significant digits for each variable of a grid cell
*
*/
std::vector<int> dht_signif_vector;
std::vector<uint32_t> dht_signif_vector;
std::vector<uint32_t> dht_prop_type_vector;
/**
* @brief Defines the type of a variable
*
*/
std::vector<std::string> dht_prop_type_vector;
/**
* @brief Absolute path to a DHT snapshot
*
*/
std::string dht_file;
/**
* @brief Absolute path to R file containing simulation definitions
*
*/
std::string filesim;
/**
* @brief Absolute path to output dir
*
*/
std::string out_dir;
};
} // namespace poet

12
src/BaseChemModule.cpp
View File

@ -1,12 +0,0 @@
#include <poet/ChemSimSeq.hpp>
poet::BaseChemModule::BaseChemModule(SimParams &params, RInside &R_,
Grid &grid_)
: R(R_), grid(grid_) {
t_simparams tmp = params.getNumParams();
this->world_rank = tmp.world_rank;
this->world_size = tmp.world_size;
this->prop_names = this->grid.GetPropNames();
this->n_cells_per_prop = this->grid.GetTotalCellCount();
}

8
src/CMakeLists.txt
View File

@ -7,14 +7,10 @@ file(GLOB poet_lib_SRC
find_library(MATH_LIBRARY m)
find_library(CRYPTO_LIBRARY crypto)
option(POET_DHT_DEBUG "Build with DHT debug info" OFF)
add_library(poet_lib ${poet_lib_SRC})
target_include_directories(poet_lib PUBLIC ${PROJECT_SOURCE_DIR}/include)
target_link_libraries(poet_lib PUBLIC
MPI::MPI_C ${MATH_LIBRARY} ${CRYPTO_LIBRARY} RRuntime tug PhreeqcRM DataStructures)
MPI::MPI_CXX ${MATH_LIBRARY} ${CRYPTO_LIBRARY} RRuntime tug PhreeqcRM DataStructures ChemistryModule)
target_compile_definitions(poet_lib PUBLIC STRICT_R_HEADERS OMPI_SKIP_MPICXX)
if(POET_DHT_DEBUG)
target_compile_definitions(poet_lib PRIVATE DHT_STATISTICS)
endif()
add_subdirectory(ChemistryModule)

445
src/ChemMaster.cpp
View File

@ -1,445 +0,0 @@
/*
** Copyright (C) 2018-2021 Alexander Lindemann, Max Luebke (University of
** Potsdam)
**
** Copyright (C) 2018-2022 Marco De Lucia, Max Luebke (GFZ Potsdam)
**
** POET is free software; you can redistribute it and/or modify it under the
** terms of the GNU General Public License as published by the Free Software
** Foundation; either version 2 of the License, or (at your option) any later
** version.
**
** POET is distributed in the hope that it will be useful, but WITHOUT ANY
** WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
** A PARTICULAR PURPOSE. See the GNU General Public License for more details.
**
** You should have received a copy of the GNU General Public License along with
** this program; if not, write to the Free Software Foundation, Inc., 51
** Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include "poet/ChemSimPar.hpp"
#include "poet/ChemSimSeq.hpp"
#include "poet/DiffusionModule.hpp"
#include "poet/Grid.hpp"
#include "poet/SimParams.hpp"
#include <Rcpp.h>
#include <array>
#include <cstdint>
#include <cstring>
#include <iostream>
#include <string>
#include <vector>
using namespace poet;
using namespace std;
using namespace Rcpp;
ChemMaster::ChemMaster(SimParams &params, RInside &R_, Grid &grid_)
: BaseChemModule(params, R_, grid_) {
t_simparams tmp = params.getNumParams();
this->wp_size = tmp.wp_size;
this->dht_enabled = tmp.dht_enabled;
uint32_t grid_size = grid.GetTotalCellCount() * this->prop_names.size();
/* allocate memory */
this->workerlist = new worker_struct[this->world_size - 1];
std::memset(this->workerlist, '\0', sizeof(worker_struct) * (world_size - 1));
this->send_buffer =
new double[this->wp_size * this->prop_names.size() + BUFFER_OFFSET];
this->mpi_buffer = new double[grid_size];
/* calculate distribution of work packages */
uint32_t mod_pkgs = grid.GetTotalCellCount() % this->wp_size;
uint32_t n_packages = (uint32_t)(grid.GetTotalCellCount() / this->wp_size) +
(mod_pkgs != 0 ? 1 : 0);
Rcpp::Function wp_f("GetWorkPackageSizesVector");
this->wp_sizes_vector = Rcpp::as<std::vector<uint32_t>>(
wp_f(n_packages, this->wp_size, grid.GetTotalCellCount()));
this->state = this->grid.RegisterState(
poet::BaseChemModule::CHEMISTRY_MODULE_NAME, this->prop_names);
std::vector<double> &field = this->state->mem;
field.resize(this->n_cells_per_prop * this->prop_names.size());
for (uint32_t i = 0; i < this->prop_names.size(); i++) {
std::vector<double> prop_vec =
this->grid.GetSpeciesByName(this->prop_names[i]);
std::copy(prop_vec.begin(), prop_vec.end(),
field.begin() + (i * this->n_cells_per_prop));
}
}
ChemMaster::~ChemMaster() {
delete this->workerlist;
delete this->send_buffer;
delete this->mpi_buffer;
}
void ChemMaster::Simulate(double dt) {
/* declare most of the needed variables here */
double chem_a, chem_b;
double seq_a, seq_b, seq_c, seq_d;
double worker_chemistry_a, worker_chemistry_b;
double sim_e_chemistry, sim_f_chemistry;
int pkg_to_send, pkg_to_recv;
int free_workers;
int i_pkgs;
uint32_t iteration;
/* start time measurement of whole chemistry simulation */
chem_a = MPI_Wtime();
/* start time measurement of sequential part */
seq_a = MPI_Wtime();
std::vector<double> &field = this->state->mem;
// HACK: transfer the field into R data structure serving as input for phreeqc
R["TMP_T"] = field;
R.parseEvalQ("mysetup$state_T <- setNames(data.frame(matrix(TMP_T, "
"ncol=length(mysetup$grid$props), nrow=" +
std::to_string(this->n_cells_per_prop) +
")), mysetup$grid$props)");
/* shuffle grid */
// grid.shuffleAndExport(mpi_buffer);
this->shuffleField(field, this->n_cells_per_prop, this->prop_names.size(),
mpi_buffer);
/* retrieve needed data from R runtime */
iteration = (int)R.parseEval("mysetup$iter");
// dt = (double)R.parseEval("mysetup$requested_dt");
/* setup local variables */
pkg_to_send = wp_sizes_vector.size();
pkg_to_recv = wp_sizes_vector.size();
double *work_pointer = mpi_buffer;
free_workers = world_size - 1;
i_pkgs = 0;
/* end time measurement of sequential part */
seq_b = MPI_Wtime();
seq_t += seq_b - seq_a;
/* start time measurement of chemistry time needed for send/recv loop */
worker_chemistry_a = MPI_Wtime();
/* start send/recv loop */
// while there are still packages to recv
while (pkg_to_recv > 0) {
// print a progressbar to stdout
printProgressbar((int)i_pkgs, (int)wp_sizes_vector.size());
// while there are still packages to send
if (pkg_to_send > 0) {
// send packages to all free workers ...
sendPkgs(pkg_to_send, i_pkgs, free_workers, work_pointer, dt, iteration);
}
// ... and try to receive them from workers who has finished their work
recvPkgs(pkg_to_recv, pkg_to_send > 0, free_workers);
}
// Just to complete the progressbar
cout << endl;
/* stop time measurement of chemistry time needed for send/recv loop */
worker_chemistry_b = MPI_Wtime();
worker_t = worker_chemistry_b - worker_chemistry_a;
/* start time measurement of sequential part */
seq_c = MPI_Wtime();
/* unshuffle grid */
// grid.importAndUnshuffle(mpi_buffer);
this->unshuffleField(mpi_buffer, this->n_cells_per_prop,
this->prop_names.size(), field);
/* do master stuff */
/* start time measurement of master chemistry */
sim_e_chemistry = MPI_Wtime();
// HACK: We don't need to call master_chemistry here since our result is
// already written to the memory as a data frame
// R.parseEvalQ("mysetup <- master_chemistry(setup=mysetup, data=result)");
R["TMP_T"] = Rcpp::wrap(field);
R.parseEval(std::string(
"mysetup$state_C <- setNames(data.frame(matrix(TMP_T, nrow=" +
to_string(this->n_cells_per_prop) + ")), mysetup$grid$props)"));
/* end time measurement of master chemistry */
sim_f_chemistry = MPI_Wtime();
chem_master += sim_f_chemistry - sim_e_chemistry;
/* end time measurement of sequential part */
seq_d = MPI_Wtime();
seq_t += seq_d - seq_c;
/* end time measurement of whole chemistry simulation */
chem_b = MPI_Wtime();
chem_t += chem_b - chem_a;
/* advise workers to end chemistry iteration */
for (int i = 1; i < world_size; i++) {
MPI_Send(NULL, 0, MPI_DOUBLE, i, TAG_DHT_ITER, MPI_COMM_WORLD);
}
this->current_sim_time += dt;
}
inline void ChemMaster::sendPkgs(int &pkg_to_send, int &count_pkgs,
int &free_workers, double *&work_pointer,
const double &dt, const uint32_t iteration) {
/* declare variables */
double master_send_a, master_send_b;
int local_work_package_size;
int end_of_wp;
/* start time measurement */
master_send_a = MPI_Wtime();
/* search for free workers and send work */
for (int p = 0; p < world_size - 1; p++) {
if (workerlist[p].has_work == 0 && pkg_to_send > 0) /* worker is free */ {
/* to enable different work_package_size, set local copy of
* work_package_size to pre-calculated work package size vector */
local_work_package_size = (int)wp_sizes_vector[count_pkgs];
count_pkgs++;
/* note current processed work package in workerlist */
workerlist[p].send_addr = work_pointer;
/* push work pointer to next work package */
end_of_wp = local_work_package_size * grid.GetSpeciesCount();
work_pointer = &(work_pointer[end_of_wp]);
// fill send buffer starting with work_package ...
std::memcpy(send_buffer, workerlist[p].send_addr,
(end_of_wp) * sizeof(double));
// followed by: work_package_size
send_buffer[end_of_wp] = (double)local_work_package_size;
// current iteration of simulation
send_buffer[end_of_wp + 1] = (double)iteration;
// size of timestep in seconds
send_buffer[end_of_wp + 2] = dt;
// current time of simulation (age) in seconds
send_buffer[end_of_wp + 3] = current_sim_time;
// placeholder for work_package_count
send_buffer[end_of_wp + 4] = 0.;
/* ATTENTION Worker p has rank p+1 */
MPI_Send(send_buffer, end_of_wp + BUFFER_OFFSET, MPI_DOUBLE, p + 1,
TAG_WORK, MPI_COMM_WORLD);
/* Mark that worker has work to do */
workerlist[p].has_work = 1;
free_workers--;
pkg_to_send -= 1;
}
}
master_send_b = MPI_Wtime();
send_t += master_send_b - master_send_a;
}
inline void ChemMaster::recvPkgs(int &pkg_to_recv, bool to_send,
int &free_workers) {
/* declare most of the variables here */
int need_to_receive = 1;
double master_recv_a, master_recv_b;
double idle_a, idle_b;
int p, size;
MPI_Status probe_status;
master_recv_a = MPI_Wtime();
/* start to loop as long there are packages to recv and the need to receive
*/
while (need_to_receive && pkg_to_recv > 0) {
// only of there are still packages to send and free workers are available
if (to_send && free_workers > 0)
// non blocking probing
MPI_Iprobe(MPI_ANY_SOURCE, TAG_WORK, MPI_COMM_WORLD, &need_to_receive,
&probe_status);
else {
idle_a = MPI_Wtime();
// blocking probing
MPI_Probe(MPI_ANY_SOURCE, TAG_WORK, MPI_COMM_WORLD, &probe_status);
idle_b = MPI_Wtime();
master_idle += idle_b - idle_a;
}
/* if need_to_receive was set to true above, so there is a message to
* receive */
if (need_to_receive) {
p = probe_status.MPI_SOURCE;
MPI_Get_count(&probe_status, MPI_DOUBLE, &size);
MPI_Recv(workerlist[p - 1].send_addr, size, MPI_DOUBLE, p, TAG_WORK,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);
workerlist[p - 1].has_work = 0;
pkg_to_recv -= 1;
free_workers++;
}
}
master_recv_b = MPI_Wtime();
recv_t += master_recv_b - master_recv_a;
}
void ChemMaster::printProgressbar(int count_pkgs, int n_wp, int barWidth) {
/* visual progress */
double progress = (float)(count_pkgs + 1) / n_wp;
cout << "[";
int pos = barWidth * progress;
for (int iprog = 0; iprog < barWidth; ++iprog) {
if (iprog < pos)
cout << "=";
else if (iprog == pos)
cout << ">";
else
cout << " ";
}
std::cout << "] " << int(progress * 100.0) << " %\r";
std::cout.flush();
/* end visual progress */
}
void ChemMaster::End() {
R["simtime_chemistry"] = chem_t;
R.parseEvalQ("profiling$simtime_chemistry <- simtime_chemistry");
Rcpp::NumericVector phreeqc_time;
Rcpp::NumericVector dht_get_time;
Rcpp::NumericVector dht_fill_time;
Rcpp::IntegerVector phreeqc_counts;
Rcpp::NumericVector idle_worker;
int phreeqc_tmp;
std::array<double, 3> timings;
std::array<int, 3> dht_perfs;
int dht_hits = 0;
int dht_miss = 0;
int dht_evictions = 0;
double idle_worker_tmp;
/* loop over all workers *
* ATTENTION Worker p has rank p+1 */
for (int p = 0; p < world_size - 1; p++) {
/* Send termination message to worker */
MPI_Send(NULL, 0, MPI_DOUBLE, p + 1, TAG_FINISH, MPI_COMM_WORLD);
/* ... and receive all timings and metrics from each worker */
MPI_Recv(timings.data(), 3, MPI_DOUBLE, p + 1, TAG_TIMING, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
phreeqc_time.push_back(timings[0], "w" + to_string(p + 1));
MPI_Recv(&phreeqc_tmp, 1, MPI_INT, p + 1, TAG_TIMING, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
phreeqc_counts.push_back(phreeqc_tmp, "w" + to_string(p + 1));
MPI_Recv(&idle_worker_tmp, 1, MPI_DOUBLE, p + 1, TAG_TIMING, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
idle_worker.push_back(idle_worker_tmp, "w" + to_string(p + 1));
if (dht_enabled) {
dht_get_time.push_back(timings[1], "w" + to_string(p + 1));
dht_fill_time.push_back(timings[2], "w" + to_string(p + 1));
MPI_Recv(dht_perfs.data(), 3, MPI_INT, p + 1, TAG_DHT_PERF,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);
dht_hits += dht_perfs[0];
dht_miss += dht_perfs[1];
dht_evictions += dht_perfs[2];
}
}
/* distribute all data to the R runtime */
R["simtime_chemistry"] = chem_t;
R.parseEvalQ("profiling$simtime_chemistry <- simtime_chemistry");
R["simtime_workers"] = worker_t;
R.parseEvalQ("profiling$simtime_workers <- simtime_workers");
R["simtime_chemistry_master"] = chem_master;
R.parseEvalQ(
"profiling$simtime_chemistry_master <- simtime_chemistry_master");
R["seq_master"] = seq_t;
R.parseEvalQ("profiling$seq_master <- seq_master");
R["idle_master"] = master_idle;
R.parseEvalQ("profiling$idle_master <- idle_master");
R["idle_worker"] = idle_worker;
R.parseEvalQ("profiling$idle_worker <- idle_worker");
R["phreeqc_time"] = phreeqc_time;
R.parseEvalQ("profiling$phreeqc <- phreeqc_time");
R["phreeqc_count"] = phreeqc_counts;
R.parseEvalQ("profiling$phreeqc_count <- phreeqc_count");
if (dht_enabled) {
R["dht_hits"] = dht_hits;
R.parseEvalQ("profiling$dht_hits <- dht_hits");
R["dht_miss"] = dht_miss;
R.parseEvalQ("profiling$dht_miss <- dht_miss");
R["dht_evictions"] = dht_evictions;
R.parseEvalQ("profiling$dht_evictions <- dht_evictions");
R["dht_get_time"] = dht_get_time;
R.parseEvalQ("profiling$dht_get_time <- dht_get_time");
R["dht_fill_time"] = dht_fill_time;
R.parseEvalQ("profiling$dht_fill_time <- dht_fill_time");
}
}
void ChemMaster::shuffleField(const std::vector<double> &in_field,
uint32_t size_per_prop, uint32_t prop_count,
double *out_buffer) {
uint32_t wp_count = this->wp_sizes_vector.size();
uint32_t write_i = 0;
for (uint32_t i = 0; i < wp_count; i++) {
for (uint32_t j = i; j < size_per_prop; j += wp_count) {
for (uint32_t k = 0; k < prop_count; k++) {
out_buffer[(write_i * prop_count) + k] =
in_field[(k * size_per_prop) + j];
}
write_i++;
}
}
}
void ChemMaster::unshuffleField(const double *in_buffer, uint32_t size_per_prop,
uint32_t prop_count,
std::vector<double> &out_field) {
uint32_t wp_count = this->wp_sizes_vector.size();
uint32_t read_i = 0;
for (uint32_t i = 0; i < wp_count; i++) {
for (uint32_t j = i; j < size_per_prop; j += wp_count) {
for (uint32_t k = 0; k < prop_count; k++) {
out_field[(k * size_per_prop) + j] =
in_buffer[(read_i * prop_count) + k];
}
read_i++;
}
}
}
double ChemMaster::getSendTime() { return this->send_t; }
double ChemMaster::getRecvTime() { return this->recv_t; }
double ChemMaster::getIdleTime() { return this->master_idle; }
double ChemMaster::getWorkerTime() { return this->worker_t; }
double ChemMaster::getChemMasterTime() { return this->chem_master; }
double ChemMaster::getSeqTime() { return this->seq_t; }

112
src/ChemSeq.cpp
View File

@ -1,112 +0,0 @@
/*
** Copyright (C) 2018-2021 Alexander Lindemann, Max Luebke (University of
** Potsdam)
**
** Copyright (C) 2018-2022 Marco De Lucia, Max Luebke (GFZ Potsdam)
**
** POET is free software; you can redistribute it and/or modify it under the
** terms of the GNU General Public License as published by the Free Software
** Foundation; either version 2 of the License, or (at your option) any later
** version.
**
** POET is distributed in the hope that it will be useful, but WITHOUT ANY
** WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
** A PARTICULAR PURPOSE. See the GNU General Public License for more details.
**
** You should have received a copy of the GNU General Public License along with
** this program; if not, write to the Free Software Foundation, Inc., 51
** Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include "poet/DiffusionModule.hpp"
#include "poet/SimParams.hpp"
#include <poet/ChemSimSeq.hpp>
#include <poet/Grid.hpp>
#include <Rcpp.h>
#include <algorithm>
#include <bits/stdint-uintn.h>
#include <iostream>
#include <string>
#include <vector>
using namespace Rcpp;
using namespace poet;
ChemSeq::ChemSeq(SimParams &params, RInside &R_, Grid &grid_)
: BaseChemModule(params, R_, grid_) {
this->state = this->grid.RegisterState(
poet::BaseChemModule::CHEMISTRY_MODULE_NAME, this->prop_names);
std::vector<double> &field = this->state->mem;
field.resize(this->n_cells_per_prop * this->prop_names.size());
for (uint32_t i = 0; i < this->prop_names.size(); i++) {
std::vector<double> prop_vec =
this->grid.GetSpeciesByName(this->prop_names[i]);
std::copy(prop_vec.begin(), prop_vec.end(),
field.begin() + (i * this->n_cells_per_prop));
}
}
poet::ChemSeq::~ChemSeq() {
if (this->phreeqc_rm) {
delete this->phreeqc_rm;
}
}
void poet::ChemSeq::InitModule(poet::ChemistryParams &chem_params) {
this->phreeqc_rm = new PhreeqcWrapper(this->n_cells_per_prop);
this->phreeqc_rm->SetupAndLoadDB(chem_params);
this->phreeqc_rm->InitFromFile(chem_params.input_script);
}
void ChemSeq::Simulate(double dt) {
double chem_a, chem_b;
/* start time measuring */
chem_a = MPI_Wtime();
std::vector<double> &field = this->state->mem;
// HACK: transfer the field into R data structure serving as input for phreeqc
R["TMP_T"] = field;
R.parseEvalQ("mysetup$state_T <- setNames(data.frame(matrix(TMP_T, "
"ncol=length(mysetup$grid$props), nrow=" +
std::to_string(this->n_cells_per_prop) +
")), mysetup$grid$props)");
this->phreeqc_rm->SetInternalsFromWP(field, this->n_cells_per_prop);
this->phreeqc_rm->SetTime(0);
this->phreeqc_rm->SetTimeStep(dt);
this->phreeqc_rm->RunCells();
// HACK: we will copy resulting field into global grid field. Maybe we will
// find a more performant way here ...
std::vector<double> vecSimResult;
this->phreeqc_rm->GetWPFromInternals(vecSimResult, this->n_cells_per_prop);
std::copy(vecSimResult.begin(), vecSimResult.end(), field.begin());
R["TMP_C"] = field;
R.parseEvalQ("mysetup$state_C <- setNames(data.frame(matrix(TMP_C, "
"ncol=length(mysetup$grid$props), nrow=" +
std::to_string(this->n_cells_per_prop) +
")), mysetup$grid$props)");
/* end time measuring */
chem_b = MPI_Wtime();
chem_t += chem_b - chem_a;
}
void ChemSeq::End() {
R["simtime_chemistry"] = this->chem_t;
R.parseEvalQ("profiling$simtime_chemistry <- simtime_chemistry");
}
double ChemSeq::getChemistryTime() { return this->chem_t; }

305
src/ChemWorker.cpp
View File

@ -1,305 +0,0 @@
/*
** Copyright (C) 2018-2021 Alexander Lindemann, Max Luebke (University of
** Potsdam)
**
** Copyright (C) 2018-2021 Marco De Lucia (GFZ Potsdam)
**
** POET is free software; you can redistribute it and/or modify it under the
** terms of the GNU General Public License as published by the Free Software
** Foundation; either version 2 of the License, or (at your option) any later
** version.
**
** POET is distributed in the hope that it will be useful, but WITHOUT ANY
** WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
** A PARTICULAR PURPOSE. See the GNU General Public License for more details.
**
** You should have received a copy of the GNU General Public License along with
** this program; if not, write to the Free Software Foundation, Inc., 51
** Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include "poet/ChemSimPar.hpp"
#include "poet/DHT_Wrapper.hpp"
#include "poet/SimParams.hpp"
#include <Rcpp.h>
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <iostream>
#include <mpi.h>
#include <ostream>
#include <string>
#include <vector>
using namespace poet;
using namespace std;
using namespace Rcpp;
ChemWorker::ChemWorker(SimParams &params, RInside &R_, Grid &grid_,
MPI_Comm dht_comm)
: BaseChemModule(params, R_, grid_) {
t_simparams tmp = params.getNumParams();
this->wp_size = tmp.wp_size;
this->out_dir = params.getOutDir();
this->dt_differ = tmp.dt_differ;
this->dht_enabled = tmp.dht_enabled;
this->dht_size_per_process = tmp.dht_size_per_process;
this->dht_snaps = tmp.dht_snaps;
this->dht_file = params.getDHTFile();
this->mpi_buffer =
new double[(this->wp_size * this->prop_names.size()) + BUFFER_OFFSET];
this->mpi_buffer_results = new double[wp_size * this->prop_names.size()];
if (world_rank == 1)
cout << "CPP: Worker: DHT usage is " << (dht_enabled ? "ON" : "OFF")
<< endl;
if (this->dht_enabled) {
uint32_t iKeyCount = this->prop_names.size() + (dt_differ);
uint32_t iDataCount = this->prop_names.size();
if (world_rank == 1)
cout << "CPP: Worker: data count: " << iDataCount << " entries" << endl
<< "CPP: Worker: key count: " << iKeyCount << " entries" << endl
<< "CPP: Worker: memory per process "
<< params.getNumParams().dht_size_per_process / std::pow(10, 6)
<< " MByte" << endl;
dht = new DHT_Wrapper(params, dht_comm,
params.getNumParams().dht_size_per_process, iKeyCount,
iDataCount);
if (world_rank == 1)
cout << "CPP: Worker: DHT created!" << endl;
if (!dht_file.empty())
readFile();
// set size
dht_flags.resize(wp_size, true);
}
this->timing.fill(0.0);
}
ChemWorker::~ChemWorker() {
delete this->mpi_buffer;
delete this->mpi_buffer_results;
if (dht_enabled)
delete this->dht;
if (this->phreeqc_rm) {
delete this->phreeqc_rm;
}
}
void ChemWorker::loop() {
MPI_Status probe_status;
while (1) {
double idle_a = MPI_Wtime();
MPI_Probe(0, MPI_ANY_TAG, MPI_COMM_WORLD, &probe_status);
double idle_b = MPI_Wtime();
/* there is a work package to receive */
if (probe_status.MPI_TAG == TAG_WORK) {
idle_t += idle_b - idle_a;
doWork(probe_status);
}
/* end of iteration */
else if (probe_status.MPI_TAG == TAG_DHT_ITER) {
postIter();
}
/* end of simulation */
else if (probe_status.MPI_TAG == TAG_FINISH) {
finishWork();
break;
}
}
}
void poet::ChemWorker::InitModule(poet::ChemistryParams &chem_params) {
this->phreeqc_rm = new PhreeqcWrapper(this->wp_size);
this->phreeqc_rm->SetupAndLoadDB(chem_params);
this->phreeqc_rm->InitFromFile(chem_params.input_script);
}
void ChemWorker::doWork(MPI_Status &probe_status) {
int count;
int local_work_package_size = 0;
static int counter = 1;
double dht_get_start, dht_get_end;
double phreeqc_time_start, phreeqc_time_end;
double dht_fill_start, dht_fill_end;
double dt;
bool bNoPhreeqc = false;
/* get number of doubles to be received */
MPI_Get_count(&probe_status, MPI_DOUBLE, &count);
/* receive */
MPI_Recv(mpi_buffer, count, MPI_DOUBLE, 0, TAG_WORK, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
/* decrement count of work_package by BUFFER_OFFSET */
count -= BUFFER_OFFSET;
/* check for changes on all additional variables given by the 'header' of
* mpi_buffer */
// work_package_size
local_work_package_size = mpi_buffer[count];
// current iteration of simulation
this->iteration = mpi_buffer[count + 1];
// current timestep size
dt = mpi_buffer[count + 2];
// current simulation time ('age' of simulation)
current_sim_time = mpi_buffer[count + 3];
/* 4th double value is currently a placeholder */
// placeholder = mpi_buffer[count+4];
std::vector<double> vecCurrWP(
mpi_buffer,
mpi_buffer + (local_work_package_size * this->prop_names.size()));
std::vector<int32_t> vecMappingWP(this->wp_size);
DHT_ResultObject DHT_Results;
for (uint32_t i = 0; i < local_work_package_size; i++) {
vecMappingWP[i] = i;
}
if (local_work_package_size != this->wp_size) {
std::vector<double> vecFiller(
(this->wp_size - local_work_package_size) * this->prop_names.size(), 0);
vecCurrWP.insert(vecCurrWP.end(), vecFiller.begin(), vecFiller.end());
// set all remaining cells to inactive
for (int i = local_work_package_size; i < this->wp_size; i++) {
vecMappingWP[i] = -1;
}
}
if (dht_enabled) {
/* check for values in DHT */
dht_get_start = MPI_Wtime();
DHT_Results = dht->checkDHT(local_work_package_size, dt, vecCurrWP);
dht_get_end = MPI_Wtime();
DHT_Results.ResultsToMapping(vecMappingWP);
}
phreeqc_time_start = MPI_Wtime();
this->phreeqc_rm->RunWorkPackage(vecCurrWP, vecMappingWP, current_sim_time,
dt);
phreeqc_time_end = MPI_Wtime();
if (dht_enabled) {
DHT_Results.ResultsToWP(vecCurrWP);
}
/* send results to master */
MPI_Request send_req;
MPI_Isend(vecCurrWP.data(), count, MPI_DOUBLE, 0, TAG_WORK, MPI_COMM_WORLD,
&send_req);
if (dht_enabled) {
/* write results to DHT */
dht_fill_start = MPI_Wtime();
dht->fillDHT(local_work_package_size, DHT_Results, vecCurrWP);
dht_fill_end = MPI_Wtime();
timing[1] += dht_get_end - dht_get_start;
timing[2] += dht_fill_end - dht_fill_start;
}
timing[0] += phreeqc_time_end - phreeqc_time_start;
MPI_Wait(&send_req, MPI_STATUS_IGNORE);
}
void ChemWorker::postIter() {
MPI_Recv(NULL, 0, MPI_DOUBLE, 0, TAG_DHT_ITER, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
if (dht_enabled) {
dht->printStatistics();
if (dht_snaps == 2) {
writeFile();
}
}
// synchronize all processes
MPI_Barrier(MPI_COMM_WORLD);
}
void ChemWorker::writeFile() {
cout.flush();
std::stringstream out;
out << out_dir << "/iter_" << setfill('0') << setw(3) << this->iteration
<< ".dht";
int res = dht->tableToFile(out.str().c_str());
if (res != DHT_SUCCESS && world_rank == 2)
cerr << "CPP: Worker: Error in writing current state of DHT to file."
<< endl;
else if (world_rank == 2)
cout << "CPP: Worker: Successfully written DHT to file " << out.str()
<< endl;
}
void ChemWorker::readFile() {
int res = dht->fileToTable((char *)dht_file.c_str());
if (res != DHT_SUCCESS) {
if (res == DHT_WRONG_FILE) {
if (world_rank == 1)
cerr << "CPP: Worker: Wrong file layout! Continue with empty DHT ..."
<< endl;
} else {
if (world_rank == 1)
cerr << "CPP: Worker: Error in loading current state of DHT from "
"file. Continue with empty DHT ..."
<< endl;
}
} else {
if (world_rank == 2)
cout << "CPP: Worker: Successfully loaded state of DHT from file "
<< dht_file << endl;
std::cout.flush();
}
}
void ChemWorker::finishWork() {
/* before death, submit profiling/timings to master*/
MPI_Recv(NULL, 0, MPI_DOUBLE, 0, TAG_FINISH, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
// timings
MPI_Send(timing.data(), timing.size(), MPI_DOUBLE, 0, TAG_TIMING,
MPI_COMM_WORLD);
MPI_Send(&phreeqc_count, 1, MPI_INT, 0, TAG_TIMING, MPI_COMM_WORLD);
MPI_Send(&idle_t, 1, MPI_DOUBLE, 0, TAG_TIMING, MPI_COMM_WORLD);
if (dht_enabled) {
// dht_perf
int dht_perf[3];
dht_perf[0] = dht->getHits();
dht_perf[1] = dht->getMisses();
dht_perf[2] = dht->getEvictions();
MPI_Send(dht_perf, 3, MPI_INT, 0, TAG_DHT_PERF, MPI_COMM_WORLD);
}
if (dht_enabled && dht_snaps > 0)
writeFile();
}

32
src/ChemistryModule/CMakeLists.txt Normal file
View File

@ -0,0 +1,32 @@
option(POET_USE_PRM_BACKEND "Use PhreeqcRM parallelization instead of poet's." OFF)
option(POET_DHT_DEBUG "Build with DHT debug info" OFF)
list(APPEND CHEM_MODEL_SRC "ChemistryModule.cpp" )
mark_as_advanced(PHREEQCRM_BUILD_MPI PHREEQCRM_DISABLE_OPENMP)
set(PHREEQCRM_DISABLE_OPENMP ON CACHE BOOL "" FORCE)
if(POET_USE_PRM_BACKEND)
set(PHREEQCRM_BUILD_MPI ON CACHE BOOL "" FORCE)
target_compile_definitions(poet_lib PRIVATE POET_USE_PRM)
else()
set(PHREEQCRM_BUILD_MPI OFF CACHE BOOL "" FORCE)
list(APPEND CHEM_MODEL_SRC "WorkerFunctions.cpp" "MasterFunctions.cpp" "DHT.c" "DHT_Wrapper.cpp" "HashFunctions.cpp")
endif()
add_library(ChemistryModule ${CHEM_MODEL_SRC})
target_include_directories(ChemistryModule PUBLIC ${PROJECT_SOURCE_DIR}/include)
target_link_libraries(ChemistryModule
PUBLIC MPI::MPI_CXX PhreeqcRM
PRIVATE ${MATH_LIBRARY} ${CRYPTO_LIBRARY}
)
target_compile_definitions(ChemistryModule PUBLIC OMPI_SKIP_MPICXX)
if(POET_DHT_DEBUG)
target_compile_definitions(ChemistryModule PRIVATE DHT_STATISTICS)
endif()
if(POET_USE_PRM_BACKEND)
target_compile_definitions(ChemistryModule PUBLIC POET_USE_PRM)
endif()

362
src/ChemistryModule/ChemistryModule.cpp Normal file
View File

@ -0,0 +1,362 @@
#include "poet/ChemistryModule.hpp"
#include "PhreeqcRM.h"
#include "poet/DHT_Wrapper.hpp"
#include <cassert>
#include <cstdint>
#include <mpi.h>
#include <stdexcept>
#include <string>
#include <vector>
#ifndef POET_USE_PRM
poet::ChemistryModule::ChemistryModule(uint32_t nxyz, uint32_t wp_size,
MPI_Comm communicator)
: PhreeqcRM(nxyz, 1), group_comm(communicator), wp_size(wp_size) {
MPI_Comm_size(communicator, &this->comm_size);
MPI_Comm_rank(communicator, &this->comm_rank);
this->is_sequential = (this->comm_size == 1);
this->is_master = (this->comm_rank == 0);
if (!is_sequential && is_master) {
MPI_Bcast(&wp_size, 1, MPI_UINT32_T, 0, this->group_comm);
}
}
poet::ChemistryModule::~ChemistryModule() {
if (dht) {
delete dht;
}
}
poet::ChemistryModule
poet::ChemistryModule::createWorker(MPI_Comm communicator) {
uint32_t wp_size;
MPI_Bcast(&wp_size, 1, MPI_UINT32_T, 0, communicator);
return ChemistryModule(wp_size, wp_size, communicator);
}
#endif
void poet::ChemistryModule::RunInitFile(const std::string &input_script_path) {
#ifndef POET_USE_PRM
if (this->is_master) {
int f_type = CHEM_INIT;
PropagateFunctionType(f_type);
int count = input_script_path.size();
ChemBCast(&count, 1, MPI_INT);
ChemBCast(const_cast<char *>(input_script_path.data()), count, MPI_CHAR);
}
#endif
this->RunFile(true, true, false, input_script_path);
this->RunString(true, false, false, "DELETE; -all; PRINT; -warnings 0;");
this->FindComponents();
std::vector<std::string> props;
this->speciesPerModule.reserve(this->MODULE_COUNT);
std::vector<std::string> curr_prop_names = this->GetComponents();
props.insert(props.end(), curr_prop_names.begin(), curr_prop_names.end());
this->speciesPerModule.push_back(curr_prop_names.size());
curr_prop_names = this->GetEquilibriumPhases();
props.insert(props.end(), curr_prop_names.begin(), curr_prop_names.end());
char equilibrium = (curr_prop_names.empty() ? -1 : 1);
this->speciesPerModule.push_back(curr_prop_names.size());
curr_prop_names = this->GetExchangeNames();
props.insert(props.end(), curr_prop_names.begin(), curr_prop_names.end());
char exchange = (curr_prop_names.empty() ? -1 : 1);
this->speciesPerModule.push_back(curr_prop_names.size());
curr_prop_names = this->GetSurfaceNames();
props.insert(props.end(), curr_prop_names.begin(), curr_prop_names.end());
char surface = (curr_prop_names.empty() ? -1 : 1);
this->speciesPerModule.push_back(curr_prop_names.size());
// curr_prop_names = this->GetGasComponents();
// props.insert(props.end(), curr_prop_names.begin(), curr_prop_names.end());
// char gas = (curr_prop_names.empty() ? -1 : 1);
// this->speciesPerModule.push_back(curr_prop_names.size());
// curr_prop_names = this->GetSolidSolutionNames();
// props.insert(props.end(), curr_prop_names.begin(), curr_prop_names.end());
// char ssolutions = (curr_prop_names.empty() ? -1 : 1);
// this->speciesPerModule.push_back(curr_prop_names.size());
curr_prop_names = this->GetKineticReactions();
props.insert(props.end(), curr_prop_names.begin(), curr_prop_names.end());
char kinetics = (curr_prop_names.empty() ? -1 : 1);
this->speciesPerModule.push_back(curr_prop_names.size());
this->prop_count = props.size();
#ifdef POET_USE_PRM
std::vector<int> ic1;
ic1.resize(this->nxyz * 7, -1);
// TODO: hardcoded reaction modules
for (int i = 0; i < nxyz; i++) {
ic1[i] = 1; // Solution 1
ic1[nxyz + i] = equilibrium; // Equilibrium 1
ic1[2 * nxyz + i] = exchange; // Exchange none
ic1[3 * nxyz + i] = surface; // Surface none
ic1[4 * nxyz + i] = -1; // Gas phase none
ic1[5 * nxyz + i] = -1; // Solid solutions none
ic1[6 * nxyz + i] = kinetics; // Kinetics 1
}
this->InitialPhreeqc2Module(ic1);
#else
if (!this->is_master || this->is_sequential) {
std::vector<int> ic1;
ic1.resize(this->nxyz * 7, -1);
// TODO: hardcoded reaction modules
for (int i = 0; i < nxyz; i++) {
ic1[i] = 1; // Solution 1
ic1[nxyz + i] = equilibrium; // Equilibrium 1
ic1[2 * nxyz + i] = exchange; // Exchange none
ic1[3 * nxyz + i] = surface; // Surface none
ic1[4 * nxyz + i] = -1; // Gas phase none
ic1[5 * nxyz + i] = -1; // Solid solutions none
ic1[6 * nxyz + i] = kinetics; // Kinetics 1
}
this->InitialPhreeqc2Module(ic1);
}
#endif
this->prop_names = props;
}
#ifndef POET_USE_PRM
void poet::ChemistryModule::InitializeField(
uint32_t n_cells, const poet::ChemistryModule::SingleCMap &mapped_values) {
if (this->is_master) {
this->field.reserve(this->prop_count * n_cells);
for (const std::string &prop_name : this->prop_names) {
const auto m_it = mapped_values.find(prop_name);
if (m_it == mapped_values.end()) {
throw std::domain_error(
"Prop names vector does not match any key in given map.");
}
const std::vector<double> field_row(n_cells, m_it->second);
const auto chem_field_end = this->field.end();
this->field.insert(chem_field_end, field_row.begin(), field_row.end());
}
this->n_cells = n_cells;
}
}
void poet::ChemistryModule::InitializeField(
const poet::ChemistryModule::VectorCMap &mapped_values) {
if (this->is_master) {
for (const std::string &prop_name : this->prop_names) {
const auto m_it = mapped_values.find(prop_name);
if (m_it == mapped_values.end()) {
throw std::domain_error(
"Prop names vector does not match any key in given map.");
}
const auto field_row = m_it->second;
const auto chem_field_end = this->field.end();
this->field.insert(chem_field_end, field_row.begin(), field_row.end());
}
this->n_cells = mapped_values.begin()->second.size();
}
}
void poet::ChemistryModule::SetDHTEnabled(bool enable, uint32_t size_mb) {
constexpr uint32_t MB_FACTOR = 1E6;
if (this->is_master) {
int ftype = CHEM_DHT_ENABLE;
PropagateFunctionType(ftype);
ChemBCast(&enable, 1, MPI_CXX_BOOL);
ChemBCast(&size_mb, 1, MPI_UINT32_T);
}
this->dht_enabled = enable;
if (enable) {
MPI_Comm dht_comm;
if (this->is_master) {
MPI_Comm_split(this->group_comm, MPI_UNDEFINED, this->comm_rank,
&dht_comm);
return;
}
MPI_Comm_split(this->group_comm, 1, this->comm_rank, &dht_comm);
if (this->dht) {
delete this->dht;
}
const uint32_t dht_size = size_mb * MB_FACTOR;
this->dht =
new DHT_Wrapper(dht_comm, dht_size, this->prop_count, this->prop_count);
}
}
void poet::ChemistryModule::SetDHTSnaps(int type, const std::string &out_dir) {
if (this->is_master) {
int ftype = CHEM_DHT_SNAPS;
PropagateFunctionType(ftype);
int str_size = out_dir.size();
ChemBCast(&type, 1, MPI_INT);
ChemBCast(&str_size, 1, MPI_INT);
ChemBCast(const_cast<char *>(out_dir.data()), str_size, MPI_CHAR);
}
this->dht_file_out_dir = out_dir;
}
void poet::ChemistryModule::SetDHTSignifVector(
std::vector<uint32_t> signif_vec) {
if (this->is_master) {
if (signif_vec.size() != this->prop_count) {
throw std::runtime_error(
"Significant vector sizes mismatches prop count.");
}
int ftype = CHEM_DHT_SIGNIF_VEC;
PropagateFunctionType(ftype);
ChemBCast(signif_vec.data(), signif_vec.size(), MPI_UINT32_T);
return;
}
this->dht->SetSignifVector(signif_vec);
}
void poet::ChemistryModule::SetDHTPropTypeVector(
std::vector<uint32_t> proptype_vec) {
if (this->is_master) {
if (proptype_vec.size() != this->prop_count) {
throw std::runtime_error("Prop type vector sizes mismatches prop count.");
}
int ftype = CHEM_DHT_PROP_TYPE_VEC;
PropagateFunctionType(ftype);
ChemBCast(proptype_vec.data(), proptype_vec.size(), MPI_UINT32_T);
return;
}
this->dht->SetPropTypeVector(proptype_vec);
}
void poet::ChemistryModule::ReadDHTFile(const std::string &input_file) {
if (this->is_master) {
int ftype = CHEM_DHT_READ_FILE;
PropagateFunctionType(ftype);
int str_size = input_file.size();
ChemBCast(&str_size, 1, MPI_INT);
ChemBCast(const_cast<char *>(input_file.data()), str_size, MPI_CHAR);
}
if (!this->dht_enabled) {
throw std::runtime_error("DHT file cannot be read. DHT is not enabled.");
}
if (!this->is_master) {
WorkerReadDHTDump(input_file);
}
}
void poet::ChemistryModule::GetWPFromInternals(std::vector<double> &vecWP,
uint32_t wp_size) {
std::vector<double> vecCurrOutput;
vecWP.clear();
vecWP.reserve(this->prop_count * wp_size);
this->GetConcentrations(vecCurrOutput);
vecWP.insert(vecWP.end(), vecCurrOutput.begin(), vecCurrOutput.end());
if (this->speciesPerModule[1] != 0) {
this->GetPPhaseMoles(vecCurrOutput);
vecWP.insert(vecWP.end(), vecCurrOutput.begin(), vecCurrOutput.end());
}
// NOTE: Block for 'Surface' and 'Exchange' is missing because of missing
// Getters @ PhreeqcRM
// ...
// BLOCK_END
if (this->speciesPerModule[4] != 0) {
this->GetKineticsMoles(vecCurrOutput);
vecWP.insert(vecWP.end(), vecCurrOutput.begin(), vecCurrOutput.end());
}
}
void poet::ChemistryModule::SetInternalsFromWP(const std::vector<double> &vecWP,
uint32_t wp_size) {
uint32_t iCurrElements;
auto itStart = vecWP.begin();
auto itEnd = itStart;
// this->SetMappingForWP(iCurrWPSize);
int nchem = this->GetChemistryCellCount();
iCurrElements = this->speciesPerModule[0];
itEnd += iCurrElements * wp_size;
this->SetConcentrations(std::vector<double>(itStart, itEnd));
itStart = itEnd;
// Equlibirum Phases
if ((iCurrElements = this->speciesPerModule[1]) != 0) {
itEnd += iCurrElements * wp_size;
this->SetPPhaseMoles(std::vector<double>(itStart, itEnd));
itStart = itEnd;
}
// // NOTE: Block for 'Surface' and 'Exchange' is missing because of missing
// // setters @ PhreeqcRM
// // ...
// // BLOCK_END
if ((iCurrElements = this->speciesPerModule[4]) != 0) {
itEnd += iCurrElements * wp_size;
this->SetKineticsMoles(std::vector<double>(itStart, itEnd));
itStart = itEnd;
}
}
#else //POET_USE_PRM
inline void poet::ChemistryModule::PrmToPoetField(std::vector<double> &field) {
GetConcentrations(field);
int col = GetSelectedOutputColumnCount();
int rows = GetSelectedOutputRowCount();
field.reserve(field.size() + 3 * rows);
std::vector<double> so;
GetSelectedOutput(so);
for (int j = 0; j < col; j += 2) {
const auto start = so.begin() + (j * rows);
const auto end = start + rows;
field.insert(field.end(), start, end);
}
}
void poet::ChemistryModule::RunCells() {
PhreeqcRM::RunCells();
PrmToPoetField(this->field);
}
#endif

0
src/DHT.c → src/ChemistryModule/DHT.c
View File

70
src/DHT_Wrapper.cpp → src/ChemistryModule/DHT_Wrapper.cpp
View File

@ -18,18 +18,19 @@
** Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include "poet/DHT_Wrapper.hpp"
#include "poet/DHT_Types.hpp"
#include "poet/HashFunctions.hpp"
#include <algorithm>
#include <cmath>
#include <cstddef>
#include <cstdint>
#include <cstring>
#include <openssl/evp.h>
#include <poet/DHT_Wrapper.hpp>
#include <math.h>
#include <iostream>
#include <math.h>
#include <openssl/evp.h>
#include <stdexcept>
#include <vector>
using namespace poet;
@ -67,9 +68,8 @@ void poet::DHT_ResultObject::ResultsToWP(std::vector<double> &curr_wp) {
}
}
DHT_Wrapper::DHT_Wrapper(const poet::SimParams &params, MPI_Comm dht_comm,
uint32_t dht_size, uint32_t key_count,
uint32_t data_count)
DHT_Wrapper::DHT_Wrapper(MPI_Comm dht_comm, uint32_t dht_size,
uint32_t key_count, uint32_t data_count)
: key_count(key_count), data_count(data_count) {
poet::initHashCtx(EVP_md5());
// initialize DHT object
@ -80,20 +80,26 @@ DHT_Wrapper::DHT_Wrapper(const poet::SimParams &params, MPI_Comm dht_comm,
&poet::hashDHT);
// extract needed values from sim_param struct
t_simparams tmp = params.getNumParams();
// t_simparams tmp = params.getNumParams();
this->dt_differ = tmp.dt_differ;
this->dht_log = tmp.dht_log;
// this->dt_differ = tmp.dt_differ;
// this->dht_log = tmp.dht_log;
this->dht_signif_vector = params.getDHTSignifVector();
this->dht_prop_type_vector = params.getDHTPropTypeVector();
// this->dht_signif_vector = params.getDHTSignifVector();
// this->dht_prop_type_vector = params.getDHTPropTypeVector();
this->dht_signif_vector.resize(key_size, DHT_KEY_SIGNIF_DEFAULT);
this->dht_signif_vector[0] = DHT_KEY_SIGNIF_TOTALS;
this->dht_signif_vector[1] = DHT_KEY_SIGNIF_TOTALS;
this->dht_signif_vector[2] = DHT_KEY_SIGNIF_CHARGE;
this->dht_prop_type_vector.resize(key_size, DHT_TYPE_DEFAULT);
}
DHT_Wrapper::~DHT_Wrapper() {
// free DHT
DHT_free(dht_object, NULL, NULL);
// free fuzzing buffer
free(fuzzing_buffer);
poet::freeHashCtx();
}
auto DHT_Wrapper::checkDHT(int length, double dt,
@ -184,17 +190,11 @@ void DHT_Wrapper::printStatistics() {
}
}
uint64_t DHT_Wrapper::getHits() { return this->dht_hits; }
uint64_t DHT_Wrapper::getMisses() { return this->dht_miss; }
uint64_t DHT_Wrapper::getEvictions() { return this->dht_evictions; }
std::vector<DHT_Keyelement> DHT_Wrapper::fuzzForDHT(int var_count, void *key,
double dt) {
constexpr double zero_val = 10E-14;
std::vector<DHT_Keyelement> vecFuzz(var_count);
std::vector<DHT_Keyelement> vecFuzz(var_count + 1);
std::memset(&vecFuzz[0], 0, sizeof(DHT_Keyelement) * var_count);
unsigned int i = 0;
@ -203,10 +203,11 @@ std::vector<DHT_Keyelement> DHT_Wrapper::fuzzForDHT(int var_count, void *key,
for (i = 0; i < (unsigned int)var_count; i++) {
double &curr_key = ((double *)key)[i];
if (curr_key != 0) {
if (curr_key < zero_val && this->dht_prop_type_vector[i] == "act") {
if (curr_key < zero_val &&
this->dht_prop_type_vector[i] == DHT_TYPE_ACT) {
continue;
}
if (this->dht_prop_type_vector[i] == "ignore") {
if (this->dht_prop_type_vector[i] == DHT_TYPE_IGNORE) {
continue;
}
vecFuzz[i] = round_key_element(curr_key, dht_signif_vector[i]);
@ -214,9 +215,24 @@ std::vector<DHT_Keyelement> DHT_Wrapper::fuzzForDHT(int var_count, void *key,
}
// if timestep differs over iterations set current current time step at the
// end of fuzzing buffer
if (dt_differ) {
vecFuzz[var_count] = round_key_element(dt, 55);
}
vecFuzz[var_count] = round_key_element(dt, 55);
return vecFuzz;
}
void poet::DHT_Wrapper::SetSignifVector(std::vector<uint32_t> signif_vec) {
if (signif_vec.size() != this->key_count) {
throw std::runtime_error(
"Significant vector size mismatches count of key elements.");
}
this->dht_signif_vector = signif_vec;
}
void poet::DHT_Wrapper::SetPropTypeVector(std::vector<uint32_t> prop_type_vec) {
if (prop_type_vec.size() != this->key_count) {
throw std::runtime_error(
"Prop type vector size mismatches count of key elements.");
}
this->dht_prop_type_vector = prop_type_vec;
}

0
src/HashFunctions.cpp → src/ChemistryModule/HashFunctions.cpp
View File

360
src/ChemistryModule/MasterFunctions.cpp Normal file
View File

@ -0,0 +1,360 @@
#include "PhreeqcRM.h"
#include "poet/ChemistryModule.hpp"
#include <algorithm>
#include <cstdint>
#include <mpi.h>
#include <stdexcept>
#include <vector>
std::vector<uint32_t>
poet::ChemistryModule::MasterGatherWorkerMetrics(int type) const {
MPI_Bcast(&type, 1, MPI_INT, 0, this->group_comm);
uint32_t dummy;
std::vector<uint32_t> metrics(this->comm_size);
MPI_Gather(&dummy, 1, MPI_UINT32_T, metrics.data(), 1, MPI_UINT32_T, 0,
this->group_comm);
metrics.erase(metrics.begin());
return metrics;
}
std::vector<double>
poet::ChemistryModule::MasterGatherWorkerTimings(int type) const {
MPI_Bcast(&type, 1, MPI_INT, 0, this->group_comm);
double dummy;
std::vector<double> timings(this->comm_size);
MPI_Gather(&dummy, 1, MPI_DOUBLE, timings.data(), 1, MPI_DOUBLE, 0,
this->group_comm);
timings.erase(timings.begin());
return timings;
}
std::vector<double> poet::ChemistryModule::GetWorkerPhreeqcTimings() const {
int type = CHEM_PERF;
MPI_Bcast(&type, 1, MPI_INT, 0, this->group_comm);
return MasterGatherWorkerTimings(WORKER_PHREEQC);
}
std::vector<double> poet::ChemistryModule::GetWorkerDHTGetTimings() const {
int type = CHEM_PERF;
MPI_Bcast(&type, 1, MPI_INT, 0, this->group_comm);
return MasterGatherWorkerTimings(WORKER_DHT_GET);
}
std::vector<double> poet::ChemistryModule::GetWorkerDHTFillTimings() const {
int type = CHEM_PERF;
MPI_Bcast(&type, 1, MPI_INT, 0, this->group_comm);
return MasterGatherWorkerTimings(WORKER_DHT_FILL);
}
std::vector<double> poet::ChemistryModule::GetWorkerIdleTimings() const {
int type = CHEM_PERF;
MPI_Bcast(&type, 1, MPI_INT, 0, this->group_comm);
return MasterGatherWorkerTimings(WORKER_IDLE);
}
std::vector<uint32_t> poet::ChemistryModule::GetWorkerDHTHits() const {
int type = CHEM_PERF;
MPI_Bcast(&type, 1, MPI_INT, 0, this->group_comm);
return MasterGatherWorkerMetrics(WORKER_DHT_HITS);
}
std::vector<uint32_t> poet::ChemistryModule::GetWorkerDHTMiss() const {
int type = CHEM_PERF;
MPI_Bcast(&type, 1, MPI_INT, 0, this->group_comm);
return MasterGatherWorkerMetrics(WORKER_DHT_MISS);
}
std::vector<uint32_t> poet::ChemistryModule::GetWorkerDHTEvictions() const {
int type = CHEM_PERF;
MPI_Bcast(&type, 1, MPI_INT, 0, this->group_comm);
return MasterGatherWorkerMetrics(WORKER_DHT_EVICTIONS);
}
inline std::vector<double> shuffleField(const std::vector<double> &in_field,
uint32_t size_per_prop,
uint32_t prop_count,
uint32_t wp_count) {
std::vector<double> out_buffer(in_field.size());
uint32_t write_i = 0;
for (uint32_t i = 0; i < wp_count; i++) {
for (uint32_t j = i; j < size_per_prop; j += wp_count) {
for (uint32_t k = 0; k < prop_count; k++) {
out_buffer[(write_i * prop_count) + k] =
in_field[(k * size_per_prop) + j];
}
write_i++;
}
}
return out_buffer;
}
inline void unshuffleField(const std::vector<double> &in_buffer,
uint32_t size_per_prop, uint32_t prop_count,
uint32_t wp_count, std::vector<double> &out_field) {
uint32_t read_i = 0;
for (uint32_t i = 0; i < wp_count; i++) {
for (uint32_t j = i; j < size_per_prop; j += wp_count) {
for (uint32_t k = 0; k < prop_count; k++) {
out_field[(k * size_per_prop) + j] =
in_buffer[(read_i * prop_count) + k];
}
read_i++;
}
}
}
inline void printProgressbar(int count_pkgs, int n_wp, int barWidth = 70) {
/* visual progress */
double progress = (float)(count_pkgs + 1) / n_wp;
std::cout << "[";
int pos = barWidth * progress;
for (int iprog = 0; iprog < barWidth; ++iprog) {
if (iprog < pos)
std::cout << "=";
else if (iprog == pos)
std::cout << ">";
else
std::cout << " ";
}
std::cout << "] " << int(progress * 100.0) << " %\r";
std::cout.flush();
/* end visual progress */
}
inline void poet::ChemistryModule::MasterSendPkgs(
worker_list_t &w_list, workpointer_t &work_pointer, int &pkg_to_send,
int &count_pkgs, int &free_workers, double dt, uint32_t iteration,
const std::vector<uint32_t> &wp_sizes_vector) {
/* declare variables */
int local_work_package_size;
/* search for free workers and send work */
for (int p = 0; p < this->comm_size - 1; p++) {
if (w_list[p].has_work == 0 && pkg_to_send > 0) /* worker is free */ {
/* to enable different work_package_size, set local copy of
* work_package_size to pre-calculated work package size vector */
local_work_package_size = (int)wp_sizes_vector[count_pkgs];
count_pkgs++;
/* note current processed work package in workerlist */
w_list[p].send_addr = work_pointer.base();
/* push work pointer to next work package */
const uint32_t end_of_wp = local_work_package_size * this->prop_count;
std::vector<double> send_buffer(end_of_wp + this->BUFFER_OFFSET);
std::copy(work_pointer, work_pointer + end_of_wp, send_buffer.begin());
work_pointer += end_of_wp;
// fill send buffer starting with work_package ...
// followed by: work_package_size
send_buffer[end_of_wp] = (double)local_work_package_size;
// current iteration of simulation
send_buffer[end_of_wp + 1] = (double)iteration;
// size of timestep in seconds
send_buffer[end_of_wp + 2] = dt;
// current time of simulation (age) in seconds
send_buffer[end_of_wp + 3] = this->simtime;
// placeholder for work_package_count
send_buffer[end_of_wp + 4] = 0.;
/* ATTENTION Worker p has rank p+1 */
// MPI_Send(send_buffer, end_of_wp + BUFFER_OFFSET, MPI_DOUBLE, p + 1,
// LOOP_WORK, this->group_comm);
MPI_Send(send_buffer.data(), send_buffer.size(), MPI_DOUBLE, p + 1,
LOOP_WORK, this->group_comm);
/* Mark that worker has work to do */
w_list[p].has_work = 1;
free_workers--;
pkg_to_send -= 1;
}
}
}
inline void poet::ChemistryModule::MasterRecvPkgs(worker_list_t &w_list,
int &pkg_to_recv,
bool to_send,
int &free_workers) {
/* declare most of the variables here */
int need_to_receive = 1;
double idle_a, idle_b;
int p, size;
MPI_Status probe_status;
// master_recv_a = MPI_Wtime();
/* start to loop as long there are packages to recv and the need to receive
*/
while (need_to_receive && pkg_to_recv > 0) {
// only of there are still packages to send and free workers are available
if (to_send && free_workers > 0)
// non blocking probing
MPI_Iprobe(MPI_ANY_SOURCE, LOOP_WORK, MPI_COMM_WORLD, &need_to_receive,
&probe_status);
else {
idle_a = MPI_Wtime();
// blocking probing
MPI_Probe(MPI_ANY_SOURCE, LOOP_WORK, MPI_COMM_WORLD, &probe_status);
idle_b = MPI_Wtime();
this->idle_t += idle_b - idle_a;
}
/* if need_to_receive was set to true above, so there is a message to
* receive */
if (need_to_receive) {
p = probe_status.MPI_SOURCE;
MPI_Get_count(&probe_status, MPI_DOUBLE, &size);
MPI_Recv(w_list[p - 1].send_addr, size, MPI_DOUBLE, p, LOOP_WORK,
this->group_comm, MPI_STATUS_IGNORE);
w_list[p - 1].has_work = 0;
pkg_to_recv -= 1;
free_workers++;
}
}
}
void poet::ChemistryModule::RunCells() {
if (this->is_sequential) {
MasterRunSequential();
return;
}
MasterRunParallel();
}
void poet::ChemistryModule::MasterRunSequential() {
SetInternalsFromWP(this->field, this->nxyz);
PhreeqcRM::RunCells();
GetWPFromInternals(this->field, this->nxyz);
}
void poet::ChemistryModule::MasterRunParallel() {
/* declare most of the needed variables here */
double chem_a, chem_b;
double seq_a, seq_b, seq_c, seq_d;
double worker_chemistry_a, worker_chemistry_b;
double sim_e_chemistry, sim_f_chemistry;
int pkg_to_send, pkg_to_recv;
int free_workers;
int i_pkgs;
int ftype = CHEM_WORK_LOOP;
PropagateFunctionType(ftype);
MPI_Barrier(this->group_comm);
double dt = this->PhreeqcRM::GetTimeStep();
static uint32_t iteration = 0;
/* start time measurement of whole chemistry simulation */
chem_a = MPI_Wtime();
/* start time measurement of sequential part */
seq_a = MPI_Wtime();
const std::vector<uint32_t> wp_sizes_vector =
CalculateWPSizesVector(this->n_cells, this->wp_size);
/* shuffle grid */
// grid.shuffleAndExport(mpi_buffer);
std::vector<double> mpi_buffer = shuffleField(
this->field, this->n_cells, this->prop_count, wp_sizes_vector.size());
/* setup local variables */
pkg_to_send = wp_sizes_vector.size();
pkg_to_recv = wp_sizes_vector.size();
workpointer_t work_pointer = mpi_buffer.begin();
worker_list_t worker_list(this->comm_size - 1);
free_workers = this->comm_size - 1;
i_pkgs = 0;
/* end time measurement of sequential part */
seq_b = MPI_Wtime();
seq_t += seq_b - seq_a;
/* start time measurement of chemistry time needed for send/recv loop */
worker_chemistry_a = MPI_Wtime();
/* start send/recv loop */
// while there are still packages to recv
while (pkg_to_recv > 0) {
// print a progressbar to stdout
printProgressbar((int)i_pkgs, (int)wp_sizes_vector.size());
// while there are still packages to send
if (pkg_to_send > 0) {
// send packages to all free workers ...
MasterSendPkgs(worker_list, work_pointer, pkg_to_send, i_pkgs,
free_workers, dt, iteration, wp_sizes_vector);
}
// ... and try to receive them from workers who has finished their work
MasterRecvPkgs(worker_list, pkg_to_recv, pkg_to_send > 0, free_workers);
}
// Just to complete the progressbar
std::cout << std::endl;
/* stop time measurement of chemistry time needed for send/recv loop */
worker_chemistry_b = MPI_Wtime();
this->send_recv_t += worker_chemistry_b - worker_chemistry_a;
/* start time measurement of sequential part */
seq_c = MPI_Wtime();
/* unshuffle grid */
// grid.importAndUnshuffle(mpi_buffer);
unshuffleField(mpi_buffer, this->n_cells, this->prop_count,
wp_sizes_vector.size(), this->field);
/* do master stuff */
/* start time measurement of master chemistry */
sim_e_chemistry = MPI_Wtime();
/* end time measurement of sequential part */
seq_d = MPI_Wtime();
this->seq_t += seq_d - seq_c;
/* end time measurement of whole chemistry simulation */
/* advise workers to end chemistry iteration */
for (int i = 1; i < this->comm_size; i++) {
MPI_Send(NULL, 0, MPI_DOUBLE, i, LOOP_END, this->group_comm);
}
chem_b = MPI_Wtime();
this->chem_t += chem_b - chem_a;
this->simtime += dt;
iteration++;
}
void poet::ChemistryModule::MasterLoopBreak() {
int type = CHEM_BREAK_MAIN_LOOP;
MPI_Bcast(&type, 1, MPI_INT, 0, this->group_comm);
}
std::vector<uint32_t>
poet::ChemistryModule::CalculateWPSizesVector(uint32_t n_cells,
uint32_t wp_size) const {
bool mod_pkgs = (n_cells % wp_size) != 0;
uint32_t n_packages = (uint32_t)(n_cells / wp_size) + mod_pkgs;
std::vector<uint32_t> wp_sizes_vector(n_packages, 0);
for (int i = 0; i < n_cells; i++) {
wp_sizes_vector[i % n_packages] += 1;
}
return wp_sizes_vector;
}

422
src/ChemistryModule/WorkerFunctions.cpp Normal file
View File

@ -0,0 +1,422 @@
#include "IrmResult.h"
#include "poet/ChemistryModule.hpp"
#include <cstddef>
#include <cstdint>
#include <iomanip>
#include <iostream>
#include <mpi.h>
#include <stdexcept>
#include <string>
#include <vector>
inline std::string get_string(int root, MPI_Comm communicator) {
int count;
MPI_Bcast(&count, 1, MPI_INT, root, communicator);
char *buffer = new char[count + 1];
MPI_Bcast(buffer, count, MPI_CHAR, root, communicator);
buffer[count] = '\0';
std::string ret_str(buffer);
delete[] buffer;
return ret_str;
}
void poet::ChemistryModule::WorkerLoop() {
struct worker_s timings;
// HACK: defining the worker iteration count here, which will increment after
// each CHEM_ITER_END message
uint32_t iteration = 1;
bool loop = true;
while (loop) {
int func_type;
PropagateFunctionType(func_type);
switch (func_type) {
case CHEM_INIT: {
RunInitFile(get_string(0, this->group_comm));
break;
}
case CHEM_DHT_ENABLE: {
bool enable;
ChemBCast(&enable, 1, MPI_CXX_BOOL);
uint32_t size_mb;
ChemBCast(&size_mb, 1, MPI_UINT32_T);
SetDHTEnabled(enable, size_mb);
break;
}
case CHEM_DHT_SIGNIF_VEC: {
std::vector<uint32_t> input_vec(this->prop_count);
ChemBCast(input_vec.data(), this->prop_count, MPI_UINT32_T);
SetDHTSignifVector(input_vec);
break;
}
case CHEM_DHT_PROP_TYPE_VEC: {
std::vector<uint32_t> input_vec(this->prop_count);
ChemBCast(input_vec.data(), this->prop_count, MPI_UINT32_T);
SetDHTPropTypeVector(input_vec);
break;
}
case CHEM_DHT_SNAPS: {
int type;
ChemBCast(&type, 1, MPI_INT);
SetDHTSnaps(type, get_string(0, this->group_comm));
break;
}
case CHEM_DHT_READ_FILE: {
ReadDHTFile(get_string(0, this->group_comm));
break;
}
case CHEM_WORK_LOOP: {
WorkerProcessPkgs(timings, iteration);
break;
}
case CHEM_PERF: {
int type;
ChemBCast(&type, 1, MPI_INT);
if (type < WORKER_DHT_HITS) {
WorkerPerfToMaster(type, timings);
break;
}
WorkerMetricsToMaster(type);
break;
}
case CHEM_BREAK_MAIN_LOOP: {
WorkerPostSim(iteration);
loop = false;
break;
}
default: {
throw std::runtime_error("Worker received unknown tag from master.");
}
}
}
}
void poet::ChemistryModule::WorkerProcessPkgs(struct worker_s &timings,
uint32_t &iteration) {
MPI_Status probe_status;
bool loop = true;
MPI_Barrier(this->group_comm);
while (loop) {
double idle_a = MPI_Wtime();
MPI_Probe(0, MPI_ANY_TAG, this->group_comm, &probe_status);
double idle_b = MPI_Wtime();
switch (probe_status.MPI_TAG) {
case LOOP_WORK: {
timings.idle_t += idle_b - idle_a;
int count;
MPI_Get_count(&probe_status, MPI_DOUBLE, &count);
WorkerDoWork(probe_status, count, timings);
break;
}
case LOOP_END: {
WorkerPostIter(probe_status, iteration);
iteration++;
loop = false;
break;
}
}
}
}
void poet::ChemistryModule::WorkerDoWork(MPI_Status &probe_status,
int double_count,
struct worker_s &timings) {
int local_work_package_size = 0;
static int counter = 1;
double dht_get_start, dht_get_end;
double phreeqc_time_start, phreeqc_time_end;
double dht_fill_start, dht_fill_end;
uint32_t iteration;
double dt;
double current_sim_time;
const uint32_t n_cells_times_props = this->prop_count * this->wp_size;
std::vector<double> vecCurrWP(n_cells_times_props + BUFFER_OFFSET);
int count = double_count;
/* receive */
MPI_Recv(vecCurrWP.data(), count, MPI_DOUBLE, 0, LOOP_WORK, this->group_comm,
MPI_STATUS_IGNORE);
/* decrement count of work_package by BUFFER_OFFSET */
count -= BUFFER_OFFSET;
/* check for changes on all additional variables given by the 'header' of
* mpi_buffer */
// work_package_size
local_work_package_size = vecCurrWP[count];
// current iteration of simulation
iteration = vecCurrWP[count + 1];
// current timestep size
dt = vecCurrWP[count + 2];
// current simulation time ('age' of simulation)
current_sim_time = vecCurrWP[count + 3];
/* 4th double value is currently a placeholder */
// placeholder = mpi_buffer[count+4];
// std::vector<double> vecCurrWP(
// mpi_buffer,
// mpi_buffer + (local_work_package_size * this->prop_names.size()));
vecCurrWP.resize(n_cells_times_props);
std::vector<int32_t> vecMappingWP(this->wp_size);
DHT_ResultObject DHT_Results;
for (uint32_t i = 0; i < local_work_package_size; i++) {
vecMappingWP[i] = i;
}
if (local_work_package_size != this->wp_size) {
// std::vector<double> vecFiller(
// (this->wp_size - local_work_package_size) * prop_count, 0);
// vecCurrWP.insert(vecCurrWP.end(), vecFiller.begin(), vecFiller.end());
// set all remaining cells to inactive
for (int i = local_work_package_size; i < this->wp_size; i++) {
vecMappingWP[i] = -1;
}
}
if (dht_enabled) {
/* check for values in DHT */
dht_get_start = MPI_Wtime();
DHT_Results = dht->checkDHT(local_work_package_size, dt, vecCurrWP);
dht_get_end = MPI_Wtime();
DHT_Results.ResultsToMapping(vecMappingWP);
}
phreeqc_time_start = MPI_Wtime();
WorkerRunWorkPackage(vecCurrWP, vecMappingWP, current_sim_time, dt);
phreeqc_time_end = MPI_Wtime();
if (dht_enabled) {
DHT_Results.ResultsToWP(vecCurrWP);
}
/* send results to master */
MPI_Request send_req;
MPI_Isend(vecCurrWP.data(), count, MPI_DOUBLE, 0, LOOP_WORK, MPI_COMM_WORLD,
&send_req);
if (dht_enabled) {
/* write results to DHT */
dht_fill_start = MPI_Wtime();
dht->fillDHT(local_work_package_size, DHT_Results, vecCurrWP);
dht_fill_end = MPI_Wtime();
timings.dht_get += dht_get_end - dht_get_start;
timings.dht_fill += dht_fill_end - dht_fill_start;
}
timings.phreeqc_t += phreeqc_time_end - phreeqc_time_start;
MPI_Wait(&send_req, MPI_STATUS_IGNORE);
}
void poet::ChemistryModule::WorkerPostIter(MPI_Status &prope_status,
uint32_t iteration) {
MPI_Recv(NULL, 0, MPI_DOUBLE, 0, LOOP_END, this->group_comm,
MPI_STATUS_IGNORE);
if (this->dht_enabled) {
this->dht->printStatistics();
if (this->dht_snaps_type == DHT_FILES_ITEREND) {
WorkerWriteDHTDump(iteration);
}
}
}
void poet::ChemistryModule::WorkerPostSim(uint32_t iteration) {
/* before death, submit profiling/timings to master*/
// double timings_serialized[4];
// timings_serialized[0] = timings.phreeqc_t;
// timings_serialized[1] = timings.dht_get;
// timings_serialized[2] = timings.dht_fill;
// timings_serialized[3] = timings.idle_t;
// // timings
// MPI_Send(timings_serialized, 4, MPI_DOUBLE, 0, 0, this->group_comm);
// // MPI_Send(&phreeqc_count, 1, MPI_INT, 0, TAG_TIMING, MPI_COMM_WORLD);
// // MPI_Send(&idle_t, 1, MPI_DOUBLE, 0, TAG_TIMING, MPI_COMM_WORLD);
// if (this->dht_enabled) {
// // dht_perf
// int dht_perf[3];
// dht_perf[0] = dht->getHits();
// dht_perf[1] = dht->getMisses();
// dht_perf[2] = dht->getEvictions();
// MPI_Send(dht_perf, 3, MPI_INT, 0, 0, this->group_comm);
// }
if (this->dht_enabled && this->dht_snaps_type > DHT_FILES_SIMEND) {
WorkerWriteDHTDump(iteration);
}
}
void poet::ChemistryModule::WorkerWriteDHTDump(uint32_t iteration) {
std::stringstream out;
out << this->dht_file_out_dir << "/iter_" << std::setfill('0') << std::setw(4)
<< iteration << ".dht";
int res = dht->tableToFile(out.str().c_str());
if (res != DHT_SUCCESS && this->comm_rank == 2)
std::cerr
<< "CPP: Worker: Error in writing current state of DHT to file.\n";
else if (this->comm_rank == 2)
std::cout << "CPP: Worker: Successfully written DHT to file " << out.str()
<< "\n";
}
void poet::ChemistryModule::WorkerReadDHTDump(
const std::string &dht_input_file) {
int res = dht->fileToTable((char *)dht_input_file.c_str());
if (res != DHT_SUCCESS) {
if (res == DHT_WRONG_FILE) {
if (this->comm_rank == 1)
std::cerr
<< "CPP: Worker: Wrong file layout! Continue with empty DHT ...\n";
} else {
if (this->comm_rank == 1)
std::cerr << "CPP: Worker: Error in loading current state of DHT from "
"file. Continue with empty DHT ...\n";
}
} else {
if (this->comm_rank == 2)
std::cout << "CPP: Worker: Successfully loaded state of DHT from file "
<< dht_input_file << "\n";
}
}
IRM_RESULT
poet::ChemistryModule::WorkerRunWorkPackage(std::vector<double> &vecWP,
std::vector<int32_t> &vecMapping,
double dSimTime, double dTimestep) {
if (this->wp_size != vecMapping.size()) {
return IRM_INVALIDARG;
}
if ((this->wp_size * this->prop_count) != vecWP.size()) {
return IRM_INVALIDARG;
}
// check if we actually need to start phreeqc
bool bRunPhreeqc = false;
for (const auto &aMappingNum : vecMapping) {
if (aMappingNum != -1) {
bRunPhreeqc = true;
break;
}
}
if (!bRunPhreeqc) {
return IRM_OK;
}
std::vector<double> vecCopy;
vecCopy = vecWP;
for (uint32_t i = 0; i < this->prop_count; i++) {
for (uint32_t j = 0; j < this->wp_size; j++) {
vecWP[(i * this->wp_size) + j] = vecCopy[(j * this->prop_count) + i];
}
}
IRM_RESULT result;
this->PhreeqcRM::CreateMapping(vecMapping);
this->SetInternalsFromWP(vecWP, this->wp_size);
this->PhreeqcRM::SetTime(dSimTime);
this->PhreeqcRM::SetTimeStep(dTimestep);
result = this->PhreeqcRM::RunCells();
this->GetWPFromInternals(vecWP, this->wp_size);
vecCopy = vecWP;
for (uint32_t i = 0; i < this->prop_count; i++) {
for (uint32_t j = 0; j < this->wp_size; j++) {
vecWP[(j * this->prop_count) + i] = vecCopy[(i * this->wp_size) + j];
}
}
return result;
}
void poet::ChemistryModule::WorkerPerfToMaster(int type,
const struct worker_s &timings) {
switch (type) {
case WORKER_PHREEQC: {
MPI_Gather(&timings.phreeqc_t, 1, MPI_DOUBLE, NULL, 1, MPI_DOUBLE, 0,
this->group_comm);
break;
}
case WORKER_DHT_GET: {
MPI_Gather(&timings.dht_get, 1, MPI_DOUBLE, NULL, 1, MPI_DOUBLE, 0,
this->group_comm);
break;
}
case WORKER_DHT_FILL: {
MPI_Gather(&timings.dht_fill, 1, MPI_DOUBLE, NULL, 1, MPI_DOUBLE, 0,
this->group_comm);
break;
}
case WORKER_IDLE: {
MPI_Gather(&timings.idle_t, 1, MPI_DOUBLE, NULL, 1, MPI_DOUBLE, 0,
this->group_comm);
break;
}
default: {
throw std::runtime_error("Unknown perf type in master's message.");
}
}
}
void poet::ChemistryModule::WorkerMetricsToMaster(int type) {
uint32_t value;
switch (type) {
case WORKER_DHT_HITS: {
value = dht->getHits();
break;
}
case WORKER_DHT_MISS: {
value = dht->getMisses();
break;
}
case WORKER_DHT_EVICTIONS: {
value = dht->getEvictions();
break;
}
default: {
throw std::runtime_error("Unknown perf type in master's message.");
}
}
MPI_Gather(&value, 1, MPI_UINT32_T, NULL, 1, MPI_UINT32_T, 0,
this->group_comm);
}

1
src/DiffusionModule.cpp
View File

@ -25,7 +25,6 @@
#include <algorithm>
#include <bits/stdint-intn.h>
#include <cstdint>
#include <poet/ChemSimSeq.hpp>
#include <poet/DiffusionModule.hpp>
#include <poet/Grid.hpp>

19
src/Grid.cpp
View File

@ -19,7 +19,9 @@
*/
#include "poet/SimParams.hpp"
#include <RInside.h>
#include <Rcpp.h>
#include <Rcpp/internal/wrap.h>
#include <algorithm>
#include <cassert>
#include <cstdint>
@ -253,3 +255,20 @@ auto poet::Grid::GetSpeciesByName(std::string name,
return std::vector<double>(module_memory->mem.begin() + begin_vec,
module_memory->mem.begin() + end_vec);
}
// HACK: Helper function
void poet::Grid::WriteFieldsToR(RInside &R) {
for (auto const &elem : this->state_register) {
std::string field_name = elem.first;
StateMemory *field = elem.second;
const uint32_t n_cells_per_prop = field->mem.size() / field->props.size();
R["TMP"] = Rcpp::wrap(field->mem);
R["TMP_PROPS"] = Rcpp::wrap(field->props);
R.parseEval(std::string(
"mysetup$" + field_name + " <- setNames(data.frame(matrix(TMP, nrow=" +
std::to_string(n_cells_per_prop) + ")), TMP_PROPS)"));
}
}

336
src/PhreeqcWrapper.cpp
View File

@ -1,336 +0,0 @@
#include "poet/PhreeqcWrapper.hpp"
#include "IPhreeqc.hpp"
#include "IrmResult.h"
#include "PhreeqcRM.h"
#include <algorithm>
#include <cstdint>
#include <iterator>
#include <stdexcept>
#include <string>
#include <vector>
poet::PhreeqcWrapper::PhreeqcWrapper(uint32_t inxyz)
: PhreeqcRM(inxyz, 1), iWPSize(inxyz) {
this->vecDefMapping.resize(inxyz);
for (uint32_t i = 0; i < inxyz; i++) {
this->vecDefMapping[i] = i;
}
}
void poet::PhreeqcWrapper::SetupAndLoadDB(
const poet::ChemistryParams &chemPar) {
// TODO: hardcoded options ...
this->SetErrorHandlerMode(1);
this->SetComponentH2O(false);
this->SetRebalanceFraction(0.5);
this->SetRebalanceByCell(true);
this->UseSolutionDensityVolume(false);
this->SetPartitionUZSolids(false);
// Set concentration units
// 1, mg/L; 2, mol/L; 3, kg/kgs
this->SetUnitsSolution(2);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
this->SetUnitsPPassemblage(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
this->SetUnitsExchange(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
this->SetUnitsSurface(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
this->SetUnitsGasPhase(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
this->SetUnitsSSassemblage(1);
// 0, mol/L cell; 1, mol/L water; 2 mol/L rock
this->SetUnitsKinetics(1);
// Set representative volume
std::vector<double> rv;
rv.resize(this->iWPSize, 1.0);
this->SetRepresentativeVolume(rv);
// Set initial porosity
std::vector<double> por;
por.resize(this->iWPSize, 1);
this->SetPorosity(por);
// Set initial saturation
std::vector<double> sat;
sat.resize(this->iWPSize, 1.0);
this->SetSaturation(sat);
// Load database
this->LoadDatabase(chemPar.database_path);
}
void poet::PhreeqcWrapper::InitFromFile(const std::string &strInputFile) {
this->RunFile(true, true, false, strInputFile);
// MDL: this is run only by the workers
this->RunString(true, false, true, "DELETE; -all; PRINT; -warnings 0;");
this->FindComponents();
std::vector<int> ic1;
ic1.resize(this->iWPSize * 7, -1);
// TODO: hardcoded reaction modules
for (int i = 0; i < nxyz; i++) {
ic1[i] = 1; // Solution 1
ic1[nxyz + i] = 1; // Equilibrium 1
ic1[2 * nxyz + i] = -1; // Exchange none
ic1[3 * nxyz + i] = -1; // Surface none
ic1[4 * nxyz + i] = -1; // Gas phase none
ic1[5 * nxyz + i] = -1; // Solid solutions none
ic1[6 * nxyz + i] = 1; // Kinetics 1
}
this->InitialPhreeqc2Module(ic1);
// Initial equilibration of cells
// double time = 0.0;
// double time_step = 0.0;
// this->SetTime(time);
// this->SetTimeStep(time_step);
// this->RunCells();
this->InitInternals();
}
void poet::PhreeqcWrapper::InitInternals() {
uint32_t sum = 0, curr_count;
std::vector<std::string> vecCurrSpecies;
this->vecSpeciesPerModule.reserve(this->MODULE_COUNT);
vecCurrSpecies = this->GetComponents();
this->vecSpeciesNames.insert(this->vecSpeciesNames.end(),
vecCurrSpecies.begin(), vecCurrSpecies.end());
this->vecSpeciesPerModule.push_back(vecCurrSpecies.size());
vecCurrSpecies = this->GetEquilibriumPhases();
this->vecSpeciesNames.insert(this->vecSpeciesNames.end(),
vecCurrSpecies.begin(), vecCurrSpecies.end());
this->vecSpeciesPerModule.push_back(vecCurrSpecies.size());
vecCurrSpecies = this->GetExchangeSpecies();
this->vecSpeciesNames.insert(this->vecSpeciesNames.end(),
vecCurrSpecies.begin(), vecCurrSpecies.end());
this->vecSpeciesPerModule.push_back(vecCurrSpecies.size());
vecCurrSpecies = this->GetSurfaceSpecies();
this->vecSpeciesNames.insert(this->vecSpeciesNames.end(),
vecCurrSpecies.begin(), vecCurrSpecies.end());
this->vecSpeciesPerModule.push_back(vecCurrSpecies.size());
vecCurrSpecies = this->GetKineticReactions();
this->vecSpeciesNames.insert(this->vecSpeciesNames.end(),
vecCurrSpecies.begin(), vecCurrSpecies.end());
this->vecSpeciesPerModule.push_back(vecCurrSpecies.size());
this->iSpeciesCount = this->vecSpeciesNames.size();
}
auto poet::PhreeqcWrapper::GetSpeciesCount() -> uint32_t {
return this->iSpeciesCount;
}
auto poet::PhreeqcWrapper::GetSpeciesCountByModule()
-> const std::vector<uint32_t> & {
return this->vecSpeciesPerModule;
}
auto poet::PhreeqcWrapper::GetSpeciesNamesFull()
-> const std::vector<std::string> & {
return this->vecSpeciesNames;
}
void poet::PhreeqcWrapper::SetInternalsFromWP(const std::vector<double> &vecWP,
uint32_t iCurrWPSize) {
uint32_t iCurrElements;
auto itStart = vecWP.begin();
auto itEnd = itStart;
// this->SetMappingForWP(iCurrWPSize);
int nchem = this->GetChemistryCellCount();
iCurrElements = this->vecSpeciesPerModule[0];
itEnd += iCurrElements * this->iWPSize;
std::vector<double> out;
this->GetConcentrations(out);
this->SetConcentrations(std::vector<double>(itStart, itEnd));
itStart = itEnd;
// Equlibirum Phases
if ((iCurrElements = this->vecSpeciesPerModule[1]) != 0) {
itEnd += iCurrElements * this->iWPSize;
this->GetPPhaseMoles(out);
this->SetPPhaseMoles(std::vector<double>(itStart, itEnd));
itStart = itEnd;
}
// // NOTE: Block for 'Surface' and 'Exchange' is missing because of missing
// // setters @ PhreeqcRM
// // ...
// // BLOCK_END
if ((iCurrElements = this->vecSpeciesPerModule[4]) != 0) {
itEnd += iCurrElements * this->iWPSize;
this->GetKineticsMoles(out);
this->SetKineticsMoles(std::vector<double>(itStart, itEnd));
itStart = itEnd;
}
}
void poet::PhreeqcWrapper::GetWPFromInternals(std::vector<double> &vecWP,
uint32_t iCurrWPSize) {
std::vector<double> vecCurrOutput;
vecWP.clear();
vecWP.reserve(this->iSpeciesCount * this->iWPSize);
this->GetConcentrations(vecCurrOutput);
vecWP.insert(vecWP.end(), vecCurrOutput.begin(), vecCurrOutput.end());
if (this->vecSpeciesPerModule[1] != 0) {
this->GetPPhaseMoles(vecCurrOutput);
vecWP.insert(vecWP.end(), vecCurrOutput.begin(), vecCurrOutput.end());
}
// NOTE: Block for 'Surface' and 'Exchange' is missing because of missing
// Getters @ PhreeqcRM
// ...
// BLOCK_END
if (this->vecSpeciesPerModule[4] != 0) {
this->GetKineticsMoles(vecCurrOutput);
vecWP.insert(vecWP.end(), vecCurrOutput.begin(), vecCurrOutput.end());
}
}
auto poet::PhreeqcWrapper::ReplaceTotalsByPotentials(
const std::vector<double> &vecWP, uint32_t iCurrWPSize)
-> std::vector<double> {
uint32_t iPropsPerCell = this->vecSpeciesNames.size();
int iphreeqcResult;
std::vector<double> vecOutput;
vecOutput.reserve((iPropsPerCell - 1) * iCurrWPSize);
auto itCStart = vecWP.begin();
auto itCEnd = itCStart + (this->vecSpeciesPerModule[0]);
uint32_t iDiff = iPropsPerCell - this->vecSpeciesPerModule[0];
for (uint32_t i = 0; i < iCurrWPSize; i++) {
std::vector<double> vecCIn(itCStart, itCEnd);
double pH, pe;
// FIXME: Hardcoded temperatures and pressures here!
IPhreeqc *util_ptr = this->Concentrations2Utility(
vecCIn, std::vector<double>(1, 25.0), std::vector<double>(1, 1.0));
std::string sInput = "SELECTED_OUTPUT " + std::to_string(this->DHT_SELOUT) +
"; -pH; -pe;RUN_CELLS; -cells 1; -time_step 0";
iphreeqcResult = util_ptr->RunString(sInput.c_str());
if (iphreeqcResult != 0) {
throw std::runtime_error("IPhreeqc Utility returned non-zero value: " +
std::to_string(iphreeqcResult));
}
util_ptr->SetCurrentSelectedOutputUserNumber(this->DHT_SELOUT);
int vtype;
static std::string svalue(100, '\0');
iphreeqcResult = util_ptr->GetSelectedOutputValue2(
1, 0, &vtype, &pH, svalue.data(), svalue.capacity());
if (iphreeqcResult != 0) {
throw std::runtime_error("IPhreeqc Utility returned non-zero value: " +
std::to_string(iphreeqcResult));
}
iphreeqcResult = util_ptr->GetSelectedOutputValue2(
1, 1, &vtype, &pe, svalue.data(), svalue.capacity());
if (iphreeqcResult != 0) {
throw std::runtime_error("IPhreeqc Utility returned non-zero value: " +
std::to_string(iphreeqcResult));
}
vecOutput.insert(vecOutput.end(), vecCIn.begin() + 3, vecCIn.end());
vecOutput.push_back(pH);
vecOutput.push_back(pe);
vecOutput.insert(vecOutput.end(), itCEnd, itCEnd + iDiff);
itCStart = itCEnd + iDiff;
itCEnd = itCStart + (this->vecSpeciesPerModule[0]);
}
return vecOutput;
}
IRM_RESULT
poet::PhreeqcWrapper::RunWorkPackage(std::vector<double> &vecWP,
std::vector<int32_t> &vecMapping,
double dSimTime, double dTimestep) {
if (this->iWPSize != vecMapping.size()) {
return IRM_INVALIDARG;
}
if ((this->iWPSize * this->iSpeciesCount) != vecWP.size()) {
return IRM_INVALIDARG;
}
// check if we actually need to start phreeqc
bool bRunPhreeqc = false;
for (const auto &aMappingNum : vecMapping) {
if (aMappingNum != -1) {
bRunPhreeqc = true;
break;
}
}
if (!bRunPhreeqc) {
return IRM_OK;
}
std::vector<double> vecCopy;
vecCopy = vecWP;
for (uint32_t i = 0; i < this->iSpeciesCount; i++) {
for (uint32_t j = 0; j < this->iWPSize; j++) {
vecWP[(i * this->iWPSize) + j] = vecCopy[(j * this->iSpeciesCount) + i];
}
}
IRM_RESULT result;
this->CreateMapping(vecMapping);
this->SetInternalsFromWP(vecWP, this->iWPSize);
this->SetTime(dSimTime);
this->SetTimeStep(dTimestep);
result = this->PhreeqcRM::RunCells();
this->GetWPFromInternals(vecWP, this->iWPSize);
vecCopy = vecWP;
for (uint32_t i = 0; i < this->iSpeciesCount; i++) {
for (uint32_t j = 0; j < this->iWPSize; j++) {
vecWP[(j * this->iSpeciesCount) + i] = vecCopy[(i * this->iWPSize) + j];
}
}
return result;
}
inline void poet::PhreeqcWrapper::SetMappingForWP(uint32_t iCurrWPSize) {
if (iCurrWPSize == this->iWPSize) {
this->CreateMapping(this->vecDefMapping);
return;
}
std::vector<int> vecCurrMapping(this->vecDefMapping);
for (uint32_t i = iCurrWPSize; i < vecCurrMapping.size(); i++) {
vecCurrMapping[i] = -1;
}
this->CreateMapping(vecCurrMapping);
}

43
src/SimParams.cpp
View File

@ -18,6 +18,7 @@
** Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include "poet/DHT_Types.hpp"
#include <bits/stdint-uintn.h>
#include <poet/SimParams.hpp>
@ -130,7 +131,7 @@ int SimParams::parseFromCmdl(char *argv[], RInside &R) {
// cout << "CPP: DHT logarithm before rounding: " << ( dht_logarithm ? "ON"
// : "OFF" ) << endl;
cmdl("dht-size", DHT_SIZE_PER_PROCESS) >> simparams.dht_size_per_process;
cmdl("dht-size", DHT_SIZE_PER_PROCESS_MB) >> simparams.dht_size_per_process;
// cout << "CPP: DHT size per process (Byte) = " << dht_size_per_process <<
// endl;
@ -196,17 +197,30 @@ void SimParams::initVectorParams(RInside &R, int col_count) {
/*Load significance vector from R setup file (or set default)*/
bool signif_vector_exists = R.parseEval("exists('signif_vector')");
if (signif_vector_exists) {
dht_signif_vector = as<std::vector<int>>(R["signif_vector"]);
} else {
dht_signif_vector.assign(col_count, simparams.dht_significant_digits);
dht_signif_vector = as<std::vector<uint32_t>>(R["signif_vector"]);
}
/*Load property type vector from R setup file (or set default)*/
bool prop_type_vector_exists = R.parseEval("exists('prop_type')");
if (prop_type_vector_exists) {
dht_prop_type_vector = as<std::vector<string>>(R["prop_type"]);
} else {
dht_prop_type_vector.assign(col_count, "act");
std::vector<std::string> prop_type_R =
as<std::vector<string>>(R["prop_type"]);
this->dht_prop_type_vector.clear();
this->dht_prop_type_vector.reserve(prop_type_R.size());
for (const auto &type : prop_type_R) {
if (type == "act") {
this->dht_prop_type_vector.push_back(DHT_TYPE_ACT);
continue;
}
if (type == "ignore") {
this->dht_prop_type_vector.push_back(DHT_TYPE_IGNORE);
continue;
}
this->dht_prop_type_vector.push_back(DHT_TYPE_DEFAULT);
}
}
if (simparams.world_rank == 0) {
@ -228,21 +242,6 @@ void SimParams::initVectorParams(RInside &R, int col_count) {
}
}
void SimParams::setDtDiffer(bool dt_differ) { simparams.dt_differ = dt_differ; }
t_simparams SimParams::getNumParams() const { return this->simparams; }
std::vector<int> SimParams::getDHTSignifVector() const {
return this->dht_signif_vector;
}
std::vector<std::string> SimParams::getDHTPropTypeVector() const {
return this->dht_prop_type_vector;
}
std::string_view SimParams::getDHTFile() const { return this->dht_file; }
std::string_view SimParams::getFilesim() const { return this->filesim; }
std::string_view SimParams::getOutDir() const { return this->out_dir; }
std::list<std::string> SimParams::validateOptions(argh::parser cmdl) {
/* store all unknown parameters here */
std::list<std::string> retList;