{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "### Content:\n", "\n", "* Installation\n", "* Preparing model input parameters and dataset\n", "* ***Preparing DRYP simulations***\n", "* Post processing model outputs\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Preparing DRYP model simulations" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1. Preparing model parameter and setting files\n", "* Running DRYP\n", "* Preparing multiple simulation\n", "* Runing pipeline DRYP\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Local_directory\n", "|__training\n", " |__basin\n", " | |__datasets\n", " | | |__csv\n", " | | |__shp\n", " | |__model\n", " | | |__inputs\n", " | |__outputs\n", " |__regional\n" ] } ], "source": [ "print(\"Local_directory\\n|__training\\n |__basin\\n | |__datasets\\n | | |__csv\\n | | |__shp\\n | |__model\\n | | |__inputs\\n | |__outputs\\n |__regional\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Specifiy the location of training files, once the training files are downloaded change the paths below to access them through this traning material." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "\n", "training_general_path = \"D:/HAD/training/historical/\"\n", "regional_path = \"D:/HAD/training/historical/regional/\"\n", "basin_path = \"D:/HAD/training/historical/basin/\"\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 3.1. Preparing model parameter and setting files" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "DRYP requires at leas two files for running, this is the input-paramter-file and the setting-paramter file. Additional\n", "files may be required depending on the model settings and dataset properties.\n", "\n", "The folowing are the list of DRYP model files:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* input parameter file (essential)\n", "* setting-paramter file (essential)\n", "* GW parameter file (optional)\n", "* riparian input paramter file (optional)\n", "* data projection setting file (optional)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Model input and settins files are simple plain text format files, so they can be easily updated using any text editor.\n", "\n", "For conveniency we use here the library pandas to edit these files." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* Input parameter file" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The *input-parameter-file* is used for specifying path names of all names model files, including input parameters, forcings, and setting files, as well as model outputs. You can also specified the model name as well as any additional model file." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "default_input = {\n", "\n", " \"model_name\": \"DRYP_model_sim\",\n", " \"path_input\": \"\",\n", "\n", " \"TERRAIN\": {\n", " \"path_dem\": \"required\", # Digital elevation model\n", " \"path_Qo\": None, # initial channel storage\n", " \"path_fdl\": None, # flow direction in landlab format\n", " \"path_riv_decay\": None, # river flow velocity\n", " \"path_mask\": None, # basin mask. model active domain\n", " \"path_riv_len\": None, # river lenght\n", " \"path_riv_width\": None, # river width\n", " \"path_riv_elev\": None, # tiver stream bottom elevation\n", " \"path_of_bc_flux\": None, # flux boundary condition overland flow\n", " \n", " },\n", "\n", " \"VEGETATION\": {\n", " \"path_veg_kc\": None,\n", " \"path_veg_lulc\": None,\n", " #\"path_veg_nn\": None\n", " },\n", "\n", " \"UNSATURATED\": {\n", " \"path_uz_theta_sat\": None,\n", " \"path_uz_theta_res\": None,\n", " \"path_uz_theta_awc\": None,\n", " \"path_uz_theta_wp\": None,\n", " \"path_uz_rootdepth\": None,\n", " \"path_uz_lambda\": None,\n", " \"path_uz_psi\": None,\n", " \"path_uz_ksat\": None,\n", " \"path_uz_sigmaksat\": None,\n", " \"path_uz_theta\": None, # initial conditions\n", " \"path_riv_ksat\": None,\n", " \"path_uz_bottomksat\": None,\n", " \"path_uz_bc_flux\": None,\n", " },\n", "\n", " \"SATURATED\": {\n", " \"path_sz_mask\": None,\n", " \"path_sz_ksat\": None,\n", " \"path_sz_sy\": None,\n", " \"path_sz_wte\": None,\n", " \"path_sz_bc_flux\": None,\n", " \"path_sz_bc_head\": None,\n", " \"path_sz_bottom\": None,\n", " \"path_sz_depth\": None,\n", " \"path_sz_bdd\": None,\n", " \"path_sz_type\": None, # aquifer type\n", " },\n", "\n", " \"METEO\": {\n", " \"path_pre\": None,\n", " \"path_pet\": None,\n", " \"path_aof\": None,\n", " \t\t\"path_lai\": None,\n", "\t\t\"path_savi\": None,\n", "\t\t\"path_kc\": None,\n", " \t\"path_TSOF\": None,\n", " \t\"path_TSUZ\": None,\n", " \t\"path_TSSZ\": None,\n", " \t\"path_TSav\": None, # vegetation cover fraction\n", " \"path_TSWB\": None, # Variable flux for water bodies and dams\n", " },\n", "\n", " \"OUTPUT\": {\n", " \"path_out_sz\": None,\n", " \"path_out_uz\": None,\n", " \"path_out_oz\": None,\n", " \"path_output\": None,\n", " \"path_setting\": None,\n", " \"path_store_settings\": None\n", " },\n", "\n", " \"RIPARIAN\": {\n", " \"path_rp_theta_sat\": None,\n", " \"path_rp_theta_res\": None,\n", " \"path_rp_theta_awc\": None,\n", " \"path_rp_theta_wp\": None,\n", " \"path_rp_rootdepth\": None,\n", " \"path_rp_lambda\": None,\n", " \"path_rp_psi\": None,\n", " \"path_rp_ksat\": None,\n", " \"path_rp_sigmaksat\": None,\n", " \"path_rp_theta\": None,\n", " \"path_rp_width\": None\n", " },\n", "\n", " \"INTERCEPTION\": {\n", " \"path_veg_lulc_frac\": None,\n", " \"path_veg_hs_savi\": None,\n", " \"path_veg_hs_par_a\": None,\n", " \"path_veg_hs_par_b\": None,\n", " \"path_veg_hs_savi_min\": None,\n", " \"path_veg_hs_savi_max\": None,\n", " \"path_veg_hs_lai\": None,\n", " \"path_veg_hs_tap_depth\": None,\n", " \"path_veg_hs_extinction_depth\": None,\n", " \"path_veg_hs_fcw\": None,\n", " \"path_veg_hs_sca\": None,\n", " \"path_veg_rp_par_a\": None,\n", " \"path_veg_rp_par_b\": None,\n", " \"path_veg_rp_savi_min\": None,\n", " \"path_veg_rp_savi_max\": None,\n", " \"path_veg_rp_lai\": None,\n", " \"path_veg_rp_tap_depth\": None,\n", " \"path_veg_rp_extinction_depth\": None,\n", " \"path_veg_rp_fcw\": None,\n", " \"path_veg_rp_sca\": None\n", " },\n", "\n", " \"GROUNDWATER\": {\n", " # \"path_gw_depth\": None,\n", " # \"path_gw_bdd\": None,\n", " \"path_gw_2l_bottom\": None,\n", " \"path_gw_2l_ksat\": None,\n", " \"path_gw_2l_sy\": None,\n", " \"path_gw_2l_ss\": None,\n", " \"path_gw_2l_wte\": None, \n", " # \"path_gw_type\": None, # aquifer type\n", " # \"path_gw_lake_elev\": None, # lakes bathymetry\n", " # \"path_pnds_hmax\": None, # ponds max depth\n", " # \"path_pnds_Amax\": None, # ponds maximum extend\n", " },\n", "\n", " \"WATER_BODIES\": {\n", " \"path_lake_depth\": None, # lakes bathymetry\n", " \"path_pnd_hmax\": None, # ponds max depth\n", " \"path_pnd_Amax\": None, # ponds maximum extend\n", " \"path_pnd_Vo\": None, # ponds volume of water\n", " \"path_wb_bc_flux\": None, # water bodies boundary conditions\n", " }\n", "\n", "}\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* Model setting file: Reference and projection system" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "default_settings = {\n", " \"SIMULATION_PERIOD\": {\n", " \"start_date\": \"2000 1 1\",\n", " \"end_date\": \"2002 1 3\",\n", " },\n", "\n", " \"PROJECTION\" : None,\n", "\n", " \"TIMESTEP_SETTINGS\": {\n", " \"dt_of\": 60,\n", " \"dt_uz\": 60,\n", " \"dt_gw\": 60,\n", " },\n", "\n", " \"READING\": {\n", " \n", " \"data_reading\": {\n", " \"pre\": 0,\n", " \"pet\": 0,\n", " \"abs\": 0,\n", " \"kc\": 0,\n", " \"fluxOF\": 0,\n", " \"fluxUZ\": 0,\n", " \"fluxSZ\": 0,\n", " \"fluxWB\": 0,\n", " \"savi\": 0,\n", " \"savi_min\": 0,\n", " \"savi_max\": 0,\n", " \"lai\": 0,\n", " \"av\": 0,\n", " },\n", " \"data_step\": {\n", " \"pre\": 60,\n", " \"pet\": 60,\n", " \"abs\": 60,\n", " \"kc\": 60,\n", " \"fluxOF\": 60,\n", " \"fluxUZ\": 60,\n", " \"fluxSZ\": 60,\n", " \"fluxWB\": 60,\n", " \"savi\": 60,\n", " \"savi_min\": 60,\n", " \"savi_max\": 60,\n", " \"lai\": 60,\n", " \"av\": 60,\n", " },\n", " \"data_reproject\": {\n", " \"pre\": True,\n", " \"pet\": True,\n", " \"abs\": True,\n", " \"kc\": True,\n", " \"fluxOF\": False,\n", " \"fluxUZ\": False,\n", " \"fluxSZ\": False,\n", " \"fluxWB\": False,\n", " \"savi\": True,\n", " \"savi_min\": True,\n", " \"savi_max\": True,\n", " \"lai\": True,\n", " \"av\": False,\n", " },\n", " \"data_interp\": {\n", " \"pre\": True,\n", " \"pet\": True,\n", " \"abs\": True,\n", " \"kc\": True,\n", " \"fluxOF\": False,\n", " \"fluxUZ\": False,\n", " \"fluxSZ\": False,\n", " \"fluxWB\": False,\n", " \"savi\": True,\n", " \"savi_min\": True,\n", " \"savi_max\": True,\n", " \"lai\": True,\n", " \"av\": True,\n", " },\n", " \"data_projection\" : {\n", "\t\t\t\"pre\": \"EPSG:4326\",\n", "\t\t\t\"pet\": \"EPSG:4326\",\n", "\t\t\t\"abs\": \"EPSG:4326\",\n", "\t\t\t\"kc\": \"EPSG:4326\",\n", "\t\t\t\"fluxOF\": \"EPSG:4326\",\n", "\t\t\t\"fluxUZ\": \"EPSG:4326\",\n", "\t\t\t\"fluxSZ\": \"EPSG:4326\",\n", "\t\t\t\"fluxWB\": \"EPSG:4326\",\n", "\t\t\t\"savi\":\"EPSG:4326\",\n", "\t\t\t\"savi_min\": \"EPSG:4326\",\n", " \"savi_max\": \"EPSG:4326\",\n", " \"lai\": \"EPSG:4326\",\n", " \"av\": \"EPSG:4326\",\n", " },\n", " },\n", "\n", " \"COMPONENTS\": {\n", " \"method_inf\": 1, # choose infiltration method\n", " \"run_gw\" : True, # activate groundwater component\n", " \"method_gw\": 0, # choose groundwater transimissivity approach\n", " },\n", "\n", " \"OUTPUT\": {\n", " \"output_csv\": True, # activate save model outputs (only csv files)\n", " \"output_grid\": False, # activate save model outputs (grid files)\n", " \"output_dt_csv\": \"1M\",\n", " \"output_dt\": \"1M\",\n", " },\n", "\n", " \"GLOBAL_FACTORS\": {\n", " \"uz_kdt\": 1.0,\n", " \"uz_kdroot\": 1.0,\n", " \"uz_kawc\": 1.0,\n", " \"uz_kkast\": 1.0,\n", " \"uz_ksigma\": 1.0,\n", " \"riv_kksat\": 1.0,\n", " \"riv_kdecay\": 1.0,\n", " \"riv_kwidth\": 1.0,\n", " \"sz_kksat\": 1.0,\n", " \"sz_ksy\": 1.0,\n", " \"of_kflow\": 1.0\n", " }\n", "}\n", "\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* Changing model input paths\n", "\n", "Creating a new model requires to specify the new paths of the model parameter files, it is a task that takes time and can be prone to errors. To avoid any mistake when changing paths, we can use the replace text option availble in python." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "from shutil import copyfile\n", "import fileinput\n", "import sys" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "def replaceAll(file, searchExp, replaceExp):\n", " for line in fileinput.input(file, inplace=1):\n", " if searchExp in line:\n", " line = line.replace(searchExp,replaceExp)\n", " sys.stdout.write(line)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "fname_file = regional_path + \"/model/HAD_IMERGcv_input_sim85.json\"\n", "fname_file_update = regional_path + \"/model/HAD_IMERGcv_input_sim85_cuwalid.json\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create a copy avoid the lost of information" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'D:/HAD/training/historical/regional//model/HAD_IMERGcv_input_sim85_cuwalid.json'" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "copyfile(fname_file, fname_file_update)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "#pd.read_csv(fname_file_update)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Specify the which text you want to change and the new text" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "searchExpresion = \"/home/c1755103/HAD/input/\"\n", "replaceExpresion = \"/home/cuwalid/HAD/input/\"\n", "#replaceAll(fname_file_update, searchExpresion, replaceExpresion)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "searchExpresion = \"/home/c1755103/HAD/HAD_\"\n", "replaceExpresion = \"/home/cuwalid/HAD/HAD_\"\n", "#replaceAll(fname_file_update, searchExpresion, replaceExpresion)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 3.2. Runing DRYP" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**WARNING**: If you are using Windows you have to use the docker for running DRYP" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are two main ways of runing DRYP, you can even add more option that are more convenient for yor analysis:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "a) Running DRYP from the command line" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# python run_model_input.py " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "b) Running dryp within the python environment" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import sys\n", "sys.path.append('C:/Users/Edisson/Documents/GitHub/DRYPv2.0.1')\n", "\n", "from cuwalid.dryp.main_DRYP import run_DRYP\n", "#run_DRYP(inputfile)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 3.3. Preparing and running multiple DRYP simulations" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is useful for performing a Monte Carlo analysis or stochastic forecasting.\n", "\n", "We can create a script to modify the model files that changes specific fiels of each of the all model files, below ther is\n", "an example of the script to genereate multiple simulation files for a calibration pourposes." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "import os\n", "import json\n", "import numpy as np\n", "import calendar\n", "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "def write_JSON_dryp_files(json_template, destination, model_name=None, path_pre=None, path_pet=None,\n", "\t\t\t\t\t\t start_date=None, end_date=None, new_setting_file=None,\n", "\t\t\t\t\t\t path_Qo=None, path_uz_theta=None, path_sz_wte=None, path_rp_theta=None,\n", "\t\t\t\t\t\t path_pnd_Vo=None, path_outputs=None, parameter_factors=None):\n", "\t\"\"\" This function create the simulation and setting file for running DRYP. New\n", "\tfiles are created based on files provided as original files, this function\n", "\tchanges the model name, precipitation and potential evapotranspiration\n", "\tpaths.\n", "\tWARNING: if no new filename is provided it will replace the original file\n", "\t\n", "\tParameters:\n", "\t-----------\n", "\tjson_template : string\n", "\t\t\tmodel input, as a dictionary\n", "\tmodel_name : string\n", "\t\t\tmodel name for the new file\n", "\tpath_pre : string\n", "\t\t\tprecipitation dataset name, including path\n", "\tpath_pet : string\n", "\t\t\tpotential evapotranspiration dataset name, including path\n", "\tstart_date : string\n", "\t\t\tdate in the following format \"YYYY-MM-DD\" (e.g. 2002-01-01)\n", "\tend_date : string\n", "\t\t\tdate in the following format \"YYYY-MM-DD\" (e.g. 2002-03-01)\n", "\tnew_setting_file : bool\n", "\t\t\tIf True it create the setting dryp file\n", "\t\"\"\"\n", " \n", "\t# Change necesarry variables in template\n", "\tif model_name is not None:\n", "\t\tjson_template[\"model_name\"] = model_name\n", "\tif path_pre is not None:\n", "\t\tjson_template[\"METEO\"][\"path_pre\"] = path_pre\n", "\tif path_pet is not None:\n", "\t\tjson_template[\"METEO\"][\"path_pet\"] = path_pet\n", "\n", "\tif path_Qo is not None:\n", "\t\tjson_template[\"TERRAIN\"][\"path_Qo\"] = path_Qo\n", "\tif path_uz_theta is not None:\n", "\t\tjson_template[\"UNSATURATED\"][\"path_uz_theta\"] = path_uz_theta\n", "\tif path_sz_wte is not None:\n", "\t\tjson_template[\"SATURATED\"][\"path_sz_wte\"] = path_sz_wte\n", "\tif path_rp_theta is not None:\n", "\t\tjson_template[\"RIPARIAN\"][\"path_rp_theta\"] = path_rp_theta\n", "\tif path_pnd_Vo is not None:\n", "\t\tjson_template[\"WATER_BODIES\"][\"path_pnd_Vo\"] = path_pnd_Vo\n", "\tif path_outputs is not None:\n", "\t\tjson_template[\"OUTPUT\"][\"path_output\"] = path_outputs\n", "\n", "\t# create new settings file only if new setting file does not exist\n", "\tif new_setting_file is not None:\n", "\n", "\t\t# Get input file as dictionary\n", "\t\tsettings_file_path = json_template[\"OUTPUT\"][\"path_setting\"]\n", "\t\twith open(settings_file_path, 'r') as file:\n", "\t\t\tsettings_file_template = json.load(file)\n", "\t\t# Change settings file location\n", "\t\tjson_template[\"OUTPUT\"][\"path_setting\"] = new_setting_file\n", "\t\t\n", "\t\t## Change variables in settings file\n", "\t\tif start_date is not None:\n", "\t\t\tsettings_file_template[\"SIMULATION_PERIOD\"][\"start_date\"] = start_date\n", "\t\tif end_date is not None:\n", "\t\t\tsettings_file_template[\"SIMULATION_PERIOD\"][\"end_date\"] = end_date\n", "\n", "\t\t# update parameter factors only if provided\n", "\t\tif parameter_factors is not None:\n", "\t\t\tfor ikey in parameter_factors.keys():\n", "\t\t\t\tsettings_file_template[\"GLOBAL_FACTORS\"][ikey] = parameter_factors[ikey]\n", "\t\n", "\t# Save the `dryp` part to the destination file\n", "\twith open(destination, \"w\") as dest_file:\n", "\t\tjson.dump(json_template, dest_file, indent=4)\n", "\n", "\t# Save the `dryp_settings` part to the new settings file\n", "\tif new_setting_file is not None:\n", "\t\twith open(new_setting_file, \"w\") as settings_file:\n", "\t\t\tjson.dump(settings_file_template, settings_file, indent=4)" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [], "source": [ "def gen_array_input_files(fname_input, fname_parameter_sets, model_name, nmax=100):\n", " # open input file\n", "\twith open(fname_input, 'r') as file:\n", "\t\tinput_template = json.load(file)\n", " # read parameter sets\n", "\tparameter = pd.read_csv(fname_parameter_sets)\n", "\t\n", "\t#Create a copy of inputfile\n", "\tfname_root = fname_input.split('.')[0]\n", "\tfname_ext = fname_input.split('.')[1]\n", "\tfor npar in range(0, nmax):\n", "\t\t# new input file name\n", "\t\tnewfname_input = fname_root+'_'+str(npar)+'.json'\n", "\t\timodel_name = model_name + \"_\" + str(npar)\n", "\t\t# replace all new values in dataset\n", "\t\tprint(imodel_name)\n", "\t\twrite_JSON_dryp_files(input_template, newfname_input, model_name=imodel_name,\n", "\t\t\t\t\t\tparameter_factors=dict(parameter.loc[npar]))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Write multiple input model files" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "HAD_IMERGcv_input_sim85_0\n", "HAD_IMERGcv_input_sim85_1\n" ] } ], "source": [ "# ========================================================\n", "# LOOP FOR CREATING MULTIPLE IMPORT FILES FOR RUNNING IN HPC\n", "fname = [\n", "#basin_path +'model/HAD_IMERGcv_input_sim85.json',\n", "regional_path +'model/HAD_IMERGcv_input_sim85.json',\n", "]\n", "fname_parameter_sets = training_general_path + \"/basin/dataset/csv/test_parameter_set.csv\"\n", "for ifname in fname:\n", "\tgen_array_input_files(ifname, fname_parameter_sets, \"HAD_IMERGcv_input_sim85\", nmax=2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For running the script above you need to create a parameter set file, an example is shown below:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fname_parameter_sets = training_general_path + \"/regional/datasets/csv/test_parameter_set.csv\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pd.read_csv(fname_parameter_sets).head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**TASK**: modify the script to create model files for running multiples simulation with different forcing\n", "datasets (e.g. stochastic forecasting).\n", "\n", "**HINT**: lines 66 and 68 needs to be modified in the input model parameters file, add lines in script to modify this lines" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 3.4. Runing pipeline DRYP simulations" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For large simulations output files can become very large which may result in consuming cosiderable amount of\n", "memory that can even stop the model. This problem is very challenging to address in python, requiring significan changes\n", "in the code or even considering parallelisation. Here, we simply split a continuos simulation (in time, not in space)\n", "to store model output files in specified simulation periods (e.g. years).\n", "\n", "For this, we can create a series of model files that read and save inital conditions for the previous and subsequente\n", "simulations. An example of a python script to generate pipeline simulaiton is presented below:" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [], "source": [ "def gen_inital_end_simulation_dates(year, month, day, dtyear=1, dtmonth=0, dtday=0):\n", "\t\"\"\"This function gets the initial and end date of a specified period\n", "\t\"\"\"\n", "\tdate_ini = str(year+dtyear) + ' ' + str(month+dtmonth) + ' ' + str(day+dtday)\n", "\tdate_end = str(year+dtyear+1) + ' ' + str(month+dtmonth) + ' ' + str(day+dtday)\n", "\t\n", "\tname = str(year)\n", "\tname_end = str(year+dtyear)\n", "\t\n", "\treturn date_ini, date_end, name, name_end\n", "\n", "def gen_pipeline_dryp_files(path_json_template, path_outputs=None, model_name=None, start_year=2000, end_year=2023):\n", "\t\n", "\t# Get input file as dictionary\n", "\twith open(path_json_template, 'r') as file:\n", "\t\tjson_template = json.load(file)\n", "\tif path_outputs is None:\n", "\t\tpath_outputs = json_template[\"OUTPUT\"][\"path_output\"]\n", "\t\n", "\t# get path, filename and model name\n", "\tpath, file = os.path.split(path_json_template)\n", "\t\n", "\t# identify filename of paramter file\n", "\tif model_name is None:\n", "\t\tmodel_name = file.split('.')[0]\n", "\t\n", "\t# avoid updating forcing dataset\n", "\tpath_pre, path_pet = None, None\t\n", "\t\n", "\t# iterate over the simulatio period\n", "\tfor i, iyear in enumerate(range(start_year, end_year)):\n", "\t\tmname = model_name + \"_\" + str(iyear)\n", "\t\tpath_new_input = path + \"/\" + mname + \".json\"\n", "\t\tpath_new_settings = path + \"/\" + mname + \"_settings.json\"\n", "\t\t# get name of initial conditions\n", "\t\t# skip if the initial year\n", "\t\tif i != 0:\n", "\t\t\tmname0 = model_name + \"_\" + str(iyear-1)\n", "\t\t\tpath_Qo = path_outputs + '/'+ mname0 +'_avg_Q_ini.asc'\n", "\t\t\tpath_uz_theta = path_outputs + '/'+ mname0 +'_avg_tht_ini.asc'\n", "\t\t\tpath_sz_wte = path_outputs + '/'+ mname0 +'_avg_wte_ini.asc'\n", "\t\t\tpath_rp_theta = path_outputs + '/'+ mname0 + '_avg_tht_rp_ini.asc'\n", "\t\t\tpath_pnd_Vo = path_outputs + '/' + mname0 + '_avg_V_pnd_ini.asc'\n", "\t\telse:\n", "\t\t\tpath_Qo=None\n", "\t\t\tpath_uz_theta=None\n", "\t\t\tpath_sz_wte=None\n", "\t\t\tpath_rp_theta=None\n", "\t\t\tpath_pnd_Vo=None\n", "\t\t\n", "\t\t# get simulation period\n", "\t\tif i == 0:\n", "\t\t\tdate_ini, date_end = None, None\n", "\t\telse:\n", "\t\t\tdate_ini, date_end, name, name_end = gen_inital_end_simulation_dates(iyear-1, 1, 1)\n", "\n", "\t\t# write json files\n", "\t\twrite_JSON_dryp_files(json_template, mname, path_pre, path_pet, path_new_input,\n", "\t\t\t\t\t\t start_date=date_ini, end_date=date_end, new_setting_file=path_new_settings,\n", "\t\t\t\t\t\t path_Qo=path_Qo,\n", "\t\t\t\t\t\t path_uz_theta=path_uz_theta,\n", "\t\t\t\t\t\t path_sz_wte=path_sz_wte,\n", "\t\t\t\t\t\t path_rp_theta=path_rp_theta,\n", "\t\t\t\t\t\t path_pnd_Vo=path_pnd_Vo\n", "\t\t\t\t\t\t )\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Generate pipeline simulation files" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ========================================================\n", "# LOOP FOR CREATING MULTIPLE IMPORT FILES FOR RUNNING IN HPC\n", "fname = [\n", "training_general_path + \"model/HAD_IMERG_Tana_input_sim.json\",\n", "]\n", "\n", "path_outputs = \"/home/c1755103/HAD/HAD_output\"\n", "# WARNINGS\n", "# The initial file should specify the initial conditions at the begining of\n", "# the simulation of the entire period, therefore, the name of inital model\n", "# must be according to the name required to the next simulation\n", "# subsequent period must conside\n", "for ifname_input in fname:\n", "\tgen_pipeline_dryp_files(ifname_input, path_outputs)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import glob\n", "files = glob.glob(regional_path + \"inputs/*.asc\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "new_path = \"D:/\"\n", "fnames = []\n", "for ifile in files:\n", " fnames.append(ifile.split(\"\\\\\")[-1])\n", " new_name = new_path + ifile.split(\"\\\\\")[-1]\n", " print(new_name)\n", " #new_name = new_path + fnames[0]\n", " #process clip raster" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#fnames" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#new_path = \"D:/\"\n", "#new_name = new_path + fnames[0]" ] } ], "metadata": { "kernelspec": { "display_name": "Python [conda env:Anaconda3]", "language": "python", "name": "conda-env-Anaconda3-py" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }