import pytest29 Tests
Acknowledgements
This section is based on the tests in the Python DES RAP Template developed by Amy Heather 
and Tom Monks from the PenCHORD team at the University of Exeter. This is a template for running SimPy DES models within a reproducible analytical pipeline, and the model structure in the template was based on this book, among other sources.
Testing is the process of evaluating a model to ensure it works as expected, gives reliable results, and can handle different conditions. By systematically checking for errors, inconsistencies, or unexpected behaviors, testing helps improve the quality of a model, catch errors and prevent future issues.
29.1 Pytest
When you create a model, you will naturally carry out tests, with simple manual checks where you observe outputs and ensure they look right. These checks can be formalised and automated so that you can run them after any changes, and catch any issues that arise.
A popular framework for testing in python is pytest.
29.1.1 Simple pytest example
Each test in pytest is a function that contains an assertion statement to check a condition (e.g. number > 0). If the condition fails, pytest will return an error message (e.g. “The number should be positive”).
Tests are typically stored in a folder called tests, with filenames starting with the prefix test_. This naming convention allows pytest to automatically discover and run all the tests in the folder.
Here’s an example of a simple test using pytest:
def test_positive():
"""
Confirm that the number is positive.
"""
number = 5
assert number > 0, "The number should be positive"29.1.2 Running the tests
Tests are typically run from the terminal. Commands include:
pytest- runs all tests.pytest tests/test_example_simple.py- runs tests from a specific file.
When you run a test, you’ll see an output like this in the terminal:
29.1.3 Parametrise
We can execute the same test on different parameters using pytest.mark.parametrize.
Here’s an example:
@pytest.mark.parametrize("number", [1, 2, 3, -1])
def test_positive_param(number):
"""
Confirm that the number is positive.
Arguments:
number (float):
Number to check.
"""
assert number > 0, f"The number {number} is not positive."In this example, we’re testing the same logic with four different values: 1, 2, 3, and -1. The last value, -1, will cause the test to fail. The error message includes the failed value for easy debugging.
29.2 Coding the model
Tip
Throughout the code, anything new that’s been added will be followed by the comment ##NEW - so look out for that in the following code chunks.
We will design tests for the model from Chapter 13. However, we will modify the model so that, instead of modifying a global class of parameter values, we create instances of this class and use it in our model.
By using class instances, each test has isolated parameters, preventing interference and ensuring consistency. This improves flexibility for independent test scenarios, simplifies debugging, and supports parallel execution by avoiding shared state, making the model more robust.
29.2.1 Param class
As these are no longer “global” parameters, we will rename g to Param.
# Class to store parameter values.
class Param: ##NEW
patient_inter = 5
mean_reception_time = 2
mean_n_consult_time = 6
mean_d_consult_time = 20
number_of_receptionists = 1
number_of_nurses = 1
number_of_doctors = 2
prob_seeing_doctor = 0.6
sim_duration = 600
number_of_runs = 100- 1
-
Renamed
gtoParam.
29.2.2 Patient class
This remains unchanged.
29.2.3 Model class
Set parameters as an input to the class. Each instance of g is changed to param (which refers to the parameter instance provided to the class).
class Model:
def __init__(self, param, run_number): ##NEW
self.param = param ##NEW
self.env = simpy.Environment()
self.patient_counter = 0
self.receptionist = simpy.Resource(
self.env, capacity=self.param.number_of_receptionists) ##NEW
self.nurse = simpy.Resource(
self.env, capacity=self.param.number_of_nurses) ##NEW
self.doctor = simpy.Resource(
self.env, capacity=self.param.number_of_doctors) ##NEW
...
self.patient_inter_arrival_dist = Exponential(
mean = self.param.patient_inter, ##NEW
random_seed = self.run_number*2)
self.patient_reception_time_dist = Exponential(
mean = self.param.mean_reception_time, ##NEW
random_seed = self.run_number*3)
self.nurse_consult_time_dist = Exponential(
mean = self.param.mean_n_consult_time, ##NEW
random_seed = self.run_number*4)
self.doctor_consult_time_dist = Exponential(
mean = self.param.mean_d_consult_time, ##NEW
random_seed = self.run_number*5)
...
def attend_clinic(self, patient):
...
if random.uniform(0,1) < self.param.prob_seeing_doctor: ##NEW
...
def run(self):
self.env.process(self.generator_patient_arrivals())
self.env.run(until=self.param.sim_duration) ##NEW
...- 1
-
Set
paramas an input to theModel, and made a model attribute. - 2
-
Replaced all
gwithself.param.
29.2.4 Trial class
Set parameters as an input to the class, and renamed g to param. Also, disabled printing sections.
class Trial:
def __init__(self, param): ##NEW
self.param = param ##NEW
self.df_trial_results = pd.DataFrame()
self.df_trial_results["Run Number"] = [0]
self.df_trial_results["Arrivals"] = [0]
self.df_trial_results["Mean Q Time Recep"] = [0.0]
self.df_trial_results["Mean Q Time Nurse"] = [0.0]
self.df_trial_results["Mean Q Time Doctor"] = [0.0]
self.df_trial_results.set_index("Run Number", inplace=True)
def print_trial_results(self):
print ("Trial Results")
print (self.df_trial_results.round(2))
print(self.df_trial_results.mean().round(2))
def run_trial(self):
# print(f"{self.param.number_of_receptionists} receptionists, " +
# f"{self.param.number_of_nurses} nurses, " +
# f"{self.param.number_of_doctors} doctors") ##NEW - no printing
# print("")
for run in range(self.param.number_of_runs): ##NEW
random.seed(run)
my_model = Model(param=self.param, run_number=run) ##NEW
patient_level_results = my_model.run()
self.df_trial_results.loc[run] = [
len(patient_level_results),
my_model.mean_q_time_recep,
my_model.mean_q_time_nurse,
my_model.mean_q_time_doctor
]
##NEW - no printing
# self.print_trial_results()- 1
-
Set
paramas an input to theTrial, and made a trial attribute. - 2
- Disabled printing.
- 3
-
Replaced all
gwithself.param. - 4
- Disabled printing.
29.2.5 The full code
The full updated code for the model is given below.
Click here to view the code
import simpy
import random
import pandas as pd
from sim_tools.distributions import Exponential
# Class to store parameter values.
class Param: ##NEW
patient_inter = 5
mean_reception_time = 2
mean_n_consult_time = 6
mean_d_consult_time = 20
number_of_receptionists = 1
number_of_nurses = 1
number_of_doctors = 2
prob_seeing_doctor = 0.6
sim_duration = 600
number_of_runs = 100
# Class representing patients coming in to the clinic.
class Patient:
def __init__(self, p_id):
self.id = p_id
self.q_time_recep = 0
self.q_time_nurse = 0
self.q_time_doctor = 0
# Class representing our model of the clinic.
class Model:
# Constructor to set up the model for a run. We pass in a run number and
# instance of the parameter class when we create a new model.
def __init__(self, param, run_number): ##NEW
# Store the passed in parameters
self.param = param ##NEW
# Create a SimPy environment in which everything will live
self.env = simpy.Environment()
# Create a patient counter (which we'll use as a patient ID)
self.patient_counter = 0
# Create our resources
self.receptionist = simpy.Resource(
self.env, capacity=self.param.number_of_receptionists) ##NEW
self.nurse = simpy.Resource(
self.env, capacity=self.param.number_of_nurses) ##NEW
self.doctor = simpy.Resource(
self.env, capacity=self.param.number_of_doctors) ##NEW
# Store the passed in run number
self.run_number = run_number
# Create a new Pandas DataFrame that will store some results against
# the patient ID (which we'll use as the index).
self.results_df = pd.DataFrame()
self.results_df["Patient ID"] = [1]
self.results_df["Q Time Recep"] = [0.0]
self.results_df["Time with Recep"] = [0.0]
self.results_df["Q Time Nurse"] = [0.0]
self.results_df["Time with Nurse"] = [0.0]
self.results_df["Q Time Doctor"] = [0.0]
self.results_df["Time with Doctor"] = [0.0]
self.results_df.set_index("Patient ID", inplace=True)
# Create an attribute to store the mean queuing times across this run of
# the model
self.mean_q_time_recep = 0
self.mean_q_time_nurse = 0
self.mean_q_time_doctor = 0
self.patient_inter_arrival_dist = Exponential(
mean = self.param.patient_inter, ##NEW
random_seed = self.run_number*2)
self.patient_reception_time_dist = Exponential(
mean = self.param.mean_reception_time, ##NEW
random_seed = self.run_number*3)
self.nurse_consult_time_dist = Exponential(
mean = self.param.mean_n_consult_time, ##NEW
random_seed = self.run_number*4)
self.doctor_consult_time_dist = Exponential(
mean = self.param.mean_d_consult_time, ##NEW
random_seed = self.run_number*5)
# A generator function that represents the DES generator for patient
# arrivals
def generator_patient_arrivals(self):
# We use an infinite loop here to keep doing this indefinitely whilst
# the simulation runs
while True:
# Increment the patient counter by 1 (this means our first patient
# will have an ID of 1)
self.patient_counter += 1
# Create a new patient - an instance of the Patient Class we
# defined above. Remember, we pass in the ID when creating a
# patient - so here we pass the patient counter to use as the ID.
p = Patient(self.patient_counter)
# Tell SimPy to start up the attend_clinic generator function with
# this patient (the generator function that will model the
# patient's journey through the system)
self.env.process(self.attend_clinic(p))
# Randomly sample the time to the next patient arriving. Here, we
# sample from an exponential distribution (common for inter-arrival
# times), and pass in a lambda value of 1 / mean. The mean
# inter-arrival time is stored in the g class.
sampled_inter = self.patient_inter_arrival_dist.sample()
# Freeze this instance of this function in place until the
# inter-arrival time we sampled above has elapsed. Note - time in
# SimPy progresses in "Time Units", which can represent anything
# you like (just make sure you're consistent within the model)
yield self.env.timeout(sampled_inter)
# A generator function that represents the pathway for a patient going
# through the clinic.
# The patient object is passed in to the generator function so we can
# extract information from / record information to it
def attend_clinic(self, patient):
start_q_recep = self.env.now
with self.receptionist.request() as req:
yield req
end_q_recep = self.env.now
patient.q_time_recep = end_q_recep - start_q_recep
sampled_recep_act_time = self.patient_reception_time_dist.sample()
self.results_df.at[patient.id, "Q Time Recep"] = (
patient.q_time_recep
)
self.results_df.at[patient.id, "Time with Recep"] = (
sampled_recep_act_time
)
yield self.env.timeout(sampled_recep_act_time)
# Here's where the patient finishes with the receptionist, and starts
# queuing for the nurse
# Record the time the patient started queuing for a nurse
start_q_nurse = self.env.now
# This code says request a nurse resource, and do all of the following
# block of code with that nurse resource held in place (and therefore
# not usable by another patient)
with self.nurse.request() as req:
# Freeze the function until the request for a nurse can be met.
# The patient is currently queuing.
yield req
# When we get to this bit of code, control has been passed back to
# the generator function, and therefore the request for a nurse has
# been met. We now have the nurse, and have stopped queuing, so we
# can record the current time as the time we finished queuing.
end_q_nurse = self.env.now
# Calculate the time this patient was queuing for the nurse, and
# record it in the patient's attribute for this.
patient.q_time_nurse = end_q_nurse - start_q_nurse
# Now we'll randomly sample the time this patient with the nurse.
# Here, we use an Exponential distribution for simplicity, but you
# would typically use a Log Normal distribution for a real model
# (we'll come back to that). As with sampling the inter-arrival
# times, we grab the mean from the g class, and pass in 1 / mean
# as the lambda value.
sampled_nurse_act_time = self.nurse_consult_time_dist.sample()
# Here we'll store the queuing time for the nurse and the sampled
# time to spend with the nurse in the results DataFrame against the
# ID for this patient. In real world models, you may not want to
# bother storing the sampled activity times - but as this is a
# simple model, we'll do it here.
# We use a handy property of pandas called .at, which works a bit
# like .loc. .at allows us to access (and therefore change) a
# particular cell in our DataFrame by providing the row and column.
# Here, we specify the row as the patient ID (the index), and the
# column for the value we want to update for that patient.
self.results_df.at[patient.id, "Q Time Nurse"] = (
patient.q_time_nurse)
self.results_df.at[patient.id, "Time with Nurse"] = (
sampled_nurse_act_time)
# Freeze this function in place for the activity time we sampled
# above. This is the patient spending time with the nurse.
yield self.env.timeout(sampled_nurse_act_time)
# When the time above elapses, the generator function will return
# here. As there's nothing more that we've written, the function
# will simply end. This is a sink. We could choose to add
# something here if we wanted to record something - e.g. a counter
# for number of patients that left, recording something about the
# patients that left at a particular sink etc.
# Conditional logic to see if patient goes on to see doctor
# We sample from the uniform distribution between 0 and 1. If the value
# is less than the probability of seeing a doctor (stored in g Class)
# then we say the patient sees a doctor.
# If not, this block of code won't be run and the patient will just
# leave the system (we could add in an else if we wanted a branching
# path to another activity instead)
if random.uniform(0,1) < self.param.prob_seeing_doctor: ##NEW
start_q_doctor = self.env.now
with self.doctor.request() as req:
yield req
end_q_doctor = self.env.now
patient.q_time_doctor = end_q_doctor - start_q_doctor
sampled_doctor_act_time = self.nurse_consult_time_dist.sample()
self.results_df.at[patient.id, "Q Time Doctor"] = (
patient.q_time_doctor
)
self.results_df.at[patient.id, "Time with Doctor"] = (
sampled_doctor_act_time
)
yield self.env.timeout(sampled_doctor_act_time)
# This method calculates results over a single run. Here we just calculate
# a mean, but in real world models you'd probably want to calculate more.
def calculate_run_results(self):
# Take the mean of the queuing times across patients in this run of the
# model.
self.mean_q_time_recep = self.results_df["Q Time Recep"].mean()
self.mean_q_time_nurse = self.results_df["Q Time Nurse"].mean()
self.mean_q_time_doctor = self.results_df["Q Time Doctor"].mean()
# The run method starts up the DES entity generators, runs the simulation,
# and in turns calls anything we need to generate results for the run
def run(self):
# Start up our DES entity generators that create new patients. We've
# only got one in this model, but we'd need to do this for each one if
# we had multiple generators.
self.env.process(self.generator_patient_arrivals())
# Run the model for the duration specified in g class
self.env.run(until=self.param.sim_duration) ##NEW
# Now the simulation run has finished, call the method that calculates
# run results
self.calculate_run_results()
# Print the run number with the patient-level results from this run of
# the model
return (self.results_df)
# Class representing a Trial for our simulation - a batch of simulation runs.
class Trial:
# The constructor sets up a pandas dataframe that will store the key
# results from each run against run number, with run number as the index.
def __init__(self, param): ##NEW
# Store the model parameters
self.param = param ##NEW
self.df_trial_results = pd.DataFrame()
self.df_trial_results["Run Number"] = [0]
self.df_trial_results["Arrivals"] = [0]
self.df_trial_results["Mean Q Time Recep"] = [0.0]
self.df_trial_results["Mean Q Time Nurse"] = [0.0]
self.df_trial_results["Mean Q Time Doctor"] = [0.0]
self.df_trial_results.set_index("Run Number", inplace=True)
# Method to print out the results from the trial. In real world models,
# you'd likely save them as well as (or instead of) printing them
def print_trial_results(self):
print ("Trial Results")
print (self.df_trial_results.round(2))
print(self.df_trial_results.mean().round(2))
# Method to run a trial
def run_trial(self):
# print(f"{self.param.number_of_receptionists} receptionists, " +
# f"{self.param.number_of_nurses} nurses, " +
# f"{self.param.number_of_doctors} doctors") ##NEW - no printing
# print("")
# Run the simulation for the number of runs specified in g class.
# For each run, we create a new instance of the Model class and call its
# run method, which sets everything else in motion. Once the run has
# completed, we grab out the stored run results (just mean queuing time
# here) and store it against the run number in the trial results
# dataframe.
for run in range(self.param.number_of_runs): ##NEW
random.seed(run)
my_model = Model(param=self.param, run_number=run) ##NEW
patient_level_results = my_model.run()
self.df_trial_results.loc[run] = [
len(patient_level_results),
my_model.mean_q_time_recep,
my_model.mean_q_time_nurse,
my_model.mean_q_time_doctor
]
##NEW - no printing
# Once the trial (ie all runs) has completed, print the final results
# self.print_trial_results()29.3 Testing our model
There are many different ways of categorising tests. We will focus on three types:
- Functional testing
- Unit testing
- Back testing
29.3.1 Functional tests
Functional tests verify that the system or components perform their intended functionality.
For example, we expect that the mean wait time for a nurse should decrease if:
- The number of nurses increases.
- The patient inter-arrival time increases (so there are fewer arrivals).
- The length of the nurse consultation decreases.
For simplicity, this test just focuses the nurse waiting times, but this idea can be expanded to other resources and metrics in the model as well.
import pytest
from full_model import Param, Trial
@pytest.mark.parametrize('param_name, initial_value, adjusted_value', [
('number_of_nurses', 3, 9),
('patient_inter', 2, 15),
('mean_n_consult_time', 30, 3),
])
def test_waiting_time_utilisation(param_name, initial_value, adjusted_value):
"""
Test that adjusting parameters decreases the waiting time and utilisation.
Arguments:
param_name (string):
Name of parameter to change in the Param() class.
initial_value (float|int):
Value with which we expect longer waiting times.
adjusted_value (float|int):
Value with which we expect shorter waiting time.
"""
# Define helper function for the test
def helper_param(param_name, value):
"""
Helper function to set a specific parameter value, run the model for
a single replication and return the results of that run.
Arguments:
param_name (string):
Name of the parameter to modify.
value (float|int):
Value to assign to the parameter.
Returns:
dataframe:
Dataframe with the trial-level results.
"""
# Create instance of parameter class with default values but one run
param = Param()
param.number_of_runs = 1
# Modify specific parameter
setattr(param, param_name, value)
# Run replications and return the results from the run as a series
trial = Trial(param)
trial.run_trial()
return trial.df_trial_results.iloc[0]
# Run model with initial and adjusted values
initial_results = helper_param(param_name, initial_value)
adjusted_results = helper_param(param_name, adjusted_value)
# Check that nurse waiting times from adjusted model are lower
initial_wait = initial_results["Mean Q Time Nurse"]
adjusted_wait = adjusted_results["Mean Q Time Nurse"]
assert initial_wait > adjusted_wait, (
f"Changing '{param_name}' from {initial_value} to {adjusted_value} " +
"did not decrease waiting time for the nurse as expected: observed " +
f"waiting times of {initial_wait} and {adjusted_wait}, respectively."
)These tests pass.
29.3.2 Unit tests
Unit tests are a type of functional testing that focuses on individual components (e.g. methods, classes) and tests them in isolation to ensure they work as intended.
For example, we expect that our model should fail if the number of doctors or the patient inter-arrival time were set to 0. This is tested using test_zero_inputs.
import pytest
from full_model import Param, Trial
@pytest.mark.parametrize("param_name, value", [
("number_of_doctors", 0),
("patient_inter", 0)
])
def test_zero_inputs(param_name, value):
"""
Check that the model fails when inputs that are zero are used.
Arguments:
param_name (string):
Name of parameter to change in the Param() class.
value (float|int):
Invalid value for parameter.
"""
# Create parameter class with an invalid value
param = Param()
setattr(param, param_name, value)
# Verify that initialising the model raises an error
with pytest.raises(ValueError):
Trial(param)When we run the test, we see that both fail.
These tests fail as we do not have an error handling for these values. If we had proceeded to run_trial()…
- Number of doctors = 0: The model would’ve stopped, as SimPy has built in functionality requiring that the capacity of resources must be greater than 0, and so it raises a ValueError and stops execution.
- Patient inter-arrival time = 0: The model would have run infinitely, as it would just constantly generating new patients.
To address this, we could add error handling which raises an error for users if they try to input a value of 0. For example, we could add the following code to our Model __init__ method:
# Loop through the specified parameters
for param_name in ["sim_duration", "patient_inter"]:
# Get the value of that parameter by its name
param_value = getattr(self.param, param_name)
# Raise an error if it is 0 or less
if param_value <= 0:
raise ValueError(
f"Parameter '{param_name}' must be greater than 0, but has been" +
f"set to {param_value:.3f}.)29.3.3 Back tests
Back tests check that the model code produces results consistent with those generated historically/from prior code.
First, we’ll generate a set of expected results, with a specific set of parameters. Although this may seem unnecessary in this case, as they match our default parameters in our Param class, these are still specified to ensure that we are testing on the same parameters, even if defaults change in Param class.
param = Param()
param.patient_inter = 5
param.mean_reception_time = 2
param.mean_n_consult_time = 6
param.mean_d_consult_time = 20
param.number_of_receptionists = 1
param.number_of_nurses = 1
param.number_of_doctors = 2
param.prob_seeing_doctor = 0.6
param.sim_duration = 600
param.number_of_runs = 100We’ll then run the model and save the results to .csv files.
# Run trial
trial = Trial(param)
trial.run_trial()
# Preview and save results to csv
print(trial.df_trial_results.head())
trial.df_trial_results.to_csv("tests/exp_results/trial.csv") Arrivals Mean Q Time Recep Mean Q Time Nurse Mean Q Time Doctor
Run Number
0 102.0 0.000000 57.193168 1.146714
1 125.0 1.842723 144.688786 0.018501
2 112.0 0.845851 15.299135 1.131864
3 120.0 1.082325 82.669429 0.037010
4 132.0 1.943043 107.474373 0.506261In the test, we’ll run the same model parameters, then import and compare against the saved .csv file to check for any differences.
from pathlib import Path
import pandas as pd
from full_model import Param, Trial
def test_reproduction():
"""
Check that results from particular run of the model match those previously
generated using the code.
"""
# Define model parameters
param = Param()
param.patient_inter = 5
param.mean_reception_time = 2
param.mean_n_consult_time = 6
param.mean_d_consult_time = 20
param.number_of_receptionists = 1
param.number_of_nurses = 1
param.number_of_doctors = 2
param.prob_seeing_doctor = 0.6
param.sim_duration = 600
param.number_of_runs = 100
# Run trial
trial = Trial(param)
trial.run_trial()
# Compare the trial results
exp_trial = pd.read_csv(
Path(__file__).parent.joinpath("exp_results/trial.csv"), index_col=0)
pd.testing.assert_frame_equal(trial.df_trial_results, exp_trial)This test passes.
We generate the expected results for our backtest in a seperate Python file or Jupyter notebook, rather than within the test itself. We then would generally run tests using the same pre-generated .csv files, without regenerating them. However, the test will fail if the model logic is intentionally changed, leading to different results from the same parameters. In that case, if we are certain that these changes are the reason for differing results, we should re-run the Python file or notebook to regenerate the .csv. It is crucial to exercise caution when doing this, to avoid unintentionally overwriting correct expected results.
29.3.4 Further testing examples
For more inspiration, check out the Python DES RAP Template. Examples of other tests it includes are:
- Functional tests for the impact of high demand on utilisation.
- Functional tests checking for expected decreases in the number of arrivals.
- Functional tests for an interval auditor.
- Functional tests for parallel execution.
- Functional tests for a warm-up period.
- Unit tests for the exponential class.
- Unit tests for a logging class.
- Unit tests for a modified parameter class which has functionality designed to prevent the addition of new attributes.