At the main page of the web aplication, there are links to two tools: the batch processing tool (demo/demoexpertsform.jsp) and the Expert Aggregation tool, which implements a Modified Delphi Process (MDP). (coord/index.jsp).
The batch processing tool allows a coordinator to process experts' responses that are stored in a text file (in CSV format), along with the "ground truth" data, and the rules for weighting experts. See the data entry page for explanation of the format of both these types of data. The application then computes the NJSD (based on Jensen-Shannon distance) and weights for the experts, and the weighted averages of their predictions.
The MDP management tool allows a coordinator to create and manage surveys, and to process survey results. The coordinator's pages are password-protected, while the experts (respondents) each receive special unique URLs, with their access code embedded, to access the input screens. In what follows we refer to the Delphi process as either "MDP" or "survey", interchangeably.
MDP life cycle.
Each MDP has four stages in its life cycle:
|
|
|
|
Creating and preparing a survey. When creating the survey, the coordinator will have to specify the number of observation periods K, and the number of experts whom he or she wants to invite. (It does not hurt to reserve a greater number of experts -- nothing bad will happen if some of them will never respond). The newly created survey is now in the PREPARE stage; here, the coordinator can specify the names for the K observation periods, how they are to be presented to the experts in the survey form. The experts also needs to enter names, aliases, or IDs of some kind to be assigned to experts; this is primarily needed for listing experts in various tables later. In a true MDP, this stage may include a conversation among experts, to agree on the time periods, and the specific demographic for which they are making estimates. The demographic can be mentioned in the name given to the survey, which serves as a reminder to the respondents.
First collection period. Once the coordinator has activated the survey (i.e. started the first collection period), he or she needs to invite the experts to participate. The web application generates a unique participation URL for each expert. These URLs are displayed on the survey's status page. The coordinator must send to each expert the appropriate participation URL associated with that expert, using an appropriate communication channel (such as email or snail mail). The system does not send email on its own, so you don't ever need to enter the experts' email addresses into the system, nor are they stored on the server. To preserve anonymity, in a face-to-face meeting those codes may be printed on paper, cut into tickets, and literally drawn from a hat or bowl. In a synchronous distributed discussion, they may be posted to a Google doc document. Using an "anyone with the link can edit", the experts are invited to each visit that page, cut one of the codes from the document, and paste it into his or her own browser. With this process, each expert gets a distinct code, but no one can tell who took which code.
As a coordinator, you can preview the survey form, as it would be shown to each individual expert, by clicking on this expert's response URL link in the survey management form. However, the system will be aware that you are the coordinator, and not one of the participating experts, because you are currently logged in as a coordinator. It therefore won't allow you to actually submit a response. If you have created a test survey and want to test the complete experience of a responder, you will need to open the response URL in a different web browser (or on a different computer), or in an "incognito" or "private" window of the same browser.
As the experts enter their responses, the coordinator can revisit the survey management page to see how many experts have entered their responses so far. If the coordinator has access to the "ground truth" information, s/he can now compute weights, averages, etc., for the accumulated responses. (See Processing the experts' estimates for details on the mathematics involved). The coordinator can also save the experts' responses into a CSV-format file.
Once the coordinator sees that all invited experts have entered their estimates, or s/he has decided not to wait for those who have not responded, he or she can end the first collection period and start the second collection period.
Second collection period.
During the second collection period, that is, the second round of the modified Delphi Process, experts are given the option to revise their earlier estimates. Each expert can do this at the same personal participation URL that was used during the first round. The system does not automatically notify the experts about the beginning of the second collection period; it is up to the coordinator to either send a reminder email to all experts, or to tell them in advance to go to the participation URL twice (e.g., "do this once on Monday, and once on Tuesday"). In a synchronous session the reminder can be given using an open telephone bridge.
During the second collection period, each expert is shown not only the estimate s/he has entered during the first round, but also the (anonymized) estimates supplied by all other experts. This is the Delphi aspect of the process.
As in the first round, the coordinator can monitor the experts' responses as they are submitted, and can close the survey (that is, move it to the "Finished") stage at any time.
Finished surveys.
Once a survey is finished, no more responses can be submitted. The coordinator can view the collected responses from both rounds of the survey, but cannot make any modifications. The coordinator can then move to compute the weighted aggregations of the expert inputs. From the Coordinator’s Main Page, select the [View and Manage] action, for that survey. From the Survey Details page, select the button labeled [View and Analyze Responses.]