How to Create a Workflow

To create a Workflow head over to the Workflow Catalogue page. To start click the "Create workflow" button, outlined in orange in the image below.

Workflow Metadata

This will take you to a new page that asks you to fill in some metadata for the Workflow you are creating as in the image below.

The Workflow name field is required as this will be the name of your Workflow when it is run on DAFNI. The workflow engine DAFNI uses to run your workflow requires that this name only contain lower case alphanumeric characters and dashes

The Display name field is what you and other users (if you choose to share your Workflow) will see as the name of your Workflow in the catalogue and on the details page. Give your Workflow a sensible name and we'll move to the next field.

The Publisher field will be visible on the details page and is an opportunity for you to let other users know who made this Workflow (again only if you choose to share the Workflow). Usually, you would enter your name into this field, but you can also enter your Organisation, Company name, or Project name if you would prefer.

The Summary field should be given a short description of what your Workflow does. This will appear next to the Display name in the catalogue.

The Description field is an opportunity for you to give a long-form description of what your Workflow accomplishes.

The final field, Version message, should be a short note to let other users know what you have changed in this version of your Workflow. Since this is the first version of your Workflow it will be filled in for you, feel free to change this if you would like to.

Workflow Steps

Once you have filled in the metadata for your Workflow, click the "Continue" button in the bottom right-hand corner of the page to go to the next stage, which is shown in the image below.

Let's quickly look over what is on this page. In the top right-hand corner, you can see a key that shows the colours of the step types we support in DAFNI, green for model steps, purple for publisher steps and pink for visualisation steps. Each of these has a different purpose that will be explained later. To the right of the key is the "Reset position" button, this button resets the position of your Workflow in the canvas so that if you accidentally lose it in the canvas you can easily bring it back.

Since we've mentioned the canvas, let's cover that next. In the centre of the page, you can see the canvas, a white area that contains 3 circles connected by arrows, a "Start", "+", and "End". The "Start" and "End" circles are only there to show you where the start and end of your Workflow is. The "+" circle will be used a lot, clicking on this circle will allow you to create the first step in your Workflow.

If you create a long enough Workflow you'll notice that some steps may go off the screen, to continue creating your Workflow and to see those steps, click and drag anywhere in the white area surrounding your Workflow in the direction you would like the view box to move and you should see your Workflow move within the canvas. As mentioned above, if you accidentally drag your Workflow completely out of the view box and can't find it again, click "Reset position" to return it to its proper position.

Step Types

DAFNI currently supports 3 step types in Workflows. Below we'll cover what those step types are and what each of them does.

Model Steps

This is the most important step type for Workflows, for a Workflow to do something it needs a model step. A model step tells the Workflow which Model you want to run and specifically what version of a Model you want to run in your Workflow.

Publisher Steps

These steps take the output data produced by one or more of the model steps in your Workflow, combine it with some metadata that is defined in the step configuration and upload that to DAFNI's Data catalogue.

Visualisation Steps

Visualisation steps are very similar to publisher steps, currently, they require a dataset to be uploaded to DAFNI before they create a visualisation. In addition to the metadata for the dataset that is uploaded to DAFNI, they also require the metadata for the visualisation that will be created from the step. The visualisation step will create the dataset in the Data catalogue and spin up the visualisation automatically for you when the Workflow is executed.

Creating Steps

Now that we know what each step type does let's start by creating a model step. Click on the "+" step in the Workflow in front of you, this will hide the "Back" and "Continue" buttons and reveal some configuration options for your new step, as shown in the image below.

The first option available to you in this configuration panel is choosing what step type you want to create, model has already been selected for us so we'll just ignore this field for now.

Model Step

Below the step type, you are asked to give this step a name. The step name must be unique within the Workflow. We recommend entering a step name that tells you what the step will be doing, for example naming it after the Model that will be run in this step. Next, we need to choose the Model that we will run in this step, to do this click the "Select model" button. A dialog box should open up showing the Model catalogue with checkboxes down the left -hand side of the table, check one of these to select your Model and click "Select model" in the bottom right-hand corner of the dialog.

This is the minimum information that is required to create a model step, the last field, "Steps to include data from" allows you to send the output data from one model step to the inputs folder of another model step (at the path /data/inputs/{STEP-NAME} where {STEP-NAME} is the name of the step you are taking data from). Since this Workflow doesn't have any other model steps yet, the list of steps you can take data from is empty. Press "Create step" to confirm the creation of your step and it will be added to the Workflow.

Publisher Step

Press the "+" step again to add a new step, this time we will create a publisher step so select "Publisher" from the step type buttons. You should notice the fields in the configuration panel have changed to show those required for the publisher step type as below.

As with the model step, the first field to fill in is the step name, for a publisher step the name of the dataset that will be uploaded to the catalogue could be a useful name.

Below the step name, there is a switch that lets you select whether you want to specify the metadata for the dataset that will be produced manually or to include a JSON file from a model step that has the metadata in. Initially, the switch will be turned on and you will be prompted to use a metadata file. You will see two fields to the right of the switch, the first "Step to include metadata from" lets you choose which model step to take the JSON file from. The second "File path" needs a path to the file that is produced by the model step. For now, let's click on the switch to change to the manual metadata entry form that should appear as below.

Note: If you are using a metadata file, the Model, from which the metadata file is generated, must put it into the /data/outputs/ folder or a child folder within that. The field itself automatically adds the /data/ prefix so you only need to enter the path from that folder e.g. outputs/metadata.json or outputs/folder/metadata.json.

This metadata form is the same as the one shown on dataset upload so it won't be covered in detail here. A couple of things to note though - the organisation fields are disabled because they are automatically filled by DAFNI, the organisation name will be filled with DAFNI Workflows ({WORKFLOW-NAME}) where {WORKFLOW-NAME} will be replaced by the display name of your Workflow and we will add a link to the Workflow instance that created the dataset when you execute the Workflow. The creation date for the dataset will be set to the current time when the dataset gets created too, so that field is not present in the form.

Below the metadata form is an empty table called "Data files to publish". Here, you select the files you want to upload as part of the dataset. Click "New data file" to add a new set of files to upload, this will open a dialog with two fields "Step to include data file from" and "File path". "Step to include data file from" works the same as the similarly named field for metadata file selection, choose a model step from the list to choose the step to take files from. As with the "File path" field for metadata, this takes files from the /data/outputs/ folder or a child folder within that. It's up to you how verbose you want to be here, you can either add a new entry into the table for each file you want to include or, if you want to include an entire folder worth of files, you can do so by using the * wild card character e.g. outputs/*. This will only grab the files in that folder though, any child folders will be ignored, to recursively collect files from the folder and all child folders within it use **/* e.g. outputs/**/*. If you would like to include files from multiple different model steps you need to add a new entry into the table for each model step you want to include files from.

Once you have entered all the files you want to publish, click "Create step" to add the step to your Workflow.

Visualisation Step

Press the "+" step again to add a new step, this time we will create a visualisation step so select "Visualisation" from the step type buttons. You should notice the fields in the configuration panel have changed to show those required for the visualisation step type as below.

As with the other steps, the first field to fill in is the step name. For a visualisation step, the name of the visualisation that will be created could be a useful name.

As mentioned earlier, a lot of the configuration of the visualisation step is the same as the publisher step because they both currently upload a dataset to the Data catalogue. As such, we won't cover the metadata or "Data files to visualise" here as they work in the same way as with the publisher steps.

There are some additional fields that visualisation steps need though, Visualisation name is one of those. This is the name that will be given to the visualisation that is created by this step, so name it something sensible and relevant. Next, you need to select the Visualisation type. Currently, we support Jupyter Notebooks and Drag and Drop visualisations but in the future, we hope to support more. If you are comfortable with programming in Python/R then you might want to select Jupyter Notebook, if not or if you would rather produce the visualisation using a web tool that we created in collaboration with Oxford University then select Drag and Drop. The final additional field is the Visualisation description, here you can describe exactly what you plan to do in this visualisation - don't worry too much if you're not sure about that now, this can always be updated manually on the visualisation later.

Once you have entered this information, filled in the metadata and selected the files you want to add to your visualisation, click "Create step" to add it to your Workflow.

Editing a Step

If you made a mistake or want to change any of the information you entered into any of the steps that you have already created you can do that by clicking on the step in your Workflow. You should see that the appropriate configuration form is shown below the Workflow, similar to when you add a new step except the "Choose a step type" field has gone as it is not possible to change the type of a step once it has been created. The form should be pre-populated with all the information that you previously entered, change any fields that you want to and then click "Apply changes" to save your changes.

Note: Clicking "Cancel", anywhere on the canvas around your Workflow or onto another step will lose any changes that you have made to the step.

Wrapping Up

You should now have a Workflow that looks something like the image below. Next, we will create the first Parameter Set for your Workflow, so click the "Continue button" and go to How to Create a Parameter Set.